System and method for integrated control of 3D visualization by a surgical robotic system

Abstract

A surgical robotic system includes a control tower, a mobile cart, and a surgical console. The mobile cart is coupled to the control tower and includes a surgical robotic arm. The surgical robotic arm includes a surgical instrument and an image capture device. The surgical instrument is actuatable in response to user input and is configured to treat a target tissue in real time. The image capture device is to capture at least one of an image or a video of the target tissue in real time. The surgical console includes a user input device to generate user input and a controller. The controller is operably coupled to the user input device and is configured to switch from a first mode to a second mode based on the user input and from the second mode to the first mode.

Description

Technical Field

[0001] This disclosure relates in its entirety to a robotic surgical system, and more particularly to a system and method for integrated control of a surgical robotic system and 3D visualization of a surgical site to register a 3D model with target tissue in the surgical site. Background Technology

[0002] Surgical robotic systems are currently used in minimally invasive medical procedures. Some surgical robotic systems include a surgical console that controls a surgical robotic arm and surgical instruments, which have end effectors (e.g., clamps or gripping instruments) that are coupled to and actuated by the robotic arm.

[0003] Typically, during surgical procedures using a surgical robotic system, clinicians develop surgical plans using a 3D model of the surgical site. In most cases, the clinician will view the 3D model on a separate interactive monitor. This requires the clinician to switch from the surgical robotic system controls to separate controls to interact with the 3D model on a separate interactive monitor, thus distracting the clinician. Summary of the Invention

[0004] This disclosure relates in its entirety to a surgical robot system including a user interface controller for providing integrated control of a 3D model via controls of the surgical robot system to register the 3D model with target tissue at a surgical site.

[0005] In one aspect, this disclosure provides a surgical robot system including a control tower, a mobile cart, and a surgical console. The mobile cart is coupled to the control tower and includes a surgical robot arm. The surgical robot arm includes surgical instruments and an image capture device. The surgical instruments are actuated in response to user input and are configured to treat target tissue in real time. The image capture device is configured to capture at least one of an image or video of the target tissue in real time. The surgical console is coupled to the control tower and includes a display, a memory, a user input device, and a controller. The memory is configured to store preoperative images of the target tissue. The user input device is configured to generate user input. The controller is operatively coupled to the display, the memory, and the user input device and is configured to switch from a first mode to a second mode and from the second mode to the first mode based on user input. The first mode includes real-time display of the target tissue and control of the surgical robot arm activated by the user input device, and the second mode includes displaying a 3D model of the target tissue and interactive control of the 3D model of the target tissue activated by the user input device.

[0006] In various aspects, the surgical console may also include an image processor configured to display or generate a 3D model of the target tissue based on stored preoperative images of the target tissue at the surgical site.

[0007] In various aspects, interactive control of the 3D model of the target tissue may include adjusting at least one of the scale, position, orientation, or at least one heuristic factor of the 3D model of the target tissue.

[0008] In various aspects, the controller can be further configured to maintain or retain the user input device from its actual position in the first mode.

[0009] In various aspects, the controller can be further configured to determine the desired position of the user input device by sensing the pointing force on the user input device.

[0010] In various aspects, the controller can be further configured to analyze the surface of the target tissue in real time.

[0011] In various aspects, the controller can be further configured to determine at least one of the tissue characteristics, mechanical properties, shape, reference marks, or heuristic factors of the target tissue.

[0012] In various aspects, the controller can be further configured to manipulate the 3D model of the target tissue based on at least one of the tissue characteristics, mechanical properties, shape, reference marks, or heuristic factors determined in real time, so as to further register the 3D model of the target tissue with the target tissue in real time.

[0013] In various aspects, the controller can be further configured to determine the completion of the real-time registration of the 3D model of the target tissue with the target tissue.

[0014] In various aspects, the controller can be further configured to complete the real-time registration of the 3D model of the target tissue with the target tissue, and to overlay the 3D model of the target tissue on top of the target tissue in real time.

[0015] In various respects, the controller can be further configured to synchronize at least one of the position or orientation of the image capture device with the viewpoint of the 3D model of the target tissue.

[0016] In another aspect, this disclosure provides a method for registering a 3D model of a target tissue with a real-time image of the target tissue via a user input device of a surgical robotic system. The method includes generating or displaying a 3D model of the target tissue based on stored preoperative images of the target tissue; capturing at least one of an image or video of the target tissue in real time by an image capture device; activating a first mode to display the target tissue in real time, and treating the target tissue in response to a user input device; switching from the first mode to a second mode to display the 3D model of the target tissue; manipulating the 3D model of the target tissue via the user input device to register the 3D model of the target tissue with the target tissue in real time; and switching from the second mode to the first mode to further register the 3D model of the target tissue with the target tissue in real time.

[0017] In various aspects, manipulating a 3D model of a target tissue to register the 3D model of the target tissue with the target tissue in real time may include activating interactive control of the 3D model of the target tissue via a user input device.

[0018] In various aspects, interactive control of the 3D model of the target tissue may include adjusting at least one of the scale, position, orientation, or at least one heuristic factor of the 3D model of the target tissue.

[0019] In various aspects, switching from the first mode to the second mode may include maintaining the actual position of the user input device and sensing the pointing force on the user input device to determine the desired position of the user input device.

[0020] In various aspects, switching from the second mode to the first mode to further register the 3D model of the target tissue with the target tissue may include real-time analysis of the surface of the target tissue.

[0021] In various aspects, real-time analysis of the surface of a target tissue may include determining at least one of the tissue characteristics, mechanical properties, shape, reference marks, or heuristic factors of the target tissue in real time.

[0022] In all respects, determining at least one of the tissue properties, mechanical properties, shape, or reference marks of the target tissue in real time may include manipulating a 3D model of the target tissue based on at least one of the tissue properties, mechanical properties, shape, reference marks, or heuristic factors determined in real time.

[0023] In various aspects, the method may also include determining the completion of real-time registration between the 3D model of the target tissue and the target tissue; and, based on the completion of real-time registration between the 3D model of the target tissue and the target tissue, superimposing the 3D model of the target tissue onto the target tissue in real time.

[0024] In various respects, the method may also include synchronizing at least one of the position or orientation of the image capturing device with the viewpoint of the 3D model of the target tissue.

[0025] Details of one or more aspects of this disclosure are set forth in the following drawings and description. Other features, objectives, and advantages of the technology described in this disclosure will be apparent from the specification, drawings, and claims. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a surgical robotic system including a control tower, a console, and one or more surgical robotic arms, according to various aspects of this disclosure.

[0027] Figure 2 yes Figure 1 A perspective view of the surgical robotic arm of a surgical robotic system;

[0028] Figure 3 It has Figure 1 Perspective view of the setup of the surgical robotic arm in a surgical robotic system;

[0029] Figure 4 yes Figure 1 A schematic diagram of the computer architecture of a surgical robotic system, which includes a surgical console;

[0030] Figure 5 yes Figure 4 A schematic diagram of the surgical control console;

[0031] Figure 6 This is an exemplary view showing the target tissue at the surgical site in the first mode;

[0032] Figure 7 This is an exemplary view showing a 3D model of the target tissue in the second mode;

[0033] Figure 8 This is an exemplary view of a 3D model of the target tissue superimposed on a display of the target tissue at the surgical site; and

[0034] Figure 9 This is a flowchart of a method according to the present disclosure for registering a 3D model with target tissue at a surgical site through integrated control of 3D visualization. Detailed Implementation

[0035] The surgical robotic system of this disclosure is described in detail with reference to the accompanying drawings, wherein similar reference numerals in each of the several views represent the same or corresponding elements.

[0036] As will be described in detail below, this disclosure relates to a surgical robotic system including a surgical console, a control tower, and one or more movable trolleys having surgical robotic arms coupled to a setup arm. The surgical console is configured to select between a first mode and a second mode based on user input via a user input device, and to activate control of the robotic surgical arm or the display of the surgical console. The clinician controls the display of the robotic surgical arm or the surgical console.

[0037] The term "application" can include computer programs designed to perform functions, tasks, or activities for the benefit of clinicians. For example, an application can refer to software that runs as a standalone program or locally or remotely in a web browser, or other software that a person skilled in the art would understand as an application. Applications can run on controllers or user devices, including, for example, on mobile devices, IoT devices, or server systems.

[0038] As used herein, the term "network," whether singular or plural, refers to a data network, including but not limited to the Internet, intranet, wide area network, or local area network, and is not limited to the full scope of the definition of communication networks covered by this disclosure. Suitable protocols include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Datagram Protocol / Internet Protocol (UDP / IP), Ethernet / Ethernet, and / or Datagram Congestion Control Protocol (DCCP). Wireless communication may be implemented via one or more wireless configurations, such as radio frequencies, optics, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances from fixed and mobile devices using shortwave radio waves to create personal area networks (PANs)), etc. (A set of high-level communication protocol specifications using small, low-power digital radios based on the IEEE 802.15.4-2003 standard for Wireless Personal Area Networks (WPANs).)

[0039] refer to Figure 1 The surgical robot system 10 includes a control tower 20 connected to all components of the surgical robot system 10. The surgical robot system includes a surgical console 30 and one or more robotic arms 40. Each robotic arm 40 includes a surgical instrument 50 removably coupled thereto. Each robotic arm 40 is also coupled to a movable trolley 60.

[0040] Surgical instrument 50 is configured for use during minimally invasive surgical procedures. Alternatively, surgical instrument 50 may be configured for use in open surgical procedures. In various aspects, surgical instrument 50 may be an endoscope configured to provide video feed to a user, an electrosurgical clamp configured to seal tissue by compressing tissue between jaw members and applying an electrosurgical current thereto, or a surgical stapler comprising a pair of jaws configured to grasp and hold tissue while deploying multiple tissue fasteners (e.g., staples) and cutting sutures.

[0041] Each robotic arm in the robotic arms 40 may include an input capture device, or be configured to capture surgical site 310. Figure 6 The surgical console 30 includes a first display 32 and a second display device 34, the first display showing the surgical site 310 as seen by the camera 51 mounted on the robotic arm 40. Figure 6 The second display device displays a user interface for controlling the surgical robot system 10. The surgical console 30 also includes multiple user interface devices, such as a foot pedal 36 and a pair of handle controllers 38a and 38b used by the user to remotely control the robotic arm 40 (e.g., remote operation of the robotic arm 40 via the surgical console).

[0042] The control tower 20 includes a display 23, which may be a touchscreen and outputs on a graphical user interface (GUI). The control tower 20 also acts as an interface between the surgical console 30 and one or more robotic arms 40. Specifically, the control tower 20 is configured to control the robotic arms 40 to move the robotic arms 40 and corresponding surgical instruments 50, for example, based on a set of programmable instructions and / or input commands from the surgical console 30, such that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from the foot pedals 36 and handle controllers 38a and 38b. In some cases, the control tower 20 may not be included in the surgical robot system 10, but rather the control tower 20 of the surgical robot system 10 may be implemented in the surgical console 30 and / or the movable cart 60.

[0043] Each of the control tower 20, surgical console 30, and robotic arm 40 includes a corresponding computer 21, 31, or 41. Computers 21, 31, and 41 are interconnected using any suitable communication network based on wired or wireless communication protocols.

[0044] Computers 21, 31, and 41 may include a suitable processor (not shown) operatively connected to a memory (not shown), which may include one or more volatile, non-volatile, magnetic, optical, or electronic media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to perform the operations, calculations, and / or instruction sets described herein, including but not limited to hardware processors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs), microprocessors, and combinations thereof. Those skilled in the art will understand that the processor may be replaced by any logical processor (e.g., control circuitry) adapted to perform the algorithms, calculations, and / or instruction sets described herein.

[0045] refer to Figure 2 Each robotic arm 40 may include multiple connectors 42a, 42b, and 42c, which are interconnected at engagements 44a, 44b, and 44c, respectively. Engagement 44a is configured to secure the robotic arm 40 to a movable trolley 60 and define a first longitudinal axis. (Reference) Figure 3 The movable cart 60 includes a lift 61 and a mounting arm 62, which provides a base for mounting the robotic arm 40. The lift 61 allows the mounting arm 62 to move vertically. The movable cart 60 also includes a display 69 for displaying information related to the robotic arm 40.

[0046] The setup arm 62 includes a first connector 62a, a second connector 62b, and a third connector 62c, which provide lateral maneuverability of the robotic arm 40. Connectors 62a, 62b, and 62c are interconnected at joints 63a and 63b, each joint including an actuator (not shown) for rotating connectors 62b and 62b relative to each other and connector 62c. Specifically, connectors 62a, 62b, and 62c are movable in their respective lateral planes parallel to each other, thereby allowing the robotic arm 40 to extend relative to a patient (e.g., a surgical table). In some instances, the robotic arm 40 may be coupled to a surgical table (not shown). The setup arm 62 includes a controller 65 for adjusting the movement of connectors 62a, 62b, and 62c, as well as the lift 61.

[0047] The third connector 62c includes a rotatable base 64 with two degrees of freedom. Specifically, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first fixed arm axis perpendicular to the plane defined by the third connector 62c, and the second actuator 64b is rotatable about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for full three-dimensional orientation of the robot arm 40.

[0048] The robotic arm 40 also includes a plurality of manual control buttons 53 disposed on the instrument drive unit 52 and the setting arm 62, which can be used in manual mode. The user can press one or more of the buttons 53 to move the component associated with the button 53.

[0049] refer to Figure 2 The robotic arm 40 also includes a retainer 46 that defines a second longitudinal axis and is configured to receive the instrument drive unit 52 of the surgical instrument 50. Figure 1 The instrument drive unit 52 is configured to be coupled to the actuation mechanism of the surgical instrument 50. The instrument drive unit 52 transmits actuating force from its actuator to the surgical instrument 50 to actuate components of the surgical instrument 50 (e.g., end effectors). The retainer 46 includes a sliding mechanism 46a configured to move the instrument drive unit 52 along a second longitudinal axis defined by the retainer 46. The retainer 46 also includes an engagement portion 46b that allows the retainer 46 to rotate relative to the connector 42c.

[0050] The joints 44a and 44b include actuators 48a and 48b, which are configured to drive the joints 44a, 44b, and 44c relative to each other via a series of belts 45a and 45b or other mechanical links (such as drive rods, cables, or bars). Specifically, actuator 48a is configured to rotate the robot arm 40 about a longitudinal axis defined by connector 42a.

[0051] Actuator 48b of joint 44b is coupled to joint 44c via belt 45a, and joint 44c is further coupled to joint 46c via belt 45b. Joint 44c may include a transfer case connecting belts 45a and 45b, such that actuator 48b is configured to rotate each of connectors 42b, 42c relative to retainer 46. More specifically, connectors 42b, 42c and retainer 46 are passively coupled to actuator 48b, which forces rotation about a pivot point “P” located at the intersection of a first axis defined by connector 42a and a second axis defined by retainer 46. Thus, actuator 48b controls the angle θ between the first and second axes, thereby allowing the orientation of surgical instrument 50. Since the connectors 42a, 42b, 42c and the retainer 46 are interconnected via belts 45a and 45b, the angle between the connectors 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle θ. Some or all of the engagements 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0052] refer to Figure 4 Each of the computers 21, 31, and 41 of the surgical robot system 10 may include multiple controllers, which may be embodied in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety observer 21b. The controller 21a receives data from the computer 31 of the surgical console 30 regarding the current position and / or orientation of the handle controllers 38a and 38b, as well as the status of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each engagement of the robotic arm 40 and / or the instrument drive unit 52, and transmits these commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual engagement angle and uses this information to determine force feedback commands transmitted back to the computer 31 of the surgical console 30 to provide tactile feedback via the handle controllers 38a and 38b. The safety observer 21b performs validity checks on data entering and leaving the controller 21a, and if an error is detected in the data transmission, notifies the system fault processor to place the computer 21 and / or the surgical robot system 10 into a safe state.

[0053] Computer 41 includes multiple controllers: a main trolley controller 41a, a setup arm controller 41b, a robot arm controller 41c, and an instrument drive unit (IDU) controller 41d. The main trolley controller 41a receives and processes engagement commands from controller 21a of computer 21 and transmits these commands to the setup arm controller 41b, robot arm controller 41c, and IDU controller 41d. The main trolley controller 41a also manages instrument exchange and the overall status of the movable trolley 60, robot arm 40, and instrument drive unit 52. The main trolley controller 41a also transmits the actual engagement angle back to controller 21a.

[0054] The arm controller 41b controls each of the joints 63a and 63b, and sets the rotatable base 64 of the arm 62, calculating the desired motor movement command (e.g., motor torque) for the pitch axis and controlling the brakes. The robot arm controller 41c controls each of the joints 44a and 44b of the robot arm 40, calculating the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robot arm 40. The robot arm controller 41c calculates the movement command based on the calculated torque. The calculated motor command is then transmitted to one or more actuators 48a and 48b in the robot arm 40. The actual engagement position is then transmitted back to the robot arm controller 41c via actuators 48a and 48b.

[0055] The IDU controller 41d receives the desired engagement angle of the surgical instruments 50, such as the wrist and jaw angles, and calculates the desired current of the motor in the instrument drive unit 52. The IDU controller 41d calculates the actual angle based on the motor position and transmits the actual angle back to the trolley controller 41a.

[0056] The robotic arm 40 is controlled as follows. First, the pose of the handle controller (e.g., handle controller 38a) controlling the robotic arm 40 is converted into the desired pose of the robotic arm 40 via a hand-eye conversion function executed by controller 21a. The hand-eye function, as well as other functions described herein, are embodied in software that can be executed by controller 21a or any other suitable controller described herein. The pose of one of the handle controllers in handle controller 38a can be represented as a coordinate position and roll-pitch-yaw (“RPY”) orientation relative to a coordinate reference system fixed to the surgical console 30. The desired pose of the instrument 50 is relative to a fixed reference system on the robotic arm 40. The pose of the handle controller 38a is then scaled by a scaling function executed by controller 21a. In some instances, the coordinate position can be scaled down and the orientation can be scaled up by the scaling function. Additionally, controller 21a also executes a disengagement function that disengages the handle controller 38a from the robotic arm 40. Specifically, if certain movement limits or other thresholds are exceeded, the main trolley controller 21a stops transmitting movement commands from the handle controller 38a to the robot arm 40, and essentially acts as a virtual clutch mechanism, for example, limiting the mechanical input from affecting the mechanical output.

[0057] The desired pose of the robot arm 40 is based on the pose of the handle controller 38a and then transmitted via an inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, and 44c of the robot arm 40, which are scaled and adjusted by the pose input from the handle controller 38a. The calculated angles are then transmitted to the robot arm controller 41c, which includes a joint axis controller with a proportional-derivative (PD) controller, a friction estimator module, a gravity compensator module, and a double-sided saturation block. This joint axis controller is configured to limit the commanded torque of the motors of the joints 44a, 44b, and 44c.

[0058] Continue to refer to Figure 1 , Figure 4 and Figure 5The surgical console 30 also includes an image processor 100, a user interface controller 200, and a memory 300. The user interface controller 200 is configured to receive input commands from the surgical console 30 via handle controllers 38a and 38b and / or foot pedals 36, and to select a user interface mode. In some cases, the surgical robot system 10 may also include a microphone configured to receive voice commands as input commands from a clinician. User interface modes include a first interface mode 202, a second interface mode 204, and a third interface mode. In response to receiving an output command as an input command, the user interface controller 200 may return to a previous user interface mode. The user interface controller 200 may also deliver auditory or tactile feedback based on the input command as confirmation of the user interface mode and / or output command to the handle controllers 38a and 38b and / or foot pedals 36.

[0059] refer to Figure 6 The first mode 202 is configured to display a video feed of the surgical site 310 provided by the camera 51 on the first display 32, and to allow control of the robotic arm 40 via the handle controllers 38a and 38b and / or the foot pedal 36.

[0060] refer to Figure 7 The second mode 204 is configured to display a 3D model 320 (of the target tissue / organ) on the first display 32 and allow interactive control of the 3D model 320 via handheld controllers 38a and 38b and / or foot pedal 36 to align the 3D model with the video feed. Interactive control of the 3D model 320 includes adjusting the scale, position, and / or orientation of the 3D model to match the scale, position, and / or orientation of the target tissue (living or in vivo) at the surgical site 310 in the video feed.

[0061] During the second mode 204, the interface controller 200 is further configured to maintain the actual positions of the handle controllers 38a and 38b and / or the foot pedal 36 from the first interface mode 202, and sense the pointing force on the handle controllers 38a and 38b and / or the foot pedal 36 to determine the desired position, thereby allowing the handle controllers 38a and 38b to seamlessly transition between the first interface mode 202 and the second interface mode 204. The interface controller 200 may be further configured to retain the actual positions of the handle controllers 38a and 38b and / or the foot pedal 36 from the first interface mode 202, and allow further movement of the handle controllers 38a and 38b and / or the foot pedal 36. Therefore, when the interface controller 200 switches from the second interface mode 204 back to the first interface mode 202, the reserved actual positions of the handle controllers 38a and 38b and / or the foot pedal 36 are used to return the positions of the handle controllers 38a and 38b and / or the foot pedal 36 to the reserved actual positions before the handle controllers 38a and 38b and / or the foot pedal 36 move in the second interface mode 204.

[0062] The image processor 100 is configured to receive images of the target tissue at the surgical site 310 from the memory 300 based on preoperative diagnostic images (e.g., CT, MRI, fluorescence microscopy) of the target tissue taken at the surgical site 310. Figure 6 320 preoperative 3D model at the location Figure 7 The 3D model 320 can be reconstructed using any suitable method to generate different 3D model 320 renders, such as surfaces, voxels, and / or incisions. Other suitable methods for generating the 3D model 320 from preoperative diagnostic images are also envisioned. The image processor 100 can store the 3D model 320 and various renders of the 3D model 320 in the memory 300.

[0063] The image processor 100 is further configured to analyze the surface of the target tissue at the surgical site 310 by determining at least one of the tissue properties, mechanical properties, and / or shape of the target tissue at the surgical site 310. The tissue properties and mechanical properties may include, for example, airflow, tissue age, tissue thickness, gravity acting on the tissue, perfusion, vascular distribution, etc. Based on the determined tissue properties, mechanical properties, and / or the shape of the target tissue, the image processor 100 further manipulates the 3D model 320 to register the 3D model 320 with the target tissue at the surgical site 310.

[0064] In some cases, memory 300 may store a virtual container corresponding to a mathematical rule that replicates the effects of mechanical properties of the in vivo target tissue at the surgical site 310, and image processor 100 may apply the virtual container to 3D model 320 to assist in manipulating 3D model 320 consistent with the in vivo target tissue at the surgical site 310.

[0065] Additionally and / or alternatively, the image processor 100 may identify reference markers set on the in vivo target tissue and / or the 3D model 320 at the surgical site 310 to assist in the registration of the 3D model 320 with the in vivo target tissue at the surgical site 310 by matching the reference markers of the in vivo target tissue with the reference markers of the 3D model 320. The image processor 100 may be configured to generate additional 3D models based on in vivo video feeds to further assist in manipulating the 3D model 320 to register the 3D model 320 with the target tissue at the surgical site 310.

[0066] The image processor 100 is further configured to indicate the progress of the registration between the 3D model 320 and the target tissue at the surgical site 310 based on the proportional volume of the registration between the surface of the manipulated 3D model 320 and the target tissue at the surgical site 310.

[0067] refer to Figure 8 Once registration is complete, the interface controller 200 switches to a third interface mode, in which the image processor 100 overlays the target tissue at the surgical site 310 onto the 3D model 320 to create an enhanced view, and the clinician continues operation remotely via the surgical console through the robotic arm 40 using handle controllers 38a and 38b and / or foot pedals. The image processor 100 is configured to control the transparency of the 3D model 320 in the enhanced view. The transparency level of the 3D model 320 in the enhanced view is controlled via the handle controllers 38a and 38b. The image processor 100 is further configured to synchronize the viewpoint of the 3D model 320 with the position and / or orientation of the camera 51 at the in vivo target tissue at the surgical site 310 in the in vivo video feed. Therefore, when the position and / or orientation of the camera 51 is adjusted, for example, moved up or down, rotated, and / or zoomed in or out, the adjusted position and / or orientation is translated into the corresponding viewpoint of the 3D model 320 to maintain registration with the target tissue at the surgical site 310.

[0068] refer to Figure 9In operation, at step 700, the image processor 100 generates a 3D model 320 based on preoperative diagnostic images of the target tissue taken prior to surgery at the surgical site 310. At step 704, a camera 51 mounted on the robotic arm 40 captures images or videos of the target tissue in real time. At step 708, the surgical robot system 10 activates a first mode 202 to display the target tissue in real time and allow treatment of the target tissue in response to handle controllers 38a and 38b and / or foot pedals 36. While displaying video feeds of the surgical site 310 on the first display 32, the user interface controller 200 activates remote control of the robotic arm 40, for example, remote operation of the robotic arm 40 via foot pedals 36 and / or handle controllers 38a, 38b on the surgical console 30. The clinician remotely operates the robotic arm 40 to perform surgical procedures.

[0069] At step 712, the user interface controller 200 switches from a first mode 202 to a second mode 204 to display the 3D model 320. When switching from the first mode 202 to the second mode 204, the user interface controller 200 receives exit commands via handle controllers 38a and 38b and / or foot pedal 36, or in some cases, voice commands. The robotic surgical system 10 exits the first mode 202 and disables remote control of the robotic arm 40. In some cases, the user interface mode can switch to the second mode 204, the previous mode, or a non-selection mode. Once the second mode 204 is selected, the user interface controller 200 displays the 3D model 320 of the target tissue on the first display 32. While displaying the 3D model 320, the user interface controller 200 activates interactive control of the 3D model 320 on the first display 32 via handle controllers 38a and 38b and / or foot pedal 36 on the surgical console 30.

[0070] In step 714, the clinician manipulates the 3D model 320 via handle controllers 38a and 38b and / or foot pedal 36 to register the 3D model 320 with the target tissue. By allowing interactive control of the 3D model 320 via the foot pedal 36 and / or handle controllers 38a, 38b, the clinician can adjust at least one of the following: scale, position, and / or orientation of the 3D model 320 (e.g., by...). Figure 7 and Figure 8 (As indicated by the arrow). At step 718, the interface controller 200 switches from the second mode 204 to the first mode 202, and the image processor 100 analyzes the surface of the target tissue and determines at least one of the tissue characteristics, mechanical properties, shape, or reference marks of the target tissue.

[0071] Additionally, taking into account or incorporating heuristic computational model control, methods, and / or techniques, it is conceivable that the 3D model 320 can be generated and / or manipulated / controlled by foot pedals 36 and / or handle controllers 38a and 38b. Generally, heuristics are methods used to solve problems quickly, delivering sufficiently useful results within given time constraints.

[0072] As a non-limiting example, the generated 3D model 320 may be adjusted via handle controllers 38a and 38b, etc., to take into account at least some or all of the following factors (i.e., heuristic factors): patient age, patient sex, patient height, patient weight, patient orientation on the operating table, gravity effect based on patient orientation, time / duration spent in orientation, patient / organ hydration, organ density, patient / organ temperature, patient respiratory rate, blood pressure, patient body mass index (BMI), percentage of abdominal fat, and percentage of vascular distribution in the organ. Taking into account the heuristic factors, or during their generation, and / or while controlling the 3D model 320 via foot pedal 36 and / or handle controllers 38a and 38b, the numerical values ​​of each of these factors may be measured, observed, calculated, or otherwise obtained, and these values ​​may be added to the 3D model 320.

[0073] At step 722, based on at least one of the determined tissue characteristics, mechanical properties, shape, or reference markers of the target tissue, the 3D model 320 is further manipulated to register the 3D model 320 with the target tissue. At step 726, the image processor 100 compares the surface volume of the 3D model 320 with the surface volume of the target tissue to determine the progress of the registration of the 3D model 320 with the target tissue at the surgical site 310. When the real-time registration of the 3D model 320 with the target tissue is completed, at step 730, the 3D model 320 is superimposed on the in vivo target tissue at the surgical site 310 to create an enhanced view. Figure 8 While maintaining registration between the 3D model 320 and the in vivo target tissue at the surgical site 310, at least one of the position or orientation of the camera 51 is synchronized with the viewpoint of the 3D model 320. Furthermore, the surgical robot system 10 switches from a second mode 204 to a first mode 202 to display an enhanced view and activate remote control of the robotic arm 40, for example, remote operation of the robotic arm 40 via the foot pedal 36 and / or handle controllers 38a, 38b on the surgical console 30.

[0074] It should be understood that the various aspects disclosed herein can be combined in combinations different from those specifically given in the specification and drawings. It should also be understood that, depending on the example, certain actions or events of any process or method described herein may be performed in a different order, or may be completely added, combined, or omitted (e.g., performing the described technique may not require all the described actions or events). Furthermore, although for clarity some aspects of this disclosure are described as being performed by a single module or unit, it should be understood that the techniques of this disclosure can be performed by combinations of units or modules associated with, for example, a medical device.

[0075] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which correspond to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer).

[0076] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, as used herein, the term "processor" can refer to any of the foregoing structures or any other physical structures suitable for implementing the described techniques. Furthermore, this technique can be fully implemented in one or more circuit or logic elements.

Claims

1. A surgical robot system, the surgical robot system comprising: Control tower; A mobile trolley, connected to the control tower, includes a surgical robotic arm comprising: Surgical instruments, said surgical instruments being actuated in response to user input and configured to treat target tissue in real time; and An image capture device, the image capture device being configured to capture at least one of an image or video of the target tissue in real time; A surgical console, coupled to the control tower, includes: monitor; A memory configured to store preoperative images of the target tissue; User input device, the user input device being configured to generate the user input; and A controller operatively coupled to the display, the memory, and the user input device, the controller being configured to switch from a first mode to a second mode and from the second mode to the first mode based on the user input. The first mode includes displaying the target tissue in real time in at least one of an image or video on the display and activating control of the surgical robotic arm by the user input device, and the second mode includes displaying a 3D model of the target tissue and activating interactive control of the 3D model of the target tissue by the user input device, and registering the 3D model of the target tissue with the target tissue in at least one of the images or videos, wherein the controller is further configured to determine the completion of the real-time registration of the 3D model with the target tissue, and based on the completion of the real-time registration of the 3D model with the target tissue, to overlay the 3D model on the target tissue in real time to create an enhanced view, and to switch from the second mode to the first mode to display the enhanced view.

2. The surgical robot system of claim 1, wherein the surgical console further includes an image processor configured to generate the 3D model of the target tissue based on stored preoperative images of the target tissue.

3. The surgical robot system of claim 1, wherein the interactive control of the 3D model of the target tissue includes adjusting at least one of the scale, position, orientation, or at least one heuristic of the 3D model of the target tissue.

4. The surgical robot system of claim 1, wherein when switching from the first mode to the second mode, the controller is further configured to maintain or retain the actual position of the user input device from the first mode.

5. The surgical robot system of claim 4, wherein, while maintaining the actual position of the user input device from the first mode, the controller is further configured to determine the desired position of the user input device by sensing pointing forces on the user input device.

6. The surgical robot system of claim 1, wherein when switching from the second mode to the first mode, the controller is further configured to analyze the surface of the target tissue in real time.

7. The surgical robot system of claim 6, wherein when analyzing the surface of the target tissue in real time, the controller is further configured to determine at least one of the tissue characteristics, mechanical properties, shape, or reference marks of the target tissue.

8. The surgical robot system of claim 7, wherein the controller is further configured to manipulate the 3D model of the target tissue based on at least one of the tissue characteristics, mechanical properties, shape, reference marks, or heuristic factors of the target tissue determined in real time, so as to further register the 3D model of the target tissue with the target tissue in real time.

9. The surgical robot system of claim 1, wherein the controller is further configured to synchronize at least one of the position or orientation of the image capture device with the viewpoint of the 3D model of the target tissue.

Images