Surgical robotic systems and methods with automatic low visibility control

Abstract

The invention relates to a surgical robotic system and method with automatic low visibility control, and specifically discloses a surgical robotic system that includes an instrument drive unit coupled to and configured to actuate a surgical instrument and a laparoscopic camera. An image processing device is coupled to the laparoscopic camera and is configured to operate in a plurality of imaging modes having a low visibility mode. A surgeon console having a handle controller receives a user input for moving a surgical instrument. The controller operates the surgical instrument in a first mode of operation when the low visibility mode is active and in a second mode of operation when the low visibility mode is inactive.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 519,873, filed on August 16, 2023. The entire contents of the foregoing application are incorporated herein by reference. Background Art

[0003] Surgical robotic systems are currently being used for a variety of surgical procedures, including minimally invasive surgery. Some surgical robotic systems include a surgeon's console that controls a surgical robotic arm and a surgical instrument having an end effector (e.g., a clamp or grasping instrument) coupled to and actuated by the robotic arm. In operation, the robotic arm moves to a position above the patient and then guides the surgical instrument into a small incision via a surgical port or natural orifice in the patient to position the end effector at a working site in the patient's body. A laparoscopic camera (which is also held by one of the robotic arms) is inserted into the patient to image the surgical site.

[0004] Laparoscopic cameras can be operated in a variety of imaging modes, including conventional color or white light mode and fluorescence mode. In conventional white light mode, light in the visible spectrum is used to illuminate the surface of the tissue being viewed. The light reflected by the tissue passes through a suitable lens system and is incident on an image sensor built into or attached to the endoscope. The electrical signals from the image sensor are processed into a full-color video image that can be displayed on a video monitor or stored in memory.

[0005] In fluorescence mode, fluorescence excitation optically excites fluorophores in the tissue, which emit fluorescence at an emission wavelength, which is usually greater than the excitation wavelength. The fluorescence from the tissue passes through a suitable lens system and is incident on the image sensor. The electrical signal from the image sensor is processed into a fluorescence video image, which can be displayed on a video monitor alone or in combination with a color video image.

[0006] Fluorescence excitation wavelength and emission wavelength depend on the type of fluorophore being excited. In the case of exogenously applied fluorophores, the excitation wavelength band can be anywhere from ultraviolet (UV) to near infrared (NIR), while the emission wavelength band can be anywhere from visible light to NIR. For endogenous fluorophores of tissue, the excitation wavelength band and emission wavelength band are more limited (excitation wavelength from UV to the green part of the visible spectrum, emission wavelength from blue / green light to NIR). Fluorescence imaging can be used to identify blood vessels, cancer cells, and other tissue types. White light and fluorescence imaging modes can be combined in various ways. Camera manufacturers offer a variety of imaging modes to provide surgeons with additional insight into structures and tools used during laparoscopy or surgery. Certain modes result in lower visibility, during which instrument movement can lead to unexpected patient outcomes and inappropriate instrument use. Similarly, using energy in a limited visibility mode may also result in adverse reactions. Therefore, a system and method for controlling a surgical robot system in such a low visibility mode is needed. Summary of the invention

[0007] The present disclosure provides a surgical robot system, which includes an imaging system that can operate in multiple imaging modes. The robot system may include one or more robotic arms, each of which holds an instrument or laparoscopic camera of the imaging system. The imaging system is configured to obtain white images and NIR images of tissues using fluorophores from fluorescent dyes such as indocyanine green (ICG). The imaging system is configured to combine white light images with NIR images in an overlay mode, during which a conventional white light image is combined with NIR / ICG data to generate an overlay image. In overlay mode, the imaging system can be configured to display NIR images using visible light depending on user preferences and applications, for example, NIR / ICG data can be displayed as a green or blue overlay. In intensity map mode, the imaging system displays the intensity of the NIR / ICG signal using a color scale in the overlay image. In monochrome mode, the NIR / ICG signal is displayed in white alone on a black background to achieve maximum possible distinction.

[0008] In a mode that limits visibility of the lumen and / or instrument (e.g., monochrome mode), the surgical system prevents the user from moving the instrument(s), camera, or activating the instrument (e.g., delivering electrosurgical energy, ejecting staples, etc.). Additionally, the system prevents the user from entering a low-visibility mode when controlling the instrument / camera or when an instrument function (e.g., energy delivery or stapling) is selected.

[0009] Specific use cases include, but are not limited to, preventing energy selection while in limited visibility mode, preventing position control while in limited visibility mode, preventing entry into limited visibility mode when selecting energy, preventing entry into limited visibility mode when controlling an instrument / camera, and providing a notification to the user explaining the behavior if the user attempts to perform any of the above.

[0010] The surgical robotic system also provides additional risk reduction. If the above conditions are not met (e.g., a major system failure has occurred), a system inoperable error is reported and certain functions of the instrument or the entire system are disabled. The imaging system can also switch to a non-limited visibility mode (e.g., white imaging) when a user attempts to access energy delivery or enter position control while in limited visibility mode.

[0011] Specifically, in monochrome mode, the system blocks position control and energy selection. If a user attempts to select energy or enter position control while in monochrome mode, the system will ignore the user selection and provide the user with a notification that position control and / or energy delivery are not available in monochrome mode.

[0012] When position control or energy delivery is selected, the system prevents the user from selecting monochrome mode. Specifically, the monochrome mode option is unavailable during this time, e.g., the selection button is "greyed out" in the graphical user interface (GUI). If the user attempts to enter monochrome mode via physical input (e.g., tapping a pedal that controls the camera), the system will ignore the input and display a notification to the user.

[0013] The surgical robotic system also includes various inputs (e.g., foot pedal, GUI, etc.) that can be used to switch between limited visibility mode and high visibility mode without having to look away from the screen. This allows the user to quickly switch between limited visibility mode and high visibility mode to take advantage of limited visibility while also safely providing position control and energy delivery.

[0014] According to one embodiment of the present disclosure, a surgical robot system is disclosed. The surgical robot system includes an instrument drive unit, which is connected to and configured to actuate a surgical instrument and a laparoscopic camera. The system also includes an image processing device, which is connected to the laparoscopic camera and is configured to operate in multiple imaging modes with a low-visibility mode. The system further includes a surgeon's console, which has a handle controller configured to receive user input for moving the surgical instrument. The system further includes a controller, which is configured to operate the surgical instrument in a first operating mode when the low-visibility mode is active, and to operate the surgical instrument in a second operating mode when the low-visibility mode is not active.

[0015] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the laparoscopic camera is configured to capture white light images and near infrared (NIR) light images. The low visibility mode may be a monochrome NIR mode. The surgical robot system may also include a display screen configured to display a video captured by the laparoscopic camera and processed by the image processing device based on an imaging mode selected from the multiple imaging modes. When the surgical instrument is in the first operating mode, the controller may be configured to prevent activation of the surgical instrument. The activation of the surgical instrument may include supplying electrosurgical energy, ejecting a nail, or advancing a knife. The controller may be further configured to output a prompt indicating that activation of the surgical instrument is prevented. When in the first operating mode, the controller may be further configured to switch from the low visibility mode to the default visibility imaging mode in response to a user input for moving the surgical instrument.

[0016] According to another embodiment of the present disclosure, a method for controlling a surgical robotic system is disclosed. The method includes operating an image processing device coupled to a laparoscopic camera in a plurality of imaging modes, one of which is a low-visibility mode. The method also includes receiving user input for moving a surgical instrument at a surgeon's console having a handle controller. The method further includes controlling the surgical instrument in a first operating mode when the low-visibility mode is active. The method further includes controlling the surgical instrument in a second operating mode when the low-visibility mode is active.

[0017] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may include capturing white light images and near infrared (NIR) light images at a laparoscopic camera. The low visibility mode may be a monochrome NIR mode. The method may further include processing, by the image processing device, the video captured by the laparoscopic camera based on the imaging mode selected from the multiple imaging modes and displaying it on a display screen. The method may further include preventing activation of the surgical instrument when the surgical instrument is in a first operating mode. Activation of the surgical instrument may include supplying electrosurgical energy, ejecting nails, or advancing a knife. The method may also include displaying a prompt indicating that activation of the surgical instrument is blocked. The method may further include automatically switching from a low visibility mode to a default visibility imaging mode in response to a user input for moving the surgical instrument when the surgical instrument is in the first operating mode.

[0018] According to another embodiment of the present disclosure, a method for controlling a surgical robotic system is disclosed. The method includes selecting an imaging mode for an image processing device from a plurality of imaging modes, including a low-visibility mode. The method also includes capturing an image using a laparoscopic camera and processing the image using the selected imaging mode at the image processing device. The method further includes controlling a surgical robotic instrument in a first operating mode when the low-visibility mode is active, during which the surgical robotic instrument is prevented from being activated. The method further includes controlling a surgical robotic instrument in a second operating mode when the low-visibility mode is not active, during which the surgical robotic instrument is able to move and can be activated to perform one or more functions.

[0019] Implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may also include receiving user input for moving the surgical robotic instrument at a surgeon's console (which may include a handle controller). The low visibility mode may be a monochrome NIR mode. The function may be to supply electrosurgical energy, eject nails, or advance the knife. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0021] Figure 1 is a perspective view of a surgical robot system according to an embodiment of the present disclosure, the surgical robot system comprising a control tower, a control console, and one or more surgical robot arms, each of the one or more surgical robot arms being disposed on a mobile cart;

[0022] Figure 2 According to the embodiments of the present disclosure Figure 1 A three-dimensional diagram of a surgical robot arm of a surgical robot system;

[0023] Figure 3 is a perspective view of a mobile cart having a mounting arm with a mounting arm according to an embodiment of the present disclosure Figure 1 The surgical robot arm of the surgical robot system;

[0024] Figure 4 According to the embodiments of the present disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system;

[0025] Figure 5 According to the embodiments of the present disclosure Figure 1 A schematic plan view of a surgical robot system, the surgical robot system being positioned around an operating table;

[0026] Figure 6is a schematic diagram of a system for determining stages of a surgical procedure according to an embodiment of the present disclosure;

[0027] Figure 7 is a perspective view of an imaging system according to an embodiment of the present disclosure;

[0028] Figure 8 is a screenshot of a graphical user interface (GUI) for selecting an imaging mode according to an embodiment of the present disclosure;

[0029] Fig. 9 is a screenshot of a surgeon's display screen in monochrome imaging mode according to an embodiment of the present disclosure;

[0030] Fig.10 is a screenshot of a surgeon's display screen in color imaging mode according to an embodiment of the present disclosure; and

[0031] Fig.11 According to an embodiment of the present disclosure, Figure 1 Flow chart of a method for automatic low-visibility control of a surgical robotic system. DETAILED DESCRIPTION

[0032] Embodiments of the surgical robotic system disclosed herein are described in detail with reference to the accompanying drawings, in which like reference numerals represent like or corresponding elements in each of the several views. As used herein, the term "coupled to" refers to a connection between components, which may be direct or indirect (i.e., through one or more components), and may be electronic, electrical, mechanical, or a combination thereof.

[0033] Reference Figure 1 , the surgical robotic system 10 includes a control tower 20 that is connected to all components of the surgical robotic system 10, including a surgeon's console 30 and one or more mobile carts 60. Each mobile cart 60 includes a robotic arm 40 to which a surgical instrument 50 is removably coupled. The robotic arm 40 is also coupled to the mobile cart 60. The robotic system 10 may include any number of mobile carts 60 and / or robotic arms 40.

[0034] The surgical instrument 50 is configured to be used during minimally invasive surgery. In an embodiment, the surgical instrument 50 can be configured for open surgery. In another embodiment, the surgical instrument 50 can be an electrosurgical or ultrasonic instrument (such as a clamp) configured to seal the tissue by pressing the tissue between the jaw members and applying an electrosurgical current to the tissue or ultrasonic vibration via an ultrasonic transducer. In another embodiment, the surgical instrument 50 can be a surgical stapler, which includes a pair of jaws configured to grasp and clamp the tissue, deploy multiple tissue fasteners (e.g., suture nails) at the same time, and cut the stapled tissue. In another embodiment, the surgical instrument 50 can be a surgical clip applicator, which includes a pair of jaws configured to apply a surgical clip to the tissue. The system also includes an electrosurgical generator 57, which is configured to output electrosurgery (e.g., monopolar or bipolar) or ultrasonic energy in various operating modes such as coagulation, cutting, and sealing. Suitable generators include the Valleylab, available from Medtronic, Minneapolis, Minnesota TM FT10 Energy Platform.

[0035] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture a video of the surgical site. The laparoscopic camera 51 may be a stereo camera configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be disposed within the control tower 20. The image processing device 56 may be any computing device configured to receive a video feed from the laparoscopic camera 51 and output a processed video stream.

[0036] The surgeon console 30 includes a first (i.e., surgeon) screen 32 that displays a video feed of the surgical site provided by a camera 51 of a surgical instrument 50 disposed on the robotic arm 40, and a second screen 34 that displays a user interface for controlling the surgical robotic system 10. The first screen 32 and the second screen 34 may be touch screens that allow various graphical user inputs to be displayed.

[0037] The surgeon console 30 also includes a plurality of user interface devices, such as a foot pedal 36 and a pair of hand controls 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgeon console further includes an armrest 33 for supporting the clinician's arms while operating the hand controls 38a and 38b.

[0038] The control tower 20 includes a screen 23 (which may be a touch screen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon's console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as moving the robotic arms 40 and corresponding surgical instruments 50 based on a set of programmable instructions and / or input commands from the surgeon's console 30, in a manner such that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from the foot pedals 36 and hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, reposition camera movement, and electrosurgical activation / deactivation. In particular, the foot pedals 36 may be used to perform clutch actions on the hand controllers 38a and 38b. The clutch is activated by pressing one of the foot pedals 36, which disconnects the hand controllers 38a and / or 38b from the robotic arm 40 and the corresponding instrument 50 or camera 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and instruments 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0039] Each of the control tower 20, the surgeon's console 30, and the robotic arm 40 includes a corresponding computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other using any suitable communication network based on wired or wireless communication protocols. The term "network" as used herein, whether plural or singular, means a data network, including but not limited to the Internet, an intranet, a wide area network, or a local area network, and is not limited to the full scope of the definition of a communication network covered by the present disclosure. Suitable protocols include, but are not limited to, the Transmission Control Protocol / Internet Protocol (TCP / IP), the Datagram Protocol / Internet Protocol (UDP / IP), and / or the Datagram Congestion Control Protocol (DC). Wireless communication can be achieved via one or more wireless configurations, such as radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data from fixed and mobile devices over short distances using short-length radio waves to create a personal area network (PAN)), (Specification of a suite of advanced communications protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for Wireless Personal Area Networks (WPANs).

[0040] Computer 21,31,41 may include any suitable processor (not shown), which is operably connected to a memory (not shown), which may include one or more of a volatile, nonvolatile, magnetic, optical or electrical medium, such as a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM (EEPROM), a nonvolatile RAM (NVRAM) or a flash memory. The processor may be any suitable processor (e.g., a control circuit) suitable for performing the operations, calculations and / or instruction sets described in the present disclosure, including but not limited to a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor and a combination thereof. It will be appreciated by those skilled in the art that a processor may be replaced by using any logical processor (e.g., a control circuit) suitable for performing the algorithms, calculations and / or instruction sets described herein.

[0041] refer to Figure 2 Each robot arm 40 may include a plurality of links 42a, 42b, 42c that are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be used as known to those skilled in the art. Joint 44a is configured to secure the robot arm 40 to the mobile cart 60 and defines a first longitudinal axis. Figure 3 , the mobile cart 60 includes a lift 67 and a mounting arm 61 that provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The mobile cart 60 also includes a screen 69 that is used to display information about the robotic arm 40. In an embodiment, the robotic arm 40 may include any type and / or number of joints.

[0042] The installation arm 61 includes a first link 62a, a second link 62b and a third link 62c, which provide lateral maneuverability of the robot arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, 62c can move in their corresponding lateral planes parallel to each other, thereby allowing the robot arm 40 to extend relative to the patient (e.g., an operating table). In an embodiment, the robot arm 40 can be connected to an operating table (not shown). The installation arm 61 includes a control device 65, which is used to adjust the movement of the links 62a, 62b, 62c and the elevator 67. In an embodiment, the installation arm 61 can include any type and / or number of joints.

[0043] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a can rotate around a first fixed arm axis perpendicular to a plane defined by the third link 62c, while the second actuator 64b can rotate around a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow the robot 40 to achieve full three-dimensional orientation.

[0044] The actuator 48b of the joint 44b is connected to the joint 44c via the belt 45a, and the joint 44c is further connected to the joint 46b via the belt 45b. The joint 44c can include a transfer case connecting the belts 45a and 45b, so that the actuator 48b is configured to rotate each of the links 42b, 42c and the holder 46 relative to each other. More specifically, the links 42b, 42c and the holder 46 are passively connected to the actuator 48b, which is forced to rotate around the pivot point "P", which is located at the intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point "P" is the remote center of motion (RCM) of the robot 40. Thus, the actuator 48b controls the angle θ between the first axis and the second axis, thereby allowing the surgical instrument 50 to be oriented. Due to the interconnection of the links 42a, 42b, 42c and the retainer 46 via the straps 45a and 45b, the angle between the links 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle θ. In an embodiment, some or all of the joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0045] The joints 44a and 44b include actuators 48a and 48b that are configured to drive the joints 44a, 44b, 44c relative to each other through a series of belts 45a and 45b or other mechanical linkages such as drive rods, cables or levers. In particular, the actuator 48a is configured to rotate the robotic arm 40 about the longitudinal axis defined by the link 42a.

[0046] refer to Figure 2 The holder 46 defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1). The IDU 52 is configured to be coupled to the actuation mechanisms of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. The IDU 52 transmits the actuation force from its actuator to the surgical instrument 50 to actuate components of the end effector 49 of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a that is configured to move the IDU 52 along a second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b that rotates the holder 46 relative to the connecting rod 42c. During laparoscopic surgery, the instrument 50 can be inserted through a laparoscopic access port 55 ( Figure 3 ) is inserted. The holder 46 also includes a port lock 46c ( Figure 2 ).

[0047] The robot arm 40 also includes a plurality of manual override buttons 53 ( Figure 1 ), these manual override buttons can be used in manual mode. The user can press one or more of these buttons 53 to move the component associated with the button 53.

[0048] refer to Figure 4 , each of the computers 21, 31, 41 of the surgical robot system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety viewer 21b. The controller 21a receives data from the computer 31 of the surgeon's console 30 about the current position and / or orientation of the hand controllers 38a and 38b and the state of the foot pedal 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint and / or IDU 52 of the robotic arm 40, and transmits these commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback through the hand controllers 38a and 38b. The safety observer 21b performs validity checks on data entering and exiting the controller 21a and, if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or surgical robot system 10 into a safe state.

[0049] The controller 21a is coupled to a memory 22a, which may be a non-transitory computer readable medium configured to store any suitable computer data, such as software instructions executable by the controller 21a. The controller 21a also includes a transient memory 22b for loading instructions and other computer readable data during execution of the instructions. In an embodiment, other controllers of the system 10 include similar configurations.

[0050] The computer 41 includes a plurality of controllers, namely, a cart master controller 41a, a placement arm controller 41b, a robot arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart master controller 41a receives and processes joint commands from the controller 21a of the computer 21, and transmits them to the placement arm controller 41b, the robot arm controller 41c, and the IDU controller 41d. The master cart controller 41a also manages instrument exchange and moves the overall status of the cart 60, the robot arm 40, and the IDU 52. The cart master controller 41a also transmits the actual joint angles back to the controller 21a.

[0051] Each of the joints 63a and 63b and the rotatable base 64 of the installation arm 61 is a passive joint (i.e., there is no actuator therein) that allows manual adjustment by the user. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to construct the installation arm 61. When the brake is engaged, the installation arm controller 41b monitors the sliding of each of the joints 63a and 63b and the rotatable base 64 of the installation arm 61, or when the brake is disengaged, the installation arm controller can be freely moved by the operator without affecting the control of other joints. The robot arm controller 41c controls each joint 44a and 44b of the robot arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robot arm 40. The robot arm controller 41c calculates a movement command based on the calculated torque. The calculated motor command is then transmitted to one or more of the actuators 48a and 48b in the robot arm 40. The actual joint positions are then transmitted by actuators 48a and 48b back to the robot controller 41c.

[0052] The IDU controller 41d receives the desired joint angles (such as wrist angle and jaw angle) of the surgical instrument 50 and calculates the desired current for the motors in the IDU 52. The IDU controller 41d calculates the actual angle based on the motor position and transmits the actual angle back to the cart main controller 41a.

[0053] Reference Figure 5, the surgical robotic system 10 is set up around an operating table 90. The system 10 includes mobile carts 60a to 60d, which may be numbered "1" to "4". During set up, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depends on a number of factors, such as the placement of a number of access ports 55a-d, which in turn depends on the surgery being performed. Once the port placement is determined, the access ports 55a to 55d are inserted into the patient and the carts 60a to 60d are positioned to insert the instruments 50 and laparoscopic camera 51 into the corresponding ports 55a to 55d.

[0054] During use, each of the mechanical arms 40a-d engages the latch 46c ( Figure 2 ) is attached to the entry port 55 ( Figure 3 ) is attached to one of the access ports 55a-d inserted into the patient's body. The IDU 52 is attached to the holder 46, and then the SIM 43 is attached to the distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts that are configured to transmit the rotation of each motor of the IDU 52 to the instrument 50, thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and other components of the robotic arm 40 (including the IDU 52). The SIM 43 is also configured to fix a sterile sheet (not shown) to the IDU 52.

[0055] A surgical procedure may include multiple stages, and each stage may include one or more surgical actions. As used herein, the term "stage" refers to a surgical event (e.g., closure) consisting of a series of steps. A "surgical action" may include incision, compression, anastomosis, clamping, suturing, cauterization, sealing, or any other such action performed to complete a stage in a surgical procedure. A "step" refers to completing a specified surgical goal (e.g., hemostasis). During each step, certain surgical instruments 50 (e.g., clamps) are used to achieve a specific goal by performing one or more surgical actions.

[0056] refer to Figure 6, the surgical robot system 10 may include a machine learning (ML) processing system 310 that processes surgical data using one or more ML models to identify one or more features in the surgical data, such as surgical stages, instruments, anatomical structures, etc. The ML processing system 310 includes an ML training system 325, which may be a separate device (e.g., a server) that stores its output as one or more trained ML models 330. The ML models 330 may be accessed by an ML execution system 340. The ML execution system 340 may be separate from the ML training system 325, i.e., the device that “trains” the model is separate from the device that “infers”, i.e., uses the trained ML model 330 to process the surgical data in real time.

[0057] System 10 includes a data receiving system 305 that collects surgical data, including video data and surgical instrument data. Data receiving system 305 may include one or more devices (e.g., one or more user devices and / or servers) located in and / or associated with a surgical operating room and / or a control center. Data receiving system 305 may receive surgical data in real time, i.e., receive surgical data while a surgical procedure is being performed.

[0058] In some examples, the ML processing system 310 may further include a data generator 315 for generating simulated surgical data (such as a virtual image set) or recording video data from the image processing device 56 to train the ML model 330, as well as other data sources, such as user input, arm movement, etc. The data generator 315 may access (read / write) a data storage 320 to record data, including multiple images and / or multiple videos.

[0059] The ML processing system 310 also includes a phase detector 350 that uses the ML model to identify phases within a surgical procedure. The phase detector 350 uses a specific surgical tracking data structure 355 from the list of surgical tracking data structures. The phase detector 350 selects the surgical tracking data structure 355 based on the type of surgical procedure being performed. In one or more examples, the type of surgical procedure is predetermined or input by a user. The surgical tracking data structure 355 identifies a set of potential phases that may correspond to a portion of a particular type of surgical procedure.

[0060] In some examples, the surgical tracking data structure 355 can be a graph including a set of nodes and a set of edges, where each node corresponds to a potential stage. These edges can provide directional connections between nodes, which indicate (via directions) the expected order of the stages to be encountered throughout the surgical iteration. The surgical tracking data structure 355 may include one or more branch nodes that feed into multiple next nodes and / or may include one or more divergence points and / or convergence points between nodes. In some instances, the stage indicates a surgical action (e.g., surgical action) being performed or already performed and / or indicates a combination of actions that have been performed. In some instances, the stage is related to the biological state of the patient undergoing surgery. For example, the biological state may indicate complications (e.g., blood clots, arterial / venous obstruction, etc.), prerequisites (e.g., lesions, polyps, etc.). In some examples, the ML model 330 is trained to detect "abnormal conditions" such as bleeding, arrhythmias, vascular abnormalities, etc.

[0061] The stage detector 350 outputs a stage prediction associated with the portion of the video data analyzed by the ML processing system 310. The stage prediction is associated with the portion of the video data analyzed by the ML execution system 340 by identifying the start time and the end time of the video portion. The output stage prediction may include an identification of the surgical stage as detected by the stage detector 350 based on the output of the ML execution system 340. Further, in one or more examples, the stage prediction may include an identification of a structure (e.g., an instrument, an anatomical structure, etc.) identified by the ML execution system 340 in the analyzed video portion. The stage prediction may also include a confidence score for the prediction. Other examples may include various other types of information in the output stage prediction. The controller 21a may use the predicted stage to determine when to switch between various visibility modes, as described below.

[0062] refer to Figure 7, the surgical robot system 10 also includes an imaging system 400, wherein the laparoscope camera 51 is connected to the image processing device 56. The laparoscope camera 51 includes a laparoscope 402 having a longitudinal axis 414 with a plurality of optical components (not shown), such as lenses, reflectors, prisms, etc., disposed in the longitudinal axis 414. The laparoscope 402 is connected to a combined light source 406 via an optical cable 408. The light source 406 may include a white light source (not shown) and a NIR light source (not shown), which may be a light emitting diode or any other suitable light source. The NIR light source may be a laser or any other suitable light source. The optical cable 408 may include one or more optical fibers for transmitting white light and NIR light, which illuminate the tissue under observation by the laparoscope 402. The laparoscope 402 collects the reflected white light and NIR light and transmits the reflected light to a camera assembly 410, which is connected to the proximal portion of the laparoscope 402. Laparoscope 402 may be any conventional laparoscope configured to transmit and collect white light and NIR light.

[0063] The camera assembly 410 includes a sensor configured to separate the fluorescent wavelengths from the undesired components of the spectrum to specific sensors. In particular, the camera assembly includes a white light (e.g., visible light) (VIS) sensor and an IR sensor, and is configured to separate the white light and transmit it to the VIS sensor and separate the fluorescent IR light and transmit it to the IR sensor. The VIS sensor and the IR sensor can be complementary metal oxide semiconductor (CMOS) image sensors with any desired resolution (which can be 4K, UHD, etc. in embodiments).

[0064] The camera assembly 410 is connected to the image processing device 56 via a transmission cable 412. The image processing device 56 is configured to receive the image data signal, process the raw image data from the camera assembly 410, and generate a mixed white light and NIR image for recording and / or real-time display. The image processing device 56 also processes the image data signal and displays it through a DISPLAYPORT TM , Any suitable video output port outputs these image data signals to any one of the display screens 23, 32, 34 of the surgical robot system 10, and the video output port can transmit the processed images at any desired resolution, display rate and / or bandwidth.

[0065] Figure 8A GUI 500 for controlling the imaging system 400 is shown, which can be displayed on the display screens 23, 32, 34 of the surgical robot system 10. The GUI 500 includes options for controlling fluorescence settings, including turning fluorescence (e.g., NIR detection) on or off via a toggle key 502. Once fluorescence is selected, the user can also select from a variety of imaging modes that visualize NIR light. Overlay mode can be selected via button 504. In overlay mode, the imaging system 400 combines white light images with NIR images, during which conventional white light images are combined with NIR / ICG data to generate an overlay image. In overlay mode, the imaging system can be configured to display NIR light in visible light depending on user preferences and applications. NIR / ICG data can be displayed as a green or blue overlay, which can be selected via menu 505. In intensity map mode, which can be selected via button 506, the imaging system displays the intensity of the NIR / ICG signal using a color scale in the overlay image. In monochrome mode, selectable via button 508, the NIR / ICG signal is displayed alone in white on a black background to allow for the greatest possible differentiation, such as Fig. 9 In an embodiment, the mode may also be selected via one or more foot pedals 36 associated with mode selection, for example, one foot pedal cycles through each of the NIR modes. Fig.10 A white light video feed from the camera 51 is shown displayed on the first screen 32 along with an indicator that the imaging system 400 is switching between imaging modes.

[0066] The imaging system 400 may also provide a marking capability to allow the user to mark landmarks on the video so that the surgeon can identify the landmark using limited visibility mode, switch to high visibility mode to move the instrument and deliver energy, and use the marker from limited visibility mode. If the camera moves, the marker can be removed from the screen so that the marker is not applied to a different part of the interior. Alternatively, an artificial intelligence or machine learning (AI / ML) computer vision algorithm may be used to identify the landmark and allow the landmark to transition with the camera.

[0067] The user or the surgical robotic system can generate so-called "bookmarks" for one or more camera positions, which can then be selected by the user to return to any of the bookmarks. The imaging system also allows the user to take screenshots in the limited visibility view so that the limited visibility view can be compared side-by-side on one or more displays of the surgeon's console when the user is in the high visibility view.

[0068] Fig.11 A method of operating the surgical robotic system 10 in conjunction with the imaging system 400 is shown, wherein the selected imaging mode is affected by the operating mode of the robotic arms 40a to 40d, and vice versa. Fig.11The method may be embodied as software instructions executable by any controller of the system 10 (e.g., the main controller 21a). At step 600, the controller 21a determines whether the system 10 is in position control and / or whether the instrument 50 is activated. As used herein, position control includes any movement command input on the handle controllers 38a and 38b or other input device, which is used to move the robotic arms 40a to 40d and their attachments (e.g., the instrument 50, the camera 51, etc.). Activation of the instrument 50 is different from movement, and the activation includes enabling or switching electrosurgical energy delivery or another function of the instrument 50, for example, for ejecting nails, advancing the knife to cut tissue, etc. If the system 10 is in any of these operating modes, the imaging system 400 is in any high imaging mode, which can be a white light mode or any of the above-mentioned combined white light / NIR modes, for example, an overlay mode or a thermal map mode.

[0069] If the system 10 is not in either of these two operating modes, the controller 21a enables imaging mode selection at step 602. At step 604, the user can select a desired imaging mode (including a low visibility mode, such as a monochrome mode) to observe the surgical site. In an embodiment, the imaging mode selection can be done automatically based on the particular stage of the surgical procedure provided by the stage detector 350.

[0070] Once in the low-visibility mode, the controller 21a operates in a first operating mode and performs a verification of the user input. The first verification is completed at step 606, during which the controller 21a determines whether the user attempts to activate the instrument 50 by monitoring the user input, for example, pressing a button on the handle controller 38a and 38b or pressing one of the foot pedals 36 to energize the instrument 50 via the generator 57. While the low-visibility mode is active, if an activation input is detected, then at step 608, the controller 21a ignores the command. At step 610, the controller 21a also provides an indication of the input just received that was not executed due to being in the current imaging mode. The indication can be a prompt on the GUI 500 displayed on any of the display screens 23, 32, 34.

[0071] In addition to limiting the activation of the instrument 50, the controller 21a also reacts to movement input for position control of the instrument 50 and / or the camera 51. At step 612, when the low-visibility mode is active, the controller 21a determines whether the user attempts to move the instrument 50 and / or the camera 51 by monitoring user input (e.g., movement of the handle controllers 38a and 38b). If no movement is detected, the controller 21a places the imaging system 400 in the selected low-visibility mode (e.g., monochrome imaging mode) and returns to step 604. If movement is detected, at step 603, the controller 21a automatically exits the low-visibility mode and enters a default or normal imaging mode, which can be a white light mode or any of the above-mentioned combined white light / NIR modes, such as an overlay mode or a thermal map mode. The user can also choose to switch to one of these imaging modes. In an embodiment, the switch can be automatically performed based on the detected stage of the surgical procedure. When the default imaging mode is active, the controller 21 a operates the surgical robotic system 10 in the second (ie, normal) operating mode without restricting user input regarding activation or movement of the instrument 50 and / or the camera 51 .

[0072] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be interpreted as limiting, but merely as an illustration of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A surgical robot system, comprising: Device drive unit; A surgical instrument actuated by the instrument drive unit; Laparoscopic camera; an image processing device coupled to the laparoscopic camera and capable of operating in a plurality of imaging modes including a low visibility mode; a surgeon console including a handle controller that receives user input for moving the surgical instrument; as well as A controller operates the surgical instrument in a first mode of operation when the low-visibility mode is active and in a second mode of operation when the low-visibility mode is not active.

2. The surgical robot system according to claim 1, wherein: The laparoscopic camera is configured to capture white light images and near infrared (NIR) light images.

3. The surgical robot system according to claim 2, wherein: The low visibility mode is a monochromatic NIR mode.

4. The surgical robot system according to claim 1, further comprising: A display screen is configured to display a video captured by the laparoscopic camera and processed by the image processing device based on an imaging mode selected from the plurality of imaging modes.

5. The surgical robot system according to claim 4, wherein: When the surgical instrument is in the first operating mode, the controller is configured to prevent activation of the surgical instrument.

6. The surgical robot system according to claim 5, wherein: Activation of the surgical instrument includes at least one of supplying electrosurgical energy, expelling a staple, or advancing a blade.

7. The surgical robot system according to claim 5, wherein: The controller is further configured to output a prompt indicating that activation of the surgical instrument is prevented.

8. The surgical robot system according to claim 1, wherein: When the surgical instrument is in the first operating mode, the controller is configured to switch from the low-visibility mode to a default visibility imaging mode in response to a user input for moving the surgical instrument.

9. A method for controlling a surgical robot system, the method comprising: operating an image processing device coupled to a laparoscopic camera in a plurality of imaging modes including a low visibility mode; receiving user input for moving a surgical instrument at a surgeon's console including a handle controller; controlling the surgical instrument in a first operating mode while the low-visibility mode is active; as well as The surgical instrument is controlled in a second operating mode when the low-visibility mode is inactive.

10. The method according to claim 9, further comprising: White light images and near infrared (NIR) light images are captured at the laparoscopic camera.

11. The method according to claim 10, wherein: The low visibility mode is a monochromatic NIR mode.

12. The method according to claim 9, further comprising: Processing, by the image processing device, the video captured by the laparoscopic camera based on the imaging mode selected from the plurality of imaging modes; as well as The processed video is displayed on a display screen.

13. The method according to claim 12, further comprising: When the surgical instrument is in the first operating mode, activation of the surgical instrument is prevented.

14. The method according to claim 13, wherein: Activation of the surgical instrument includes at least one of supplying electrosurgical energy, expelling a staple, or advancing a blade.

15. The method according to claim 13, further comprising: A prompt is displayed indicating that activation of the surgical instrument is blocked.

16. The method according to claim 9, further comprising: Automatically switching from the low-visibility mode to a default visibility imaging mode in response to the user input for moving the surgical instrument when the surgical instrument is in the first operating mode.

17. A method for controlling a surgical robot system, the method comprising: Images were captured using a laparoscopic camera; processing the images using a low visibility mode at an image processing device; controlling the surgical robotic instrument in a first operating mode while the low-visibility mode is active, the surgical robotic instrument being prevented from being activated during the first operating mode; as well as The surgical robotic instrument is controlled in a second operating mode while the low-visibility mode is inactive, during which the surgical robotic instrument is movable and activatable to perform at least one function.

18. The method according to claim 17, further comprising: User input for moving the surgical robotic instrument at a surgeon's console including a handle controller is received.

19. The method according to claim 17, further comprising: White light images and near infrared (NIR) light images are captured at the laparoscopic camera, wherein the low visibility mode is a monochromatic NIR mode.

20. The method according to claim 17, wherein: The at least one function is selected from the group consisting of supplying electrosurgical energy, expelling staples, or advancing a blade.

Images