Surgical robotic systems and methods for instrument tracking and visualization in near-infrared imaging mode

Abstract

A surgical robotic system, the surgical robotic system comprising: a surgeon console having a handle controller configured to receive a user input; and a robotic arm having an instrument drive unit coupled to the surgical instrument and configured to actuate the surgical instrument in response to a user input. The system also includes a laparoscopic camera configured to capture a video feed of the surgical instrument; and an image processing device coupled to the laparoscopic camera and configured to operate in a plurality of imaging modes, including a low visibility imaging mode. The system additionally includes a controller configured to process a video feed of the surgical instrument and generate an overlay of the surgical instrument based on the processed video feed; and a screen configured to display the video feed in a low visibility imaging mode in which the overlay layer is on a portion of the video feed that may include the surgical instrument.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 520,117, filed August 17, 2023, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Surgical robotic systems are currently used in a variety of surgical procedures, including minimally invasive surgery. Some surgical robotic systems include a surgeon's console that controls a robotic arm and surgical instruments with end effectors (e.g., clamps or gripping instruments) that are coupled to and actuated by the robotic arm. During operation, the robotic arm moves to a position above the patient and then guides the surgical instruments through a small incision via the patient's surgical port or natural orifice to position the end effector at the work site within the patient's body. A laparoscopic camera (also held by one of the robotic arms) is inserted into the patient to image the surgical site.

[0003] Laparoscopic cameras can operate in various imaging modes, including conventional color or white light modes and fluorescence modes. In conventional white light mode, the observed tissue surface is illuminated using light within the visible spectrum. Light reflected from the tissue passes through a suitable lens system and is incident on an image sensor built into or attached to the endoscope. The electrical signals from the image sensor are processed into a full-color video image, which can be displayed on a video monitor or stored in memory.

[0004] In fluorescence mode, fluorescence excitation photoexcites fluorophores in the tissue that emit fluorescence at an emission wavelength, typically longer than the excitation wavelength. The fluorescence from the tissue passes through a suitable lens system and is incident on an image sensor. The electrical signal from the image sensor is processed into a fluorescence video image, which can be displayed alone on a video monitor or in combination with a color video image.

[0005] The excitation and emission wavelengths of fluorescence depend on the type of fluorophore being excited. In the case of exogenous fluorophores, the excitation wavelength band can lie anywhere in the ultraviolet (UV) to near-infrared (NIR) range, while the emission wavelength band can lie anywhere in the visible to NIR range. For endogenous fluorophores within tissue, the excitation and emission wavelength bands are more limited (excitation wavelengths from UV to the green portion of the visible spectrum, emission wavelengths from blue / green to NIR). Fluorescence imaging can be used to identify blood vessels, cancer cells, and other tissue types. White light and fluorescence imaging modes can be combined in various ways. Camera manufacturers offer a variety of imaging modes to provide surgeons with additional insight into structures and instruments used during laparoscopy or surgical procedures. Some modes that enhance NIR light result in low visibility of non-fluorescent objects, such as instruments. Because instruments are not sufficiently visible in monochromatic imaging modes, especially when repositioning is required, users are forced to switch imaging modes to correctly position instruments relative to fluorescent tissue. Mode switching requires adjustments to the endoscopic system parameters and can cause video to freeze for extended periods, such as 50 frames or more. This imaging freeze makes it difficult for surgeons to quickly switch between imaging modes to operate the surgical robotic system while simultaneously focusing on instruments or fluorescent tissue structures. Therefore, there is a need for a surgical robotic system with imaging modes that allow for the simultaneous imaging of fluorescent tissue structures and surgical instruments. Summary of the Invention

[0006] This disclosure provides a surgical robotic system comprising an imaging system operable in multiple imaging modes. The robotic system may include one or more robotic arms, each holding an instrument or laparoscopic camera of the imaging system. The imaging system is configured to acquire white and NIR images of tissue using fluorophores from a fluorescent dye such as indocyanine green (ICG). The imaging system is configured to combine the white light image with the NIR image in an overlay mode, during which a conventional white light image is combined with NIR / ICG data to generate an overlay image. In overlay mode, the imaging system can be configured to display the NIR image using visible light, depending on user preference and application; for example, the NIR / ICG data may be displayed as a green or blue overlay. In intensity map mode, the imaging system uses a color scale in the overlay image to display the intensity of the NIR / ICG signal. In monochrome mode, the NIR / ICG signal is displayed individually in white on a black background for maximum possible distinction.

[0007] The combination of a surgical robotic system and an imaging system allows for instrument visualization during monochrome imaging mode. Since the instruments are not fluorescent, instrument visualization is provided by a machine learning / artificial intelligence (ML / AI) algorithm that runs in the background and detects the instruments based on white light imaging data. In one embodiment, an overlay or representation of the instruments is superimposed on an image from the monochrome imaging mode. In another embodiment, the algorithm is capable of tracking robotic arm and instrument control information while in white light imaging mode and continuing to update the instrument overlay as the robotic arm and instrument / end-effector are updated using kinematic data when the imaging system switches to monochrome imaging mode, providing a better instrument position reference. The systems and methods according to this disclosure provide better visualization and understanding of instrument position while obtaining high-quality fluorescence imaging information, which in turn enables safe manipulation of surgical instruments while using low-visibility modes.

[0008] According to one embodiment of this disclosure, a surgical robot system is disclosed. The system includes: a surgeon's console having a handle controller configured to receive user input; and a robotic arm having an instrument actuation unit coupled to a surgical instrument and configured to actuate the surgical instrument in response to user input. The system also includes: a laparoscopic camera configured to capture video feeds from the surgical instrument; and an image processing unit coupled to the laparoscopic camera and configured to operate in multiple imaging modes, including a low-visibility imaging mode. The system further includes: a controller configured to process the video feeds from the surgical instrument and generate an overlay of the surgical instrument based on the processed video feeds; and a screen configured to display the video feeds in a low-visibility imaging mode, wherein the overlay is on a portion of the video feeds that may include the surgical instrument.

[0009] The implementation of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the laparoscopic camera can be configured to capture white light images and near-infrared (NIR) light images. The low-visibility imaging mode may be a monochromatic NIR mode. The overlay layer may include a line model, a mesh model, or a 3D surface model. The controller is further configured to determine the position of the surgical instruments based on kinematic data from the robotic arm and instrument drive unit. The controller is further configured to generate the overlay layer based on the position of the surgical instruments. The controller is further configured to determine the position of the surgical instruments by processing the video feed of the surgical instruments using a machine learning computer vision algorithm. The controller is further configured to track the movement of the surgical instruments and update the overlay layer based on the movement of the surgical instruments. The controller is further configured to track the movement of the surgical instruments based on kinematic data from the robotic arm and instrument drive unit. The controller is further configured to track the movement of the surgical instruments by processing the video feed of the surgical instruments using a machine learning computer vision algorithm.

[0010] According to another embodiment of this disclosure, a method for controlling a surgical robot system is disclosed. The method includes: receiving user input at a surgeon's console including a handle controller; and actuating surgical instruments coupled to a robotic arm via an instrument drive unit in response to the user input. The method further includes: capturing video feeds of the surgical instruments via a laparoscopic camera; and operating an image processing device coupled to the laparoscopic camera in multiple imaging modes having a low-visibility imaging mode. The method further includes: processing the video feeds of the surgical instruments; generating an overlay of the surgical instruments based on the processed video feeds; and displaying the video feeds on a screen in a low-visibility imaging mode, wherein the overlay is on the portion of the video feed including the surgical instruments.

[0011] The implementation of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may include capturing white light images and near-infrared (NIR) light images at a laparoscopic camera. The low-visibility imaging mode may be a monochromatic NIR mode. The overlay layer may include a line model, a mesh model, or a 3D surface model. The method may further include determining the position of surgical instruments based on kinematic data from a robotic arm and instrument drive unit. The method may further include generating the overlay layer based on the position of the surgical instruments. The method may additionally include determining the position of the surgical instruments by processing video feeds of the surgical instruments using machine learning computer vision algorithms. The method may further include tracking the movement of the surgical instruments and updating the overlay layer based on the movement of the surgical instruments. Tracking the movement of the surgical instruments may be based on kinematic data from a robotic arm and instrument drive unit. Tracking the movement of the surgical instruments may also be performed by processing video feeds of the surgical instruments using machine learning computer vision algorithms. Attached Figure Description

[0012] Different embodiments of the disclosure are described herein with reference to the accompanying drawings, in which: Figure 1 This is a perspective view of a surgical robot system according to an embodiment of the present disclosure, the surgical robot system including a control tower, a console, and one or more surgical robotic arms, each of the one or more surgical robotic arms being mounted on a mobile cart; Figure 2 Examples based on this disclosure Figure 1 A 3D view of the surgical robotic arm of a surgical robot system; Figure 3 This is a perspective view of a mobile cart with a mounting arm according to an embodiment of the present disclosure. Figure 1 Surgical robotic arms in surgical robot systems; Figure 4 Examples based on this disclosure Figure 1 A schematic diagram of the computer architecture of a surgical robot system; Figure 5 Examples based on this disclosure Figure 1 A plan view of a surgical robot system positioned around the operating table; Figure 6 This is a schematic diagram of a system for determining the stages of a surgical procedure according to an embodiment of this disclosure; Figure 7 This is a perspective view of an imaging system according to an embodiment of the present disclosure; Figure 8 These are screenshots of a graphical user interface (GUI) for selecting an imaging mode according to embodiments of this disclosure; Figure 9 This is a flowchart of a method for tracking and visualizing an instrument in near-infrared imaging mode, according to an embodiment of this disclosure; Figure 10 This is a screenshot of a surgeon's screen showing video captured in color imaging mode by a laparoscopic camera of a surgical instrument, according to an embodiment of this disclosure. Figure 11 This is a perspective view of a surgeon's screen according to an embodiment of the present disclosure, showing a screenshot of a video in monochrome NIR imaging mode; Figure 12 This is a perspective view of a surgeon's screen according to an embodiment of the present disclosure, showing a screenshot of a video in monochrome NIR imaging mode, wherein the surgical instruments have a joint overlay layer. Figure 13This is a perspective view of a surgeon's screen according to an embodiment of the present disclosure, showing a screenshot of a video in monochrome NIR imaging mode, wherein the surgical instruments have a wireframe mesh overlay; and Figure 14 This is a perspective view of a surgeon's screen according to an embodiment of the present disclosure, showing a screenshot of a video in monochrome NIR imaging mode, wherein the surgical instruments have a 3D model overlay layer. Detailed Implementation

[0013] Embodiments of the surgical robot system disclosed herein are described in detail with reference to the accompanying drawings, in which the same reference numerals denote the same or corresponding elements in each of several views.

[0014] refer to Figure 1 The surgical robot system 10 includes a control tower 20 connected to all components of the surgical robot system 10, including a surgeon's console 30 and one or more trolleys 60. Each trolley 60 includes a robotic arm 40 removably coupled to surgical instruments 50. The robotic arm 40 is also coupled to the trolley 60. The robot system 10 may include any number of trolleys 60 and / or any number of robotic arms 40.

[0015] Surgical instrument 50 is configured for use during minimally invasive surgical procedures. In one embodiment, surgical instrument 50 may be configured for open surgical procedures. In another embodiment, surgical instrument 50 may be an electrosurgical or ultrasonic instrument (such as a clamp) configured to seal tissue by pressing tissue between clamp members and applying an electrosurgical current or ultrasonic vibration via an ultrasonic transducer. In yet another embodiment, surgical instrument 50 may be a surgical stapler including a pair of clamps configured to hold and clamp tissue, simultaneously deploy multiple tissue fasteners (e.g., suture staples) and cut the stapled tissue. In yet another embodiment, surgical instrument 50 may be a surgical clip applicator including a pair of clamps configured to apply a surgical clip to tissue. The system also includes an electrosurgical generator 57 configured to output electrosurgical (e.g., monopolar or bipolar) or ultrasonic energy in various operating modes such as coagulation, cutting, and sealing. Suitable generators include the Valleylab™ FT10 energy platform from Medtronic, Minneapolis, Minnesota.

[0016] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereo camera configured to capture two side-by-side (i.e., left and right) images of the surgical site to generate a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing unit 56, which may be located within the control tower 20. The image processing unit 56 may be any computing device configured to receive video feeds from the laparoscopic camera 51 and output a processed video stream.

[0017] The surgeon's console 30 includes a first (i.e., surgeon's) screen 32 and a second screen 34. The first screen displays a video feed of the surgical site provided by a camera 51 of the surgical instrument 50 mounted on the robotic arm 40, and the second screen displays a user interface for controlling the surgical robot system 10. The first screen 32 and the second screen 34 may be touchscreens that allow the display of various graphical user inputs.

[0018] The surgeon's console 30 also includes multiple user interface devices, such as a foot pedal 36 and a pair of hand controllers 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgeon's console further includes a handrail 33 for supporting the clinician's arm when operating the hand controllers 38a and 38b.

[0019] The control tower 20 includes a screen 23, which may be a touchscreen and output on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgeon's console 30 and one or more robotic arms 40. Specifically, the control tower 20 is configured to control the robotic arms 40, for example, to move the robotic arms 40 and corresponding surgical instruments 50 based on a set of programmable instructions and / or input commands from the surgeon's console 30, in such a way that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to inputs from foot pedals 36 and hand controllers 38a and 38b. Foot pedals 36 can be used to enable and lock hand controllers 38a and 38b, reposition camera movement, and activate / deactivate electrosurgical devices. Specifically, foot pedals 36 can be used to perform a clutch action on the hand controllers 38a and 38b. Engaging the clutch by depressing one of the foot pedals 36 disconnects the hand controllers 38a and / or 38b from the robotic arms 40 and their attached corresponding instruments 50 or cameras 51 (i.e., prevents movement input). This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arms 40 and instruments 50 and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0020] Each of the control tower 20, the surgeon's console 30, and the robotic arm 40 includes a corresponding computer 21, 31, or 41. Computers 21, 31, and 41 are interconnected using any suitable communication network based on wired or wireless communication protocols. As used herein, the term "network," whether plural or singular, means a data network, including but not limited to the Internet, intranet, wide area network, or local area network, and is not limited to the full scope of the definition of communication networks covered by this disclosure. Suitable protocols include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DC). Wireless communication can be achieved via one or more wireless configurations, such as radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol used to exchange data between fixed and mobile devices over short distances using short-wavelength radio waves to create personal area networks (PANs)), and ZigBee® (a set of specifications for advanced communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 Wireless Personal Area Network (WPAN) standard).

[0021] Computers 21, 31, and 41 may include any suitable processor (not shown) operatively connected to a memory (not shown) that may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to perform the operations, calculations, and / or instruction sets described in this disclosure, including but not limited to hardware processors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs), microprocessors, and combinations thereof. Those skilled in the art will understand that a processor may be used instead of a logic processor (e.g., control circuitry) adapted to perform the algorithms, calculations, and / or instruction sets described herein.

[0022] refer to Figure 2 Each robotic arm 40 may include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be used, as is known to those skilled in the art. Joint 44a is configured to secure the robotic arm 40 to a movable trolley 60 and defines a first longitudinal axis. (Reference) Figure 3The mobile cart 60 includes a lift 67 and a mounting arm 61 that provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The mobile cart 60 also includes a screen 69 for displaying information about the robotic arm 40. In embodiments, the robotic arm 40 may include any type and / or any number of joints.

[0023] The mounting arm 61 includes a first link 62a, a second link 62b, and a third link 62c, which provide lateral maneuverability of the robotic arm 40. Links 62a, 62b, and 62c are interconnected at joints 63a and 63b, each joint including an actuator (not shown) for rotating links 62b and 62b relative to each other and relative to link 62c. Specifically, links 62a, 62b, and 62c are movable in their corresponding parallel lateral planes, thereby allowing the robotic arm 40 to extend relative to a patient (e.g., an operating table). In an embodiment, the robotic arm 40 may be coupled to an operating table (not shown). The mounting arm 61 includes controls 65 for adjusting the movement of links 62a, 62b, and 62c, as well as the lift 67. In an embodiment, the mounting arm 61 may include any type and / or any number of joints.

[0024] The third link 62c may include a rotatable base 64 having two degrees of freedom. Specifically, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b is rotatable about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for full three-dimensional orientation of the robotic arm 40.

[0025] Actuator 48b of joint 44b is coupled to joint 44c via belt 45a, and joint 44c is further coupled to joint 46b via belt 45b. Joint 44c may include a transfer case connecting belts 45a and 45b, such that actuator 48b is configured to rotate each of links 42b, 42c and retainer 46 relative to each other. More specifically, links 42b, 42c and retainer 46 are passively coupled to actuator 48b, which forces rotation about a pivot point “P” located at the intersection of a first axis defined by link 42a and a second axis defined by retainer 46. In other words, pivot point “P” is the remote center of motion (RCM) of robotic arm 40. Thus, actuator 48b controls the angle θ between the first and second axes, thereby allowing orientation of surgical instrument 50. Because links 42a, 42b, 42c and retainer 46 are interconnected via belts 45a and 45b, the angle between links 42a, 42b, 42c and retainer 46 is also adjusted to achieve a desired angle θ. In an embodiment, some or all of joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0026] Joints 44a and 44b include actuators 48a and 48b configured to drive joints 44a, 44b, and 44c relative to each other via a series of links 45a and 45b or other mechanical linkages (such as drive rods, cables, or levers). Specifically, actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by link 42a.

[0027] refer to Figure 2 The retainer 46 defines a second longitudinal axis and is configured to receive the instrument drive unit (IDU) 52. Figure 1 IDU 52 is configured to be coupled to the actuation mechanism of surgical instrument 50 and camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or camera 51. IDU 52 transmits actuating force from its actuator to surgical instrument 50 to actuate components of the end effector 49 of surgical instrument 50. Holder 46 includes a sliding mechanism 46a configured to move IDU 52 along a second longitudinal axis defined by holder 46. Holder 46 also includes a joint 46b that rotates holder 46 relative to link 42c. During laparoscopic surgery, instrument 50 can be accessed through laparoscopic access port 55 provided by holder 46. Figure 3 Insertion. The retainer 46 also includes a port latch 46c for securing the inlet port 55 to the retainer 46. Figure 2 ).

[0028] The robotic arm 40 also includes multiple manual control buttons 53. Figure 1These manual control buttons are located on IDU 52 and mounting arm 61 and can be used in manual mode. The user can press one or more of these buttons 53 to move the component associated with the button 53.

[0029] refer to Figure 4 Each of the computers 21, 31, and 41 in the surgical robot system 10 may include multiple controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety observer 21b. The controller 21a receives data from the computer 31 of the surgeon's console 30 regarding the current position and / or orientation of the hand controllers 38a and 38b, as well as the status of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine force feedback commands, which are transmitted back to the computer 31 of the surgeon's console 30 to provide tactile feedback via the hand controllers 38a and 38b. The safety observer 21b performs validity checks on the data entering and exiting the controller 21a, and if an error is detected in the data transmission, it notifies the system fault handler to put the computer 21 and / or the surgical robot system 10 into a safe state.

[0030] Controller 21a is coupled to storage device 22a, which may be a non-transitory computer-readable medium configured to store any suitable computer data, such as software instructions executable by controller 21a. Controller 21a also includes transient memory 22b for loading instructions and other computer-readable data during instruction execution. In embodiments, other controllers of system 10 include similar configurations.

[0031] Computer 41 includes multiple controllers: a trolley main controller 41a, a mounting arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The trolley main controller 41a receives and processes joint commands from controller 21a on computer 21 and transmits them to the mounting arm controller 41b, robotic arm controller 41c, and IDU controller 41d. The trolley main controller 41a also manages instrument changes and the overall status of the moving trolley 60, robotic arm 40, and IDU 52. The trolley main controller 41a also transmits actual joint angles back to controller 21a.

[0032] Each of joints 63a and 63b, and the rotatable base 64 of the mounting arm 61, is a passive joint that allows manual adjustment by the user (i.e., where no actuator is present). Joints 63a and 63b, and the rotatable base 64, include brakes that are disengaged by the user to configure the mounting arm 61. When the brakes are engaged, the mounting arm controller 41b monitors slippage of each of joints 63a and 63b, and the rotatable base 64 of the mounting arm 61; or when the brakes are disengaged, the mounting arm controller can be freely moved by the operator without affecting the control of other joints. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robotic arm 40. The robotic arm controller 41c calculates a movement command based on the calculated torque. The calculated motor command is then transmitted to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint position is then transmitted back to the robotic arm controller 41c by actuators 48a and 48b.

[0033] IDU controller 41d receives the desired joint angles (such as wrist angle and gripper angle) from surgical instrument 50 and calculates the desired current of the motor in IDU 52. IDU controller 41d calculates the actual angles based on the motor position and transmits these actual angles back to trolley main controller 41a.

[0034] refer to Figure 5 The surgical robot system 10 is positioned around the operating table 90. System 10 includes mobile trolleys 60a-d, which can be numbered "1" through "4". During setup, each of the trolleys 60a-d is positioned around the operating table 90. The position and orientation of the trolleys 60a-d depend on several factors, such as the placement of multiple access ports 55a-d, which in turn depend on the surgery being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient, and the trolleys 60a-d are positioned to insert instruments 50 and a laparoscopic camera 51 into their corresponding ports 55a-d.

[0035] During use, each of the robotic arms 40a-d uses lock 46c ( Figure 2 ) Attached to inlet port 55 ( Figure 3The IDU 52 is attached to one of the access ports 55a-d for insertion into the patient. The IDU 52 is attached to the retainer 46, and subsequently, the SIM 43 is attached to the distal portion of the IDU 52. The instrument 50 is then attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the retainer 46. The SIM 43 includes a plurality of drive shafts configured to transmit rotation of the respective motors of the IDU 52 to the instrument 50, thereby actuating the instrument 50. Furthermore, the SIM 43 provides a sterile barrier between the instrument 50 and the other components of the robotic arm 40, including the IDU 52. The SIM 43 is also configured to secure a sterile drape (not shown) to the IDU 52.

[0036] Surgical procedures may include multiple phases, and each phase may include one or more surgical actions. As used herein, the term "phase" refers to a surgical event consisting of a series of steps (e.g., closure). A "surgical action" may include incision, compression, staples, clamping, suturing, cauterization, sealing, or any other such action performed to complete a phase of a surgical procedure. A "step" refers to achieving a specified surgical goal (e.g., hemostasis). During each step, certain surgical instruments 50 (e.g., clamps) are used to achieve a specific goal by performing one or more surgical actions.

[0037] refer to Figure 6 The surgical robot system 10 may include a machine learning (ML) processing system 310 that uses one or more ML models to process surgical data to identify one or more features in the surgical data, such as surgical stage, instruments, anatomical structures, etc. The ML processing system 310 includes an ML training system 325, which may be a separate device (e.g., a server) that stores its output as one or more trained ML models 330. The ML models 330 may be accessed by an ML execution system 340. The ML execution system 340 may be separate from the ML training system 325; that is, the device for “training” the model is separate from the device for “inference,” i.e., real-time processing of the surgical data using the trained ML models 330.

[0038] System 10 includes a data receiving system 305 that collects surgical data, including video data and surgical instrument data. The data receiving system 305 may include one or more devices (e.g., one or more user devices and / or servers) located in and / or associated with the operating room and / or control center. The data receiving system 305 can receive surgical data in real time, i.e., while the surgery is being performed.

[0039] In some examples, the ML processing system 310 may further include a data generator 315 for generating simulated surgical data (such as a virtual image set) or recording video data from the image processing device 56 and other data sources (e.g., user input, arm movements, etc.) to train the ML model 330. The data generator 315 may access (read / write) the data storage 320 to record data, including multiple images and / or multiple videos.

[0040] The ML processing system 310 also includes a stage detector 350 that uses an ML model to identify stages within a surgical procedure. The stage detector 350 uses a specific surgical tracking data structure 355 from a list of surgical tracking data structures. The stage detector 350 selects the surgical tracking data structure 355 based on the type of surgery being performed. In one or more examples, the type of surgery is predetermined or entered by the user. The surgical tracking data structure 355 identifies a set of potential stages that may correspond to a portion of a particular type of surgery.

[0041] In some examples, the surgical tracking data structure 355 may be a graph comprising a set of nodes and a set of edges, where each node corresponds to a potential stage. These edges can provide directional connections between nodes, indicating (via direction) the expected sequence of stages to be encountered throughout the surgical iterations. The surgical tracking data structure 355 may include one or more branch nodes that feed to multiple next nodes and / or may include one or more divergence points and / or convergence points between nodes. In some instances, a stage indicates a surgical action being performed or already performed (e.g., a surgical procedure) and / or a combination of actions already performed. In some instances, a stage is related to the biological state of the patient undergoing surgery. For example, the biological state may indicate complications (e.g., blood clots, arterial / venous obstruction, etc.), prerequisites (e.g., lesions, polyps, etc.). In some examples, the ML model 330 is trained to detect “abnormalities,” such as bleeding, arrhythmias, vascular abnormalities, etc.

[0042] The stage detector 350 outputs a stage prediction associated with a portion of the video data analyzed by the ML processing system 310. This stage prediction is associated with the portion of video data by identifying the start and end times of the video portion analyzed by the ML execution system 340. The output stage prediction may include an identifier of the surgical stage as detected by the stage detector 350 based on the output of the ML execution system 340. Further, in one or more examples, the stage prediction may include identifiers of structures (e.g., instruments, anatomical structures, etc.) identified by the ML execution system 340 in the analyzed video portion. The stage prediction may also include a confidence score for the prediction. Other examples may include various other types of information in the output stage prediction. The controller 21a can use the predicted stage to determine when to switch between various imaging modalities, as described below.

[0043] refer to Figure 7 The surgical robot system 10 also includes an imaging system 400, in which a laparoscopic camera 51 is coupled to an image processing unit 56. The laparoscopic camera 51 includes a laparoscope 402 having a longitudinal axis 414 with multiple optical components (not shown), such as lenses, mirrors, prisms, etc., disposed in the longitudinal axis 414. The laparoscope 402 is coupled to a combined light source 406 via an optical fiber 408. The light source 406 may include a white light source (not shown) and a NIR light source (not shown), which may be light-emitting diodes or any other suitable light source. The NIR light source may be a laser or any other suitable light source. The optical fiber 408 may include one or more optical fibers for transmitting white light and NIR light that illuminate tissue under the observation of the laparoscope 402. The laparoscope 402 collects the reflected white light and NIR light and transmits this reflected light to a camera assembly 410 coupled to the proximal portion of the laparoscope 402. Laparoscope 402 can be any conventional laparoscope configured to transmit and collect white light and NIR light.

[0044] Camera assembly 410 includes components configured to separate fluorescence wavelengths from unwanted components of the spectrum and direct them to a specific sensor. Specifically, the camera assembly includes a white light (e.g., visible light) (VIS) sensor and an IR sensor, and is configured to separate the white light and transmit it to the VIS sensor, and to separate the fluorescent IR light and transmit it to the IR sensor. The VIS and IR sensors can be complementary metal-oxide-semiconductor (CMOS) image sensors with any desired resolution (in embodiments, this could be 4K, UHD, etc.).

[0045] Camera assembly 410 is connected to image processing unit 56 via transmission cable 412. Image processing unit 56 is configured to receive image data signals, process raw image data from camera assembly 410, and generate a mixed white light and NIR image for recording and / or real-time display. Image processing unit 56 also processes image data signals and outputs these image data signals to any of the displays 23, 32, and 34 of surgical robot system 10 via any suitable video output port such as DISPLAYPORT™ or HDMI®, which is capable of transmitting the processed image at any desired resolution, display rate, and / or bandwidth.

[0046] Figure 8 A GUI 500 for controlling the imaging system 400 is shown, which can be displayed on the displays 23, 32, and 34 of the surgical robot system 10. The GUI 500 includes options for controlling fluorescence settings, including turning fluorescence (e.g., NIR detection) on or off via a toggle key 502. Once fluorescence is selected, the user can also select from a variety of imaging modes that visualize NIR light. An overlay mode can be selected via button 504. In overlay mode, the imaging system 400 combines a white light image with an NIR image; during this mode, a regular white light image is combined with NIR / ICG data to generate an overlay image. In this mode, the imaging system can be configured to display NIR light in visible light depending on user preference and application; the NIR / ICG data can be displayed as a green or blue overlay, selectable via menu 505. In intensity map mode, selectable via button 506, the imaging system displays the intensity of the NIR / ICG signal using a color scale in the overlay image. In monochrome mode, selectable via button 508, the NIR / ICG signal is displayed separately in white on a black background to achieve maximum possible distinction, such as... Figure 11 As shown. In an embodiment, a mode can also be selected via one or more foot pedals 36 associated with mode selection, for example, a foot pedal cyclically traversing each mode in the NIR modes. Figure 10 The white light video feed from camera 51 and the indicator 510 for switching between imaging modes of imaging system 400 are shown on the first screen 32.

[0047] Figure 9 A method for operating the surgical robot system 10 and the imaging system 400 is shown to enable instrument tracking and visualization in near-infrared imaging mode. Figure 9 The method can be embodied as software instructions executable by any controller of system 10 (e.g., main controller 21a). In step 600, imaging system 400, as... Figure 10The system operates in either the white light mode or the overlay / thermal imaging mode shown. In this mode, in step 602, the controller 21a continuously stores position data for the instrument 50, the camera 51, and the corresponding robotic arm 40. The position data may include the coordinates of the joints of the robotic arm 40 in the world coordinate system, as well as joint angles, arm height, angles relative to the worktable 90, etc. Additionally, the position data also includes the motor angular positions of one or more motors in the IDU 52 used to control the camera 51 and / or the instrument 50. Subsequently, the controller 21a uses the position data to generate an overlay to highlight the instrument 50 in the monochrome video feed.

[0048] In step 604, the imaging system 400 enters a state such as... in response to user input or in response to a specific stage of the surgery detected by the stage detector 350. Figure 11 The monochrome imaging mode is shown. Once the monochrome imaging mode is enabled, the controller 21a displays a prompt on the GUI 500 in step 606, asking the user whether to display an overlay on the instrument 50. If the user selects "No", the controller 21a does not display the overlay in step 608. If the user selects "Yes", the overlay is displayed on one or more instruments 50 in step 610.

[0049] The overlay layer is generated in step 612, and can be as follows: Figure 12 The line model 700 shown is as follows: Figure 13 The wireframe or mesh model shown is 702, such as Figure 14 The 3D surface model 704 is shown. In this embodiment, the overlay can be reconstructed from a white light image of the instrument 50 captured in parallel with the NIR image. The type of overlay displayed can be selected by the user via GUI 500 or via default settings.

[0050] The overlay layer is generated based on determining the position, shape, orientation, and size of the device 50. The position of the device 50 may be based on position data collected in step 602 and / or determined by an ML / AI computer vision algorithm that runs in the background and detects the device based on white light imaging data. The ML / AI algorithm also determines the size and orientation of the device 50.

[0051] ML / AI algorithms can be based on statistical ML, which is configured to develop statistical models and derive inferences from them. As more training data is provided, the system adjusts the statistical model and improves its analytical or predictive capabilities. Suitable statistical ML models include, but are not limited to, linear regression, logistic regression, decision trees, random forests, Naive Bayes, ensemble methods, support vector machines, K-nearest neighbors, etc. In a further embodiment, the ML / AI algorithm can be a deep learning algorithm that combines neural networks in successive layers. Suitable deep learning models include, but are not limited to, convolutional neural networks, recurrent neural networks, deep reinforcement networks, deep belief networks, transformer networks, etc. The input provided for training the model can be previously collected data, including images of device 50.

[0052] The overlays can be color-coded, such that one device 50 includes an overlay of one color (e.g., red), and another device 50 includes an overlay of a different color (e.g., blue). Figure 12 and Figure 13 As shown, the ML / AI computer vision algorithm is configured to identify joints labeled as circles, spheres, or any other geometric shape. The identified joints can be interconnected with line segments, such as... Figure 12 As shown. In the mesh model, the edges of different surfaces are also highlighted, as shown with respect to the gripper component of instrument 50.

[0053] In one embodiment, the overlay can be reconstructed from a white light image of the device 50 captured simultaneously with the NIR image. A portion of the white light image corresponding to the device 50 is extracted using an ML / AI computer vision algorithm and then used to generate a color overlay displayed on the NIR video feed of the device 50, thereby highlighting the device 50 in the NIR video feed.

[0054] Once the overlay is displayed, in step 614, controller 21a determines whether instrument 50 can be moved. This is performed as a safety verification during certain phases of the surgical procedure to avoid potential harm to the patient from movement of instrument 50. If instrument 50 cannot be moved, in step 616, controller 21a locks instrument 50 in place until the conditions preventing movement are released, for example, instrument 50 is moved away from a critical structure. In an embodiment, controller 21a may enable movement of instrument 50, but may also exit low-visibility mode and enter a default or normal imaging mode, which may be a white light mode, or any combination of white light / NIR modes described above, such as overlay mode or thermal image mode.

[0055] If the device 50 is movable, then in step 618, the controller 21a enables the device 50 to move while in low-visibility monochrome imaging mode. In step 620, the controller 21a determines whether any device 50 has moved. The controller 21a continuously determines whether the user is attempting to move the device 50 by monitoring user input (e.g., movement of handle controllers 38a and 38b). Once movement input is detected, then in step 622, the controller 21a displays and updates the overlay. Thus, as the device 50 moves within the view of the camera 51, the overlay is continuously updated, i.e., adjusted and refreshed. In step 624, the overlay update is performed using ML / AI computer vision algorithms and kinematic data from joint sensors and position sensors of the motors of the robotic arm 40 and / or IDU 52.

[0056] Kinematic data, including movement vectors and distances, is used to transform and update the overlay. Therefore, since the overlay was previously mapped to a known coordinate system, in step 626, the original overlay generated in step 612 is transformed and generated based on the movement of the device 50. The kinematic data can be used in conjunction with an ML / AI computer vision algorithm that detects changes in the position of the device 50 based on what is captured by the camera 51. The overlay update process is continuously performed while the device 50 moves until the overlay process is disabled (e.g., by exiting low-visibility monochrome mode).

[0057] Objects lacking fluorescent markers (such as instrument 50) are not clearly visible in monochrome NIR imaging mode. The overlay feature provided in this disclosure combines ML / AI computer vision with the kinematic data inherent to the surgical robot system to provide a visual overlay when using low-visibility imaging modes. The overlay is provided whether the instrument is stationary or in motion, enabling the robot system to operate continuously even in low-visibility modes that typically limit the safe and effective operation of instruments. This provides the user with enhanced visibility of tissue structures and other physical objects.

[0058] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the above description should not be construed as limiting, but rather as illustrative of the various embodiments only. Other modifications within the scope and spirit of the appended claims will be contemplated by those skilled in the art.

Claims

1. A surgical robot system (10), comprising: A surgical console (30) includes a handpiece controller (38a, 38b) configured to receive user input; A robotic arm (40) including a machine drive unit; Surgical instrument (50), which is connected to the instrument drive unit (52) and can be actuated by the instrument drive unit in response to user input; Laparoscopic camera (51), which is configured to capture video feeds of the surgical instruments; An image processing device (56) is connected to the laparoscopic camera and is configured to operate in multiple imaging modes, including a low-visibility imaging mode. Controller (21a), which is configured to: The video feed of the surgical instrument is processed in this low-visibility imaging mode; as well as The surgical instrument overlay (700, 702, 704) is generated based on the processed video feed. as well as Screens (32, 34) are configured to display the video feed in the low-visibility imaging mode, wherein the overlay is on the portion of the video feed that includes the surgical instrument.

2. The surgical robot system according to claim 1, wherein, The laparoscopic camera is configured to capture white light images and near-infrared (NIR) light images.

3. The surgical robot system according to any of the preceding claims, wherein, This low-visibility imaging mode is a monochrome NIR mode.

4. The surgical robot system according to any of the preceding claims, wherein, The overlay includes at least one of a line model (700), a mesh model (702), or a 3D surface model (704).

5. The surgical robot system according to any of the preceding claims, wherein, The controller is further configured to determine the position of the surgical instrument based on the kinematic data of the robotic arm and the instrument drive unit.

6. The surgical robot system according to claim 5, wherein, The controller is further configured to generate the overlay based on the position of the surgical instrument.

7. The surgical robot system according to any of the preceding claims, wherein, The controller is further configured to determine the position of the surgical instrument by processing the video feed of the surgical instrument using machine learning computer vision algorithms.

8. The surgical robot system according to any of the preceding claims, wherein, The controller is further configured to track the position of the surgical instrument and update the overlay based on the position of the surgical instrument to cover the portion of the video feed that includes the surgical instrument.

9. The surgical robot system according to claim 8, wherein, The controller is further configured to track the movement of the surgical instrument based on the kinematic data of the robotic arm and the instrument drive unit.

10. The surgical robot system according to claim 9, wherein, The controller is further configured to track the movement of the surgical instrument by processing the video feed of the surgical instrument using machine learning computer vision algorithms.

Images