Surgical robotic system and method for generating a synthetic instrument image in a low-contrast imaging mode using an auxiliary image source

Abstract

A surgical robotic system (10) includes a primary imaging source (51) and an auxiliary imaging source (511) disposed within a patient. The primary imaging source (51) is used to image a surgical site and one or more instruments (50, 50') in monochrome near-infrared (NIR) mode. The auxiliary imaging source (51') is used to image the instruments in white mode. A white overlay is extracted from the video feed of the auxiliary imaging source and used to generate a composite image with the NIR video feed (694) so that the instruments (50, 50') are visible in the monochrome video feed (694). The overlay is aligned with the NIR video feed (694) via one or more image processing techniques.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 599,035, filed November 15, 2023, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Surgical robotic systems are currently used in a variety of surgical procedures, including minimally invasive surgery. Some surgical robotic systems include a surgeon's console that controls a surgical robotic arm and surgical instruments, the instruments having end effectors (e.g., clamps or gripping instruments) coupled to and actuated by the robotic arm. During operation, the robotic arm is moved to a position above the patient, and then guides the surgical instruments through a surgical port or the patient's natural orifice into a small incision to position the end effectors at the work site within the patient's body. A laparoscopic camera, also held by one of the robotic arms, is inserted into the patient to image the surgical site.

[0003] Laparoscopic cameras can operate in various imaging modes, including conventional color or white light modes and fluorescence modes. In conventional white light mode, light within the visible spectrum is used to illuminate the surface of the tissue being observed. The light reflected from the tissue passes through a suitable lens system and is incident on an image sensor built into or attached to the endoscope. The electrical signals from the image sensor are processed into a full-color video image, which can be displayed on a video monitor or stored in memory.

[0004] In fluorescence mode, fluorescence excitation light excites fluorophores in the tissue, which emit fluorescence at an emission wavelength typically greater than the excitation wavelength. The fluorescence from the tissue passes through a suitable lens system and is incident on an image sensor. The electrical signal from the image sensor is processed into a fluorescence video image, which can be displayed on a video monitor alone or in combination with a color video image.

[0005] Fluorescence excitation and emission wavelengths depend on the type of fluorophore being excited. With exogenously applied fluorophores (such as fluorescent dyes, e.g., indocyanine green (ICG)), the excitation wavelength band can be anywhere in the ultraviolet (UV) to near-infrared (NIR) range, and the emission wavelength band can be anywhere in the visible to NIR range. For tissue-endogenous fluorophores, the excitation and emission wavelength bands are more limited (excitation from the green portion of the UV to the visible spectrum, emission from blue / green to NIR). Fluorescence imaging can be used to identify blood vessels, cancer cells, and other tissue types. White light imaging modes and fluorescence imaging modes can be combined in various ways. Camera manufacturers offer a variety of imaging modes to provide surgeons with additional insight into the structures and tools used during laparoscopic or surgical procedures. Some modes that enhance NIR light result in low visibility of non-fluorescent objects (e.g., instruments). Because instruments are not sufficiently visible in monochromatic imaging modes, users are forced to switch imaging modes to correctly position instruments relative to fluorescent tissue, especially when repositioning of instruments is required.

[0006] In monochrome mode, surgical instruments are not visible within the endoscopic field of view (FOV) due to the low fluorescence of the instrument material. However, surgeons prefer this view for visualizing anatomical structures because the high contrast of monochrome mode facilitates clear visualization of structures receiving perfusion ICG. Therefore, as a safety precaution, instruments are not allowed to be moved when displayed in monochrome mode, since they do not fluoresce under NIR light and are not visible within the endoscopic FOV. Consequently, surgeons do not have a clear understanding of the position of the instruments relative to the fluorescent structures shown in monochrome mode and will not be able to move them until returning to white light mode. Summary of the Invention

[0007] This disclosure provides a surgical robotic system including an imaging system capable of operating in multiple imaging modes. The robotic system may include one or more robotic arms, each holding an instrument or laparoscopic camera of the imaging system. The imaging system is configured to acquire white and NIR images of tissue using fluorophores from a fluorescent dye (e.g., ICG). The imaging system combines the white and NIR images in a stacked mode, during which a conventional white light image is combined with NIR / ICG data to generate a composite image. In the stacked mode, the imaging system may be configured to display the NIR image using visible light, depending on user preference and application; for example, the NIR / ICG data may be displayed as a green or blue stack. In intensity map mode, the imaging system uses color gradations in the stacked image to display the intensity of the NIR / ICG signal. In monochrome mode, the NIR / ICG signal is displayed separately as white against a black background to achieve maximum possible distinction.

[0008] NIR imaging operates using a light source juxtaposed with an imaging sensor that emits light in a narrow excitation band. This light source is filtered and tuned to sense only light in the fluorescence emission band. The fluorescence emission signal emitted by tissue in NIR is typically very weak, thus requiring amplification of the emitted signal. Therefore, any material (e.g., tissue or others) that does not emit or reflect light in the emission signal band will not appear visible in the resulting image. This is typically achieved by designing to enhance the contrast of fluorescent tissue. One drawback is that surgical instruments are virtually invisible within the field of view (FOV). This, in turn, presents challenges for operator instrument positioning, most likely prohibiting manipulation in NIR imaging mode.

[0009] This disclosure provides a second image source for imaging instruments. The image source may include a light source in the visible band and a camera capable of detecting reflected visible light, wherein the instrument is within the camera's field of view (FOV). The second image source may be a stadium-view camera located at a point remote from the region of interest (ROI), possibly located at the origin of the instrument and endoscope (e.g., attached to one of one or more surgical ports). The FOV may encompass the original NIR camera FOV and include a wider FOV. The resulting image may be displayed on another monitor available to the operator, or as a picture-in-picture (PIP) view on the same monitor view as the NIR image.

[0010] The disclosed system utilizes both image sources and kinematic data from a robotic arm that holds the camera and the apparatus. The system includes an image processing device that uses chroma keying or alpha keying to co-register masked stadium view images of the apparatus with poor contrast (i.e., low visibility), specifically apparatus images in NIR frames.

[0011] The system executes an algorithm, which can be implemented as software instructions, to co-register and overlay an image of the instrument under white light captured by a stadium camera onto a monochrome NIR image for operator positioning of the instrument. The algorithm applies several deterministic transformations to the instrument image based on the stadium camera's positioning relative to the endoscope, including edge detection and masking, rotation, and scaling. Reference points for these transformations can be shared image markers visible in both images or derived from end effector kinematic data.

[0012] Additionally, if there is occlusion of equipment in the stadium view, additional processing of the equipment outlines detected by edge detection can be used to present a complete image of the equipment by utilizing algorithms designed for pixel supplementation, such as nearest neighbor or simple pixel averaging.

[0013] The transformed masked instrument image can be synthesized from a monochrome image, possibly with a buffer delay to the instrument image, to allow for image transformation processing time and avoid delays in endoscopic image delivery. Image synthesis of the instrument image from the buffer and the monochrome image can be achieved using hardware-implemented chroma keying or alpha co-mixing. The disclosed solution provides a unique implementation of image overlay in NIR monochrome imaging mode without increasing the delay in endoscopic image delivery or introducing unwanted image artifacts, while maximizing the usability of NIR monochrome imaging.

[0014] According to one embodiment of this disclosure, an imaging system is disclosed. The imaging system includes a first laparoscopic camera disposed at a first location within a patient's body for capturing a first video feed of surgical instruments from a first perspective. The system also includes a second laparoscopic camera disposed at a second location within the patient's body for capturing a second video feed of surgical instruments from a second perspective different from the first perspective. The system further includes an image processing device coupled to the first and second laparoscopic cameras. The image processing device processes the first video feed of surgical instruments in a low-visibility imaging mode and the second video feed of surgical instruments in a white imaging mode. The system additionally includes a controller for extracting a white image from frames of the second video feed and generating a stack of surgical instruments from the white image. When in the low-visibility imaging mode, the controller also generates a composite frame including the stack and frames of the first video feed. The stack is disposed on a portion of the video feed including the surgical instruments. The system also includes a screen for displaying the composite video feed including the composite frame.

[0015] Specific implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the controller may rotate one or both of the frames of the overlay or the first video feed relative to each other to align their respective coordinate systems. The controller may also translate one or both of the frames of the overlay or the first video feed relative to each other to align their respective coordinate systems. The low-visibility imaging mode may be a monochromatic near-infrared (NIR) mode. The first laparoscopic camera may have a first field of view, and the second laparoscopic camera may have a second field of view that at least partially overlaps with the first field of view, wherein surgical instruments are within both the first and second fields of view. The second field of view may be larger than the first field of view. The controller may use alpha blending and / or chroma keying to generate composite frames.

[0016] According to another embodiment of this disclosure, a surgical robot system is disclosed. The surgical robot system includes: a first robotic arm including a first laparoscopic camera disposed at a first location within a patient for capturing a first video feed of surgical instruments from a first perspective. The system also includes a second robotic arm including a second laparoscopic camera disposed at a second location within the patient for capturing a second video feed of surgical instruments from a second perspective different from the first perspective. The system further includes an image processing device coupled to the first and second laparoscopic cameras. The image processing device processes the first video feed of surgical instruments in a low-visibility imaging mode and the second video feed of surgical instruments in a white imaging mode. The system additionally includes a controller for extracting a white image from frames of the second video feed and generating a stack of surgical instruments from the white image. When in the low-visibility imaging mode, the controller also generates a composite frame including the stack and frames of the first video feed. The stack is disposed on a portion of the video feed including the surgical instruments. The system also includes a screen for displaying the composite video feed, which includes composite frames.

[0017] Specific implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the controller may receive kinematic data from a first robotic arm and a second robotic arm. The kinematic data may include positioning information of the first and second robotic arms, as well as the first and second laparoscopic cameras. The controller may also, based on the kinematic data, rotate at least one of the overlay or the first video feed or frames relative to each other to align their respective coordinate systems. The controller may also, based on the kinematic data, translate one or both of the overlay or the first video feed or frames relative to each other to align their respective coordinate systems. The low-visibility imaging mode may be a monochromatic near-infrared (NIR) mode. The first laparoscopic camera may have a first field of view, and the second laparoscopic camera may have a second field of view that at least partially overlaps with the first field of view, wherein surgical instruments are within the first and second fields of view. The second field of view may be larger than the first field of view. The controller may use alpha blending and / or chroma keying to generate composite frames.

[0018] According to another embodiment of this disclosure, a method for generating synthetic instrument images is disclosed. The method includes: capturing a first video feed of surgical instruments from a first perspective using a first laparoscopic camera positioned at a first location within a patient's body. The method further includes: capturing a second video feed of surgical instruments from a second perspective, different from the first perspective, using a second robotic arm including a second laparoscopic camera positioned at a second location within the patient's body. The method further includes: processing the first video feed of surgical instruments in a low-visibility imaging mode and processing the second video feed of surgical instruments in a white imaging mode at an image processing device coupled to the first and second laparoscopic cameras. The method additionally includes: extracting a white image from frames of the second video feed. The method further includes: generating a stack of surgical instruments from the white image. The method further includes: generating a synthetic frame including the stack and frames of the first video feed when in low-visibility imaging mode. The stack is disposed on a portion of the video feed including the surgical instruments. The method further includes: displaying the synthetic video feed including the synthetic frame on a screen.

[0019] Specific implementations of the above embodiments may include one or more of the following features. According to one aspect of the above embodiments, the method may further include: rotating at least one of the frames in the overlay or the first video feed relative to each other to align their respective coordinate systems. The method may further include: translating at least one of the frames in the overlay or the first video feed relative to each other to align their respective coordinate systems. Alpha blending and / or chroma keying may be used to generate the composite frame. Attached Figure Description

[0020] This document describes various embodiments of the present disclosure in conjunction with the accompanying drawings, wherein: Figure 1 This is a perspective view of a surgical robot system according to an embodiment of the present disclosure, the surgical robot system including a control tower, a console and one or more surgical robot arms, each of the one or more surgical robot arms being mounted on a mobile trolley; Figure 2 It is based on the implementation scheme of this disclosure. Figure 1 A perspective view of the surgical robotic arm of a surgical robotic system; Figure 3 It is based on the implementation scheme of this disclosure and has Figure 1 A perspective view of the setting of the surgical robot arm of a surgical robot system, showing the movement of the arm on a trolley. Figure 4 It is based on the implementation scheme of this disclosure. Figure 1 A schematic diagram of the computer architecture of a surgical robotic system; Figure 5 This is based on one aspect of the disclosure regarding the positioning of the surgical operating table. Figure 1 A plan view of a surgical robotic system; Figure 6 This is a schematic diagram of a system for determining the stages of a surgical procedure according to an embodiment of this disclosure; Figure 7 This is a perspective view of an imaging system according to an embodiment of the present disclosure; Figure 8 This is a screenshot of a graphical user interface (GUI) for selecting an imaging mode according to an embodiment of this disclosure; Figure 9 A schematic diagram of a dual laparoscopic camera imaging system according to an embodiment of the present disclosure is shown; Figure 10 A flowchart is shown of a method for generating synthetic instrument images in a low-contrast imaging mode according to an embodiment of the present disclosure; Figure 11 A flowchart is shown of a method for extracting instrument images from a second camera to generate a composite image according to an embodiment of the present disclosure; Figure 12 A schematic diagram is shown illustrating the rotation of a frame captured by a second camera to align with a frame captured by a first camera, according to an embodiment of the present disclosure. Figure 13 A schematic diagram is shown illustrating a translation of a frame captured by a second camera to align with a frame captured by a first camera, according to an embodiment of the present disclosure. Figure 14 The illustration shows that, according to an embodiment of the present disclosure, extracted instrument images are superimposed on a low-visibility imaging mode frame to generate a synthetic instrument image; and Figure 15 A flowchart is shown of a method for generating synthetic instrument images in a low-contrast imaging mode according to another embodiment of the present disclosure. Detailed Implementation

[0021] The embodiments of the surgical robot system disclosed in this invention are described in detail with reference to the accompanying drawings, wherein similar reference numerals in each of the plurality of views represent the same or corresponding elements.

[0022] refer to Figure 1 The surgical robot system 10 includes a control tower 20 connected to all components of the surgical robot system 10, including a surgeon's console 30 and one or more mobile trolleys 60. Each mobile trolley 60 includes a robotic arm 40 with surgical instruments 50 removably coupled thereto. The robotic arm 40 is also coupled to the mobile trolley 60. The robot system 10 may include any number of mobile trolleys 60 and / or robotic arms 40.

[0023] Surgical instrument 50 is configured for use during minimally invasive surgical procedures. In one embodiment, surgical instrument 50 may be configured for use in open surgery. In another embodiment, surgical instrument 50 may be an electrosurgical or ultrasonic instrument, such as a clamp configured to seal tissue by compressing tissue between jaw members and applying electrosurgical current or ultrasonic vibration to the tissue via an ultrasonic transducer. In another embodiment, surgical instrument 50 may be a surgical suture device including a pair of jaws configured to grasp and clamp tissue while deploying multiple tissue fasteners (e.g., staples) and cutting the sutured tissue. In another embodiment, surgical instrument 50 may be a surgical clamp applicator including a pair of jaws configured to apply a surgical clamp to the tissue. The system also includes an electrosurgical generator configured to output electrosurgical (e.g., monopolar or bipolar) or ultrasonic energy in multiple operating modes (e.g., coagulation, cutting, sealing, etc.). Suitable generators include Valleylab from Medtronic in Minneapolis, Minnesota. ™ FT10 energy platform.

[0024] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of a surgical site. The laparoscopic camera 51 may be a stereo camera configured to capture two side-by-side (i.e., left-right) images of the surgical site to generate a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device 56, which may be located within the control tower 20. The image processing device 56 may be any computing device configured to receive video feeds from the laparoscopic camera 51 and output a processed video stream.

[0025] The surgeon's console 30 includes: a first (i.e., surgeon's) screen 32 displaying a video feed of the surgical site provided by a camera 51 of a surgical instrument 50 mounted on the robotic arm 40; and a second screen 34 displaying a user interface for controlling the surgical robot system 10. The first screen 32 and the second screen 34 may be touchscreens that allow for the display of various graphical user inputs.

[0026] The surgeon's console 30 also includes several user interface devices, such as foot pedals 36 and a pair of hand controllers 38a and 38b for the user to remotely control the robotic arm 40. The surgeon's console also includes armrests 33 for supporting the clinician's arm when operating the hand controllers 38a and 38b.

[0027] The control tower 20 includes a screen 23, which may be a touchscreen and output on a graphical user interface (GUI). The control tower 20 also acts as an interface between the surgeon's console 30 and one or more robotic arms 40. Specifically, the control tower 20 is configured to control the robotic arms 40 to move the robotic arms 40 and corresponding surgical instruments 50, such as based on a set of programmable instructions and / or input commands from the surgeon's console 30, so that the robotic arms 40 and surgical instruments 50 perform a desired sequence of movements in response to input from foot pedals 36 and handheld controllers 38a and 38b. Foot pedals 36 can be used to enable and lock the handheld controllers 38a and 38b, reposition camera movement, and activate / deactivate electrosurgery. Specifically, foot pedals 36 can be used to perform a clutch action on the handheld controllers 38a and 38b. Engaging the clutch by pressing one of the foot pedals 36 disconnects the handheld controllers 38a and / or 38b from the robotic arms 40 and the corresponding instruments 50 or cameras 51 attached thereto (i.e., prevents movement input). This allows the user to reposition the handheld controllers 38a and 38b without moving the robotic arm 40, instruments 50, and / or camera 51. This is useful when reaching the control boundaries of the surgical space.

[0028] Each of the control tower 20, the surgeon's console 30, and the robotic arm 40 includes a corresponding computer 21, 31, or 41. Computers 21, 31, and 41 are interconnected using any suitable communication network based on wired or wireless communication protocols. As used herein, the term "network," whether singular or plural, refers to a data network, including but not limited to the Internet, intranets, wide area networks, or local area networks, and is not limited to the full scope of the definition of communication networks covered by this disclosure. Suitable protocols include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DC). Wireless communication may be implemented via one or more wireless configurations, such as radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances from fixed and mobile devices using short-length radio waves to create personal area networks (PANs)), and ZigBee. ® (A set of high-level communication protocol specifications using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for Wireless Personal Area Networks (WPANs).)

[0029] Computers 21, 31, and 41 may include any suitable processor (not shown) operatively connected to a memory (not shown), which may include one or more volatile, non-volatile, magnetic, optical, or electronic media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to perform the operations, calculations, and / or instruction sets described herein, including but not limited to hardware processors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs), microprocessors, and combinations thereof. Those skilled in the art will understand that a processor may be substituted for any logical processor (e.g., control circuitry) adapted to perform the algorithms, calculations, and / or instruction sets described herein.

[0030] refer to Figure 2 Each robotic arm in the robotic arm 40 may include a plurality of connectors 42a, 42b, 42c, which are interconnected at engagements 44a, 44b, 44c, respectively. Other configurations of connectors and engagements may be used, as will be known to those skilled in the art. Engagement 44a is configured to secure the robotic arm 40 to the mobile trolley 60 and define a first longitudinal axis. (See reference...) Figure 3 The mobile cart 60 includes a lift 67 and a mounting arm 61 that provides a base for mounting the robotic arm 40. The lift 67 allows the mounting arm 61 to move vertically. The mobile cart 60 also includes a screen 69 for displaying information related to the robotic arm 40. In embodiments, the robotic arm 40 may include any type and / or number of joints.

[0031] The setup arm 61 includes a first connector 62a, a second connector 62b, and a third connector 62c, which provide lateral maneuverability of the robotic arm 40. Connectors 62a, 62b, and 62c are interconnected at joints 63a and 63b, each of which may include an actuator (not shown) for rotating connectors 62b and 62b relative to each other and connector 62c. Specifically, connectors 62a, 62b, and 62c are movable in their corresponding parallel lateral planes, thereby allowing the robotic arm 40 to extend relative to a patient (e.g., a surgical table). In an embodiment, the robotic arm 40 may be coupled to a surgical table (not shown). The setup arm 61 includes a controller 65 for adjusting the movement of connectors 62a, 62b, and 62c, as well as the lift 67. In an embodiment, the setup arm 61 may include any type and / or number of joints.

[0032] The third connector 62c may include a rotatable base 64 with two degrees of freedom. Specifically, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first fixed arm axis perpendicular to the plane defined by the third connector 62c, and the second actuator 64b is rotatable about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for full three-dimensional orientation of the robot arm 40.

[0033] Actuator 48b of joint 44b is coupled to joint 44c via belt 45a, and joint 44c is in turn coupled to joint 46b via belt 45b. Joint 44c may include a transfer case that connects belts 45a and 45b, such that actuator 48b is configured to rotate each of connectors 42b, 42c and retainer 46 relative to each other. More specifically, connectors 42b, 42c and retainer 46 are passively coupled to actuator 48b, which forces rotation about a pivot point “P” located at the intersection of a first axis defined by connector 42a and a second axis defined by retainer 46. In other words, pivot point “P” is the remote center of motion (RCM) of robotic arm 40. Thus, actuator 48b controls the angle θ between the first and second axes, thereby allowing the orientation of surgical instrument 50. Since the connectors 42a, 42b, 42c and the retainer 46 are interconnected via belts 45a and 45b, the angle between the connectors 42a, 42b, 42c and the retainer 46 is also adjusted to achieve the desired angle "θ". In an embodiment, some or all of the engagements 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0034] The joints 44a and 44b include actuators 48a and 48b configured to drive the joints 44a, 44b, and 44c relative to each other via a series of belts 45a and 45b or other mechanical linkages (such as drive rods, cables, and bars). In particular, actuator 48a is configured to rotate the robot arm 40 about a longitudinal axis defined by connector 42a.

[0035] refer to Figure 2 Holder 46 defines a second longitudinal axis and is configured to receive instrument drive unit (IDU) 52. Figure 1IDU 52 is configured to be coupled to the actuation mechanism of surgical instrument 50 and camera 51, and is configured to move (e.g., rotate) and actuate instrument 50 and / or camera 51. IDU 52 transmits actuation force from its actuator to surgical instrument 50 to actuate components of end effector 49 of surgical instrument 50. Retainer 46 includes sliding mechanism 46a configured to move IDU 52 along a second longitudinal axis defined by retainer 46. Retainer 46 also includes engagement 46b that allows retainer 46 to rotate relative to connector 42c. During laparoscopic surgery, instrument 50 can be accessed through laparoscopic entry port 55 held by retainer 46. Figure 3 Insertion. Retainer 46 also includes a port latch 46c for securing the inlet port 55 to retainer 46. Figure 2 ).

[0036] Robotic arm 40 also includes multiple manual control buttons 53 located on IDU 52 and setting arm 61. Figure 1 These multiple manual control buttons can be used in manual mode. The user can press one or more of the buttons 53 to move the component associated with button 53.

[0037] refer to Figure 4 Each of the computers 21, 31, and 41 in the surgical robot system 10 may include multiple controllers, which may be embodied in hardware and / or software. Computer 21 of control tower 20 includes controller 21a and safety observer 21b. Controller 21a receives data from computer 31 of surgeon console 30 regarding the current positioning and / or orientation of hand controllers 38a and 38b, as well as the status of foot pedal 36 and other buttons. Controller 21a processes this input positioning to determine desired drive commands for each engagement of robotic arm 40 and / or IDU 52, and transmits these commands to computer 41 of robotic arm 40. Controller 21a also receives actual engagement angles measured by encoders of actuators 48a and 48b and uses this information to determine force feedback commands transmitted back to computer 31 of surgeon console 30 to provide tactile feedback via hand controllers 38a and 38b. Safety observer 21b performs validity checks on data entering and leaving controller 21a, and if an error is detected in the data transmission, notifies the system fault processor to place computer 21 and / or surgical robot system 10 into a safe state.

[0038] Controller 21a is coupled to storage device 22a, which may be a non-transitory computer-readable medium configured to store any suitable computer data, such as software instructions executable by controller 21a. Controller 21a also includes transient memory 22b for loading instructions and other computer-readable data during instruction execution. In embodiments, other controllers of system 10 include similar configurations.

[0039] Computer 41 includes multiple controllers: a main trolley controller 41a, a setup arm controller 41b, a robot arm controller 41c, and an instrument drive unit (IDU) controller 41d. The main trolley controller 41a receives and processes engagement commands from controller 21a of computer 21 and transmits these commands to the setup arm controller 41b, robot arm controller 41c, and IDU controller 41d. The main trolley controller 41a also manages instrument exchange and the overall status of the moving trolley 60, robot arm 40, and IDU 52. The main trolley controller 41a also transmits the actual engagement angle back to controller 21a.

[0040] Each of the joints 63a and 63b, and the rotatable base 64 of the setting arm 61, is a passive joint (i.e., where no actuator is present), allowing for manual adjustment by the user. Joints 63a and 63b, and the rotatable base 64, include brakes that are disengaged by the user to configure the setting arm 61. The setting arm controller 41b monitors each of the joints 63a and 63b, and the rotatable base 64 of the setting arm 61, for slippage when the brakes are engaged or for free movement by the operator when the brakes are disengaged, without affecting control of the other joints. The robot arm controller 41c controls each of the joints 44a and 44b of the robot arm 40 and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop positioning control of the robot arm 40. The robot arm controller 41c calculates movement commands based on the calculated torque. The calculated motor commands are then transmitted to one or more actuators 48a and 48b in the robot arm 40. The actual engagement positioning is then transmitted back to the robot arm controller 41c via actuators 48a and 48b.

[0041] IDU controller 41d receives the desired engagement angles of the surgical instruments 50, such as wrist and jaw angles, and calculates the desired current of the motor in IDU 52. IDU controller 41d calculates the actual angles based on motor positioning and transmits the actual angles back to trolley controller 41a.

[0042] refer to Figure 5The surgical robot system 10 is positioned around the surgical table 90. The system 10 includes mobile trolleys 60a to 60d, which may be numbered "1" to "4". During setup, each of the trolleys 60a to 60d is positioned around the surgical table 90. The positioning and orientation of the trolleys 60a to 60d depend on various factors, such as the placement of multiple access ports 55a to 55d, which in turn depends on the ongoing surgical procedure. Once the port placement is determined, the access ports 55a to 55d are inserted into the patient, and the trolleys 60a to 60d are positioned to insert instruments 50 and a laparoscopic camera 51 into the corresponding ports 55a to 55d.

[0043] During use, by using latch 46c ( Figure 2 ) Attached to ingress port 55 ( Figure 3 Each of the robotic arms 40a to 40d is attached to one of the entry ports 55a to 55d for insertion into the patient. IDU 52 is attached to retainer 46, and then SIM 43 is attached to the distal portion of IDU 52. Instrument 50 is then attached to SIM 43. Instrument 50 is then inserted into entry port 55 by moving IDU 52 along retainer 46. SIM 43 includes a plurality of drive shafts configured to transmit rotation of the respective motors of IDU 52 to instrument 50, thereby actuating instrument 50. Furthermore, SIM 43 provides a sterile barrier between instrument 50 and other components of robotic arm 40, including IDU 52. SIM 43 is also configured to secure a sterile drape (not shown) to IDU 52.

[0044] Surgical procedures may include multiple phases, and each phase may include one or more surgical actions. As used herein, the term "phase" refers to a surgical event consisting of a series of steps (e.g., closure). "Surgical actions" may include incision, compression, staplement, clamping, suturing, cauterization, sealing, or any other such action performed to complete a surgical phase. "Step" refers to achieving a specified surgical goal (e.g., hemostasis). During each step, certain surgical instruments 50 (e.g., clamps) are used to achieve the specific goal by performing one or more surgical actions.

[0045] refer to Figure 6The surgical robotic system 10 may include a machine learning (ML) processing system 310 that uses one or more ML models to process surgical data to identify one or more features in the surgical data, such as surgical stage, instruments, anatomical structures, etc. The ML processing system 310 includes an ML training system 325, which may be a separate device (e.g., a server) that stores its output as one or more trained machine learning models 330. The ML models 330 can be accessed by an ML execution system 340. The ML execution system 340 may be separate from the ML training system 325; that is, the means of “training” the model is separate from the means of “inference” (i.e., performing real-time processing of surgical data using the trained ML models 330).

[0046] System 10 includes a data receiving system 305 for collecting surgical data, including video data and surgical instrument data. The data receiving system 305 may include one or more devices (e.g., one or more user devices and / or servers) located within and / or associated with the operating room and / or control center. The data receiving system 305 can receive surgical data in real time (i.e., while surgery is being performed).

[0047] In some examples, the ML processing system 310 may also include a data generator 315 for generating simulated surgical data, such as a set of virtual or masked images, or recording video data from the image processing device 56, to train the ML model 330 and other data sources, such as user input, arm movements, etc. The data generator 315 can access (read / write) the data storage area 320 to record data, including multiple images and / or multiple videos.

[0048] The ML processing system 310 also includes a stage detector 350 that uses an ML model to identify stages within a surgical procedure. The stage detector 350 uses a specific surgical tracking data structure 355 from a list of surgical tracking data structures. The stage detector 350 selects the surgical tracking data structure 355 based on the type of surgery being performed. In one or more examples, the type of surgery is predetermined or input by the user. The surgical tracking data structure 355 identifies a set of potential stages that may correspond to a portion of a specific type of surgery.

[0049] In some examples, the surgical tracking data structure 355 may be a graph comprising a set of nodes and a set of edges, where each node corresponds to a potential stage. Edges provide directional connections between nodes that indicate (by direction) the expected order in which stages will be encountered throughout repetitions of the surgical procedure. The surgical tracking data structure 355 may include one or more branch nodes fed to multiple next nodes, and / or may include one or more bifurcation points and / or convergence points between nodes. In some cases, a stage indicates a surgical action (e.g., a surgical procedure) being performed or already performed and / or indicates a combination of actions already performed. In some cases, a stage is associated with the biological state of the patient undergoing the surgery. For example, a biological state may indicate complications (e.g., thrombosis, arterial / venous occlusion, etc.), pre-existing conditions (e.g., lesions, polyps, etc.). In some examples, the ML model 330 is trained to detect “abnormal conditions” such as bleeding, arrhythmias, vascular abnormalities, etc.

[0050] The stage detector 350 outputs a stage prediction associated with a portion of video data analyzed by the ML processing system 310. The stage prediction is associated with that portion of the video data by identifying the start and end times of the portion analyzed by the ML execution system 340. The output stage prediction may include an identifier of the surgical stage detected by the stage detector 350 based on the output of the ML execution system 340. Furthermore, in one or more examples, the stage prediction may include identifiers of structures (e.g., instruments, anatomical structures, etc.) identified by the ML execution system 340 in the portion of the video being analyzed. The stage prediction may also include a confidence score for the prediction. Other examples may include various other types of information in the output stage prediction. The controller 21a may use the predicted stage to determine when to switch between various imaging modalities, as described below.

[0051] refer to Figure 7The surgical robot system 10 also includes an imaging system 400, in which a laparoscopic camera 51 is coupled to an image processing device 56. The laparoscopic camera 51 includes a laparoscope 402 having a longitudinal axis 414 with multiple optical components (not shown), such as lenses, mirrors, and prisms disposed in the longitudinal axis 414. The laparoscope 402 is coupled to a combined light source 406 via an optical fiber 408. The light source 406 may include a white light source (not shown) and a NIR light source (not shown), which may be light-emitting diodes or any other suitable light source. The NIR light source may be a laser or any other suitable light source. The optical fiber 408 may include one or more optical fibers for transmitting white light and NIR light that illuminate tissue observed by the laparoscope 402. The laparoscope 402 collects the reflected white light and NIR light and transmits it to a camera assembly 410 coupled to the proximal end portion of the laparoscope 402. Laparoscope 402 can be any conventional laparoscope configured to emit and collect white light and NIR light.

[0052] Camera assembly 410 is configured to separate the fluorescence wavelength from unwanted components of the spectrum to a specific sensor. Specifically, the camera assembly includes a white (e.g., visible) light (VIS) sensor and an IR sensor, and is configured to separate and transmit white light to the VIS sensor and to separate and transmit fluorescent IR light to the IR sensor. The VIS and IR sensors can be complementary metal-oxide-semiconductor (CMOS) image sensors with any desired resolution, which in this embodiment could be 4K, UHD, etc.

[0053] Camera assembly 410 is connected to image processing device 56 via transmission cable 412. Image processing device 56 is configured to receive image data signals, process raw image data from camera assembly 410, and generate mixed white light and NIR images for recording and / or real-time display. Image processing device 56 also processes image data signals and transmits the processed images through any suitable video output port (such as a DisplayPort) capable of transmitting at any desired resolution, display rate, and / or bandwidth. ™ HDMI ® (etc.) output it to any of the displays 23, 32, 34 of the surgical robot system 10.

[0054] Figure 8A GUI 500 for controlling the imaging system 400 is shown, which can be displayed on the displays 23, 32, and 34 of the surgical robotic system 10. The GUI 500 includes options for controlling fluorescence settings, including turning fluorescence (e.g., NIR detection) on or off via a toggle key 502. Once fluorescence is selected, the user can also select from several imaging modes that visualize NIR light. An overlay mode can be selected via button 504. In overlay mode, the imaging system 400 combines a white image and an NIR image, during which a regular white light image is combined with NIR / ICG data to generate an overlay image. In this mode, the imaging system can be configured to display NIR light in visible light depending on user preference and application, and the NIR / ICG data can be displayed as a green or blue overlay, selectable via menu 505. In an intensity map mode selectable via button 506, the imaging system uses color scales in the overlay image to display the intensity of the NIR / ICG signal. In monochrome mode, selectable via button 508, the NIR / ICG signal is displayed separately in white against a black background to achieve maximum possible distinction, such as... Figure 14 As shown. In the implementation, the mode can also be selected via one or more foot pedals 36 associated with the mode selection, for example, a foot pedal cycle through each NIR mode in the NIR modes.

[0055] refer to Figure 9 System 10 includes a first camera 51 inserted into and coupled to a first robotic arm 40a via a first access port 55a. The first camera 51 includes a first field of view (FOV) pointing towards a surgical site including a first instrument 50 and a second instrument 50'. Camera 51 can be a stereo or monocular camera with any suitable lens (e.g., a 30° angle lens). As described above, the first camera 51 is capable of operating in multiple imaging modes, including a low-visibility monochrome mode.

[0056] System 10 also includes a second camera 51', which is inserted via a second access port 55b such that a second field of view (FOV) of the second camera 51' covers the surgical site. The second camera 51' is coupled to a second robotic arm 40b. Furthermore, the second camera 51' can be used with [other devices] as described above. Figure 7The first camera 51, as described, is connected to the light source 46 and the image processing device 56 in the same manner. The second camera 51' may be a stadium-view camera with a wide-angle lens providing a wide field of view (FOV) covering the surgical site. Camera 51 may be a stereo or monocular camera with any suitable lens (e.g., a 0° angle lens). The second camera 51' may be capable of operating in multiple imaging modes, at least one of which is a white light (i.e., visible color) mode. The white light frames obtained by the second camera 51' are used to generate a stack on the frames obtained by the first camera 51 when imaging in a low-visibility mode on instruments 50 and 50'.

[0057] Figure 10 A method is illustrated that uses masked images of instruments 50 and 50' from a second camera 51' as an overlay on low-visibility mode frames of instruments 50 and 50' obtained from a first camera 51. This method can be implemented as software instructions stored in a non-transitory medium (e.g., storage device 22a) that can be executed by a processor (e.g., controller 21a).

[0058] At step 600, the first camera 51 is used to acquire one or more low-visibility frames of instruments 50 and 50', while the second camera 51' is used to acquire white-light frames of these instruments at step 602, a step performed in parallel with step 600. The image processing device 56 can operate in two corresponding modes, namely, a low-visibility imaging mode for camera 51 and a white-light mode for camera 51'.

[0059] At step 604, the image processing device 56 processes the monochrome frame to adjust image characteristics such as brightness, sharpness, and contrast. At step 606, the image processing device 56 performs the same operation on the white light frame.

[0060] At step 608, the image processing device 56 outputs the processed monochrome frame from the first camera 51 onto the main screen 32 of the surgeon's console 30. Similarly, at step 610, the image processing device 56 outputs the processed white light frame from the second camera 51' in PIP mode onto the second screen 34 of the surgeon's console 30 or onto the main screen 32. When the low visibility mode is active, steps 604 and 608 can be repeated continuously to continuously capture and process monochrome frames.

[0061] At step 612, the image processing device 56 extracts images of instruments 50 and 50' from the white light frame to generate a stack of instruments 50 and 50'. (See reference) Figure 11 Step 612 includes multiple sub-steps 613a to 613e. Image extraction can be performed on each frame of the video feed from the second camera 51' or a selected number of frames of the color video feed (e.g., periodically). Reference Figure 9 and Figure 12 In step 613a, image rotation is performed on the selected frame to correspond the white light frame to the low visibility frame. This is done because the second camera 51' is in a different position from the first camera 51, and therefore the viewpoints of the frames are different. (See reference...) Figure 9 Each of the first camera 51 and the second camera 51' has its own 3D coordinate frame; that is, the coordinates of the first camera 51 are x... e y e z e And the coordinates of the second camera 51' are x s y s z s Rotate the white light frame so that the coordinates of the frame captured by the second camera 51' match the coordinates of the low-visibility frame captured by the first camera 51.

[0062] At step 613b, the white frame is also scaled (enlarged or reduced) to match the size of the low-visibility frame, also due to the different perspectives of the first camera 51 and the second camera 51'. At step 613c, the rotated and scaled frame is masked to isolate specific areas of the frame, i.e., portions of the frame containing the images of instruments 50 and / or 51'. Computer vision AI / ML algorithms can be used to identify instruments 50 and 50' in the white light frame using edge detection or another image processing algorithm. The detected edges can then be used to generate a masked image.

[0063] refer to Figure 13 Then, in step 613d, the masking image is translated by moving the image along the horizontal and / or vertical directions to align the masking image with the monochrome frame. In step 613e, a transparency factor (e.g., percentage) is applied to the masking image, which can be based on a user-selected transparency factor or the default system setting. The image processing device 56 then generates an image as shown below. Figure 14 Image 690 extracted from the instruments 50 and / or 50' shown.

[0064] refer to Figure 10 In step 614, the image processing device 56 loads the extracted image 690 into the buffer. For example... Figure 14As shown, at step 616, the extracted image 690 is co-registered with the low-visibility image frame 694. Registration can be performed using the procedure described below in step 716. After registration, the extracted image 690 is overlaid on the images of instruments 50 and 50' in the low-visibility frame. A composite image can be generated by performing alpha blending and / or chroma keying on the extracted image 690 and the low-visibility frame 694. At step 618, the resulting composite image is displayed on the main screen 32 of the surgeon's console 30. The composite image can be displayed with each frame, with every other image, or as any other percentage of the low-visibility frames, depending on the processing bandwidth of the image processing device 56.

[0065] Figure 15 A method is illustrated that uses masked images of instruments 50 and 50' from a second camera 51' as an overlay on low-visibility mode frames of instruments 50 and 50' obtained from a first camera 51. This method can be implemented as software instructions stored in a non-transient medium (e.g., storage device 22a) that can be executed by a processor (e.g., controller 21a). Figure 15 The method is basically similar to Figure 10 The method, in which kinematic feedback is added by a robotic arm 40 that controls the first camera 51 and the second camera 51', and steps with similar reference numerals (e.g., 600 and 700) describe the same motion.

[0066] At step 700, the first camera 51 is used to acquire one or more low-visibility frames of instruments 50 and 50', while the second camera 51' is used to acquire white-light frames of these instruments at step 702, a step performed in parallel with step 700. The image processing device 56 can operate in two corresponding modes, namely, a low-visibility imaging mode for camera 51 and a white-light mode for camera 51'.

[0067] At step 704, the image processing device 56 processes the monochrome frame to adjust image characteristics such as brightness, sharpness, and contrast. At step 706, the image processing device 56 performs the same operation on the white light frame.

[0068] At step 708, the image processing device 56 outputs the processed monochrome frame from the first camera 51 onto the main screen 32 of the surgeon's console 30. Similarly, at step 710, the image processing device 56 outputs the processed white light frame from the second camera 51' in PIP mode (if enabled) onto the second screen 34 of the surgeon's console 30 at step 711a or onto the main screen 32 at step 711b. When low visibility mode is active, steps 704 and 708 can be repeated continuously to continuously capture and process monochrome frames. At step 709, the previously processed monochrome image can be stored at the image processing device 56.

[0069] At step 712, image processing device 56 isolates and extracts images of instruments 50 and 50' from the white light frame to generate a stack of instruments 50 and 50'. At step 713, image processing device 56 performs instrument detection using an artificial intelligence / computer vision (AI / CV) algorithm. Once identified, the instrument image is masked (e.g., using edge detection) and subsequently filtered. At step 714, image processing device 56 loads the extracted image into a buffer, which is then used to display the extracted image as a stack on a portion of the monochrome image showing instruments 50 and 50'.

[0070] In step 716, the image processing device 56 co-registers the extracted image 690 with the low-visibility image frame 694, such as... Figure 14 As shown. Multiple markers 692 found in the extracted image 690 are used to register the extracted image 690 with the low-visibility frame 694, such that the extracted image 690 is overlaid on the images of the instruments 50 and 50' in the low-visibility frame.

[0071] Registration of the two images is performed using kinematic data from each of the first robotic arms 40a and 40b, which respectively control the first camera 51 and the second camera 51'. The kinematic data includes the joint angle of each of the robotic arms 40a and 40b, as well as any other positioning information. The kinematic data is used to orient the robotic arms 40a and 40b in the same coordinate system. At step 717, the controller 21a continuously records the kinematic data and timestamps it to match each frame captured by the image processing device 56. In addition to the kinematic data, supplementary data about the first and second cameras, such as camera angles, camera rotation relative to 0° defined by the sliding mechanism 46a, camera distance, lens type, etc., may also be collected.

[0072] At step 719, affine and nonlinear transformations are used to transform the extracted image 690 to align it with the low-visibility frame 694. Specifically, the extracted image 690 can be rotated, scaled, cropped, dilated, reflected, etc., as described above. Figure 9 and Figure 12 As described, rotation can be used because each of the first camera 51 and the second camera 51' has its own 3D coordinate frame; that is, the coordinates of the first camera 51 are x... e y e z e And the coordinates of the second camera 51' are x s y s z sThe extracted image 690 is rotated so that its coordinates match the coordinates of the low-visibility frame captured by the first camera 51. The corresponding coordinates are derived from the kinematic data. The absolute reference frame is a so-called "world" frame, which represents the space in which the system 10 is set. The kinematic data is mapped to the absolute reference frame, thereby allowing the use of the absolute reference frame as a common reference frame to align images captured in different frames.

[0073] refer to Figure 13 The extracted image 690 can also be translated by moving the image along the horizontal and / or vertical direction to align the masked image with the monochrome frame. A composite image can be generated by performing alpha blending and / or chroma keying on the extracted image 690 and the low-visibility frame 694. The resulting composite image is displayed on the main screen 32 of the surgeon's console 30. The composite image can be displayed with each frame, with every other image, or as any other percentage of the low-visibility frame, depending on the processing bandwidth of the image processing device 56.

[0074] It should be understood that various modifications can be made to the embodiments disclosed in this invention. Therefore, the above description should not be construed as limiting, but rather as illustrative of various embodiments only. Those skilled in the art will be able to conceive of other modifications within the scope and spirit of the appended claims.

[0075] The following examples illustrate the techniques described herein.

[0076] Example 1. An imaging system comprising: a first laparoscopic camera disposed at a first location within a patient for capturing a first video feed of a surgical instrument from a first perspective; a second laparoscopic camera disposed at a second location within the patient for capturing a second video feed of the surgical instrument from a second perspective different from the first perspective; an image processing device coupled to the first and second laparoscopic cameras, the image processing device processing the first video feed of the surgical instrument in a low-visibility imaging mode and processing the second video feed of the surgical instrument in a white imaging mode; a controller configured to: extract a white image from frames of the second video feed; generate a stack of the surgical instrument from the white image; and, when in the low-visibility imaging mode, generate a composite frame including the stack and frames of the first video feed, the stack being disposed on a portion of the first video feed including the surgical instrument; and a screen for displaying the composite video feed including the composite frame.

[0077] Example 2. The imaging system according to Example 1, wherein the controller causes at least one of the stack or the frames of the first video feed to rotate relative to each other to align their respective coordinate systems.

[0078] Example 3. The imaging system according to Example 1, wherein the controller causes at least one of the stack or the frame of the first video feed to translate relative to each other to align their respective coordinate systems.

[0079] Example 4. The imaging system according to Example 1, wherein the low visibility imaging mode is a monochromatic near-infrared (NIR) mode.

[0080] Example 5. The imaging system according to Example 1, wherein the first laparoscopic camera has a first field of view, the second laparoscopic camera has a second field of view that at least partially overlaps with the first field of view, and the surgical instruments are within the first field of view and the second field of view.

[0081] Example 6. The imaging system according to Example 5, wherein the second field of view is larger than the first field of view.

[0082] Example 7. The imaging system according to Example 1, wherein the controller uses at least one of alpha blending or chroma keying to generate the composite frame.

[0083] Example 8. A surgical robot system comprising: a first robotic arm including a first laparoscopic camera disposed at a first location within a patient for capturing a first video feed of surgical instruments from a first perspective; a second robotic arm including a second laparoscopic camera disposed at a second location within the patient for capturing a second video feed of the surgical instruments from a second perspective different from the first perspective; an image processing device coupled to the first and second laparoscopic cameras, the image processing device processing the first video feed of the surgical instruments in a low-visibility imaging mode and processing the second video feed of the surgical instruments in a white imaging mode; a controller configured to: extract a white image from frames of the second video feed; generate a stack of the surgical instruments from the white image; and, when in the low-visibility imaging mode, generate a composite frame including the stack and frames of the first video feed, the stack being disposed on a portion of the first video feed including the surgical instruments; and a screen for displaying the composite video feed including the composite frame.

[0084] Example 9. The surgical robot system according to Example 8, wherein the controller receives kinematic data from the first robotic arm and the second robotic arm.

[0085] Example 10. The surgical robot system according to Example 9, wherein the kinematic data includes positioning information of the first robotic arm and the second robotic arm, as well as the first laparoscopic camera and the second laparoscopic camera.

[0086] Example 11. A surgical robot system according to Example 9, wherein the controller rotates at least one of the stack or the frames of the first video feed relative to each other based on the kinematic data to align their respective coordinate systems.

[0087] Example 12. The surgical robot system according to Example 9, wherein the controller, based on the kinematic data, translates at least one of the stack or the frames of the first video feed relative to each other to align their respective coordinate systems.

[0088] Example 13. The surgical robot system according to Example 8, wherein the low-visibility imaging mode is a monochromatic near-infrared (NIR) mode.

[0089] Example 14. The surgical robot system according to Example 8, wherein the first laparoscopic camera has a first field of view, the second laparoscopic camera has a second field of view that at least partially overlaps with the first field of view, and the surgical instruments are within the first field of view and the second field of view.

[0090] Example 15. The surgical robot system according to Example 14, wherein the second field of view is larger than the first field of view.

[0091] Example 16. The surgical robot system according to Example 8, wherein the controller uses at least one of alpha blending or chroma keying to generate the synthesized frame.

[0092] Example 17. A method for generating a synthetic instrument image, the method comprising: capturing a first video feed of a surgical instrument from a first perspective using a first laparoscopic camera positioned at a first location within a patient; capturing a second video feed of the surgical instrument from a second perspective different from the first perspective using a second robotic arm, the second robotic arm including a second laparoscopic camera positioned at a second location within the patient; processing the first video feed of the surgical instrument in a low-visibility imaging mode and processing the second video feed of the surgical instrument in a white imaging mode at an image processing device coupled to the first laparoscopic camera and the second laparoscopic camera; extracting a white image from a frame of the second video feed; generating a stack of the surgical instrument from the white image; generating a synthetic frame including the stack and a frame of the first video feed when in the low-visibility imaging mode, the stack being disposed on a portion of the first video feed including the surgical instrument; and displaying the synthetic video feed including the synthetic frame on a screen.

[0093] Example 18. The method according to Example 17, the method further comprising: rotating at least one of the stack or the frames of the first video feed relative to each other to align their respective coordinate systems.

[0094] Example 19. The method according to Example 17, the method further comprising: translating at least one of the stack or the frames of the first video feed relative to each other to align their respective coordinate systems.

[0095] Example 20. The method according to Example 17, wherein at least one of α-blending or chroma keying is used to generate the composite frame.

Claims

1. An imaging system (400), the imaging system comprising: A first laparoscopic camera (51) is positioned at a first location within the patient's body to capture a first video feed of surgical instruments from a first perspective. A second laparoscopic camera (51') is positioned at a second location within the patient's body to capture a second video feed of the surgical instruments from a second perspective different from the first perspective. Image processing device (56), the image processing device being connected to the first laparoscopic camera and the second laparoscopic camera, the image processing device processing the first video feed of the surgical instrument in a low visibility imaging mode and processing the second video feed of the surgical instrument in a white imaging mode; Controller (21a), the controller is used for: Extract the white image (690) from the frames of the second video feed. Generate a stack of the surgical instruments from the white image; as well as When in the low-visibility imaging mode, a composite frame (694) is generated, comprising the overlay and a frame from the first video feed, the overlay being disposed on a portion of the first video feed including the surgical instrument; and Screen (32), the screen is used to display a composite video feed including the composite frame.

2. The imaging system of claim 1, wherein the controller causes at least one of the stack or the frames of the first video feed to rotate relative to each other to align their respective coordinate systems.

3. The imaging system according to any of the preceding claims, wherein the controller causes at least one of the stack or the frame of the first video feed to translate relative to each other to align their respective coordinate systems.

4. The imaging system according to any of the preceding claims, wherein the low-visibility imaging mode is a monochromatic near-infrared (NIR) mode.

5. The imaging system according to any of the preceding claims, wherein the first laparoscopic camera has a first field of view, the second laparoscopic camera has a second field of view that at least partially overlaps with the first field of view, and the surgical instruments are within the first field of view and the second field of view.

6. The imaging system according to claim 5, wherein the second field of view is larger than the first field of view.

7. The imaging system according to any of the preceding claims, wherein the controller uses at least one of alpha blending or chroma keying to generate the composite frame.

Images