Multispectral imaging camera and method of use

Abstract

A surgical robot system and method for providing simultaneous multispectral imaging are disclosed herein. In some embodiments, the system includes first and second camera assemblies having one or more LEDs, one or more lenses, one or more filter elements, and one or more imaging sensors, the first and second camera assemblies providing a stereoscopic image for viewing by a user of the system. The method includes providing an image or video that displays multiple spectra of light.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 345,800, filed May 25, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002] Laparoscopes or other minimally invasive surgical instruments or systems, such as robotic surgical systems, are used in several surgical procedures. Often, these devices and systems can include additional components such as cameras. The camera may be paired with one or more light sources depending on the desired imaging.

[0003] Fluorescence - based imaging provides the surgeon with visualization of anatomical structures and tissue activity that are not visible with normal visualization. One of the most common forms used in surgery is to use the dye indocyanine green (ICG) injected into the patient's bloodstream to image anatomical features and conditions such as tissue perfusion and blood flow. A plurality of other dyes and autofluorescence capabilities all enable potentially different visualization behaviors that can assist the surgeon in targeting the correct tissue to excise.

Summary of the Invention

Means for Solving the Problems

[0004] The present disclosure provides a multispectral camera assembly that can be used as part of a laparoscope or surgical robotic system, and a method of use whereby an operator (e.g., a surgeon) of the laparoscope or surgical robotic system can observe an internal cavity of a subject (e.g., a patient) by utilizing multispectral imaging. In some embodiments, the multispectral camera assembly enables simultaneous imaging of non - visible light, e.g., fluorescence, and visible light for visualization of internal body cavities.

[0005] The present disclosure is directed to a camera assembly configured for simultaneous multispectral imaging. According to some embodiments, the camera assembly includes a first lens assembly, a second lens assembly, a first plurality of light emitting diodes (LEDs) configured to emit light in a first wavelength range, a second plurality of LEDs configured to emit light in a second wavelength range, a plurality of LED bandpass filters, wherein each of the plurality of LED bandpass filters is positioned in front of each of the second plurality of LEDs to filter the light emitted from each of the second plurality of LEDs, a plurality of image sensors, wherein a first image sensor of the plurality of image sensors is positioned behind the first lens assembly to capture light from the first lens assembly, and a second image sensor of the plurality of image sensors is positioned behind the second lens assembly to capture light from the second lens assembly, a plurality of notch filters, wherein each notch filter is positioned between each of the plurality of image sensors and either the first lens assembly or the second lens assembly, and each notch filter is configured to filter out light in a selected wavelength range transmitted by the respective first and second lens assemblies, and a circuit board, wherein the circuit board is electronically coupled to the first and second plurality of LEDs and the plurality of image sensors, and the circuit board is configured to strobe the plurality of LEDs such that each of the plurality of image sensors simultaneously captures multiple spectra of light.

[0006] In some embodiments, the camera assembly further includes a laser. In further embodiments, the camera assembly further includes a laser bandpass filter, and the laser bandpass filter is positioned adjacent to the laser to allow light in a selected wavelength band from the laser to pass through the laser bandpass filter.

[0007] In some embodiments, the first plurality of LEDs are configured to emit light in the range of 400 nm to 700 nm, and the second plurality of LEDs are configured to emit light in the range of 800 nm to 820 nm. In some embodiments, the camera assembly further includes a third plurality of LEDs configured to emit light in the range of 475 nm to 505 nm. In some embodiments, at least one of the plurality of LED bandpass filters is configured to block all light except light having a wavelength of about 490 nm.

[0008] In some embodiments, the second plurality of LEDs are configured to excite a dye within biological tissue. In a further embodiment, the dye is a fluorescein dye. In some embodiments, at least one of the plurality of LED bandpass filters is configured to allow the passage of visible light.

[0009] The present disclosure also relates to a surgical robot system, comprising a first camera assembly having one or more LEDs, one or more lenses, one or more filter elements, and one or more imaging sensors; a second camera assembly having one or more LEDs, one or more lenses, one or more filter elements, and one or more imaging sensors, wherein the first and second camera assemblies provide a stereoscopic image for viewing by a user of the system; a memory storing one or more instructions; and a processor configured or programmed to read the one or more instructions stored in the memory, the processor being operably coupled to the first camera assembly and the second camera assembly to simultaneously capture a plurality of spectra of light from the first camera assembly and the second camera assembly.

[0010] In some embodiments, the system further includes a display operably connected to the first camera assembly and the second camera assembly, and the display is configured to depict images captured by one or more imaging sensors of each camera assembly. In a further embodiment, the processor is configured to strobe a plurality of LEDs such that the images are composed of multiple spectra of light.

[0011] In some embodiments, at least one of the first camera assembly or the second camera assembly further includes a laser. In a further embodiment, at least one of the first camera assembly or the second camera assembly further includes a laser bandpass filter, and the laser bandpass filter is positioned adjacent to the laser to allow light in a selected wavelength band from the laser to pass through the laser bandpass filter.

[0012] In some embodiments, one or more LEDs of at least one of the first camera assembly or the second camera assembly include at least one LED configured to emit light in the range of 400 nm to 700 nm and at least one LED configured to emit light in the range of 800 nm to 820 nm. In a further embodiment, one or more LEDs of at least one of the first camera assembly or the second camera assembly further include at least one LED configured to emit light in the range of 475 nm to 505 nm.

[0013] In some embodiments, one or more filter elements of at least one of the first camera assembly or the second camera assembly are configured to block all light except light having a wavelength of about 490 nm. In some embodiments, one or more LEDs of at least one of the first camera assembly or the second camera assembly are configured to excite a dye in biological tissue. In a further embodiment, the dye is a fluorescein dye. In some embodiments, one or more filter elements of at least one of the first camera assembly or the second camera assembly are configured to allow the passage of visible light.

[0014] The novel features of the present invention are described in detail in the appended claims. A better understanding of the features and advantages of the present invention can be obtained by referring to the following detailed description which describes exemplary embodiments in which the principles of the present invention are utilized. In the accompanying drawings, like reference numerals are used to identify like components, although these may not be the same components.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 11

Figure 12A

Figure 12B

Figure 13A

Figure 13B

Figure 13C

Figure 14

Figure 15A

Figure 15B

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 22C

Figure 22D

Figure 23A

Figure 23B

Figure 23C

Figure 23D

Figure 24

Figure 25

Figure 26

Figure 27A

Figure 27B

Figure 27C

Figure 27D

Figure 28

Figure 29A

Figure 29B

Figure 30A

Figure 30B

Figure 30C

Figure 31

DETAILED DESCRIPTION OF THE INVENTION

[0016] Although various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the present invention. It can be understood that various alternatives to the embodiments of the present invention described herein may be used.

[0017] As used in this specification and the claims, the singular forms "a", "an", and "the" include plural referents unless the content clearly dictates otherwise.

[0018] Before providing additional specific descriptions of the multispectral camera assemblies as taught herein with respect to FIGS. 6-31, a surgical robot system that may be employed in some embodiments will be described below with respect to FIGS. 1-5. In some embodiments, the multispectral camera assembly may be used without a surgical robot system.

[0019] One of the challenges in designing a camera system that enables the simultaneous visualization of different spectra (e.g., indicating fluorescence and visible light) is the image sensor. Previous solutions have addressed this problem by placing multiple different sensors in the camera system, each being unique to a selected subset of wavelengths. Another common approach is to modify the Bayer pattern by adding specific pixels that are sensitive to a subset of the band (IR pixels), or by using a hyperspectral imaging sensor with a unique custom pattern. However, these approaches increase the complexity and cost of the camera system, which may not be practical for surgical solutions. These approaches also reduce the sensitivity of the captured color spectrum because the approach reduces the active area of the imaging.

[0020] Fluorescence can, for example, assist in visualizing blood vessels, ureters, cancer, nerves, and tissue perfusion. All types of fluorescence, such as dyes, autofluorescence, and other types of differential visualization, may be paired with a multispectral imaging system. The disclosed imaging system functions by controlling the illumination environment, synchronizing the light source to a specific image, and selectively displaying that image to the surgeon. This enables multiple different visualizations to be used simultaneously with the overlay live color without the need for additional sensors. The system employs a filter on the camera assembly that selectively blocks specific frequencies of the emitted light.

[0021] The present disclosure provides a multispectral camera assembly, whereby an operator of the camera assembly (e.g., a surgeon) can observe an internal cavity of a subject (e.g., a patient) by utilizing the coordinated movement of the camera assembly, according to some embodiments. In some embodiments, the multispectral camera assembly enables simultaneous imaging of non-visible light, such as fluorescence, and visible light, and visualization of internal body cavities. In some embodiments, the camera assembly provides a 360-degree field of view, or at least two degrees of freedom, for changing the orientation of the direction of the camera assembly's field of view without the need to change the position (e.g., translation) or orientation (e.g., tilt) of the support of the camera assembly that extends outside the subject's body. In some embodiments, the camera assembly provides at least three degrees of freedom for changing the orientation of the direction of the camera assembly's field of view without the need to change the position (e.g., translation) or orientation (e.g., tilt) of the support of the camera assembly that extends outside the subject's body. In some embodiments, the orientation of the direction of the camera assembly's field of view can be tilted or rotated around three orthogonal axes without translating or tilting the support of the camera assembly that extends outside the subject's body.

[0022] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the disclosed subject matter. It will be apparent, however, to one skilled in the art that the disclosed subject matter may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid obscuring the complexity of the disclosed subject matter. Further, any examples provided below are merely illustrative and not to be construed in a limiting fashion. It will be understood by the inventors that other systems, devices, and / or methods may be used to practice or implement the teachings of the present invention and be within the scope of the present invention.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a,” “an,” and “the” as used herein are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises” and / or “comprising,” when used herein, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0024] Unless specifically stated or obvious from context, as used herein, the term “about” is understood to be within a normal tolerance in the art, for example within 2 standard deviations of the mean. “About” can be understood as within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise clear from context, all numerical values provided herein are modified by the term “about.”

[0025] While the camera assembly and method of the present disclosure can be designed to be used in conjunction with one or more surgical robotic systems, the robotic systems of the present disclosure can be used in connection with any type of surgical system, including, for example, robotic surgical systems, linear stick surgical systems, virtual reality surgical systems, and laparoscopic systems. Further, the camera assembly of the present disclosure can be used in other non-surgical systems where a user needs access to numerous information while controlling the device or equipment.

[0026] The camera assembly of the present disclosure assists a surgeon in controlling the movement of a robotic unit during a surgery in which the robotic unit is operable within a patient's body. Thus, the imaging features of the present disclosure enable the surgeon to minimize the risk of accidental injury to the patient during the surgery.

[0027] Similar numerical identifiers are used throughout the figures to refer to the same element.

[0028] FIG. 1 is a schematic view of a surgical robotic system 10 in which aspects of the present disclosure can be employed according to some embodiments of the present disclosure. The surgical robotic system 10 includes an operator console 11 and a robotic subsystem 20 according to some embodiments.

[0029] The surgical robot system 10 of the present disclosure uses a robot subsystem 20 that includes a robot unit 50 that can be inserted into a patient through a trocar through a single incision point or site. The robot unit 50 is small enough to be deployed in vivo at the surgical site and is sufficiently maneuverable when inserted into the patient's body so as to be able to move within the body to perform various surgical procedures at a plurality of different points or sites. The robot unit 50 includes a plurality of separate robot arms 42 that can be deployed within the patient along different or distinct axes. Further, a surgical camera assembly 44 can also be deployed along a separate axis and can form part of the robot unit 50. Thus, the robot unit 50 uses a pair of robot arms and a plurality of different components such as a surgical or robot camera assembly, each of which can be deployed along a different axis and is separately operable, maneuverable, and movable. In particular, the robot unit 50 is not limited to the robot arms and camera assemblies described herein, and additional components may be included in the robot unit. The robot arms and camera assemblies that can be disposed along separate operable axes are referred to herein as a split arm (SA) architecture. The SA architecture simplifies and enhances the efficiency of inserting robotic surgical instruments through a single trocar at a single insertion site, while also assisting in the deployment of the surgical instruments to a surgically ready state and the subsequent removal of the surgical instruments through the trocar. As an example, a surgical instrument can be inserted through the trocar to access the patient's abdominal cavity and perform surgery in vivo. In some embodiments, various surgical instruments may be utilized, including but not limited to robotic surgical instruments and other surgical instruments known in the art.

[0030] The operator console 11 includes a display 12, an image calculation module 14 (which may be a three-dimensional (3D) calculation module), a hand controller 17 having a sensing and tracking module 16, and a calculation module 18. Further, the operator console 11 may include a foot pedal array 19 including a plurality of pedals. The image calculation module 14 may include a graphical user interface 39. The graphical user interface 39, the controller 26, or the image rendering device 30, or both, may render one or more images or one or more graphical user interface elements on the graphical user interface 39. For example, a pillar box associated with a mode of operating the surgical robot system 10 or any of the various components of the surgical robot system 10 may be rendered on the graphical user interface 39. Live video footage captured by the camera assembly 44 can also be rendered on the graphical user interface 39 by the controller 26 or the image rendering device 30.

[0031] The operator console 11 may include a visualization system 9 including a display 12, which can be any selected type of display for displaying information, images, or videos generated by the image calculation module 14, the calculation module 18, and / or the robot subsystem 20. The display 12 can include, for example, a head-mounted display (HMD), an augmented reality (AR) display (e.g., an AR display, or AR glasses combined with a screen or display), a screen or display, a two-dimensional (2D) screen or display, a three-dimensional (3D) screen or display, etc., or can form a part thereof. The display 12 may also optionally include a sensing and tracking module 16A. In some embodiments, the display 12 may include an image display for outputting images from the camera assembly 44 of the robot subsystem 20.

[0032] The hand controller 17 is configured to sense the movement of the operator's hand and / or arm to operate the surgical robot system 10. The hand controller 17 may include a sensing and tracking module 16, circuitry, and / or other hardware. The sensing and tracking module 16 may include one or more sensors or detectors that sense the movement of the operator's hand. In some embodiments, one or more sensors or detectors that sense the movement of the operator's hand are disposed within the hand controller 17 that is gripped or engaged by the operator's hand. In some embodiments, one or more sensors or detectors that sense the movement of the operator's hand are coupled to the operator's hand and / or arm. For example, the sensors of the sensing and tracking module 16 may be coupled to areas of the hand and / or arm such as the fingers, wrist area, elbow area, and / or shoulder area. In some embodiments, additional sensors can also be coupled to the operator's head and / or neck area. In some embodiments, the sensing and tracking module 16 may be external and may be coupled to the hand controller 17 via electrical components and / or wearable hardware. In some embodiments, an optional sensor and tracking module 16A may sense and track the movement of one or more of the operator's head, the operator's eyes, or at least a portion of the operator's neck, based at least in part on imaging of the operator, in addition to, or instead of, sensors attached to the operator's body.

[0033] In some embodiments, the sensing and tracking module 16 can use sensors coupled to the operator's torso or any other body part. In some embodiments, the sensing and tracking module 16 can use, in addition to sensors, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of the magnetometer can reduce sensor drift around the vertical axis. In some embodiments, the sensing and tracking module 16 also includes sensors placed within surgical materials such as gloves, surgical scrubs, or surgical gowns. The sensors may be reusable or disposable. In some embodiments, the sensors can be disposed external to the operator, such as at a fixed location in a room such as an operating room. The external sensors 37 can generate external data 36 that can be processed by the computing module 18 and thus used by the surgical robot system 10.

[0034] The sensors generate position and / or orientation data indicating the position and / or orientation of the operator's hand and / or arm. The sensing and tracking module 16 and / or 16A can be utilized to control the movement (e.g., changes in position and / or orientation) of the camera assembly 44 and the robotic arm 42 of the robot subsystem 20. The tracking and position data 34 generated by the sensing and tracking module 16 can be transmitted to the computing module 18 for processing by at least one processor 22.

[0035] The computing module 18 can determine or calculate, from the tracking and position data 34 and 34A, the position and / or orientation of the operator's hand or arm, and in some embodiments, also of a part of the operator's head, and transmit the tracking and position data 34 and 34A to the robot subsystem 20. The tracking and position data 34, 34A can be processed by the processor 22 and stored, for example, in the storage 24. The tracking and position data 34 and 34A can also be used by the controller 26, which can generate control signals for controlling the movement of the robot arm 42 and / or the camera assembly 44 in response thereto. For example, the controller 26 can change the position and / or orientation of at least a part of the camera assembly 44, at least a part of the robot arm 42, or both. In some embodiments, the controller 26 can also adjust the pan and tilt of the camera assembly 44 to follow the movement of the operator's head. The computing module can further include a graphics processing unit (GPU) 52, which will be discussed in more detail below.

[0036] The robot subsystem 20 can include a robot support system (RSS) 46 having a motor 40 and a trochlear 50 or trochlear mount, a robot arm 42, and a camera assembly 44. The robot arm 42 and the camera assembly 44 can form part of a single support axis robot unit as disclosed and described in U.S. Patent No. 10,285,765, or part of a split arm (SA) architecture robot system as disclosed and described in PCT Patent Application No. PCT / US2020 / 039203, both of which are hereby incorporated by reference in their entirety.

[0037] The robotic subsystem 20 can utilize a plurality of different robotic arms that can be deployed along different or separate axes. In some embodiments, a camera assembly 44 that can utilize a plurality of different camera elements can also be deployed along a common separate axis. Thus, the surgical robotic system 10 can utilize a pair of separate robotic arms deployable along different axes and a plurality of different components such as the camera assembly 44. In some embodiments, the robotic arm assembly 42 and the camera assembly 44 are separately operable, maneuverable, and movable. The robotic subsystem 20, including the robotic arm 42 and the camera assembly 44, is disposable along separate operable axes and is referred to herein as the SA architecture. The SA architecture simplifies and enhances the efficiency of the insertion of robotic surgical instruments through a single trocar at a single insertion point or site, while also assisting in the deployment of the robotic surgical instruments into a surgically ready state and, as further described below, the subsequent removal of the robotic surgical instruments through the trocar 50.

[0038] The RSS 46 can include a motor 40 and the trocar 50 or a trocar mount. The RSS 46 can further include a support member that supports the motor 40 coupled to its distal end. The motor 40 can be coupled to each of the camera assembly 44 and the robotic arm assembly 42. The support member can be configured and controlled to operate one or more components of the robotic subsystem 20 linearly or in any other selected direction or orientation. In some embodiments, the RSS 46 can be upright. In some embodiments, the RSS 46 can include a motor 40 that is coupled to the robotic subsystem 20 at one end and to an adjustable support member or element at the opposite end.

[0039] The motor 40 can receive a control signal generated by the controller 26. The motor 40 can include gears, one or more motors, a drive train, electronics, etc. for powering and driving the robotic arm 42 and the camera assembly 44 either individually or together. The motor 40 can also provide mechanical power, power, mechanical communication, and electrical communication to the robotic arm 42, the camera assembly 44, and / or the RSS 46 and other components of the robot subsystem 20. The motor 40 can be controlled by the computing module 18. Thus, the motor 40 can generate signals for controlling one or more motors that can control and drive the robotic arm 42, including, for example, the position and orientation of the joints of each articulating joint of each arm, and the camera assembly 44. The motor 40 can further provide a translational or linear degree of freedom that is first utilized for inserting and removing each component of the robot subsystem 20 through the trocar 50. The motor 40 can also be used to adjust the insertion depth of each robotic arm 42 when inserted through the trocar 50 into the patient 300.

[0040] In some embodiments, the trocar 50 can be a medical device that can consist of a claw (which can be a sharp or bladeless tip of metal or plastic), a cannula (essentially a hollow tube), and a seal. The trocar 50 can be used to place at least a portion of the robotic subsystem 20 into an internal cavity of a subject (e.g., a patient) and can draw gas and / or fluid out of the body cavity. The robotic subsystem 20 can be inserted through the trocar 50 to access the patient's body cavity and perform surgery in vivo. In some embodiments, the robotic subsystem 20 of the present invention can be at least partially supported by the trocar 50 or a trocar mount with multiple degrees of freedom such that the robotic arm 42 and the camera assembly 44 can be maneuvered within the patient at a single location or multiple different locations. In some embodiments, the robotic arm 42 and the camera assembly 44 can be supported by the trocar 50 or a trocar mount with multiple degrees of freedom such that the robotic arm 42 and the camera assembly 44 can be maneuvered within the patient at a single location or multiple different locations.

[0041] In some embodiments, the RSS 46 can further include an optional controller for processing input data from one or more of the system components (e.g., the display 12, the sensing and tracking module 16, the robotic arm 42, the camera assembly 44, etc.) and for generating control signals in response thereto. The motor 40 can also include a storage element for storing data in some embodiments.

[0042] In some embodiments and in some operating modes, the robotic arm 42 can be controlled to follow the scaled-down movement or motion of the operator's arm and / or hand as sensed by associated sensors. The robotic arm 42 includes a first robotic arm including a first end effector having a tool tip disposed at the distal end of the first robotic arm, and a second robotic arm including a second end effector having a tool tip disposed at the distal end of the second robotic arm. In some embodiments, the robotic arm 42 can have parts or regions associated with the movement of the shoulder joint, elbow joint, and wrist joint and associated with the movement that can be associated with the operator's fingers. For example, the robotic elbow joint can follow the position and orientation of the human elbow, and the robotic wrist joint can follow the position and orientation of the human wrist. The robotic arm 42 can also have an end region associated therewith, which in some embodiments can terminate in an end effector that follows the movement of one or more fingers of the operator, such as the index finger, for example, when the user pinches the index finger and thumb together. In some embodiments, the robotic arm 42 may follow the movement of the operator's arm in some control modes, while the virtual chest of the robotic arm assembly may remain stationary (e.g., in the tool control mode). In some embodiments, the position and orientation of the operator's torso are subtracted from the position and orientation of the operator's arm and / or hand. This subtraction allows the operator to move the torso without the robotic arm moving. Further disclosure of the control of the movement of the individual arms of the robotic arm assembly is provided in International Patent Application Publication Nos. WO 2022 / 094000 A1 and WO 2021 / 231402 A1, each of which is incorporated herein by reference in its entirety.

[0043] The camera assembly 44 is configured to not only provide the operator with image data 48, such as, for example, live video feeds of a surgery or a surgical site, but also enable the operator to operate and control a camera forming part of the camera assembly 44. In some embodiments, the camera assembly 44 may include one or more cameras (e.g., a pair of cameras) whose optical axes are axially separated by a selected distance, which is known as the inter-camera distance, and which provides a stereoscopic view or image of the surgical site. In some embodiments, the operator can control the movement of the camera via hand movement, either via a sensor coupled to the operator's hand or via a hand controller 17 gripped or held by the operator's hand, and thus the operator can obtain a desired view of the surgical site in an intuitive and natural manner. In some embodiments, the operator can additionally control the movement of the camera via movement of the operator's head. The camera assembly 44 is movable in a plurality of directions with respect to the direction of the field of view, including, for example, the yaw direction, the pitch direction, and the roll direction. In some embodiments, the components of the stereoscopic camera may be configured to provide a natural and comfortable user experience. In some embodiments, the inter-axial distance between the cameras can be adjusted to accommodate the perceived depth of the surgical site by the operator.

[0044] The image or video data 48 generated by the camera assembly 44 can be displayed on the display 12. In embodiments, when the display 12 includes an HMD, the display can include an embedded sensing and tracking module 16A that obtains raw orientation data for the yaw, pitch, and roll directions of the HMD, as well as position data within the Cartesian space (x, y, z) of the HMD. In some embodiments, position and orientation data regarding the operator's head can be provided via a separate head tracking module. In some embodiments, the sensing and tracking module 16A can be used to provide supplemental position and orientation tracking data for the display, instead of or in addition to an embedded tracking system of the HMD. In some embodiments, operator head tracking is not used or employed. In some embodiments, the operator's image may be used by the sensing and tracking module 16A to track at least a portion of the operator's head.

[0045] FIG. 2A shows an exemplary robotic arm assembly 20 (also referred to herein as a robotic subsystem) of the surgical robot system 10 incorporated into or mounted on a mobile patient cart, according to some embodiments. In some embodiments, the robotic arm assembly 20 includes an RSS 46, which in turn includes a motor 40, the robotic arm 42 has an end effector 45, the camera assembly 44 has one or more cameras 47, and may also include a trocar 50 or a trocar mount.

[0046] FIG. 2B shows an example of an operator console 11 of the surgical robot system 10 of the present disclosure, according to some embodiments. The operator console 11 includes a display 12, a hand controller 17, and may also include one or more additional controllers such as a foot pedal array 19 for controlling the robotic arm 42, the camera assembly 44, and other aspects of the system.

[0047] FIG. 2B also illustrates the left hand controller subsystem 23A and the right hand controller subsystem 23B of the operator console. The left hand controller subsystem 23A includes and supports the left hand controller 17A, and the right hand controller subsystem 23B includes and supports the right hand controller 17B. In some embodiments, the left hand controller subsystem 23A may be removably connected or engaged to the left hand controller 17A, and the right hand controller subsystem 23B may be removably connected or engaged to the right hand controller 17A. In some embodiments, the connection may be both physical and electronic such that the left hand controller subsystem 23A and the right hand controller subsystem 23B can receive signals from the left hand controller 17A and the right hand controller 17B respectively, including signals conveying input received from user selection on buttons or touch input devices of the left hand controller 17A or the right hand controller 17B.

[0048] Each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may include components that enable the range of motion of their respective left hand controller 17A and right hand controller 17B, such that the left hand controller 17A and the right hand controller 17B may translate or displace in three dimensions and may additionally move in roll, pitch, and yaw directions. Further, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may register the operation of their respective left hand controller 17A and right hand controller 17B in each of the aforementioned directions and may transmit signals providing such movement information to a processor (not shown) of the surgical robot system.

[0049] In some embodiments, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may be configured to receive and connect to, or engage with, different hand controllers (not shown). For example, hand controllers having different configurations of buttons and touch input devices may be provided. Additionally, hand controllers having different shapes may be provided. The hand controller may be selected to be compatible with a particular surgical robot system or a particular surgical robot procedure, or may be selected based on the operator's preference for the buttons and input devices, or with respect to the shape of the hand controller, to provide more comfort and ease for the operator.

[0050] Figure 3A schematically shows a side view of a surgical robot system 10 for performing surgery within an internal cavity 304 of a subject 300, according to some embodiments and for some surgical procedures. Figure 3B schematically shows a perspective top view of a surgical robot system 10 for performing surgery within an internal cavity 304 of a subject 300. The subject 300 (e.g., a patient) is placed on an operating table 302 (e.g., a surgical operating table 302). In some embodiments and for some surgical procedures, an incision is made in the patient 300 to gain access to the internal cavity 304. Next, a trocar 50 is inserted into the patient 300 at a selected location to provide access to the internal cavity 304 or the surgical site. Then, the RSS 46 can be maneuvered to positions on the patient 300 and the trocar 50. In some embodiments, the RSS 46 includes a trocar mount that couples to the trocar 50. The robot arm assembly 20 can be coupled to a motor 40, and at least a portion of the robot arm assembly can be inserted into the trocar 50 and thus into the internal cavity 104 of the subject 300. For example, the camera assembly 44 and the robot arm assembly 42 can be individually and sequentially inserted into the patient 300 through the trocar 50. The camera assembly and the robot arm assembly can include some portions that remain outside the subject's body during use, but references to inserting the robot arm assembly 42 and / or the camera assembly into the subject's internal cavity and positioning the robot arm assembly 42 and / or the camera assembly 44 within the subject's internal cavity refer to the portions of the robot arm assembly 42 and the camera assembly 44 that are intended to be within the subject's internal cavity during use. The sequential insertion method has the advantage of supporting smaller trocars and thus enabling a smaller incision to be made in the patient 300, and thus reducing the trauma experienced by the patient 300. In some embodiments, the camera assembly 44 and the robot arm assembly 42 can be inserted in any order or a specific order. In some embodiments, following the camera assembly 44, a first robot arm of the robot arm assembly 42 follows, and then a second robot arm of the robot arm assembly 42 follows, all of which can be inserted into the trocar 50 and thus into the internal cavity 304.Once inserted into patient 300, the RSS 46 can move the robotic arm assembly 42 and the camera assembly 44 to the surgical site that is manually or automatically controlled by the operator console 11.

[0051] Disclosure controls regarding the management of the movement of the individual arms of the robotic arm assembly are provided in International Patent Application Publication Nos. 2022 / 094000A1 and 2021 / 231402A1, each of which is hereby incorporated by reference in its entirety.

[0052] FIG. 4A is a perspective view of a robotic arm sub-assembly 21 according to some embodiments. The robotic arm sub-assembly 21 includes a robotic arm 42A, an end effector 45 having an instrument tip 320 (e.g., monopolar forceps, a needle driver / holder, a bipolar grasper, or any other suitable tool), and a shaft 322 that supports the robotic arm 42A. The distal end of the shaft 322 is coupled to the robotic arm 42A, and the proximal end of the shaft 322 is coupled to the housing 324 of the motor 40 (as shown in FIG. 2A). At least a portion of the shaft 322 can be outside the internal cavity 304 (as shown in FIGS. 3A and 3B). At least a portion of the shaft 322 can be inserted into the internal cavity 304 (as shown in FIGS. 3A and 3B).

[0053] FIG. 4B is a side view of the robotic arm assembly 42. The robotic arm assembly 42 includes, in some embodiments, a virtual shoulder 326, a virtual elbow 328 having a position sensor 332 (e.g., a capacitive proximity sensor), a virtual wrist 330, and an end effector 45. In some embodiments, the virtual shoulder 326, the virtual elbow 328, and the virtual wrist portion 330 can include a series of hinges and rotational joints to provide seven degrees of freedom that can be positioned for each arm, along with an additional one degree of freedom of grasping with respect to the end effector 45.

[0054] FIG. 5 illustrates a perspective front view of a portion of a robotic arm subsystem 20 configured for insertion into a patient's internal body cavity. The robotic arm assembly 20 includes robotic arms 42A and 42B. In some embodiments, the two robotic arms 42A and 42B may define a virtual chest 340 of the robotic arm assembly 20. In some embodiments, the virtual chest 340 may be defined by a chest plane extending between a first pivot point 342A of the most proximal joint of the robotic arm 42A (e.g., shoulder joint 326), a second pivot point 342B of the most proximal joint of the robotic arm 42B, and a camera imaging center point 344 of the camera 47. The pivot center 346 of the virtual chest 340 is at the center of the virtual chest.

[0055] In some embodiments, sensors on one or both of the robotic arms 42A and 42B can be used by the system to determine changes in the position of at least a portion of the robotic arm within a three-dimensional space. In some embodiments, sensors on one or both of the first robotic arm and the second robotic arm can be used by the system to determine the position of at least a portion of one robotic arm relative to the position of at least a portion of the other robotic arm within a three-dimensional space.

[0056] In some embodiments, the camera assembly 44 is configured to acquire an image such that the system can determine the relative position within the three-dimensional space. For example, the camera assembly may include a plurality of cameras, at least two of which are laterally displaced from each other with respect to the imaging axis, and the system may be configured to determine the distance to a feature within the internal body cavity. Further disclosure regarding a camera assembly and related system for determining the distance to a feature, including a surgical robotic system, can be found in International Patent Application Publication No. 2021 / 159409, entitled "System and Method for Determining Depth Perception In Vivo in a Surgical Robotic System", published on August 12, 2021, which is hereby incorporated by reference in its entirety. Information regarding the distance to a feature of the camera and information regarding the optical characteristics may be used by the system to determine the relative position within the three-dimensional space.

[0057] The hand controller for a surgical robotic system described herein can be used in conjunction with any of the above-described surgical robotic systems or any other suitable surgical robotic system. Further, some embodiments of the hand controller described herein may be used with a semi-robotic endoscopic surgical system that is only partially robotic.

[0058] As described above, it may be desirable for the controller for a surgical robotic system to be provided with sufficient inputs to provide control of the system, ergonomic design, and a "natural" feel during use.

[0059] In some embodiments described herein, reference is made to a left hand controller and corresponding left robotic arm (which may be a first robotic arm), and a right hand controller and corresponding right robotic arm (which may be a second robotic arm). In some embodiments, the robotic arms considered to be the left robotic arm and the right robotic arm may vary due to the configuration of the robotic arms, and the camera assembly is adjusted such that the second robotic arm corresponds to the left robotic arm with respect to the field of view provided by the camera assembly, and the first robotic arm corresponds to the right robotic arm with respect to the field of view provided by the camera assembly. In some embodiments, during use, the surgical robotic system identifies the robotic arm as corresponding to the left hand controller and the robotic arm as corresponding to the right hand controller. In some embodiments, at least one hand controller includes one or more operator input devices for providing one or more inputs for additional control of the robotic assembly. In some embodiments, the one or more operator input devices receive one or more operator inputs for at least one of engaging in a scan mode, resetting the orientation and position of the camera assembly to align the field of view of the camera assembly with the instrument tip and the chest, displaying a menu, highlighting options or items for traversing or selecting a menu, selecting an item or option, selecting and adjusting the position of the elbow, engaging a clutch associated with an individual hand controller.

[0060] In some embodiments, additional functionality may be accessed via a menu, for example, selecting a level of gripping force (e.g., high / low), selecting an insertion mode, extraction mode, or exchange mode, adjusting focus, illumination, or gain, camera cleaning, motion scaling, rotating the camera to enable looking down, and the like.

[0061] Figures 6 and 7 illustrate a system 100 that includes a laparoscope 110 having a camera assembly 44, according to some embodiments. In some embodiments, the laparoscope 110 is configured to be held by hand. In some embodiments, the laparoscope 110 is configured to be held by, attached to, and / or mated with an external support. The system 100 may include some or all of the components that make up the system 10, depending on the type of surgical procedure. As described herein, the system 100 represents, but is not limited to, laparoscopic embodiments.

[0062] In some embodiments, system 100 also includes a display for displaying an image or image data obtained from camera assembly 44 (e.g., an image generated from a camera feed, a processed camera feed, a processed camera feed or a sensor feed, an image generated from two-dimensional sensor data). In some embodiments, the display is or includes a display screen 130 such as a monitor or a tablet. In some embodiments, the display includes a two-dimensional (2D) display and / or a three-dimensional (3D) display. In some embodiments, the display can include a virtual reality (VR) or augmented reality (AR) headset, or different forms of VR or AR devices (e.g., smart glasses, a head-up display (HUD), a holographic display, etc.). In some embodiments, system 100 includes a motion tracking headset 140 or other head motion sensing device, system, or mechanism. In some embodiments, the motion tracking headset is also used as a display or a component of a display. In some embodiments, the output regarding the operator's head movement from motion tracking headset 140 is used as an input to control the orientation of the field of view of camera assembly 44, as described in more detail below. In some embodiments, the use of motion tracking headset 140 and / or an AR headset enables the operator to change the direction of the field of view of camera assembly 44 while maintaining visual contact with the surgical tool. Some examples and aspects of incorporating a camera feed into a VR / AR headset during surgery are described in U.S. Patent No. 10,285,765, which is hereby incorporated by reference in its entirety. Some exemplary display screens are described in International Publication No. WO 2021 / 092194, which is hereby incorporated by reference in its entirety. In some embodiments, the image is presented on display 130 at 60 frames per second. In other embodiments, the image is presented on display 130 at 120 frames per second.

[0063] In some embodiments, system 100 also includes a motor unit 150 that drives the movement of at least a portion of camera assembly 44. In some embodiments, motor unit 150 can be implemented as motor 40 within system 10, as taught herein, to drive the movement of at least a portion of camera assembly 44. In some embodiments, system 100 includes a control and processing unit or system (e.g., laptop tower box 160) that receives input from an operator controller and controls the movement of camera assembly 44. In some embodiments, the control and processing unit or control and processing system also generates an output for a display (e.g., display screen 130). In some embodiments, the control and processing unit or control and processing system can be disposed in a mobile cart, tower, box, or laptop tower box.

[0064] In some embodiments, system 100 also includes one or more operator controllers for receiving input to control laparoscope 110. In some embodiments, the one or more operator controllers also control the displayed image or data. In some embodiments, the operator controller includes any of foot pedal 180, handheld controller 170, and motion tracking headset 140. In some embodiments, other or additional operator controllers can be used. Laparoscope 110 and camera assembly 44 can be controlled by operator input provided via motion tracking headset 140, foot pedal 180, handheld controller 170, one or more buttons on motor unit 150, or combinations thereof (see FIGS. 6 and 7). In some embodiments, system 100 can further include a holder 190 for motor unit 150. Holder 190 can be a stability unit configured to hold motor unit 150 in a stationary position without support from an operator or assistant. An exemplary holder 190 is the Iron Intern® manufactured by Automated Medical Products Corp.

[0065] Figures 8A and 8B illustrate opposing side views of laparoscope 110 including camera assembly 44, according to some embodiments. Camera assembly 44 is disposed at the distal end of laparoscope 110. Camera assembly 44 is supported by support 112 (e.g., support tube) in some embodiments. Support 112 is connected to motor unit 150 of laparoscope 44 in some embodiments. Motor unit 150 may be configured to be handheld. Motor unit 150 may be configured to be held, mounted, connected, and / or mated with an external support or holder by an external support or holder, according to some embodiments. In some embodiments, motor unit 150 is similar in size and weight to a soda can. In some embodiments, cassette 151 connects support tube 112 to motor unit 150 and connects an element or component extending through support tube 112 to motor unit 150.

[0066] FIG. 9 illustrates a side view of the distal end of laparoscope 110, including camera assembly 44 and support 112. In some embodiments, camera assembly 44 includes a camera unit that includes one or more camera modules 124 and / or other imaging modules. The camera assembly may have two or more camera units in some embodiments. In some embodiments, camera unit 128 includes an image sensor mount 121 for mounting an image sensor and / or camera module 124. In some embodiments, a plurality of camera modules 124 are used to generate a stereoscopic image. In some embodiments, camera unit 128 may include a plurality of cameras or imaging modules for imaging different types of light. In some embodiments, the camera unit may include a sensor or detector for sensing or detecting non-visible electromagnetic signals. In some embodiments, the camera unit includes one or more light sources (which may include any combination of one or more light emitting diodes (LEDs) 122, such as LEDs 422, 424, or 426, as discussed in more detail below). In some embodiments, camera unit 128 includes a light source that generates light in different spectra or different spectral bands. Such light sources are discussed in more detail below with respect to FIGS. 15A, 15B, and 16. In some embodiments, the camera unit includes one or more light sources that generate light outside the visible light spectrum (e.g., a light source that generates infrared (IR) light and / or a light source that generates ultraviolet (UV) light). The field of view of camera unit 128 may be illuminated by one or more light sources 122 that generate one or more types of light, such as at least one light source that generates light in the visible spectrum (e.g., an LED that emits white light) and / or at least one light source that generates light not in the visible spectrum (e.g., a light source that generates infrared (IR) light and / or a light source that generates ultraviolet (UV) light).

[0067] System 100 includes an imaging mode. In some embodiments, System 100 is configured for two or more imaging modes that may be simultaneously involved to generate multispectral imaging. For example, multispectral illumination with light in the visible spectrum and detection of light in the visible spectrum may generate a primary image output as presented in some embodiments. In some embodiments, System 100 may also generate a secondary image output based on illumination using light outside the visible spectrum (e.g., IR light or UV light). In some embodiments, the light spectrum used for illumination may be different from the light spectrum detected for image generation (e.g., fluorescence image). In some embodiments, one or more filters (e.g., digital filters and / or physical filters as discussed below) may be used to separate a specific wavelength range of interest for some image modes. In some embodiments, inputs from only specific color channels (e.g., red, green, and / or blue) of an image detector may be used. Visual output from one imaging mode may, in some embodiments, be overlaid and displayed on the output from another imaging mode. For example, in some embodiments, an invisible spectrum image output (e.g., IR imaging output) may be overlaid on a primary (e.g., visible light) image output on a display to generate a multispectral image or video. In some embodiments, green pixels may be replaced with broadband-sensitive pixels or infrared-sensitive pixels to enable better sensitivity in the multispectral operation mode.

[0068] In some embodiments, system 100 strobes between one type of light source and another type of light source (e.g., between a white light source and an IR light source) to generate a combined multi - spectral image or video. For example, system 100 may block the white light source during one frame when the infrared light source is on. In some embodiments, system 100 is configured to strobe a primary illumination source to drop frames from a primary image feed (e.g., a visible light feed or a white light feed) multiple times per second (e.g., six times per second). In such embodiments, if the primary illumination source (e.g., a visible light source or a white light source) is not fully powered, the frame is either discarded or depicted with an image from another imaging mode (e.g., an IR imaging mode). In some embodiments, system 100 is configured to strobe two or more types of light sources. For example, the system may strobe a white light LED, a blue light LED, or a combination of the two. One or more of the light sources may include a laser source, such as a laser discussed in more detail below.

[0069] In some embodiments, camera assembly 44 performs autofocus. In some embodiments, the multi - spectral camera assembly 44 is configured to automatically focus on the central area of the strobed light. In some embodiments, the multi - spectral camera assembly 44 may provide an improved field of view and / or depth of field compared to cameras used in conventional laparoscopes or robotic surgical devices.

[0070] In some embodiments, camera unit 128 may also include at least one pulsed laser light source, and system 44 may utilize optical detection and ranging (LIDAR) functionality. In some embodiments, LIDAR may be used for autofocus. In some embodiments, LIDAR may be used to acquire a three-dimensional representation or map of at least a portion of the body cavity. In some embodiments, camera unit 128 may include an additional mount 123 for a LIDAR source or a dot matrix projector. In some embodiments, camera assembly 44 may include one or more features or components for heat dissipation. For example, camera unit 128 may include one or more heat dissipation structures 125 (e.g., fins). Camera unit 44 may also provide an improved field of view and depth of field compared to cameras used in conventional laparoscopes or robotic surgical devices according to some embodiments.

[0071] In some embodiments, camera assembly 44 includes a zoom capability (e.g., a 2x zoom capability) that does not reduce the resolution of the image displayed when maximum zoom is used. For example, in some embodiments, one or more camera modules may have a higher resolution than is required for a full field of view display. In these embodiments, only a subset of the pixels from the camera module are displayed. In some embodiments, a smaller selected portion of the field of view is displayed, but a larger percentage of the pixels in the selected portion of the field of view are displayed, and a zoom may be used that results in a zoom that does not reduce the resolution of the displayed image.

[0072] The camera assembly 44 has a yaw axis 132, a pitch axis 134, and a roll axis 136 (see FIGS. 9-10C). The direction of view of the camera unit 128 of the camera assembly 44, which may be referred to as the viewing direction 138, can rotate or tilt about different axes (e.g., the yaw axis 132, the pitch axis 134, and the roll axis 136), two of which are always orthogonal to each other (e.g., the yaw axis 132 and the pitch axis 134) without requiring movement of the support 112 that extends outside the body cavity of the subject (see FIGS. 9-10C). One or more actuators, e.g., the pitch and yaw composite actuator 132, connect the camera unit 128 to the main body 129 of the camera assembly 44 (FIG. 9). In FIGS. 9, 10A, and 10C, the pitch axis 134 of the camera body 128 is parallel to the roll axis 136 of the camera assembly corresponding to the insertion or removal configuration. The camera unit 128 can, in some embodiments, be rolled so that the viewing direction 138 returns toward the insertion point. Exemplary camera assemblies can include some of the elements described in U.S. Patent No. 11,583,342, which is hereby incorporated by reference in its entirety.

[0073] Conventional robotic surgical systems are configured to yaw, pitch, and roll the camera in a manner similar to moving a camera assembly to visually identify a target area. These conventional systems require movement external to the patient to manipulate the camera and may require repositioning the system through multiple different insertion ports to capture a 360-degree view of the surgical site. Thus, some conventional systems require a large amount of unnecessary movement or the use of additional systems across the same surgical area, making the surgical area more congested and the procedure more complex. In comparison, the disclosed camera assembly 44 is configured to provide a 360-degree view without the need to move the camera 44 relative to a support 112 of the laparoscope 110 that extends outside the body cavity of the subject, according to some embodiments. For example, the operator can insert the laparoscope 110 into the surgical area and obtain a rearward view toward the insertion point, such as a trocar if a trocar is used, without movement of an external support according to some embodiments. Removing or reducing the required external movement reduces the force exerted on the trocar and reduces damage to the tissue surrounding the trocar.

[0074] Examples and descriptions of an actuator for moving one or more components of a camera assembly, a support for the camera assembly, and a motor unit for driving the movement of the camera assembly are found in U.S. Patent Application No. 11,583,342, which is hereby incorporated by reference in its entirety. In some embodiments, the camera assembly 44 is configured to move for purposes of position correction. The movement of the camera assembly 44 can be implemented as described in International Publication No. 2021 / 231402, which is hereby incorporated by reference in its entirety.

[0075] In some embodiments, the insertable portion of the laparoscope 110 has a diameter in the range of about 12 mm to 18 mm. In some embodiments, the diameter may be 15 mm to 18 mm. In some embodiments, the insertable portion of the laparoscope 110 has a diameter of 18 mm and is insertable into a trocar having a diameter of 118 mm. In some embodiments, the insertable portion of the laparoscope 110 is inserted into a flexible trocar. The flexible trocar allows the operator to create a smaller incision port. When the insertable portion of the laparoscope 110 is inserted into the cervix, an elliptical trocar may be used.

[0076] The control of the camera assembly 44 provides improved visualization of the surgical site according to some embodiments. For example, the system 100 of FIG. 6 may provide improved visibility during hysterectomy, appendectomy, resection, excision, or other surgical procedures suitable for minimally invasive techniques and instruments. Further, the camera assembly 44 of FIG. 6 may be directly controlled by the operator without relying on an assistant according to some embodiments. Exemplary positioning or movement of the camera assembly 44 is illustrated in FIGS. 10A, 10B, and 10C.

[0077] In some embodiments, the laparoscope 110 may be insertable into the surgical site without the use of a trocar. For example, the laparoscope 110 may be inserted into the vagina. Vaginal insertion may reduce the total number of 5 mm ports required for a hysterectomy to two or three. In some embodiments, the laparoscope 110 may be insertable into the surgical site using a trocar, or the camera assembly 44 associated with the surgical robot system may be insertable using a trocar.

[0078] FIG. 11 illustrates a side view of the laparoscope 110 including a camera assembly 44, a support 110, a cassette 151, and a cover of the cassette 151 and a cover of the motor unit 150 that is transparent for illustrative purposes according to some embodiments.

[0079] System 100 may further include a motor unit 150 configured to operate the camera assembly 44 using a motor (e.g., a mason motor) that drives one or more actuators 126 of the camera assembly 44. The motor unit 150 may include one or more motor control boards (MCBs) 156. The motor unit may include one or more serializer / deserializer boards 155. The motor unit 150 may also include, according to some embodiments, any one of a power connector, a universal serial bus (USB) connector, and a fiber surface mount technology (SMT) connector. The connector may interface with the camera assembly 44 and / or a control and processing unit or system (e.g., a laptop tower box 160). In some embodiments, the motor unit 150 includes one or more inertial measurement units 153.

[0080] In some embodiments, the motor unit 150 includes at least two buttons for controlling one or more aspects of the camera assembly 44. For example, one button may be involved in orienting the camera 44 for insertion or extraction. In such an example, the other button may be involved in locking (i.e., with a gimbal lock) the camera assembly 44 to allow an operator or assistant to move the motor unit 150 while the camera assembly 44 remains in a fixed position within the surgical site.

[0081] In some embodiments, the support 112 and the elements extending within the support tube (e.g., electronic cables and mechanical actuation cables) are mated with the motor unit 150 via the cassette 151. In some embodiments, the laparoscope 110 is configured to position a sterile drape or cover between the cassette 151 and the motor unit 150. In some embodiments, at least a portion of the laparoscope 110 is reusable for multiple procedures. In some embodiments, at least a portion of the laparoscope 110 can be sterilized and reused for multiple procedures, e.g., up to 10 procedures. In some embodiments, at least a portion of the laparoscope 110 can be cleaned in an autoclave. In some embodiments, at least a portion of the laparoscope 110 can be single-use. In some embodiments, the cassette 151, the support 112, and the camera assembly 44 are single-use, and the motor unit 150 is reusable. In some embodiments, the cassette 151, the support 112, and the camera assembly 44 can be sterilized for reuse a limited number of times. In some embodiments, the cassette 151, the support 112, and the camera assembly 44 are reusable a limited number of times that is less than the number of times the motor unit 150 is reusable.

[0082] In some embodiments, the motor unit 150 can be removed from the laparoscope 110 after surgery for repair or replacement.

[0083] Figures 13A - 13C are different views of the cassette 151 according to some embodiments. The cassette 151, according to some embodiments, includes a cassette housing 201, a support tube gear 202 for rotating a support tube 112 for rotating a support tube, and an intermediate gear 203. The cassette 151 may further include a direction - changing pulley 206 and a direction - changing pulley mount 205. The cassette 151 may also include a roll pulley 204 and two pulleys 208 for yaw and pitch. The cassette 151 may also include an idler pulley 207. The cassette 151 and the motor unit 150 may include some elements and aspects described in U.S. Patent No. 11,583,342, which is hereby incorporated by reference in its entirety.

[0084] The system 100 may further include a control and processing unit or system 150 (e.g., the wrap - tower box 160 or the computing module 18 described above). The wrap - tower box 160 may be sized to be placed on a wrap - tower. The computing module 18 or the box 160 may be configured to provide an output to a data recorder and / or display 130 or a headset 140. The computing module 18 or the wrap - tower box 160 may interface with and power any or all of a camera assembly 44, a display 12 (e.g., a VR / AR headset), and an operator controller (e.g., a handheld controller 170, a foot pedal 180, and / or a motion - tracking headset). The computing module 18 or the wrap - tower box 160 may include a display 130. In some embodiments, the system 100 includes a first display 130 and a second display on the wrap - tower box 160.

[0085] The handheld controller 170 (such as the handheld controller 17) can be configured to provide a command interface for the camera assembly 44. For example, an operator or assistant can use the handheld controller 170 to move the camera unit 128, orient the camera unit 128, and / or select menu options on the display 130. In some embodiments, the handheld controller 170 can be sized and shaped similar to the handheld controller 17 described above. In some embodiments, the handheld controller 170 is connected to a foot pedal 180 (such as the foot pedal array 19 described above) by a wired connection, for example.

[0086] According to some embodiments, the system 100 can be operable in a hands-free mode. For example, the headset 140 can track the movement of the operator's head when the operator looks at the edge of the bezel of the headset 140. This head movement can trigger a modality for the operator to activate using the foot pedal 180. Next, the operator can, for example, increase the pressure applied to the pedal 180 to increase the movement speed of the camera unit 128 in the selected direction. When the desired movement is complete, the operator can release the foot pedal 180 and continue with the operation. As another example, the operator can roll the camera unit 128 by slightly rotating the head. As another example, when operating in conjunction with the foot pedal 180, the surgeon can, for example, lean the head backward while engaging the foot pedal 180 to move the camera unit 128 upward, release the foot pedal 180, reset the head to the desired position, and then engage the foot pedal 180 again to lean the head backward and move the camera unit 128 further upward. Thus, the surgeon can continue to move the camera unit 128 upward by clutching in and out of the camera mode using the foot pedal 180.

[0087] Other gestures can be programmed to perform other functions such as long blinking that signals cleaning or wiping of the camera unit 128, or rotating the camera unit 128 up to 360 degrees. In some embodiments, the operator closing one or both eyes may indicate a blurred camera feed or headset misalignment. After detecting the closing of the eyes, the system 100 generates a message on the display 130 indicating the blurred feed or headset misalignment.

[0088] Head movements can be associated with depressing the foot pedal 180, releasing the foot pedal 180, or pressing the foot pedal 180. For example, double-tapping the foot pedal 180 can trigger a menu displayed and controlled by head movements and / or the foot pedal 180 to select additional options. As another example, tilting forward or backward can control the zoom feature of the camera assembly 44.

[0089] In some embodiments, the system 100 tracks the eye movements of the operator wearing the headset 140 to determine when the operator is interacting with the system 100. Eye tracking can also identify a specific area of the surgical site the operator is looking at and prompt the system 100 to focus on it. When focusing on a specific area, the camera assembly 44 can be configured to maintain the vision of the specific area while the laparoscope 110 or the camera assembly 44 is being moved by an external force.

[0090] In some embodiments, the operator can trigger different operating modes for selection using the foot pedal 180 by looking at the visual edge of the VR / AR headset 130. Additionally or alternatively, looking at the edge of the screen of the headset 140 can adjust the movement speed of the camera assembly 44 or the laparoscope 110 without using the foot pedal 180. The accuracy of eye tracking can be improved by using retroreflectors within the headset 140.

[0091] In some embodiments, system 100 may include a joystick or other input devices such as a capacitance / inductive sensing pad attached to motor unit 150. The joystick may be configured to control the movement of camera unit 128 alone or in combination with headset 140 or foot pedal 180. In some embodiments, the movement of the joystick or capacitive / inductive sensing pad enables fast, controlled movement without any additional form of confirmation from headset 140 or foot pedal 180.

[0092] System 100 may include multiple imaging modes. For example, a first imaging mode may be a live stream captured by camera unit 128 and output to the user. A secondary imaging mode may be used to provide a secondary display to the surgeon. For example, using an image overlay, a smaller display may be output based on information captured from various different sensors that can be incorporated into camera assembly 44.

[0093] As an example, in a secondary imaging mode using a dot projector, camera assembly 44 may output infrared (IR) light between each output of a certain color (white) light (e.g., light of different frequencies). Information captured during the output of the IR light may be output on the secondary display. Thus, the secondary display in this example shows the environmental dots captured by camera unit 128. This feature is advantageous for providing additional depth information to the surgeon. The information may also be captured to generate a depth map of the environment captured by camera unit 128. Methods for determining depth perception in vivo are discussed in International Publication No. WO 2021 / 159048, which is incorporated herein by reference in its entirety.

[0094] As another example, camera unit 128 may provide spectral imaging for detecting certain body features, such as the urethra or bladder. System 100 may be configured to map body features to detect and outline organs on a display. For example, camera assembly 44 may be configured to perform indocyanine green (ICG) imaging and may detect ureters treated with a dye such as methylene blue, UreterBlue, or ZW800-1. Camera unit 128 may be configured to identify cancerous tissue stained with fluorescein. Camera unit 128 may be configured to identify neurovascular formation stained with GE3111.

[0095] In some embodiments, system 100 is configured to digitally tag the identified tissue to monitor the movement of the tissue during the procedure. For example, system 100 may identify the edge of the bladder and monitor the movement of the bladder during a hysterectomy. Detecting and monitoring the edge of the bladder may notify the operator of where to make an incision for the hysterectomy. As another example, the system may identify cancerous tissue of nerve tissue stained with an appropriate dye. By identifying the tissue within the surgical site, the surgeon can also distinguish different tissues and, for example, determine which tissues should be avoided during the surgical procedure to avoid contact.

[0096] In some embodiments, an operator or assistant may measure the distance between two points within the field of view of camera assembly 44 by identifying a point using the gripper of laparoscope 110 or a robotic arm, or by identifying a point on display 130 or headset 140. System 100 may then be configured to measure the distance between the identified points and depict the calculated distance on display 130 or headset 140.

[0097] In some embodiments, an operator or assistant may prompt system 100 to capture a 360-degree scan of the surgical site. To capture the scan, camera assembly 44 may be configured to rotate in place (e.g., via a roll about a roll axis). The collected image data may be generated as a visual mesh that can be viewed on display 130 or headset 140 during the procedure or after the surgical procedure is complete. An exemplary 360-degree field of view visualization is illustrated in FIG. 14.

[0098] FIG. 15A illustrates a perspective view of a multispectral camera assembly 44 according to some embodiments. The multispectral camera assembly 44 may include a housing 410 that includes a plurality of LEDs 422, 424, 426. In some embodiments, the multispectral camera assembly 44 may further include at least one laser 428. The multispectral camera assembly 44 includes at least one lens 430. In some embodiments, the multispectral camera assembly 44 includes two lenses 430. FIG. 15B illustrates a front view of the multispectral camera assembly of FIG. 15A.

[0099] FIG. 16 illustrates an exploded view of the multispectral camera assembly of FIG. 15A. As seen in FIG. 16, the housing 410 may include a front portion 416 and a rear portion 414. The front portion 416 and the rear portion 414 may be joined by two end caps 417.

[0100] The front portion 416 and the rear portion 414 may couple a pitch and yaw assembly 418 that is positioned on a base 415. The end caps 417 may be configured to smooth the shape of the pitch and yaw assembly 418 to enable the camera assembly 44 to slide more easily through the trocar. The base 415 may be coupled to the pitch and yaw assembly 418 and configured to provide a surface for rotation about a yaw axis.

[0101] The pitch and yaw assembly 418 can fix the front part 416 while enabling the articulation of the camera assembly 44 along the pitch and yaw axes. The front part 416 can be configured to provide a pitch-axis mounting feature for articulation about the long axis of the housing 410.

[0102] The base 415 can be connectable to the support tube 412. The support tube 412 can be connected to a motor unit of a laparoscope or to a surgical robot system as described above in some embodiments. In some embodiments, the base 415 includes a cable cover 416 configured to fix a cable led to the housing 410. For example, a coaxial cable can extend through the support tube 412 and extend through the base 415 and the pitch and yaw assembly 418.

[0103] The multispectral camera assembly 44 includes a camera substrate 432, e.g., a customized printed circuit board. The camera substrate 432 can be housed within the front part 416. In some embodiments, the camera substrate 432 includes two serializer chips on the back surface of the substrate 432 that convert mobile industry processor interface data from one or more image sensors 433 into a serial form that can be transmitted via two small coaxial cables. In some embodiments, the camera substrate 432 includes two image sensors 433 on the front surface that provide visualization of the area viewed by the camera assembly 44.

[0104] Each image sensor 433 can be aligned with a lens 430. The lens 430 can be housed within a voice coil module 434 mounted on the camera substrate 432. In some embodiments, the lens 430 is positioned within the voice coil module 434 such that the lens 430 can move to change the nominal focal position of the lens 430 and optically zoom the lens 430. The combination of the image sensor 433 and the lens 430 can be equivalent to the camera module 124 discussed above.

[0105] The multispectral camera assembly 44 further includes a plurality of LEDs. In some embodiments, the plurality of LEDs includes at least one, e.g., two or four fluorescein LEDs 426 configured to emit light to excite a fluorescent dye. Exemplary dyes may include indocyanine green, fluorescein dyes, or other dyes suitable for imaging tissue or tissue structures. In some embodiments, the fluorescein LED 426 may emit light at a wavelength of about 490 nm, e.g., in the range of 475 nm to 505 nm. The fluorescein LED 426 may be located near an opposite end of the camera substrate 432.

[0106] The plurality of LEDs further includes at least one, e.g., two or four white LEDs 422 configured to emit light in the visible spectrum. The white LED 422 may emit light at wavelengths in the range of 400 nm to 700 nm. The white LED 422 may be located adjacent to the fluorescein LED 426.

[0107] The plurality of LEDs further includes at least one, e.g., two blue LEDs 424 configured to emit light in the range of 400 nm to 430 nm. Each blue LED 424 may be located between a pair of white LEDs 422. The light emitted from the blue LED 424 may compensate for the light blocked by a notch filter that blocks a portion of the blue spectrum, as discussed below.

[0108] The multispectral camera assembly 44 utilizes the same camera capable of white light detection for multiple spectra by adjusting the light supplied from the LEDs 422, 424, 426. For example, the camera assembly 44 may be configured to strobe the LEDs 422, 424, 426 in synchronization with when the frame stops integration. This approach reduces the cost and complexity of the camera assembly 44, e.g., by reducing the number of image sensors 433 required compared to other solutions in the art.

[0109] The multispectral camera assembly 44 utilizes the entire active area of the image sensor 433 for the analysis of a specific band of light. In this way, the image is generated with the maximum possible white light image and reasonable performance for a specific band of light. In some embodiments, the filter can be adjusted to allow more infrared light. Specifically, the cut-off of the filter can be adjusted to allow more or less light to enter the image sensor 433 and / or to allow more or less emitted light from the LEDs 422, 424, 426.

[0110] The multispectral camera assembly 44 may further include one or more lasers 428. Each laser can be located near the end of the camera substrate 432. In some embodiments, the laser 428 is a vertical cavity surface emitting laser. The laser 428 can be configured to emit light in the range of 800 nm to 820 nm, for example, 808 nm. The 808 nm laser light enables indocyanine green (ICG) imaging by exciting the indocyanine green dye. In some embodiments, the laser 428 can be configured to emit light in the range of 400 nm to 850 nm, or any range therebetween. In some embodiments, a filter, such as the notch filter discussed below, can be located in front of the laser 428.

[0111] The indocyanine green dye emits fluorescence in the near-infrared wavelength region when irradiated with light of a shorter wavelength. The dye molecule absorbs the excitation photon and enters the excited state, and then, when the excited state decays, emits a photon at a longer emission wavelength (lower energy) (reproduced from the "IC-GREEN (trademark)" product sheet, Akorn, Inc., as shown in FIG. 17). Since ICG has a higher probability of absorbing photons near the peak of the excitation curve, the wavelength of the light emitted from the fluorescein LED 426 is preferably close to the excitation peak wavelength to increase the signal level of the emitted photons. These excitation curves can strongly depend on the environment in which the fluorescent compound is located. This type of excitation curve is a typical characteristic of any fluorescent dye.

[0112] In some embodiments, the only light received by the image sensor 433 is fluorescence emission rather than excitation light. The fluorescence intensity may be much lower than the excitation intensity, so the notch filter 431 can be placed in front of the image sensor 433 to block the excitation light reflected back to the camera and prevent interference.

[0113] The multispectral camera assembly 44 may include a plurality of notch filters 431, each notch filter 431 being located between the image sensor 433 and the lens 430. Each notch filter 431 can be configured to filter light emitted from at least one of the plurality of LEDs. In some embodiments, the notch filter 431 filters light at about 808 nm, 490 nm, or both. In some embodiments, the notch filter 431 may include multiple notches for filtering light of multiple wavelengths. In such embodiments, a single notch filter 431 can filter light from multiple LEDs 422, 424, 426 and / or laser 428.

[0114] The multispectral camera assembly 44 may further include bandpass filters, such as a fluorescein bandpass filter 423 and a laser bandpass filter 429. The fluorescein bandpass filter 423 can be located in front of each fluorescein LED 426. In some embodiments, the fluorescein bandpass filter 423 is configured to block all light except light at a wavelength of about 490 nm. The laser bandpass filter 429 is located adjacent to the laser 428 and can be configured to block all light except light at a wavelength of about 808 nm. In some embodiments, the multispectral camera assembly 44 includes a single multi-bandpass filter positioned in front of the LEDs 422, 424, 426 and laser 428 such that light of a specific wavelength is emitted by the camera assembly 44. The LEDs 422, 424, and 426 may be compatible with each other.

[0115] When the camera assembly 44 is also used for visible band (VIS) imaging, the user can involve the laser 428 or LED with a narrow bandpass filter (e.g., having a bandwidth of 20 nm or less) in front of it so that the notch filter 431 does not need to block the wavelength band used for VIS imaging. Under these conditions, when the excitation irradiation is turned on, the generated camera image is black except for the area where fluorescence occurs. Therefore, the dye is used to distinguish the anatomical regions that preferentially contain or absorb the dye from the regions that do not.

[0116] Dyes other than ICG generally have different excitation and emission wavelengths and can be used to visualize anatomical structures and conditions that are not possible with ICG. Different dyes generally require different excitation and emission wavelengths. In some embodiments, the camera assembly 44 is designed to be used with multiple dyes and includes at least one filter having multiple blocking bands that do not significantly interfere with the emission band of other dyes or the VIS camera band. In some embodiments, the camera assembly 44 is used with multiple dyes having the same excitation wavelength but different emission wavelengths and distinguished on the image based on the generated output color.

[0117] FIG. 18 is a diagram of the excitation / emission bands of ICG and fluorescein shown together with the response curves of the red, green, and blue channels of the image sensor 433 and the emission spectrum of the white light LED 422 used for visible band imaging. Lines 510, 520, and 530 are the pixel response curves of the image sensor 433 for the blue, green, and red channels, respectively. Line 540 is the emission characteristic of the white light LED 422 for visible band imaging. The shaded area 550 in the figure is the excitation band of the dye and blocking band of the notch filter 431, and the transparent area 560 in the figure is the transmission band of the notch filter 431. The transmission band of 400 - 715 nm can also be used for visible band imaging. The indocyanine green excitation range is 720 - 845 nm, and the emission range is 850 - 880 nm. The fluorescein excitation range is 470 - 520 nm, and the emission range is 525 - 540 nm.

[0118] FIG. 19 illustrates light rays (1) collected over a wide angle by an exemplary camera comprising a window (2), a lens stack (3), a blocking filter (4), and an image sensor (5), and shows a large angular range of light incident on the filter (4). The blocking filter characteristics vary with the angle of incidence of the light passing through them. This variation is a shift in the blocking / transmission range to shorter wavelengths with increasing angle of incidence.

[0119] FIG. 20 illustrates blocking filter characteristics 610 and 620 for angles of incidence of 0 degrees and 30 degrees, respectively, and shows how the fluorescein blocking range shift interrupts during blue emission of a white light LED. Line 630 illustrates ICG excitation, and line 640 illustrates ICG emission. Line 650 illustrates fluorescein excitation, and line 660 illustrates fluorescein emission. As the angle of incidence increases, the blocking and transmission bands shift and encroach on other bands. To compensate for this, a blue-ultraviolet LED 424 can be added to the illumination from the white LED 422 to "fill in" the image sensor blue pixel response in the wavelength band 400 - 430 nm, where the white LED 422 has much less emission.

[0120] FIG. 21 illustrates the dependence of the transmittance of blue light (470 nm) on the angle from the center of the filter. As shown, the transmittance drops significantly after 20 degrees.

[0121] Figures 22A - 22D illustrate the attenuation in the blue channel of an exemplary camera assembly 44 that creates a dark region outside the central 40 - degree field of view. Figure 22B illustrates the red - channel image of a white target illuminated by white light. Figure 22C illustrates the green - channel image of a white target illuminated by white light. Figure 22D illustrates the blue - channel image of a white target illuminated by white light. The attenuation of the blue - channel image causes non - uniform color in the final image of the combined channels. Figure 22A shows the line profile 710 along the horizontal line 720 of the blue - channel image of Figure 22D. The line profile 710 shows the gray values of the image. The gray values are zero at approximately pixels 350 and 1900, resulting in the black boundaries of the blue - channel image.

[0122] Different options of illumination with increased blue intensity were investigated to obtain improved transmission within the blue channel. In some embodiments, violet LEDs (emitting at a wavelength of approximately 415 nm) are positioned on or incorporated with the white LED 422 or the blue LED 424. Ultraviolet LEDs, or any other LED suitable for illuminating specific dyes, may be included in the camera assembly 44.

[0123] Figures 23B, 23C, and 23D illustrate the channel images of a white target illuminated by white and violet light according to an embodiment incorporating violet LEDs. Figure 22B illustrates the red - channel image of a white target illuminated by white and violet light. Figure 22C illustrates the green - channel image of a white target illuminated by white and violet light. Figure 22D illustrates the blue - channel image of a white target illuminated by white and violet light. Figure 23A shows the line profile 710' along the horizontal line 720' of the blue - channel image of Figure 23D, indicating that the cut - off of the gray values along the line profile 710' is eliminated when using violet LEDs, eliminating the black boundaries of the blue - channel image.

[0124] The multi - spectral camera assembly 44 can be part of a surgical robot system, including a memory storing one or more instructions and a processor configured or programmed to read the one or more instructions stored in the memory. The processor can be operably coupled to one or more camera assemblies 44 to simultaneously capture multiple spectra of light from the camera assembly 44. The system can be operably connected to a display to depict the images captured by the system.

[0125] Digital cameras known in the art are designed to integrate a frame over a certain time, read that frame into memory, and transmit that frame to a display. One of the problems in surgical applications is the amount of data and time it takes to process or transmit each frame. In some examples, up to 128 megabytes of information are required for each frame for transmission and processing. A global shutter camera can store that information all at once, transfer that information, and then process a new frame during transmission. As such, global shutter cameras require more energy, storage, and complex electronics than other solutions. Sensors associated with global shutter cameras typically have lower resolution, lower frames / second, and higher power consumption. Such sensors are typically not used on mobile devices due to these trade - offs. However, such sensors allow for easily changing the timing of the light strobe during the integration period.

[0126] In some embodiments, the camera assembly 44 incorporates a global shutter camera, e.g., a global shutter charge - coupled device (CCD) imaging sensor. In some embodiments, the camera assembly 44 incorporates a rolling shutter camera, e.g., a rolling shutter complementary metal - oxide - semiconductor (CMOS) imaging sensor, as opposed to a global shutter camera.

[0127] FIG. 24 illustrates the integration times and operations of frames 1 and 2 of an exemplary rolling camera shutter. The graph illustrates the integration timings of the upper row 810 and the lower row 820. As shown in FIG. 24, when a frame between transitions is dropped (or adjusted so that the exposure is minimized), the controller 26 only needs to be configured to trigger a transition between images at any timing between N-2 frames that stop the integration of the lower row 820 and N frames that start the integration of the upper row 810. The transition time described above needs to be after the illustrated first vertical dotted line 830 and before the illustrated second vertical dotted line 840.

[0128] When combined with the stroboscopic lighting of one or more LEDs, the rolling shutter enables the simultaneous processing of multiple spectra of light by the camera assembly 44. FIG. 25 provides an example of a normal image, a semi-exposed image, and a transition frame combined with a spectral image.

[0129] FIG. 25 illustrates, in order, the patterns of a normal image 910, a first semi-exposed image 920, a spectral image 930, and a second semi-exposed image 922 that can be used by the camera assembly 44 in an embodiment incorporating a rolling shutter. The system 100 can be configured to transition from one spectral image 930 to another spectral image 930, or to change the number of spectral images 930 or normal images 910 to provide any desired ratio.

[0130] In some embodiments, the camera assembly 44 can provide multi-spectral video on the display 130 or the headset 140. The creation of the video can involve three components: the camera assembly 44, the controller 26 that controls the camera assembly 44, and a graphics processing unit (GPU) 52 configured to process video data, user input, and camera information to control the system 100 at a high level and display the video output to the user. The functions performed by the GPU 52 can be performed, in particular, by the graphics processing unit and / or by a visual processing unit that includes the graphics processing unit.

[0131] The image sensor 433 can be initialized to a state that provides images in the correct format for generating video. This state must also provide the metadata for each frame that enables the controller 26 and the GPU 52 to adjust the timing of the frames. Beyond these two constraints, the image sensor 433 can be considered as a data producer for the purpose of understanding software processing.

[0132] The controller 26 can be configured to initialize the electronics of the camera assembly 44 and pass commands from the GPU 52 to the camera assembly 44. The controller 26 can be configured to listen for timing information from the image sensor 433 and use that information to drive the multispectral imaging mode according to the current visualization mode. There may not be a tight coupling between the software on the controller 26 and the GPU 52. Instead, the GPU 52 can send commands for illumination and the image sensor 433 when requested and use the metadata from the sensor 433 to understand the state of the system 100.

[0133] The GPU 52 can be configured to control all image processing, user input, and display output. The GPU 52 can consume the image data, correct initial problems, and convert the image data into a format that can be processed more easily. Next, the controller 26 can send the image data to a modular parallel processing-based video pipeline that performs rectification, color correction, overlay processing, and then presents the processed image data to the user on the display 130 or the headset 140. The GPU 52 can also be configured to obtain any information from the processed image and use that information to send commands, such as changing the focus or illumination level, to the controller 26.

[0134] The controller 26 controls the light source change points by analyzing the timing provided by the frame start packet and the frame end packet of the image sensor 433 together with the frame synchronization signal. The start and end packets are intercepted so as to be transmitted by the sensor 433 via csi-2, and provide more accurate information regarding the frame blanking interval that can be configured to switch the light source used by the controller 26. The exact time used is important to ensure that each image does not overlap with other main images. The timing is even more important when timing the activation of the interlaced frames for use in high dynamic range imaging.

[0135] The image processing modules of the controller 26 and the GPU 52 can be removed or added according to the user's needs in the generation of the video output. In some embodiments, the module may include rectification, color correction, depth detection, luminance detection, overlay processing, and composition for the video output. In the depth and luminance detection stages of the image processing, the outputs of those modules can be used by the GPU 52 to send commands to the controller 26 to control the focus and illumination respectively.

[0136] A flowchart of a method 1000 for image processing to display a multi-spectral image is illustrated in FIG. 26. The method 1000 can start at step 1010, for example, when an image from the image sensor 433 is input into the system 100. The images can be stereo images, a pair of images taken side by side, and thus they can be the left view and the right view. At step 1020, the stereo images are aligned so as to be comfortable for a human viewing the images. The alignment ensures that any feature on one line of the left view image is on the same vertical pixel line of the right view image. Stereo cameras have other problems that make viewing through manufacturing defects, misalignment, VR headsets, 3D monitors, etc. uncomfortable. This process is well documented and the system 100 can implement it using standard algorithms.

[0137] In step 1030, the image can be color corrected. Color correction is the use of multiple colors of light that irradiate an object and a color selection filter attached to the image sensor 433. Exemplary color correction is illustrated in FIGS. 27A - 27D. FIG. 27A shows an image of a color calibration chart before color correction. The histogram of FIG. 27B shows two peaks, a red channel peak 1110 close to an intensity value of 50, and a blue channel peak 1120 at an intensity value of zero. As shown in FIG. 27C, non-uniformity correction, which shows an image with a weaker red channel, can be performed by dividing the image by a flat field image obtained in a uniformly illuminated field of view. The histogram of FIG. 27D illustrates peaks 1110', 1120' associated with the correction. An exemplary flat field image 1200 is illustrated in FIG. 28. A color correction matrix can be generated, for example, using the Imatest® color correction module and applied to the image after non-uniformity correction.

[0138] In step 1040, the system 100 can perform depth detection. Depth detection is a process of taking a stereoscopic image and determining the distance to each point within the image. The system 100 can use a machine learning model to determine the distance at each pixel and then use that information to select a focal length. In calibration, the system 100 determines, for example, how to set the focus for any given distance using the distance at the center of the screen in order to automatically focus on where the surgeon is looking. In some embodiments, the depth perception node of the GPU 52 can also send a callback to a messaging system when step 1040 is complete and send an appropriate focal length across the network of the system 100.

[0139] Overlay processing can be performed in step 1050. In some embodiments, overlay processing can be performed by the overlay processing module of the GPU 52. The overlay processing module can have different functions depending on the input. The fluorescence frame can be converted to an overlay and stored in a temporary buffer. The color frames can have the current overlay buffer added to them.

[0140] The fluorescent frame can be overlaid on a transparent overlay. Each fluorescent dye generates a response on the image sensor 433 that can be uniquely identified. Depending on the type of fluorescent light used for each frame, the frame can be processed such that areas with fluorescent dye are marked white and areas without fluorescent dye are made into a dark mask. These mask images can be used to add additional information to the color frame. The system 100 can store them one by one in a temporary location for each type of fluorescent frame used by the system 100.

[0141] The color frame may not be processed directly, but instead may have a mask image overlaid on it. The mask image can be a black and white image that can be colored according to the color key of the fluorescent frame. FIG. 29A illustrates a color frame 1310 without an overlay, and FIG. 29B illustrates a color frame 1310 with a fluorescein mask image 1320 overlaid on it. In some embodiments, the fluorescein mask image 1320 can be colored green to stand out against the background.

[0142] The overlaid frame can receive the most recently processed mask from all current types of fluorescence imaging. In an exemplary embodiment using ICG and fluorescein, both masks can be overlaid on the original image. The overlay enables the user to pick up information on the image (e.g., blood vessels, ureters, or other anatomical structures) that may be difficult for the surgeon to identify. In other words, the overlay provides a reduction in the surgeon's cognitive load.

[0143] In step 1060, the system can be configured to perform luminance detection to calculate how bright the image appears on the screen. By quantifying how bright the image is, the surgeon can adjust the amount of light emitted from the LED to the luminance specified by the surgeon. System 100 can be configured to receive user input regarding how bright the image is, and then system 100 can use the luminance detection to notify the control loop on GPU 52. When the luminance detection calculation is complete and the calculation is sent to the surgeon, GPU 52 can send a callback to the messaging system.

[0144] In step 1070, the image composition module of GPU 52 can use a framework called OpenGL to display an image on display 130 or headset 140. OpenGL enables basic operations such as drawing an image on the screen and incorporates methods for arranging different images (referred to as textures) on a larger screen. System 100 can use OpenGL to place the left and right lines of sight on display 130 or headset 140.

[0145] Steps 1020, 1030, 1040, 1050, 1060 may be performed in an order different from the order presented in FIG. 26.

[0146] An interlaced frame can be generated when the illumination is switched between the spectral LED and the white LED between each normal frame and spectral frame. These frames contain a mixture of normal data and spectral data. These frames have useful information that can be incorporated into the image presented to the user.

[0147] FIGS. 30A - 30C illustrate an interlaced frame 1420 between a normal frame 1410 and a spectral frame 1430. The interlaced frame 1420 results in a partially illuminated, darker, and noisier image.

[0148] In successive acquisitions, the normal frame 1410 can also be preceded by another partially illuminated interlaced frame 1420. Both interlaced frames 1420 can have different illuminations because the illumination sequences are different. As a result, both interlaced frames 1420 have different normal and noise characteristics that prevent the system 100 from using conventional methods for combining frames. The system can be adapted to the Debevec algorithm for creating HDR images. The published algorithm can be applied to conventional color and grayscale images. Conventional HDR images can be generated by using images acquired by the same image sensor 433 with different exposures under the same lighting conditions. When the interlaced frames 1420 are illuminated by different color spectra, the system 100 can introduce different schemes that adapt the algorithm to apply weights to the RGB channels and adjust the global exposure weights. These weights can be obtained by observing the color vectors of the interlaced images. Each of the interlaced frames 1420 will have a different color correction matrix that allows the system 100 to derive appropriate weights for combining the frames. FIG. 31 illustrates the result of combining two interlaced frames and one normal frame between two spectral frames of the same light source. More specifically, FIG. 31 illustrates the tone-mapped composite image 1510 of two partially illuminated frames and one normal frame. The advantage of combining images in this manner is that in addition to utilizing interlaced frames, the system 100 can recover a large amount of information that would otherwise be washed out in the normal frames.

[0149] Interlace frame 1420 is irradiated with white light and a wavelength that induces fluorescence in the injected dye. Different dyes have different fluorescence wavelengths. For example, fluorescein emission peaks at 525 nm and appears green, while ICG emission peaks at 814 nm and appears purple after passing through the filter. Depending on the choice of dye, it is theoretically possible to separate colors from the partially exposed interlace frame within the frequency domain. These color vectors will appear as peaks at other frequencies that will be more uniformly distributed. According to frequency-selective digital filtering techniques, it is possible to extract emission data from the interlace image 1420. This data can then be combined with the spectral image 1430 to reduce the noise in the spectral image 1430.

[0150] Preferred embodiments of the invention have been shown and described herein, but it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will occur to those skilled in the art without departing from the invention. It is understood that various alternatives to the embodiments of the invention described herein may be used. The following claims define the scope of the invention, and it is intended that methods and structures within these claims and their equivalents be covered thereby.

Claims

1. A camera assembly configured for simultaneous multispectral imaging, comprising: a first lens assembly; a second lens assembly; a first plurality of light-emitting diodes (LEDs) configured to emit light in a first wavelength range; a second plurality of LEDs configured to emit light in a second wavelength range; a plurality of LED bandpass filters, each LED bandpass filter of the plurality of LED bandpass filters being positioned in front of each of the second plurality of LEDs so as to filter light emitted from each of the second plurality of LEDs; a plurality of image sensors, a first image sensor of the plurality of image sensors being positioned behind the first lens assembly so as to capture light from the first lens assembly, and a second image sensor of the plurality of image sensors being positioned behind the second lens assembly so as to capture light from the second lens assembly; a plurality of notch filters, each notch filter being positioned between a respective one of the plurality of image sensors and one of the first lens assembly and the second lens assembly, and each notch filter being configured to filter out light in a selected wavelength range transmitted by the respective first and second lens assemblies; a circuit board, the circuit board being electronically coupled to the first and second pluralities of LEDs and the plurality of image sensors, the circuit board being configured to strobe the plurality of LEDs so that each of the plurality of image sensors simultaneously captures multiple spectra of light. A camera assembly.

2. The camera assembly according to claim 1, further comprising a laser.

3. The camera assembly according to claim 2, further comprising a laser bandpass filter, the laser bandpass filter being positioned adjacent to the laser so as to allow light in a selected wavelength band from the laser to pass through the laser bandpass filter.

4. The first plurality of LEDs are configured to emit light in the range of 400 nm to 700 nm, and the second plurality of LEDs are configured to emit light in the range of 800 nm to 820 nm. The camera assembly according to claim 1.

5. The camera assembly according to claim 1, further comprising a third plurality of LEDs configured to emit light in the range of 475 nm to 505 nm.

6. The camera assembly according to claim 1, wherein at least one of the plurality of LED band-pass filters is configured to block all light except light having a wavelength of about 490 nm.

7. The camera assembly according to claim 1, wherein the second plurality of LEDs are configured to excite a dye in biological tissue.

8. The camera assembly according to claim 7, wherein the dye is a fluorescein dye.

9. The camera assembly according to claim 1, wherein at least one of the plurality of LED band-pass filters is configured to allow visible light to pass through.

10. A surgical robot system, A first camera assembly having one or more LEDs, one or more lenses, one or more filter elements, and one or more imaging sensors; A second camera assembly having one or more LEDs, one or more lenses, one or more filter elements, and one or more imaging sensors, wherein the first and second camera assemblies provide a stereoscopic image for viewing by a user of the system. A second camera assembly; A memory storing one or more instructions; A processor configured or programmed to read the one or more instructions stored in the memory, the processor being operably coupled to the first camera assembly and the second camera assembly to simultaneously capture a plurality of spectra of light from the first camera assembly and the second camera assembly. A surgical robot system comprising a processor.

11. The surgical robot system according to claim 10, further comprising a display operably connected to the first camera assembly and the second camera assembly, the display being configured to depict an image captured by the one or more imaging sensors of each camera assembly.

12. The surgical robot system according to claim 11, wherein the processor is configured to cause the plurality of LEDs to strobe so that the image is composed of a plurality of spectra of light.

13. The surgical robot system according to claim 10, wherein at least one of the first camera assembly or the second camera assembly further comprises a laser.

14. The surgical robot system according to claim 13, wherein at least one of the first camera assembly or the second camera assembly further comprises a laser band-pass filter, and the laser band-pass filter is positioned adjacent to the laser so as to allow light in a selected wavelength band from the laser to pass through the laser band-pass filter.

15. The surgical robot system according to claim 10, wherein the one or more LEDs of at least one of the first camera assembly or the second camera assembly include at least one LED configured to emit light in the range of 400 nm to 700 nm and at least one LED configured to emit light in the range of 800 nm to 820 nm.

16. The surgical robot system according to claim 15, wherein the one or more LEDs of at least one of the first camera assembly or the second camera assembly further include at least one LED configured to emit light in the range of 475 nm to 505 nm.

17. The surgical robot system according to claim 10, wherein the one or more filter elements of at least one of the first camera assembly or the second camera assembly are configured to block all light except light having a wavelength of about 490 nm.

18. The surgical robot system according to claim 10, wherein the one or more LEDs of at least one of the first camera assembly or the second camera assembly are configured to excite a dye in biological tissue.

19. The surgical robot system according to claim 18, wherein the dye is a fluorescein dye.

20. The surgical robot system according to claim 10, wherein the one or more filter elements of at least one of the first camera assembly or the second camera assembly are configured to allow visible light to pass through.

Images