Determination of chromophore concentration in tissue

Abstract

The invention relates to determination of chromophore concentration in tissue. The techniques described herein may be embodied in a method that includes obtaining a plurality of images of a surgical scene using an endoscopic camera, each image of the plurality of images being obtained under irradiation of electromagnetic radiation in a corresponding wavelength range. Corresponding wavelength ranges are selected according to a set of chromophores present in the surgical scene, and any two of the corresponding wavelength ranges are partially non-overlapping. The method further includes determining, based on the plurality of images, a concentration of each of the one or more chromophores in the group at the respective portions of the surgical scene, and generating, by the one or more processing devices, a representation of the surgical scene based on information about the concentration of each of the one or more chromophores. The method also includes presenting the representation of the surgical scene on an output device associated with the surgical device.

Description

[0001] This application is a divisional application of Chinese patent application No. 201980079965.5, filed on December 4, 2019, entitled “Determination of Chromophore Concentration in Tissue”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 775,463, entitled “DETERMINING CHROMOPHORE CONCENTRATIONS IN TISSUES,” filed on December 5, 2018. The disclosure of the foregoing application is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to imaging used in endoscopic surgical systems. Background Art

[0005] Minimally invasive surgical systems are being developed to reduce the trauma experienced by patients undergoing surgical interventions. These systems require only small incisions and the surgeon uses a camera-like stick and instrument to perform the procedure. In addition to reducing trauma, this type of remote-controlled system increases the surgeon's dexterity and allows the surgeon to operate on the patient from a remote location. Teleoperation is a general term for surgical systems in which the surgeon uses some form of remote control (e.g., servomechanism, etc.) to manipulate the movement of surgical instruments rather than directly holding and moving the instruments by hand. In such teleoperation surgical systems, an image of the surgical site is provided to the surgeon via a display device. Based on the visual feedback received via the display device, the surgeon performs the surgical procedure on the patient by manipulating the main control input device (which in turn controls the movement of the telerobotic instrument). Summary of the invention

[0006] In one aspect, the document features a method of presenting feedback about the state of a surgical scene operating when using a surgical device. The method includes using an endoscopic camera to obtain multiple images of the surgical scene, each of the multiple images being obtained under illumination with electromagnetic radiation in a corresponding wavelength range. The corresponding wavelength range is selected based on a set of chromophores present in the surgical scene, and any two of the corresponding wavelength ranges are at least partially non-overlapping. The method also includes determining, based on the multiple images, the concentration of each of one or more chromophores in the set at various portions of the surgical scene, and generating, by one or more processing devices, a representation of the surgical scene based on information about the concentration of each of the one or more chromophores. The method also includes presenting a representation of the surgical scene on an output device associated with the surgical device.

[0007] In another aspect, the document features an imaging system that presents feedback about the state of a surgical scene operating while using a surgical device. The system includes an endoscopic camera that obtains a plurality of images of the surgical scene, each of the plurality of images being obtained under illumination with electromagnetic radiation in a corresponding wavelength range. The corresponding wavelength range is selected based on a set of chromophores present in the surgical scene, wherein any two of the corresponding wavelength ranges are at least partially non-overlapping. The system also includes one or more processing devices configured to determine, based on the plurality of images, a concentration of each of the one or more chromophores in the set at respective portions of the surgical scene, generate a representation of the surgical scene based on information about the concentration of each of the one or more chromophores, and cause the representation of the surgical scene to be presented on an output device associated with the surgical device.

[0008] In another aspect, the document features one or more machine-readable storage devices having computer-readable instructions encoded thereon for causing one or more processing devices to perform various operations. The operations include obtaining a plurality of images of a surgical scene from an endoscopic camera, each of the plurality of images being obtained under illumination with electromagnetic radiation in a corresponding wavelength range. The corresponding wavelength range is selected based on a set of chromophores present in the surgical scene, wherein any two of the corresponding wavelength ranges are at least partially non-overlapping. The operations also include determining a concentration of each of one or more chromophores in the set at respective portions of the surgical scene based on the plurality of images, generating a representation of the surgical scene based on information about the concentration of each of the one or more chromophores, and causing the representation of the surgical scene to be presented on an output device associated with the surgical device.

[0009] Implementations of the above aspects may include one or more of the following features.

[0010] Determining the concentration of each of the one or more chromophores may include: identifying a set of pixel values ​​across a plurality of images in a plurality of images, wherein each pixel value in the set substantially represents a specific portion of the surgical scene; and determining the concentration of each of the one or more chromophores corresponding to the set of pixel values ​​across the plurality of images based on a chromophore model. The chromophore model represents a relationship between a combination of pixel values ​​and chromophore concentrations. The chromophore model may be pre-calculated and stored in the form of a lookup table on a storage device accessible to one or more processing devices. In some cases, a set of pixel values ​​may be identified across a plurality of images in a plurality of images, wherein each pixel value in the set substantially represents a specific portion of the surgical scene. It may be determined that the set of pixel values ​​does not represent a valid concentration of at least one of the one or more chromophores based on the chromophore model representing the relationship between the combination of pixel values ​​and the chromophore concentration, and in response, an alarm may be generated indicating that heterogeneity may be present at the surgical scene. The corresponding wavelength range may be selected based on information about the absorption spectrum of the chromophores in the set of chromophores. The set of chromophores may include hemoglobin, oxyhemoglobin, fat, and water. Generating a representation of a surgical scene may include predicting a hyperspectral image of the surgical scene based on information about the concentration of each of the one or more chromophores. The hyperspectral image may include information corresponding to wavelengths outside the wavelength range corresponding to the plurality of images. Generating a representation of the surgical scene may include generating a color visible range image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the color visible range image representing the appearance of the surgical scene under illumination by visible light. The endoscopic camera may include less than three image sensors per pixel. Generating a representation of the surgical scene may include generating a narrowband image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the narrowband image representing the appearance of the surgical scene under illumination by light outside the wavelength range corresponding to the plurality of images. An amount of illumination incident on the surgical scene may be estimated for illumination at one or more of the wavelength ranges corresponding to the plurality of images. An exposure time of the endoscopic camera may be adjusted based on the information about the amount of illumination incident on the surgical scene. A tone mapping operation may be performed on one or more of the plurality of images based on the information about the amount of illumination incident on the surgical scene. An amount of scattering of various portions of a surgical scene may be estimated based on at least a subset of the plurality of images. A boundary between two different tissue types at the surgical scene may be determined based on (i) information about the amount of scattering of various portions of the surgical scene and (ii) information about the concentration of each of the one or more chromophores. A boundary of a diseased portion of tissue at the surgical scene may be determined based on (i) information about the amount of scattering of various portions of the surgical scene and (ii) information about the concentration of each of the one or more chromophores.One or more tissue types present at the surgical scene may be identified based on (i) information about the amount of scattering of various portions of the surgical scene and (ii) information about the concentration of each of the one or more chromophores. The endoscopic camera may be configured to capture multiple images without physical contact with the surgical scene. User input may be received in response to presenting a representation of the surgical scene on a display device, wherein the user input relates to operating a surgical device at the surgical scene. The camera may be disposed in the surgical device. The representation of the surgical scene may include a visual representation, and the output device includes a display device.

[0011] Some or all of the embodiments described herein may provide one or more of the following advantages. By selecting an illumination wavelength range based on the chromophores expected to be present at the surgical scene, the concentration of the chromophores at the surgical scene can be estimated based on a small number of images. The chromophore estimates thus obtained can be used to predict / estimate / generate various types of feedback information about the surgical scene. For example, chromophore concentration data can be used to accurately predict hyperspectral visualization of the surgical scene. In some cases, chromophore concentration data can be used to generate other representations (e.g., visual images, narrowband images, etc.). Representations under these modalities can potentially be generated using less complex hardware (e.g., image sensors with fewer numbers per pixel, less complex circuits) and / or lower bandwidths than required in other cases. In some cases, the information encoded in the above modalities can also be obtained directly from the chromophore concentration data. In some cases, by detecting anomalies in the combination of chromophore concentrations, the detection of the presence of heterogeneous (e.g., metal objects or other non-tissue materials) or diseased cells / tissues can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a perspective view of an example patient side cart for a computer-assisted teleoperated surgical system.

[0013] Figure 2 is a front view of an example surgeon's console for a computer-assisted teleoperated surgical system.

[0014] Figure 3 is a side view of an example robotic manipulator arm assembly of a computer-assisted teleoperated surgical system.

[0015] Figure 4 is a schematic diagram showing the effect of chromophores on imaging of a surgical scene under white light.

[0016] Figure 5 is a schematic diagram showing chromophore imaging under illumination of a set of wavelength ranges selected according to the chromophores expected to be present at the surgical scene.

[0017] Figure 6 is a graph showing the absorption characteristics of a group of chromophores.

[0018] 7A to 7F are examples of images obtained under illumination of a set of selected wavelength ranges.

[0019] Figure 8 is a set of images showing an example of changes in the concentration of an example chromophore (oxyhemoglobin) over time at a surgical scene.

[0020] Fig. 9 is an example graph illustrating a comparison between reflectance characteristics associated with an actual hyperspectral image of a surgical scene and reflectance characteristics associated with a hyperspectral image predicted from chromophore concentration data for the same surgical scene.

[0021] Fig.10 is a flowchart of an example process for presenting a representation according to the techniques described herein. DETAILED DESCRIPTION

[0022] This document describes a technique that allows a surgical scene to be evaluated based on the concentration of multiple chromophores at the scene, which is estimated from a set of images captured at a set of selected wavelength ranges. The set of wavelength ranges is selected based on the absorption characteristics of the chromophores expected to be present at the surgical scene. The concentration of each of the multiple chromophores at the surgical scene can then be calculated (possibly over a period of time) based on a pre-calculated model that links the pixel values ​​of the set of images to the concentrations of the different chromophores. The chromophore concentration data thus obtained can then be used for various purposes, including, for example, estimating the representation of a hyperspectral, visible range, or narrowband image, detecting the possible presence of foreign matter and / or diseased cells / tissues at the surgical scene, delineating boundaries between different tissue types, or obtaining other information encoded in the chromophore concentration data. In various embodiments, the techniques described herein can improve robotic surgical systems, for example, by allowing the implementation of complex visualization modes (such as hyperspectral imaging) while using significantly less complex hardware (e.g., a smaller number of image sensors per pixel, less complex circuits) and / or lower bandwidth than would otherwise be required. In some embodiments, the visualization and / or information provided by the chromophore concentration data can provide improved detection of boundaries between tissue types, identification of diseased cells / tissue, and / or identification of foreign objects (e.g., surgical tools, etc.) at the surgical scene.

[0023] Based primarily on the use of a surgical procedure developed by Intuitive Surgical, Inc. of Sunnyvale, California. (da ) surgical system embodiments are used to describe various aspects of the technology. Examples of such surgical systems are Xi TM (da Xi TM ) Surgical System (Model IS4000). It should be understood that the aspects disclosed herein can be embodied and implemented in various ways, including computer-assisted, non-computer-assisted, and hybrid combinations of manual and computer-assisted embodiments and implementations. The embodiments on the surgical system (e.g., model IS4000) are described for illustrative purposes and should not be considered to limit the scope of the aspects of the invention disclosed herein. Where applicable, aspects of the invention can be embodied and implemented in relatively small handheld manual devices and relatively large systems with additional mechanical support, as well as in other embodiments of computer-assisted remote-controlled medical devices. Although the technology is described primarily with reference to the example of a peer-in display, the technology can also be used for other types of wearable or non-wearable display devices, such as head-mounted display devices used in virtual or augmented reality (VR / AR) systems. The captured images can also be displayed on a large format display, such as a device like a 3D television, or the images are projected onto some kind of screen and viewed by a user wearing glasses, which complete the stereoscopic effect by ensuring that the correct image reaches the correct eye. Alternatively, an autostereoscopic type display can be used, such as a lens-based LCD type display, which can also incorporate the viewer's (user's) head and / or eye tracking.

[0024] refer to Figure 1 and Figure 2 , a system for minimally invasive computer-assisted telesurgery (also known as MIS) may include a patient-side cart 100 and a surgeon's console 50. Telesurgery is a general term for surgical systems in which the surgeon uses some form of remote control (e.g., servomechanisms, etc.) to manipulate the movement of surgical instruments rather than directly holding and moving the instruments by hand. Robotically manipulable surgical instruments can be inserted through small, minimally invasive surgical holes to treat tissue at a surgical site in a patient's body, thereby avoiding the trauma associated with the relatively large incisions required for open surgery. These robotic systems can move the working ends of surgical instruments with sufficient dexterity to perform fairly complex surgical tasks, typically by pivoting the shaft of the instrument at a minimally invasive hole, sliding the shaft axially through the hole, rotating the shaft within the hole, etc.

[0025] In the depicted embodiment, the patient side cart 100 includes a base 110, a first robotic manipulator arm assembly 120, a second robotic manipulator arm assembly 130, a third robotic manipulator arm assembly 140, and a fourth robotic manipulator arm assembly 150. Each robotic manipulator arm assembly 120, 130, 140, and 150 is pivotally coupled to the base 110. In some embodiments, fewer than four or more than four robotic manipulator arm assemblies may be included as part of the patient side cart 100. Although in the depicted embodiment, the base 110 includes casters to allow for easy mobility, in some embodiments, the patient side cart 100 is fixedly mounted to a floor, ceiling, operating table, structural frame, or the like.

[0026] In a typical application, two of the robotic manipulator arm assemblies 120, 130, 140, or 150 hold surgical instruments and a third holds a stereo endoscope. The remaining robotic manipulator arm assembly can be used to enable the introduction of a third instrument at the work site. Alternatively, the remaining robotic manipulator arm assembly can be used to introduce a second endoscope or another image capture device (such as an ultrasound transducer) to the work site.

[0027] Each of the robotic manipulator arm assemblies 120, 130, 140, and 150 is conventionally formed of links coupled together and manipulated by actuatable joints. Each of the robotic manipulator arm assemblies 120, 130, 140, and 150 includes a setup arm and a device manipulator. The setup arm positions the device it holds so that a pivot point occurs at an entry hole into the patient. The device manipulator can then manipulate the device it holds so that it can pivot about the pivot point, be inserted into and retracted from the entry hole, and rotate about the axis of its shaft.

[0028] In the depicted embodiment, the surgeon console 50 includes a stereoscopic peer display 45 so that the user can view the surgical work site in stereoscopic vision from images captured by a stereo camera used in conjunction with the patient side cart 100. Left and right eyepieces 46, 47 are provided in the stereoscopic peer display 45 so that the user can view the left and right display screens within the display 45 with the user's left and right eyes, respectively. While viewing the image of the surgical site, typically on a suitable viewer or display, the surgeon performs a surgical procedure on the patient by manipulating the master control input device (which in turn controls the movement of the robotic instrument).

[0029] The surgeon console 50 also includes left and right input devices 41, 42, which a user can grasp with his / her left and right hands, respectively, to manipulate, with preferably six or more degrees of freedom (“DOF”), devices (e.g., surgical instruments) held by the robotic manipulator arm assemblies 120, 130, 140, and 150 of the patient side cart 100. A foot pedal 44 with toe and heel controls is provided on the surgeon console 50 so that a user can control movement and / or actuation of a device associated with the foot pedal.

[0030] For control and other purposes, a processing device 43 is provided in the surgeon console 50. The processing device 43 performs various functions in the medical robotic system. One function performed by the processing device 43 is to convert and transmit the mechanical motion of the input devices 41, 42 to actuate their corresponding joints in their associated robotic manipulator arm assemblies 120, 130, 140 and 150 so that the surgeon can effectively manipulate the device (such as a surgical instrument). Another function of the processing device 43 is to implement the methods described herein, cross-linking control logic and controllers.

[0031] The processing device 43 may include one or more processors, digital signal processors, field programmable gate arrays (FPGAs), and / or microcontrollers, and may be implemented as a combination of hardware, software, and / or firmware. In addition, its functions as described herein may be performed by one unit or divided between multiple subunits, each of which may be implemented in turn by any combination of hardware, software, and firmware. In addition, although shown as part of the surgeon's console 50 or physically adjacent to the surgeon's console 50, the processing device 43 may also be distributed throughout the teleoperated surgical system as a subunit. One or more of the subunits may be physically remote from the teleoperated surgical system (e.g., located on a remote server).

[0032] Also refer to Figure 3, the robotic manipulator arm assemblies 120, 130, 140, and 150 can manipulate devices such as endoscopic stereo cameras and surgical instruments to perform minimally invasive surgical procedures. For example, in the depicted arrangement, the robotic manipulator arm assembly 120 is pivotally coupled to the instrument holder 122. The cannula 180 and the surgical instrument 200 are then releasably coupled to the instrument holder 122. The cannula 180 is a hollow tubular member that is located at the patient interface site during surgery. The cannula 180 defines a cavity in which an endoscopic camera (or endoscope) or an elongated shaft 220 of the surgical instrument 200 is slidably disposed. As further described below, in some embodiments, the cannula 180 includes a distal portion having a body wall retractor member. The instrument holder 122 is pivotally coupled to the distal end of the robotic manipulator arm assembly 120. In some embodiments, the pivotable connection between the instrument holder 122 and the distal end of the robotic manipulator arm assembly 120 is a motorized joint that can be actuated from the surgeon console 50 using the processing device 43 .

[0033] The instrument holder 122 includes an instrument holder frame 124, a sleeve clamp 126, and an instrument holder bracket 128. In the depicted embodiment, the sleeve clamp 126 is fixed to the distal end of the instrument holder frame 124. The sleeve clamp 126 can be actuated to couple or disengage with the sleeve 180. The instrument holder bracket 128 is movably coupled to the instrument holder frame 124. More specifically, the instrument holder bracket 128 can be linearly translated along the instrument holder frame 124. In some embodiments, the movement of the instrument holder bracket 128 along the instrument holder frame 124 is a motorized translation movement that can be actuated / controlled by the processing device 43. The surgical instrument 200 includes a transmission assembly 210, an elongated shaft 220, and an end effector 230. The transmission assembly 210 can be releasably coupled to the instrument holder bracket 128. The shaft 220 extends distally from the transmission assembly 210. The end effector 230 is disposed at the distal end of the shaft 220 .

[0034] The shaft 220 defines a longitudinal axis 222 that coincides with the longitudinal axis of the cannula 180. As the instrument holder cradle 128 translates along the instrument holder frame 124, the elongated shaft 220 of the surgical instrument 200 moves along the longitudinal axis 222. In this manner, the end effector 230 can be inserted into and / or retracted from a surgical workspace within a patient's body.

[0035] Laparoscopic surgery may require the surgeon to view the surgical site with an endoscope and perform fine motor manipulation with laparoscopic instruments to perform exploration, dissection, suturing, and other surgical tasks. These tasks often require fine two-handed interaction with tissue. In some cases, such two-handed motor tasks can often be performed more easily when the surgeon is presented with a 3D view of the surgical scene. The surgical workspace (surgical scene) within the patient's body can be presented to the surgeon as a 3D visualization via a stereoscopic display 45. Although the technology described herein primarily uses the example of a peer-to-peer stereo display, other types of stereo and non-stereo displays are also within the scope of the technology. A peer-to-peer stereo display refers to a display that allows a user to view the display without having to wear it or share it with another user at the same time. A stereo microscope may be an example of a peer-to-peer stereo display. For example, a stereo microscope may be an example of a peer-to-peer stereo display. Figure 2 The stereoscopic display 45 shown is another example of a peer-to-peer stereoscopic display.

[0036] A chromophore is a group of atoms or electrons (also called a moiety) in an organic molecule that is responsible for the color of the organic molecule. Color is exhibited when the molecule absorbs certain wavelengths of electromagnetic energy (e.g., visible light) and transmits or reflects other wavelengths. A chromophore reacts to incident electromagnetic radiation within a given wavelength range if the energy of the incident electromagnetic radiation is sufficient to overcome the difference between two different molecular orbitals of the chromophore. When electromagnetic radiation in this wavelength range strikes the chromophore, the corresponding energy can excite electrons from the ground state to an excited state. Radiation of the corresponding wavelength is thus substantially absorbed, thereby changing the composition of the wavelength range of light reflected from the molecule.

[0037] Figure 4 4 is a schematic diagram showing the effect of chromophores on the imaging of a surgical scene under white light. In this example, tissue at a surgical scene 405 is irradiated by broadband white light 408. Chromophores 410a, 410b, 410c, etc. (generally 410) absorb energy corresponding to some wavelength ranges in the white light, thereby changing the composition of the wavelength ranges in the reflected light 415, thereby potentially making the reflected light 415 appear "colored". The reflected light 415 is sensed by an imaging device 420 (e.g., a camera), and a visual representation of the surgical scene 405 can then be generated based on the information captured using the imaging device. It should be noted that although a chromophore is a moiety that causes a structural change in a particular molecule when struck by electromagnetic radiation, as used in this document, the term chromophore refers to the underlying molecule itself. For example, hemoglobin, oxyhemoglobin, fat, and water are used as examples of chromophores throughout this document.

[0038] Because each type of chromophore at the surgical scene 405 has a characteristic color signature, the absorption of the incident light 408, and therefore the wavelength composition of the reflected light 415, depends on the concentration of the various chromophores at the surgical scene 405. For example, electromagnetic radiation in the range 500-600 nm is absorbed by oxygenated hemoglobin (also abbreviated as HbO2) in significantly greater amounts than by fat tissue. Thus, the wavelength composition of the reflected light 415 due to the higher concentration of HbO2 is different from the wavelength composition of the reflected light 415 due to the higher concentration of fat. Thus, when the surgical scene is imaged under illumination of electromagnetic radiation of different wavelength ranges, the chromophores at the scene 405 affect the image differently. This document describes a technique by which the concentrations of different chromophores at the surgical scene can be estimated based on the different appearances of the same area in multiple images obtained under varying illumination conditions. The number of different illumination wavelength ranges (and therefore the minimum number of images) can depend on the number of chromophores expected to be present at the surgical scene. For example, a typical surgical scenario may be evaluated based on chromophore concentrations corresponding to four different chromophores (hemoglobin, oxyhemoglobin, fat, and water), and therefore, images may be acquired under illumination with electromagnetic radiation in four separate wavelength ranges.

[0039] Figure 5 is a schematic diagram showing chromophore imaging under illumination of a set of wavelength ranges selected according to the chromophores expected to be present at the surgical scene. Figure 5 In the illustrated system 500, the surgical scene 405 is illuminated by electromagnetic radiation in different wavelength ranges 505a, 505b, 505c, and 505d (collectively 505), and corresponding reflected light 515 is captured using an imaging device 520. In some embodiments, the surgical scene 405 is illuminated by electromagnetic radiation in different wavelength ranges 505 sequentially, and corresponding images are captured sequentially. In some embodiments, electromagnetic radiation in different wavelength ranges 505 can be irradiated on the surgical scene substantially simultaneously, and corresponding images are captured at the imaging device 520 using corresponding optical filters. For example, the imaging device 520 can have four image sensors per pixel, each image sensor being located after an optical filter that passes light corresponding to one illumination wavelength range, but substantially blocks light corresponding to the other three illumination wavelength ranges. Figure 5 The example shown shows four different wavelength ranges corresponding to four chromophores (hemoglobin, oxyhemoglobin, fat, and water). However, a greater or lesser number of wavelength ranges is also within the scope of the present disclosure.

[0040] The illumination wavelength range 505 can be selected based on the optical properties of the chromophores expected to be present at the surgical scene. In some embodiments, the wavelength range 505 can be selected based on the absorption characteristics of different chromophores, such as Figure 6 Specifically, Figure 6 is a graph showing the absorption characteristics of a group of chromophores including hemoglobin, oxyhemoglobin, fat, and water. In this embodiment, it is shown that water and fat exhibit low absorptivity (and therefore high reflectivity) to electromagnetic energy within wavelength range 605a compared to hemoglobin and oxyhemoglobin. Moreover, in wavelength range 605a, fat exhibits a higher absorptivity (and therefore a lower reflectivity) than water. In wavelength range 605b, fat exhibits a lower absorptivity (and therefore a higher reflectivity) than water, while each of water and fat exhibits a higher reflectivity than hemoglobin and oxyhemoglobin. Therefore, it can be expected that the images obtained under illumination ranges 605a and 605b mainly represent reflections due to water and fat, respectively. The other two wavelength ranges corresponding to hemoglobin and oxyhemoglobin can be similarly calculated according to Figure 6 The absorption characteristics shown, or determined based on one or more other optical characteristics. In some embodiments, the illumination wavelength range can be selected so that the effect of the illumination on the corresponding chromophore of interest is substantially different from the effect on other chromophores of interest.

[0041] Return to reference Figure 5 , the imaging device 520 is configured to obtain a separate image under illumination of each of the different wavelength ranges 505. Figure 5 In the example of , four separate wavelength ranges 505a, 505b, 505c, and 505d are obtained, and accordingly, four separate images are obtained. In some embodiments, each of the obtained images can correspond to a separate chromophore of interest. 7A to 7D Examples of these images are shown, with these particular examples showing oxygenated hemoglobin (also called saturated, Fig. 7A ), hemoglobin( Figure 7B ),Fat( Figure 7C ) and water( Fig.7D ). In some embodiments, one or more additional images may be derived from the captured image. For example, Fig. 7E shows an intensity image representing the amount of electromagnetic energy incident on a corresponding region of tissue, and Figure 7F The scattering coefficients corresponding to different parts of the surgical scene are shown.

[0042] In some embodiments, the concentration of individual chromophores at a surgical scene can be determined based on pixel values ​​in multiple images captured at different illumination wavelength ranges and / or one or more images derived from such captured images. In some embodiments, this can be done based on a chromophore model that represents the relationship between combinations of pixel values ​​and chromophore concentrations. For example, known concentrations of chromophores of interest can be imaged in an offline process using substantially the same wavelength range to generate a set of training images. A lookup table can then be created based on the set of training images that stores concentration values ​​of chromophores linked to combinations of pixel values ​​from multiple images. For example, assuming Figure 7A-7F The example images of are training images, and the pixel at position (x1, y1) in each of the six images represents the same position of the tissue. If the concentrations of the four chromophores (oxyhemoglobin, hemoglobin, fat, and water) at that position are known to be C1, C2, C3, and C4, respectively, then the six-dimensional n-tuples (p1, p2, p3, p4, p5, p6) representing the pixel values ​​from each of the six images can be stored in a lookup table linked to the four-dimensional n-tuples (C1, C2, C3, C4) representing the corresponding chromophore concentrations. Based on the complex set of training images, various combinations of chromophore concentrations can be linked to corresponding n-tuples of pixel values. Return to reference Figure 5 , such a lookup table may be stored in a storage device 530 accessible to one or more processing devices 525. In some embodiments, one or more processing devices 525 may be provided as a reference Figure 2 A portion of the processing device 43 is described.

[0043] During runtime, the one or more processing devices 525 may be configured to analyze the multiple images captured by the imaging device 520 to determine the chromophore concentration in the underlying tissue. For example, the one or more processing devices may be configured to extract pixel values ​​representing the same underlying portion of the tissue from each of the multiple images to obtain an ordered combination of pixel values. The processing device 525 may then parse the lookup table to identify an n-tuple of pixel values ​​that substantially matches the ordered combination of pixel values ​​extracted from the multiple images, and extract the corresponding n-tuple of chromophore concentrations to obtain the concentrations of different chromophores estimated to be present in the underlying portion of the tissue. The process may be repeated for other pixel values ​​in the multiple images to obtain a map of chromophore concentrations in the underlying tissue.

[0044] In some embodiments, n-tuples representing pixel values ​​from multiple images can be determined using an analytical model that represents the relationship between chromophore concentration and corresponding reflectance (which in turn indicates pixel value). For example, an analytical model that can be used is represented as:

[0045]

[0046] Where R(λ,x) represents the reflectivity λ caused by the chromophore x, I0(x) represents the incident light intensity, ua(λ,x) represents the corresponding absorption spectrum value, and p(d) represents the probability that the chromophore x is at depth d. The absorption spectrum value can be determined as:

[0047] u a (λ,x)=u a-model *w(x) (2)

[0048] where w(x) represents the concentration of chromophore x. The depth probability p(d) is determined as:

[0049]

[0050] where u s (λ,x) is the scattering spectrum value, which can be obtained as:

[0051]

[0052] Where b(x) represents the Mie scattering coefficient. In some embodiments, the analysis model can be used to generate pixel values ​​for different concentration values ​​of different chromophores. In some embodiments, the generated values ​​can be stored as a lookup table as described above.

[0053] The concentration of chromophores determined by analyzing images captured under illumination at a limited set of wavelength ranges can be used in a variety of ways. In some embodiments, the concentration data corresponding to each chromophore can be used for a variety of diagnostic and visualization purposes. For example, the distribution of oxygenated hemoglobin (as shown, for example, using Fig. 7A Saturation maps of hemoglobin (e.g., saturation maps of hemoglobin) can indicate perfusion in the tissue. In addition, because perfusion can indicate tissue health, saturation maps can also be used to determine dead or relatively unhealthy portions of tissue. In some embodiments, saturation maps associated with hemoglobin (e.g., Figure 7B as shown), fat (e.g. Figure 7C as shown) or water (as Fig.7D Concentration data associated with hemoglobin (e.g., as shown in FIG. 1 ) can be used to delineate boundaries between different tissue types. In some embodiments, concentration data associated with hemoglobin can be used to determine adhesion in tissue. In some embodiments, one or more of the derived images can also be used for diagnostic and / or visualization purposes. For example, a scatter map (e.g., as shown in FIG. 1 ) can be used to delineate boundaries between different tissue types. In some embodiments, concentration data associated with hemoglobin can be used to determine adhesion in tissue. Figure 7F ) can be used to identify the location of diseased cells / tissues such as cancer. In another example, the intensity map (e.g., Fig. 7E ) can be used to inform whether better exposure and / or dynamic contrast would be useful in obtaining the image.

[0054] Return to reference Figure 5 , the chromophore concentration data can be processed by one or more processing devices 525 to generate control signals for one or more output devices 535. In some embodiments, the output device 535 may include a display device (e.g., a stereoscopic display 45), and the chromophore data can be presented as an image on the display device. In some embodiments, the information encoded in the image data associated with the chromophore concentration can be directly extracted and presented on the output device 535. For example, one or more processing devices 525 can be configured to process the chromophore data to determine that the n-tuple of the extracted pixel value does not correspond to any valid combination of chromophore concentrations. In this case, one or more processing devices can be configured to determine that the non-compliant n-tuple is caused by the presence of non-tissue foreign matter at the surgical scene. In response to such a determination, one or more processing devices 525 may also be configured to generate an alarm (e.g., an audible or visual alarm) in addition to displaying any image on the display device. Therefore, the technology described herein can potentially be used to improve the safety of surgical procedures by improving the heterogeneous detection capabilities of surgical systems.

[0055] In some embodiments, chromophore concentration data can improve visualization of tissue during surgical procedures, for example, by allowing the display of information not available in conventional images obtained under illumination with light in the visible range. Figure 8 , which shows a set of images illustrating an example of changes in the concentration of an example chromophore (oxyhemoglobin) at a surgical scene over time. The top row 805 shows a set of images of a portion of a human intestine captured sequentially over a period of time. The bottom row 810 shows corresponding saturation images (generated based on concentration data associated with oxyhemoglobin) obtained at the same time point. During the time period in which the images were captured, a clamp was placed at the upper portion of the intestine at time 815, and then the clamp was removed at time 820. The sequence of images between the two time points 815 and 820 on row 810 shows a decrease in the concentration of oxyhemoglobin in a portion of the tissue (which is expected due to the clamping of blood flow entering the tissue), and then after the clamp is removed, the concentration of oxyhemoglobin increases again. The visible range image (as seen in row 805) remains essentially unchanged during this time period, which shows the value of the information in the image generated based on the chromophore data.

[0056] In some embodiments, chromophore data generated by irradiating tissue with electromagnetic radiation within a small range of wavelengths can be used to estimate information that would be available when irradiating within other wavelength ranges. Thus, the chromophore data can be used to estimate / generate other broadband images (such as hyperspectral images or RGB images) and / or narrowband images (such as near infrared (NIR) images). This can be accomplished, for example, via a lookup table that stores the relationship between corresponding combinations of pixel values ​​(or reflectance characteristics) of a broadband (or narrowband) image and chromophore concentrations. Fig. 9 is an example graph showing a comparison between reflectance characteristics associated with an actual hyperspectral image of a surgical scene and reflectance characteristics associated with a hyperspectral image predicted from chromophore concentration data for the same surgical scene. Specifically, curve 905 shows the reflectance values ​​predicted for the hyperspectral image over a wide frequency range, and curve 910 shows the corresponding measured reflectance values ​​for the actual hyperspectral image of the same tissue. Comparing the two curves 905 and 910, it can be seen that for most of the spectrum, the true value of the hyperspectral image can be accurately predicted from the chromophore concentration value.

[0057] By allowing prediction of multiple broadband and / or narrowband images based on the same chromophore data, the techniques described herein may allow for the implementation of a multi-purpose camera (e.g., a camera that can be used to generate hyperspectral, visible range, and NIR images) without the need to incorporate any additional hardware. By reducing hardware and / or bandwidth requirements, the techniques also facilitate the implementation of a multi-purpose or hyperspectral camera within the limited real-eatate of an endoscope, thereby allowing for the implementation of a hyperspectral or multi-purpose endoscope that significantly improves real-time visualization for surgeons and allows for the capture of images of a surgical scene or tissue, respectively, without physical contact with the surgical scene or tissue.

[0058] Fig.10 is a flow chart of an example process 1000 for presenting a representation according to the techniques described herein. Process 1000 may be at least partially described by reference Figure 5 The operation of process 1000 includes obtaining a plurality of images of a surgical scene using an endoscopic camera (1010). Each of the plurality of images can be obtained under illumination with electromagnetic radiation in a corresponding wavelength range, wherein the corresponding wavelength range is selected based on a set of chromophores present in the surgical scene. Any two of the corresponding wavelength ranges are at least partially non-overlapping. In some embodiments, the corresponding wavelength range can be selected based on information about the absorption spectrum of the chromophores in the set of chromophores.

[0059] In some embodiments, one or more additional images can be generated based on images captured using an endoscopic camera and included in the multiple images. For example, the amount of irradiance incident on the surgical scene can be estimated for irradiation at one or more wavelength ranges, and a derivative image can be generated accordingly. Such images can be used in various ways. For example, the exposure time of the endoscopic camera can be adjusted based on information about the amount of irradiance incident on the surgical scene. In another example, a tone mapping operation can be performed on one or more of the multiple images based on information about the amount of irradiance incident on the surgical scene. In some embodiments, the amount of scatter of various parts of the surgical scene can be estimated based on one or more of the images captured using the camera and the derivative images generated accordingly. The scatter image can be used, for example, to identify the location of cells / tissues suffering from a disease such as cancer.

[0060] The operation of process 1000 also includes determining a concentration of each of the one or more chromophores in the set at various portions of the surgical scene based on the plurality of images (1020). In some embodiments, this can include, for example, identifying a set of pixel values ​​across a plurality of images in the plurality of images, wherein each pixel value in the set substantially represents a particular portion of the surgical scene, and determining a concentration of each of the one or more chromophores corresponding to the set of pixel values ​​across the plurality of images based on a chromophore model. For example, n-tuples (or ordered combinations) of pixel values ​​corresponding to similar pixels in the plurality of images can be extracted, and corresponding combinations of chromophores can be determined based on the chromophore model. In some embodiments, the chromophore model represents a relationship between combinations of pixel values ​​and chromophore concentrations, and can be stored as a pre-computed lookup table, as described above. In some embodiments, the chromophore model can be represented as a set of analytical equations, such as equations (1)-(4) above.

[0061] The operation of process 1000 also includes generating a representation of the surgical scene based on the information about the concentration of each of the one or more chromophores (1030). This can include, for example, generating a color visible range image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the color visible range image representing the appearance of the surgical scene under visible light illumination. In another example, generating the representation of the surgical scene can include generating a narrowband image of the surgical scene based on the information about the concentration of each of the one or more chromophores. The narrowband image can represent the appearance of the surgical scene under light illumination outside the wavelength range corresponding to the plurality of images. In some embodiments, generating the representation of the surgical scene includes predicting a hyperspectral image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the hyperspectral image including information corresponding to wavelengths outside the wavelength range corresponding to the plurality of images.

[0062] The generated chromophore information can be used in a variety of other ways. For example, chromophore concentration information can be used to detect the presence of heterogeneity (or non-tissue material) at the surgical scene. This can include, for example, first identifying a set of pixel values ​​across a plurality of images in a plurality of images, each pixel value in the set substantially representing a specific portion of the surgical scene. Then, it can be determined that the set of pixel values ​​does not represent an effective concentration of at least one of the one or more chromophores based on a chromophore model representing a relationship between a combination of pixel values ​​and chromophore concentrations. In response to such a determination, an alarm (e.g., an audible or visual alarm) indicating that heterogeneity may exist at the surgical scene can be generated. In some embodiments, the boundary between two different tissue types at the surgical scene can be determined based on, for example, information about the amount of scattering of various portions of the surgical scene and information about the concentration of each of the one or more chromophores. In some embodiments, information about the amount of scattering and information about the concentration of each of the one or more chromophores can be used to identify the boundary of a diseased portion of tissue at the surgical scene, and / or one or more tissue types present at the surgical scene.

[0063] The functions of the teleoperated surgical system described herein, or portions thereof, and various modifications thereof (hereinafter referred to as "functions") may be implemented at least in part via a computer program product (e.g., a computer program tangibly embodied in an information carrier (such as one or more non-transitory machine-readable media or memory devices)) for execution or control of its operation by one or more data processing devices (e.g., a programmable processor, a DSP, a microcontroller, a computer, multiple computers and / or a programmable logic component).

[0064] A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed at one location or distributed across multiple locations and interconnected by a network on one or more processing devices.

[0065] The actions associated with implementing all or part of the functions may be performed by one or more programmable processors or processing devices executing one or more computer programs to perform the functions of the processes described herein. All or part of the functions may be implemented as a dedicated logic circuit, such as an FPGA and / or an ASIC (Application Specific Integrated Circuit).

[0066] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The components of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data.

[0067] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the content claimed, but rather descriptions of features that are peculiar to a particular embodiment. Other embodiments may also be within the scope of the technology described herein. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although in this article, features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0068] The elements of different embodiments described herein can be combined to form other embodiments not specifically described above. In the case where their operation will not be adversely affected, elements can be excluded from the structure described herein. In addition, various separated elements can be combined into one or more individual elements to perform functions described herein.

Claims

1. A method of presenting feedback about the status of a surgical scenario being operated while using a surgical device, the method comprising: obtaining, using an endoscopic camera, a plurality of images of a surgical scene, each image of the plurality of images being obtained under illumination with electromagnetic radiation in a corresponding wavelength range, wherein the corresponding wavelength range is selected based on a set of chromophores present in the surgical scene, and wherein any two of the corresponding wavelength ranges are at least partially non-overlapping; determining a concentration of each of the one or more chromophores in the set at respective portions of the surgical scene based on the plurality of images; generating, by one or more processing devices, a representation of the surgical scene based on the information regarding the concentration of each of the one or more chromophores; as well as The representation of the surgical scene is presented on an output device associated with the surgical device.

2. The method of claim 1, wherein determining the concentration of each of the one or more chromophores comprises: identifying a set of pixel values ​​across a plurality of the plurality of images, wherein each pixel value in the set substantially represents a particular portion of the surgical scene; as well as The concentration of each of the one or more chromophores corresponding to the set of pixel values ​​across the plurality of images is determined based on a chromophore model, wherein the chromophore model represents a relationship between combinations of pixel values ​​and chromophore concentrations.

3. The method of claim 2, wherein the chromophore model is pre-computed and stored in the form of a lookup table on a storage device accessible to the one or more processing devices.

4. The method according to claim 1, further comprising: identifying a set of pixel values ​​across a plurality of the plurality of images, wherein each pixel value in the set substantially represents a particular portion of the surgical scene; determining, based on a chromophore model representing a relationship between combinations of pixel values ​​and chromophore concentrations, that the set of pixel values ​​does not represent a valid concentration of at least one of the one or more chromophores; as well as In response to determining that the set of pixel values ​​does not represent a valid concentration of at least one of the one or more chromophores, an alert is generated indicating that inhomogeneities may be present at the surgical scene.

5. The method of claim 1, wherein the corresponding wavelength range is selected based on information about an absorption spectrum of the chromophore in the set of chromophores.

6. The method of claim 1, wherein the set of chromophores includes hemoglobin, oxyhemoglobin, fat, and water.

7. The method of claim 1 , wherein generating the representation of the surgical scene comprises predicting a hyperspectral image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the hyperspectral image comprising information corresponding to wavelengths outside of the wavelength range corresponding to the plurality of images.

8. The method of claim 1 , wherein generating the representation of the surgical scene comprises generating a color visible range image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the color visible range image representing the appearance of the surgical scene under illumination of visible light.

9. The method of claim 8, wherein the endoscopic camera comprises fewer than three image sensors per pixel.

10. The method of claim 1 , wherein generating the representation of the surgical scene comprises generating a narrow-band image of the surgical scene based on the information about the concentration of each of the one or more chromophores, the narrow-band image representing the appearance of the surgical scene under illumination with light outside of the wavelength range corresponding to the plurality of images.

Images