System and method for detecting and mitigating foreign light at scene

Abstract

The foreign light detection system may obtain a first light image of a scene illuminated with first light in a first wavelength band and captured over a time period. The first light image depicts an object located at a scene. The foreign light detection system may track pixel values of a target region of an object in a first light image over a time period. The foreign light detection system may determine whether a target region of an object is illuminated by foreign first light based on a comparison of signal levels of pixels depicting the target region and a background model representing a reflectivity of the target region.

Description

[0001] Background Information

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 397,969, filed on August 15, 2022, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] During a minimally invasive surgical procedure, an imaging system (e.g., an endoscope) can be used to capture an image of a scene illuminated with light from a light source. An image (e.g., a video stream) can be presented during a surgical procedure to assist a surgeon in performing the surgical procedure. In some scenarios, the image of the scene is a fluorescent image or is enhanced with a fluorescent image. A fluorescent image is generated based on the fluorescence emitted by a fluorophore detected when the fluorophore is excited by fluorescent excitation light (e.g., near infrared (NIR) light). For example, a fluorescent image can be used to highlight certain portions of a scene, certain types of tissue, or tissue perfusion of a surgical area in a selected color (e.g., green). Summary of the invention

[0004] The following description presents a simplified overview of one or more aspects of the systems and methods described herein. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to indicate the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed embodiments presented below.

[0005] An illustrative system may include a memory storing instructions and a processor, the processor being communicatively coupled to the memory and configured to execute the instructions to: obtain a first light image of a scene illuminated with first light in a first wavelength band and captured within a time period, the first light image depicting an object located at the scene; track pixel values ​​of a target area of ​​the object in the first light image within the time period; and determine whether the target area is illuminated by extraneous first light based on a comparison of signal levels of pixels depicting the target area of ​​the object with a background model representing reflectivity of the target area.

[0006] An illustrative method may include: obtaining a first light image of a scene illuminated by first light in a first wavelength band and captured within a time period, the first light image depicting an object located at the scene; tracking pixel values ​​of a target area of ​​the object in the first light image within the time period; and determining whether the target area is illuminated by external first light based on a comparison of signal levels of pixels depicting the target area with a background model representing reflectivity of the target area.

[0007] An illustrative system may include a memory storing instructions and a processor, the processor being communicatively coupled to the memory and configured to execute the instructions to: obtain a visible light image of a scene illuminated with visible light and fluorescent excitation light and captured within a time period, the visible light image depicting an object located at the scene; track pixel values ​​of a target region of the object in the visible light image within the time period; adjust signal levels of pixels depicting the target region in the visible light image based on an incident visible light model, the incident visible light model estimating the amount of incident visible light from a light source of the scene at each pixel based on the pixel position and the depth of the target region, the incident visible light model representing a three-dimensional spatial distribution of visible light from the light source; determine that the target region is illuminated by extraneous fluorescent excitation light based on a comparison of the adjusted signal levels of pixels depicting the target region with a background model, the background model modeling the signal levels of pixels depicting the target region illuminated with ideal light; and perform an extraneous light mitigation operation based on a determination that the target region is illuminated by extraneous fluorescent excitation light.

[0008] An illustrative system may include a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to: obtain an image of a scene illuminated with light, the image depicting an object at the scene; determine, based on the image and a background model representing reflectivity of a target area of ​​the object, that the target area is illuminated by extraneous light; and based on determining that the target area is illuminated by extraneous light, perform an extraneous light mitigation operation.

[0009] An illustrative system may include a memory storing instructions and a processor, the processor being communicatively coupled to the memory and configured to execute the instructions to: obtain a first light image of a scene illuminated with a first light and a second light image of the scene illuminated with a second light; determine, based on the first light image and a background model representing a reflectivity of a target area of ​​an object at the scene, that the target area is illuminated by extraneous second light; and adjust a signal level in the second light image corresponding to the target area based on the determination that the target area is illuminated by the extraneous second light. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are examples only and do not limit the scope of the present disclosure. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0011] Figure 1 An illustrative configuration of an imaging system configured to capture visible light images and fluorescent images of a scene where a region of an object at the scene is illuminated by extraneous light is shown.

[0012] Figure 2 A functional diagram of an illustrative extraneous light detection system is shown.

[0013] Figure 3 An illustrative method of determining whether a region of an object is illuminated by extraneous light is shown.

[0014] Figure 4 It shows that it can be Figure 2 An illustrative visible light image of an extraneous light detection system obtained and used to detect extraneous visible light and / or extraneous fluorescent excitation light.

[0015] Figure 5A and Figure 5B An illustrative graph plotting adjusted visible light signal levels corresponding to target regions as a function of time is shown.

[0016] Figure 6 An illustrative visible light image is shown with a visual notification indicating that a target area is illuminated by extraneous light.

[0017] Figure 7 An illustrative visible light image is shown with a visual notification indicating a possible cause of extraneous light.

[0018] Figure 8 An illustrative method of performing a mitigation operation on a fluorescence channel of an imaging system when a target area is illuminated with extraneous fluorescent excitation light is shown.

[0019] Fig. 9 An illustrative method of detecting and mitigating extraneous light is shown.

[0020] Fig.10 Another illustrative method of detecting and mitigating extraneous light is shown.

[0021] Fig.11 An illustrative computer-assisted surgery system is shown.

[0022] Fig.12 A functional diagram of an illustrative computing device is shown. DETAILED DESCRIPTION

[0023] In some cases, an area of ​​the scene may be illuminated by extraneous light. Extraneous light incident on the scene area may be received indirectly from the light source of the scene and / or from another source. For example, visible light and / or fluorescent excitation light from the light source may be mutually reflected by objects at the scene (e.g., the shaft of a surgical instrument at the scene) to a small surface area of ​​the tissue at the scene. However, extraneous light at the scene area may have an adverse effect on the visible light image and / or fluorescent image presented to the user. For example, extraneous fluorescent excitation light incident on the tissue area increases the intensity of the emitted fluorescence compared to the tissue area without extraneous fluorescent excitation light. It is difficult to detect extraneous fluorescent excitation light using the fluorescent channel of the imaging system because the fluorescent signal varies over time due to the time-varying concentration of the fluorophore, photobleaching of the fluorophore, and attenuation of the emitted fluorescence.

[0024] Systems and methods for detecting extraneous light at a scene (e.g., a surgical scene) are described herein. In some examples, an imaging system may capture a first light image of a scene illuminated with a first light (e.g., light in a first wavelength band, such as blue light) over a time period. The first light image depicts an object (e.g., tissue) located at the scene. The extraneous light detection system may track pixel values ​​corresponding to a target area of ​​the object in the first light image over the time period (e.g., a surface tissue area manually selected by a user, or a surface tissue area automatically selected by an external light detection system based on image segmentation and / or feature tracking). The extraneous light detection system may determine whether the target area is illuminated by extraneous first light based on a comparison of the signal levels of pixels depicting the target area with a background model.

[0025] As used herein, "first light" is light having a first wavelength band, which may be broad, continuous spectrum light (e.g., white light) or one or more narrow-band spectrum lights (e.g., one or more color components of light, such as a blue component, a green component, and / or a red component). Also as used herein, a "first light image" of a scene is generated based on illuminating the scene with the first light. For example, the first light may be visible light (e.g., blue light or white light), and the first light image (e.g., a blue light image or a full-color image) is generated based on visible light reflected from the scene.

[0026] The background model represents the reflectivity of the target area to the incident first light. For example, when the target area is illuminated with ideal light, the background model can be based on the expected or estimated first light signal level (pixel value) in the first light image corresponding to the target area. In some examples, when the target area is illuminated by the first light from the light source of the scene (e.g., from the far end of the imaging device) but not by the external first light, the target area is illuminated by the ideal light. In other examples, when the target area is illuminated by the light modeled by the incident first light model, the target area is illuminated by the ideal light. As described below, the incident first light model represents the spatial distribution of the first light emitted from one or more light sources of the scene. The background model can be generated in real time (e.g., within a time period) based on the signal levels of the pixels depicting the target area, can be pre-generated based on the first light image captured in one or more previous time periods, and / or can be pre-generated and selected based on the type area (e.g., can be selected for a specific tissue type or anatomical feature).

[0027] In some examples, the signal level of a portion of the first light image may differ from the background model due to reasons other than incident extraneous light (e.g., movement of the imaging device, light source, and / or object). Therefore, the extraneous light detection system may adjust (e.g., normalize) the signal level of pixels in the first light image depicting the target area based on the incident first light model. Thus, false positive detections of extraneous light may be avoided.

[0028] As used herein, an "incident light model" represents the three-dimensional spatial distribution of light from one or more light sources of a scene. The incident light model can be based on or configured for light of a specific band, such as a broad, continuous spectrum light (e.g., white light) or one or more narrow-band spectrum lights (e.g., one or more color components of light, such as a blue component, a green component and / or a red component, an NIR component, etc.). For example, an incident first light model (e.g., an incident visible light model) represents the three-dimensional spatial distribution of first light (e.g., visible light) from one or more light sources. The incident light model can be estimated or can be used to estimate the amount of light incident on a surface orientation based on the position of the surface orientation relative to one or more light sources. For example, the amount of first light (e.g., visible light) incident on a surface at a scene can be determined based on a first light image (e.g., a visible light image) of the scene and an incident first light model (e.g., an incident visible light model) according to the pixel position and surface depth of each pixel in the first light image (e.g., a visible light image).

[0029] When the extraneous light detection system determines that the extraneous first light is incident on the target area of ​​the object, the extraneous light detection system can perform one or more mitigation operations. For example, the extraneous light system can provide a notification that the target area is illuminated by the extraneous first light, and / or can identify and indicate an object that may be the source of the extraneous first light.

[0030] In some examples, the scene can also be illuminated with a second light (e.g., light in a second wavelength band different from the first wavelength band). For example, the second light can be fluorescence excitation light (e.g., NIR light, ultraviolet light (UV), etc.) configured to excite fluorophores present at the scene, thereby emitting fluorescence. The fluorescence can be detected by the imaging system and used to generate a fluorescence image.

[0031] As used herein, "second light" is light having a second wavelength band, which is different from the first wavelength band. The second wavelength band may include a wide, continuous spectrum light (e.g., white light) or one or more narrow-band spectrum lights (e.g., one or more color components of light). As used herein, a "second light image" of a scene is generated based on illuminating the scene with the second light. For example, the second light may be fluorescence excitation light in a visible or invisible wavelength band (e.g., UV or NIR wavelength band), and the second light image (e.g., fluorescence image) is generated based on fluorescence emitted by a fluorophore excited by the fluorescence excitation light. The emitted fluorescence may be in a visible or invisible wavelength band (e.g., UV or NIR wavelength band), which may be different from the second wavelength band.

[0032] For illustration, indocyanine green (ICG) is a fluorophore that emits fluorescence at a wavelength of about 820 nm when irradiated with fluorescence excitation light (second light) having a wavelength of about 780 nm. The imaging system can detect the emitted fluorescence and generate a fluorescence image, wherein the detected fluorescence signal is false-colored in a visible wavelength (e.g., green). As another example, fluorescein is a fluorophore that emits fluorescence at a wavelength of about 517 nm when irradiated with fluorescence excitation light (second light) having a wavelength of about 495 nm. Various endogenous fluorophores (e.g., NAD(P)H and FAD) have peak excitation wavelengths in the UV or visible light range and emit fluorescence in the UV and / or visible light range.

[0033] In some examples, the signal levels of pixels in the second light image depicting the target area can be adjusted based on the incident second light model to account for uneven spatial and / or temporal distribution of the second light, which may be caused by, for example, movement of the imaging device, light source and / or object. The incident second light model (e.g., incident fluorescence excitation light model) is an incident light model and represents the three-dimensional spatial distribution of second light (e.g., fluorescence excitation light) from one or more light sources. The amount of second light (e.g., fluorescence excitation light) incident on the surface can be determined based on the second light image (e.g., fluorescence image) and the incident second light model (e.g., fluorescence excitation light model) according to the pixel position and surface depth of each pixel within the second light image (e.g., fluorescence image).

[0034] As with the first light, the second light from the light source may be mutually reflected by objects at the scene (such as the shaft of a surgical instrument at the scene) to a small surface area of ​​the tissue at the scene. Extraneous second light at the scene area may adversely affect the second light image. However, in some cases, it is difficult to use the second light image to determine when the target area is illuminated by the extraneous second light. For example, when the second light is fluorescence excitation light and the second image is a fluorescence image, the fluorescence signal changes over time due to photobleaching of fluorophores, attenuation of emitted fluorescence, and changes in the concentration of fluorophores within the object. Therefore, the extraneous light detection system can infer from the detection of the extraneous first light that the target area is also illuminated by the extraneous second light (e.g., extraneous fluorescence excitation light). In order to mitigate the impact of the extraneous second light, the extraneous light detection system can perform a mitigation operation, such as the above-mentioned mitigation operation, when the extraneous fluorescence excitation light is detected. In addition or alternatively, the extraneous light detection system can estimate the amount of extraneous second light incident on the target area, and can adjust the second light signal level based on the estimated amount of extraneous second light incident on the target area.

[0035] Various examples of systems and methods will be described in detail with reference to the accompanying drawings. In the following examples, the first light is described as visible light (e.g., the blue component of the light), and the first light image is a visible light image generated based on the visible light reflected from the scene (e.g., a blue light image captured in the blue channel of the imaging system). In addition, the second light is described as fluorescence excitation light (e.g., NIR light or UV light), which is configured to excite a fluorophore at the scene, and the second light image is a fluorescence image generated based on the fluorescence emitted by the excited fluorophore. However, the first light and the second light may have any other suitable band or configuration suitable for a particular embodiment. It should be understood that the examples described below are provided as non-limiting examples of how to apply various novel and creative principles in various situations. In addition, it should be understood that other examples not explicitly described herein may also be covered by the scope of the claims set forth below. The systems and methods described herein may provide one or more benefits that will be described below or become apparent.

[0036] Figure 1 An illustrative configuration of an imaging system 100 is shown, which is configured to capture a visible light image (e.g., a first light image) and a fluorescent image (e.g., a second light image) of a scene, wherein an object region at the scene is illuminated by an external light. In some examples, the scene includes a region associated with an object (e.g., a body) on or within which a fluorescence-guided medical procedure is being performed (e.g., a living human or animal body, a human or animal cadaver, a portion of a human or animal anatomical structure, tissue removed from a human or animal anatomical structure, a non-tissue artifact, a training model, etc.). In other examples, the scene can be a non-medical scene, such as a scene captured for calibration or operational evaluation purposes.

[0037] like Figure 1 As shown, the imaging system 100 includes an imaging device 102 and a controller 104. The imaging system 100 may include additional or alternative components that may serve a particular embodiment, such as various optical and / or electrical signal transmission components (e.g., wires, lenses, optical fibers, choke circuits, waveguides, cables, etc.). Although the imaging system 100 shown and described herein includes a visible light imaging system integrated with a fluorescence imaging system, the imaging system 100 may also alternatively be implemented as a stand-alone visible light imaging system that is configured to capture only visible light images of a scene. Therefore, components of the imaging system 100 that are used only to capture fluorescence images may be omitted. In some examples, the visible light imaging system and the fluorescence imaging system may be physically integrated into the same physical component, or the stand-alone fluorescence imaging system may be plugged into an auxiliary port of the visible light imaging system.

[0038] The imaging device 102 can be implemented by any suitable device configured to capture visible light images and fluorescent images of a scene. The imaging device 102 includes a camera head 106 and a shaft 108, which is coupled to the camera head 106 and extends away from the camera head 106. The imaging device 102 can be manually operated and controlled (for example, by a surgeon performing a surgical procedure on a subject). Alternatively, the camera head 106 can be coupled to a manipulator arm of a computer-assisted surgical system and controlled using robotic and / or remote operation technology. The distal end of the shaft 108 can be positioned at or near the scene to be imaged by the imaging device 102. For example, the distal end of the shaft 108 can be inserted into the patient's body via a cannula. In some examples, the imaging device 102 is implemented by an endoscope. Figure 1 As shown by arrow A in , “far” refers to positioning close to or toward a scene or area of ​​interest (e.g., away from controller 104), while “near” refers to positioning away from a scene or area of ​​interest (e.g., close to or toward controller 104).

[0039] Imaging device 102 includes a visible light camera (not shown) configured to capture a two-dimensional (2D) or three-dimensional (3D) visible light image of a scene and output visible light image data representing the visible light image. Imaging device 102 also includes a fluorescence camera (not shown) configured to capture a fluorescence image of a scene and output fluorescence image data representing the fluorescence image. The field of view of imaging device 102 is represented by dashed line 110. The visible light camera and the fluorescence camera may be implemented by any one or more suitable image sensors configured to detect (e.g., capture, collect, sense, or otherwise acquire) visible light and / or non-visible light (e.g., NIR), such as a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a hyperspectral camera, a multispectral camera, a time-correlated single photon counting (TCSPC) based photodetector (e.g., a single photon counting detector, a photomultiplier tube (PMT), a single photon avalanche diode (SPAD) detector, etc.), a time-gated based photodetector (e.g., an intensified CCD), a time-of-flight sensor, a streak camera, etc. In some examples, the visible light camera and / or the fluorescence camera are positioned at the distal end of the shaft 108. In alternative examples, the visible light camera and / or the fluorescence camera are positioned closer to the proximal end of the shaft 108, inside the camera head 106, or outside the imaging device 102 (e.g., inside the controller 104), and the optics included in the shaft 108 and / or the camera head 106 transmit the captured light from the scene to the respective cameras.

[0040] The controller 104 may be implemented by any suitable combination of hardware and / or software configured to control the imaging device 102 and / or interface with the imaging device 102. For example, the controller 104 may be implemented at least in part by a computing device included in a computer-assisted surgical system. The controller 104 may include a light source 112 and a camera control unit (CCU) 114. The controller 104 may include additional or alternative components that may serve a particular embodiment. For example, the controller 104 may include a circuit configured to provide power to components included in the imaging device 102. In an alternative example, the light source 112 and / or the CCU 114 are included in the imaging device 102 (e.g., included in the camera head 106). For example, the light source 112 may be positioned at the distal end of the shaft 108, closer to the proximal end of the shaft 108, or inside the camera head 106.

[0041] The light source 112 is configured to illuminate the scene with light (e.g., visible light and fluorescence excitation light). The light emitted by the light source 112 (represented by light ray 116) propagates through an optical channel in the axis 108 (e.g., through one or more optical fibers, light guides, lenses, etc.) to the distal end of the axis 108, where the light emerges to illuminate the scene. Therefore, the light source of the scene is the distal end of the imaging device 102. The visible light from the light source 112 may include continuous spectrum light (e.g., white light) or one or more narrowband color components of the light, such as a blue component, a green component, and / or a red component. The fluorescence excitation light from the light source 112 may include one or more broadband spectrum lights (e.g., NIR light), or may include one or more narrowband light components (e.g., narrowband NIR light components). The light source 112 may be implemented by any suitable device (e.g., a flash lamp, a laser source, a laser diode, a light emitting diode (LED), etc.). Although light source 112 is shown as a single device, light source 112 may alternatively include a plurality of light sources, each configured to generate and emit a different configuration of light (eg, visible light and fluorescent excitation light).

[0042] Visible light emitted from the distal end of shaft 108 is reflected by surface 118 of an object at the scene, and the reflected visible light is detected by a visible light camera of imaging device 102. The visible light camera (and / or other circuitry included in imaging device 102) converts the detected visible light into visible light image data representing one or more visible light images of the scene. The visible light images may include full-color images, or may be captured in one or more color channels (e.g., a blue channel) of imaging system 100.

[0043] Fluorescence excitation light emitted from the distal end of the shaft 108 excites the fluorophores 120 below the surface 118, which then emit fluorescence that is detected by a fluorescence camera of the imaging device 102. The fluorescence camera (and / or other circuitry included in the imaging device 102) converts the detected fluorescence into fluorescence image data representing one or more fluorescence images of the scene. The imaging device 102 transmits the visible light image data and the fluorescence image data to the CCU 114 via a wired or wireless communication link.

[0044] The CCU 114 may be configured to control (e.g., define, adjust, configure, set, etc.) the operation of the visible light camera and the fluorescence camera, and to receive and process the visible light image data and the fluorescence image data. For example, the CCU 114 may package and / or format the visible light image data and the fluorescence image data. The CCU 114 outputs the visible light image data and the fluorescence image data to the image processor 122 for further processing. Although the CCU 114 is shown as a single unit, the CCU 114 may alternatively be implemented by a plurality of CCUs, each of which is configured to control a different image stream (e.g., a visible light image stream and a fluorescence image stream).

[0045] The image processor 122 may be implemented by one or more computing devices external to the imaging system 100 (e.g., one or more computing devices included in a computer-assisted surgical system). Alternatively, the image processor 122 may be included in the imaging system 100 (e.g., included in the controller 104). The image processor 122 may prepare visible light image data and / or fluorescent image data for display (e.g., in the form of one or more still images and / or video streams) by a display device 124 (e.g., a computer monitor, a projector, a tablet computer, or a television screen). For example, the image processor 122 may pseudo-color fluorescent areas (e.g., green, yellow, blue, etc.) and / or selectively apply gain to adjust (e.g., increase or decrease) the intensity of fluorescent areas. The image processor 122 may also generate a graphic overlay based on the fluorescent image data and combine the graphic overlay with the visible light image to form an enhanced image (e.g., a visible light image enhanced with the fluorescent image data). The image processor 122 may also adjust the visible light image to correct various image parameters, such as automatic exposure, gain, and / or white balance.

[0046] In some examples, image processor 122 may adjust the fluorescence image data to account for a non-uniform spatial and / or temporal distribution of the fluorescence excitation light. For example, such a non-uniform distribution of the fluorescence excitation light may occur when imaging device 102 changes position and / or orientation relative to surface 118. Additionally, the intensity of the fluorescence excitation light may decrease with distance and / or angle from the light source (e.g., the distal end of imaging device 102). The non-uniform distribution of the fluorescence excitation light results in a corresponding change in the detected fluorescence signal because the intensity of the emitted fluorescence is proportional to the intensity of the incident fluorescence excitation light.

[0047] To account for such variations, the image processor 122 may adjust (e.g., normalize) the detected fluorescence signal for a portion of the fluorescent image relative to a measurement of fluorescence excitation light estimated to be incident on the surface 118 corresponding to the portion of the fluorescent image. For example, such an estimated measurement of the fluorescence excitation light may be determined using one or more sensors and / or a model of the incident fluorescence excitation light. Thus, the adjusted output fluorescence signal is substantially independent of any fluorescence signal variations due to an uneven distribution of the fluorescence excitation light. Exemplary systems and methods for adjusting the fluorescence signal level of a portion of an image relative to a measurement of fluorescence excitation light estimated to be incident on a surface corresponding to the portion of the image are described in U.S. Patent Application Publication No. 2022 / 0015857, published on January 20, 2022, the entire contents of which are incorporated herein by reference.

[0048] As described above, areas of surface 118 may be illuminated with external light. Figure 1 As shown, area 128 of surface 118 is illuminated by visible light and / or NIR light directly from imaging device 102, as shown by light 130. However, object 126 (e.g., a surgical instrument) located at the scene also reflects some visible light and / or NIR light (as shown by light 132) from imaging device 102 toward area 128. Therefore, area 128 is illuminated by extraneous visible light and / or extraneous NIR fluorescence excitation light. As used herein, "extraneous light" refers to light (e.g., visible light and / or NIR light) that is incident on a tissue area and is received indirectly from a light source of the scene (e.g., by mutual reflections within the scene) or from another source (e.g., external light leakage). As will be explained in more detail below, "extraneous" light may also refer to light that is not modeled by an incident light model, which may be used when processing a visible light image to detect extraneous light.

[0049] Extraneous visible light may adversely affect the visible light image. For example, region 128 may appear to be whitewashed or saturated. In some examples, extraneous visible light at region 128 may affect video pipeline processing, such as automatic exposure processing and / or white balance processing. Extraneous visible light at region 128 may also cause an undesirable amount of light energy to be concentrated on surface 118 at region 128.

[0050] Extraneous fluorescent excitation light may also adversely affect the fluorescent image. For example, a greater amount of fluorescent excitation light incident on region 128 of surface 118 may cause fluorophores below surface 118 at region 128 to emit a greater amount of fluorescence. As a result, the detected fluorescent signal from region 128 may not accurately represent the concentration of fluorophores below region 128, and therefore may not accurately represent or indicate the state of the medium (e.g., tissue) in which the fluorophores are located.

[0051] Extraneous fluorescent excitation light may also adversely affect processing of detected fluorescent signals used to compensate for non-uniform spatial and / or temporal distribution of fluorescent excitation light. For example, extraneous fluorescent excitation light at region 128 may adversely affect measurements of fluorescent excitation light estimated to be incident on surface 118, which may result in inaccurate adjustments of the fluorescent signal. As a result, the adjusted fluorescent signal level displayed on display device 124 may not accurately represent or indicate the state of the tissue.

[0052] An extraneous light detection system is configured to determine whether a target area of ​​an object is illuminated by extraneous light. If the target area is illuminated by extraneous light, the extraneous light detection system can perform mitigation operations, such as providing a notification and / or adjusting a signal affected by the extraneous light.

[0053] Figure 2 A functional diagram of an illustrative extraneous light detection system 200 ("system 200") is shown. System 200 may be included in, implemented by, or connected to an imaging system, a surgical system, an image processor, and / or a computing system as described herein. For example, system 200 may be implemented in whole or in part by imaging system 100, image processor 122, a computer-assisted surgical system, and / or a computing system communicatively coupled to an imaging system or a computer-assisted surgical system.

[0054] As shown, system 200 includes, but is not limited to, a memory 202 and a processor 204 selectively and communicatively coupled to each other. Each of memory 202 and processor 204 may include or be implemented by hardware and / or software components (e.g., a processor, a memory, a communication interface, instructions stored in the memory for execution by the processor, etc.). For example, memory 202 and processor 204 may be implemented by any component in a computer-assisted surgical system. In some examples, memory 202 and processor 204 may be distributed between multiple devices and / or multiple locations to serve a particular implementation.

[0055] The memory 202 may maintain (e.g., store) executable data used by the processor 204 to perform any of the operations described herein. For example, the memory 202 may store instructions 206 that may be executed by the processor 204 to perform any of the operations described herein. The instructions 206 may be implemented by any suitable application, software, code, and / or other executable data instances. The memory 202 may also maintain any data received, generated, managed, used, and / or transmitted by the processor 204.

[0056] The processor 204 may be configured to perform (e.g., execute instructions 206 stored in the memory 202 to perform) various operations related to determining whether an area of ​​an object is illuminated by extraneous light. For example, the processor 204 may access a visible light image of a scene illuminated with visible light and captured within a time period, the visible light image depicting an object located at the scene. The processor 204 may track pixel values ​​of a target area of ​​the object in the visible light image within the time period. The processor 204 may determine whether the target area is illuminated by extraneous visible light based on a comparison of the signal levels of the pixels depicting the target area with a background model representing the reflectivity of the target area. Exemplary operations that may be performed by the processor 204 will be described herein. In the following description, any reference to operations performed by the system 200 may be understood to be performed by the processor 204 of the system 200.

[0057] Figure 3 An illustrative method 300 of determining whether a target area of ​​an object is illuminated by extraneous visible light is shown. Figure 3 Operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Figure 3 Any of the actions shown in . Figure 3 One or more operations shown in may be performed by system 200, any components included therein, and / or any implementation thereof. The operations of method 300 may be performed in any suitable manner, including any manner described herein.

[0058] At operation 302, the system 200 obtains a visible light image of a scene. The visible light image depicts an object located at the scene. In some examples, the visible light image is captured in a narrowband color channel (e.g., a blue channel) of the imaging system. As described below, the signal levels of pixels in the visible light image that depict a target area of ​​the object are compared to a background model representing the reflectivity of the target area to determine whether the target area is illuminated by extraneous visible light. In the following examples, the target area is a selected area of ​​the object depicted by a subset of multiple pixels of the visible light image (e.g., a portion of the visible light image). However, in other examples to be described later, the target area may be an area of ​​the object depicted by a single pixel of the visible light image, and the system 200 may use, for example, dense optical flow to process all pixels of the visible light image (or a portion of the visible light image) to track all target areas.

[0059] At operation 304, the system 200 selects a target area of ​​the object to monitor for extraneous light based on the visible light image. If a target area was previously selected, the system 200 continues with the previously selected target area. If a target area was not previously selected, or if an additional target area is to be selected, the system 200 selects a new target area. Once the target area of ​​the object has been selected, the system 200 can use image feature tracking to track the pixel values ​​of the target area in subsequently captured visible light images of the scene. The system 200 can select a new target area in any suitable manner, including based on active or passive manual input provided by a user or automatically (e.g., without any active or passive manual input).

[0060] In some examples, system 200 selects the target area based on manual input provided by a user via a visible light image. Figure 4 An illustrative visible light image 400 that can be obtained by system 200 and that can be used to select a target area is shown. As shown, visible light image 400 depicts a scene including tissue 402 and surgical instrument 404. A user can draw a box 406 (or a circle, oval, free-drawn, or other shape) on visible light image 400 to select an area 408 of tissue depicted in box 406 as a target area. Box 406 can be drawn in any suitable manner, such as with a real or virtual instrument, a cursor, or a movable box 406.

[0061] In other examples, the system 200 can select the target region based on image segmentation. For example, a user can indicate (e.g., in the form of active manual input) an object (e.g., an anatomical feature) located at a scene, and the system 200 can use image segmentation to select the indicated object as the target region. The user can indicate the object in any suitable manner, such as by drawing a box (e.g., box 406 or other shape) around the object or selecting the object with a real or virtual instrument or cursor.

[0062] In a further example, system 200 may determine a portion of a visible light image that a user is currently viewing based on the user's eye gaze (e.g., in the form of passive manual input), and may select a region of an object that corresponds to the viewed portion of the visible light image. Figure 4 To illustrate, the system 200 may select an object (eg, an anatomical feature) that the user is currently viewing as the target region 408. Alternatively, the system 200 may select an area (eg, a rectangular or circular area) of tissue 402 within the center of the user's field of view as the target region 408.

[0063] In some examples, the system 200 can automatically select objects without providing any user input (active or manual) through visible light images and / or during the current surgical procedure. For example, the system 200 can determine the type of surgical procedure being performed based on surgical procedure data provided prior to performing the current surgical procedure, and can automatically identify specific objects associated with a specific type of surgical procedure through image segmentation and image recognition.

[0064] Additionally or alternatively, the system 200 can automatically select an object based on a preoperative image (e.g., a previous endoscopic image, MRI scan, or CT scan) that is registered to the surgical scene. For example, the preoperative image can indicate the location of a tumor at the surgical scene. The preoperative image can be registered with the surgical scene in any suitable manner. Based on the preoperative image, the system 200 can automatically select an object region that corresponds to the location of the tumor indicated in the preoperative image.

[0065] Reference again Figure 3 At operation 306, system 200 adjusts signal levels of pixels depicting the target area in the visible light image based on the incident visible light model. The incident visible light model can be used with the pixel position and depth data to adjust (e.g., normalize) the visible light signal level to account for changes in the detected visible light signal level due to changes in incident light on the surface. Changes in incident visible light on the surface are typically caused by factors other than extraneous light, such as uneven distribution patterns, movement of tissue, and / or movement of the light source.

[0066] As described above, the incident visible light model represents the three-dimensional spatial distribution of visible light emitted from a light source (e.g., the distal end of the imaging device 102). For example, a theoretical and / or empirical model (e.g., a depth map) can provide information about how the intensity of visible light from a light source decreases with distance and / or angle from the light source. Such an incident visible light model can be used to determine the amount of visible light incident on a specific surface orientation within the field of view of the imaging device. The incident visible light model can be configured, for example, to take into account intensity changes due to a specific illumination mode of the light source. In some examples, the imaging system (e.g., the imaging system 100) may include a distance sensor (e.g., a distance sensor disposed at the distal end of the imaging device 102) for measuring and / or estimating the distance between the light source and the surface tissue in the field of view. The incident visible light model as well as the pixel position and distance information can be used to determine or estimate the amount of visible light incident on each surface area in the field of view of the imaging device.

[0067] System 200 can adjust the signal level of pixels depicting the target area in the visible light image based on the amount of visible light estimated to be incident on the target area. In some embodiments, this correction can be expressed using the following equation (1):

[0068] [VLS adjusted ]=[VLS detected ] / [IVL estimated ] (1)

[0069] Among them VLS detected represents a visible light signal detected (e.g., captured) by a visible light camera for one or more pixels corresponding to a target area, IVL estimated represents the estimated amount of visible light incident on the target area, and VLS adjusted Represents the adjusted visible light signal level of one or more pixels corresponding to the target area. Other adjustment and / or normalization schemes may be used that may be suitable for a particular implementation. Figure 3 Not shown, the detected visible light signal may also be adjusted to correct other image parameters, such as automatic exposure, gain, and / or white balance.

[0070] The adjusted visible light image takes into account changes in the reflected visible light signal level due to changes in the amount of incident visible light (e.g., due to different distances and / or orientations relative to the light source) and provides a more accurate representation of the reflected visible light (and tissue reflectivity). In addition, the adjusted visible light image isolates changes in the reflected visible light signal level due to extraneous light, thereby preventing false positive detections of extraneous light.

[0071] At operation 308 , system 200 compares the signal levels of pixels in the adjusted visible light image depicting the target area to a background model representing the reflectivity of the target area.

[0072] The background model can be generated in any suitable manner. In some examples, the background model is generated in real time based on visible light images captured within a time period. In some examples, the time period includes the current time (e.g., the visible light images include the current or most recent visible light images). Figure 5A and Figure 5B An illustrative graph 500 is shown that plots an adjusted visible light signal level corresponding to a target area (e.g., target area 408) as a function of time (e.g., for each visible light image in a visible light image stream). Curve 502 represents the adjusted visible light signal level corresponding to the target area over a time period from frame 1 to frame 1000. Curve 502 can be generated based on an average, median, minimum, sum, or any other statistical analysis of the signal levels of all pixels corresponding to target area 408. System 200 can generate a background model 504 based on curve 502. Background model 504 indicates a range of expected or estimated normalized signal levels corresponding to the target area. In Figure 5AIn the example of FIG. 5 , the signal level of background model 504 ranges from about 0.10 to about 0.16. System 200 can generate background model 504 in any suitable manner using any suitable statistical analysis (e.g., a signal valley detector or ordered statistics) to ensure that background model 504 is not affected by extraneous light. Therefore, in some examples, background model 504 is implicitly a reflectance model of tissue 402.

[0073] In some examples, the background model can additionally or alternatively be generated based on one or more visible light images captured prior to the current time period (e.g., during one or more procedures performed prior to the current surgical procedure). For example, background model 504 can be generated based on visible light images captured during a preoperative procedure performed on the same subject. In some examples, the preoperatively generated background model can be used as a baseline model and can be updated in real time based on visible light images captured during the current time period (e.g., during the current surgical procedure).

[0074] In a further example, a background model can be generated based on visible light images captured during multiple different procedures performed on multiple different subjects. Such a background model can be used as a baseline model and can be updated in real time based on visible light images captured during a current time period (e.g., during a current surgical procedure).

[0075] In other examples, the background model can be specific to a specific type of tissue in the target area. For example, the system 200 can determine the tissue type of the target area and select a background model related to the tissue type of the target area from a plurality of different background models (each associated with a different type of tissue). The system 200 can determine the type of tissue in any suitable manner. For example, the system 200 can use image recognition or computer vision to determine the type of anatomical features, thereby determining the tissue type based on the anatomical features. As another example, the system 200 can determine the type of tissue based on surgical procedure data, which, for example, indicates the type of procedure being performed (e.g., hysterectomy, hernia repair, biopsy, tumor resection, etc.). In these examples, the system 200 can assume that the tissue at the surgical scene is a specific type of tissue associated with a specific surgical procedure. In other examples, the system 200 can determine the type of tissue based on a preoperative image registered to the surgical scene, as described above. The background model of each specific type of tissue can be generated in any suitable manner. In some examples, the background model can be generated empirically from one or more different procedures performed on one or more objects.

[0076] Back to Figure 3At operation 308, the system 200 compares the signal levels of the pixels depicting the target area in the adjusted visible light image (adjusted at operation 306) with the background model to determine whether the target area is illuminated by extraneous light. The system 200 can determine that the target area is illuminated by extraneous light in any suitable manner. For example, when the adjusted signal level (e.g., the average, median, maximum, minimum, or sum of the signal levels) of the pixels corresponding to the target area exceeds or falls outside the background model within a threshold time period (e.g., 3 seconds, 5 seconds, 50 frames, 100 frames, etc.), and / or exceeds the background model by a threshold amount (e.g., 10%, 25%, etc.), the system 200 can determine that the target area is illuminated by extraneous light.

[0077] To illustrate, Figure 5B Graph 500 is shown for a time period from frame 1000 to a current frame (frame 1600). As indicated by curve 502, starting from frame 1400 until at least the current time (frame 1600), the adjusted visible light signal level corresponding to the target area exceeds the background model 504. When the adjusted signal level corresponding to the target area exceeds the background model 504 (e.g., starting from time 1400), exceeds the background model 504 for a threshold time period (e.g., 100 frames, starting from frame 1500), and / or exceeds the background model 504 by a threshold amount (e.g., exceeds 25%, starting from about frame 1450), the system 200 can determine that the target area is illuminated by extraneous light.

[0078] Reference again Figure 3 At operation 310, if the system 200 determines that the target area is illuminated by extraneous light, the system 200 proceeds to operation 312. At operation 312, the system 200 performs an extraneous light mitigation operation to mitigate the effects of the extraneous light at the target area. Exemplary mitigation operations will be described in more detail below.

[0079] If the system 200 determines that the target area is not illuminated by extraneous light, or after the system 200 performs extraneous light mitigation operations at operation 312 , processing of the method 300 proceeds to operation 314 .

[0080] At operation 314, system 200 provides the adjusted visible light image for display by a display device (e.g., display device 124). In some examples, as will be described in more detail below, the visible light image can be combined with the fluorescence image to produce an enhanced image (e.g., a visible light image with a fluorescence signal superimposed thereon), and the enhanced image can be provided for display by the display device. Processing of method 300 then returns to operation 302 to obtain a subsequently captured visible light image, and method 300 is repeated for the target area. Thus, system 200 can: obtain a visible light image of a scene illuminated with visible light and captured within a time period, the visible light image depicting an object located at the scene; track a target area of ​​the object in the visible light image within the time period; and determine whether the target area is illuminated by extraneous visible light based on a comparison of the signal levels of pixels depicting the target area with a background model representing the reflectivity of the target area.

[0081] An illustrative extraneous light mitigation operation will now be described. In some examples, the extraneous light mitigation operation includes providing a notification that the target area is illuminated by extraneous light. The notification can be in any form, such as a visual, audible, and / or tactile notification (which can be provided by a user control system). The visual notification can include, for example, a message, a warning icon, and / or a graphical element overlaid on the visible light image and / or within a peripheral area of ​​a graphical user interface (GUI) displaying the visible light image. Figure 6 An illustrative visible light image 600 with a visual notification is shown. Visible light image 600 is similar to visible light image 400, except that in visible light image 600, visual notification 602 is superimposed on visible light image 600 to indicate to the user that target area 408 is illuminated by extraneous visible light. Figure 6 Visible light image 600 is shown displaying frame 406, but in other examples, frame 406 may be omitted or may be hidden and may be switched on and off as desired by the user. System 200 may continue to superimpose visual notification 602 on visible light image 600 until system 200 determines that target area 408 is no longer illuminated by extraneous visible light.

[0082] In a further example, the extraneous light mitigation operation includes identifying an object at the surgical scene that is a possible cause of extraneous light at the target area and indicating the object. The system 200 can identify the object in any suitable manner. In some examples, the system 200 identifies the object based on kinematic data representing the operation of one or more objects (e.g., one or more robotic-assisted surgical instruments) located at the scene during the time period. Figure 4 and Figure 5B, system 200 can determine based on kinematic data and / or image tracking that surgical instrument 404 changed pose (e.g., position and / or orientation within the scene) just before extraneous light was detected (e.g., at frame 1400). Therefore, system 200 can determine that surgical instrument 404 may be the cause of extraneous light at target area 408. Therefore, system 200 can indicate surgical instrument 404 as a possible cause of extraneous light.

[0083] The indication of the object can be used as a supplement or alternative to the above-mentioned visual notification, and can have any suitable form. For example, the indication can include a graphical element (such as an arrow), a message and / or a false color of the object. Figure 7 Illustrative visible light image 700 is shown in which a possible cause of extraneous light is indicated. Visible light image 700 is similar to visible light image 600, except that in visible light image 700, visual indication 702 is superimposed on visible light image 700 to indicate that surgical instrument 404 is a possible cause of extraneous light.

[0084] In some examples, the system 200 may not be able to identify an object that may be the cause of the extraneous light. Therefore, the system 200 may avoid indicating any object as a possible cause of the extraneous light. Alternatively, the system 200 may provide a notification that the cause of the extraneous light cannot be determined.

[0085] Various modifications may be made to the above method 300. In some examples, if the system 200 determines that the light source (e.g., imaging device 102) and any object at the scene (e.g., object 126) have not moved and / or the intensity of visible light output by the light source has not changed, operation 306 may be omitted. For example, the system 200 may determine based on image segmentation (e.g., feature tracking) and / or kinematic data that the imaging device 102 and the object 126 have not changed their pose relative to the previous frame, or that any change in pose is less than a threshold amount. Therefore, the system 200 may infer that there is no change in the distribution of light due to the movement of the imaging device 102 and the object 126, and may omit adjusting the visible light image based on the incident visible light model.

[0086] In some examples, the system 200 may perform the method 300 for each of a plurality of different target areas. For example, the system 200 may track a plurality of different target areas of an object over a period of time to determine whether any of the target areas are illuminated by extraneous visible light. In some examples, the system 200 generates or selects a different background model for each target area. If the system 200 determines that any of the target areas are illuminated by extraneous visible light, the system 200 may perform a mitigation operation on the target area. Alternatively, the system 200 may perform a mitigation operation when a threshold number of target areas are determined to be illuminated by extraneous visible light.

[0087] In the example of method 300 described above, the target region of the object is depicted by a subset of pixels of the visible light image, and the system 200 processes the target region as a whole (e.g., based on an average pixel value of all pixels corresponding to the target region or some other combined statistical value). In an alternative example, the target region is a region of the object depicted by a single pixel of the visible light image, and the system 200 processes multiple target regions on a single pixel basis for each pixel within the visible light image or within a selected portion of the visible light image (e.g., within box 406 or within other portions of the visible light image selected as described above). For example, the system 200 can perform method 300 for each pixel of the entire visible light image or the selected portion of the visible light image by tracking each target region using dense optical flow, and the system 200 can determine for each pixel whether the corresponding target region is illuminated by extraneous visible light. In an example where the system 200 processes all pixels of the visible light image, operation 304 can be omitted because each pixel will be analyzed in method 300.

[0088] When system 300 processes multiple target areas on a single pixel basis, as previously described, system 200 can perform an extraneous light mitigation operation when any target area is determined to be illuminated by extraneous visible light, or when a threshold number of pixels have a signal level that exceeds or falls outside the corresponding background model, exceeds or falls outside the corresponding background model for a threshold time period (e.g., 3 seconds, 5 seconds, 50 frames, 100 frames, etc.), and / or exceeds or falls outside the corresponding background model by a threshold amount (e.g., 10%, 25%, etc.).

[0089] In some examples, system 200 can use the above principles to measure the reflectivity of tissue. For example, system 200 can perform operations 302 to 306 of method 300 and use the adjusted visible light image to build a model of tissue reflectivity. As described above, the visible light signal level of the visible light image is adjusted (e.g., normalized) based on the incident visible light model to account for changes in the visible light information level caused by the three-dimensional spatial changes in the visible light distribution at the scene.

[0090] In the above example, system 200 can perform method 300 to determine whether the target area is illuminated by extraneous visible light. When system 200 determines that the target area is illuminated by extraneous visible light, system 200 can infer that the target area is also illuminated by extraneous fluorescent excitation light. Therefore, the visible light color channel of the imaging system can be used to detect extraneous fluorescent excitation light at the target area of ​​the scene. The advantage of using the visible light color channel of the imaging system to detect extraneous fluorescent excitation light is that the detection of extraneous fluorescent excitation light is based on the reflectivity of the tissue, which is generally stable over time. In contrast, it is difficult to detect extraneous fluorescent excitation light using the fluorescent channel of the imaging system because the emitted fluorescence is not fixed, but varies over time due to photobleaching of the fluorophore, attenuation of the emitted fluorescence, and changes in the concentration of the fluorophore in the object.

[0091] When the system 200 determines that the target area is illuminated by extraneous fluorescent excitation light, the system 200 can perform any mitigation operations described above with reference to extraneous visible light. In addition, the system 200 can perform mitigation operations configured to mitigate the effects of the extraneous fluorescent excitation light. For example, as will now be described, the mitigation operations can include estimating the amount of extraneous fluorescent excitation light incident on the target area, and adjusting the fluorescent image based on the estimated amount of extraneous fluorescent excitation light incident on the target area.

[0092] Figure 8 An illustrative method 800 is shown for performing a mitigation operation on a fluorescence channel of an imaging system when a target area is illuminated by extraneous fluorescent excitation light. Figure 8 Operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Figure 8 Any of the actions shown in . Figure 8 One or more operations shown in may be performed by system 200, any components included therein, and / or any implementation thereof. The operations of method 800 may be performed in any suitable manner, including any manner described herein.

[0093] At operation 802, system 200 obtains a fluorescent image and a visible light image of a scene. The fluorescent image is captured based on fluorescent light emitted by an object located at the scene, and the visible light image is captured based on visible light reflected from the object. In some examples, the fluorescent image and the visible light image are captured substantially simultaneously so that they represent the same state of the scene.

[0094] At operation 804, the system 200 estimates the amount of fluorescence excitation light incident on the object corresponding to each pixel of the fluorescence image. As described above, the uneven spatial and / or temporal distribution of the fluorescence excitation light may cause corresponding changes in the detected fluorescence signal. The system 200 can estimate the amount of fluorescence excitation light incident on the object in any suitable manner. For example, as described above, the amount of fluorescence excitation light incident on the object is sensed using one or more sensors and / or based on a model of the incident fluorescence excitation light.

[0095] At operation 806, the system 200 adjusts the fluorescence signal level of the fluorescence image in the fluorescence image based on the estimated amount of fluorescence excitation light incident on the object. By adjusting the fluorescence signal level in this manner, the fluorescence signals of the portions of the fluorescence image corresponding to the portions of the fluorescence image can be normalized relative to the amount of fluorescence excitation light estimated to be incident on the object. The normalized fluorescence image accounts for changes in the fluorescence signal due to changes in the amount of incident fluorescence excitation energy (e.g., due to different distances and / or orientations relative to the fluorescence excitation light source, etc.), and can provide a more accurate representation of the underlying fluorescence. Therefore, the adjusted fluorescence signal level is substantially independent of changes caused by uneven distribution of the fluorescence excitation light over time.

[0096] In some embodiments, the adjustment may be expressed using the following equation (2):

[0097] [FS adjusted ]=[FS detected ] / [IFEL estimated ] (2)

[0098] Among them FS detected represents the fluorescence signal detected (e.g., captured) by the fluorescence camera for one or more pixels corresponding to the target area, IFEL estimated represents the estimated amount of fluorescence excitation light incident on the target area, and FS adjusted represents the adjusted fluorescence signal level for one or more pixels corresponding to the target area. Other adjustment and / or normalization schemes suitable for a particular embodiment may be used. Although Figure 8 Not shown, but the fluorescence signal level can also be adjusted to correct for other image parameters such as gain.

[0099] At operation 808, system 200 determines whether the target area of ​​the object is illuminated by extraneous fluorescent excitation light based on the visible light image obtained in operation 802. Operation 808 may be performed in any suitable manner, such as by executing method 300 (eg, operations 302 to 310).

[0100] At operation 810, if system 200 determines that the target area is illuminated by extraneous visible light, processing of method 800 proceeds to operation 812. If system 200 determines that the target area is not illuminated by extraneous visible light, processing of method 800 skips operations 812 to 816 and proceeds to operation 818.

[0101] At operation 812, the system 200 determines that the target area is also illuminated by extraneous fluorescent excitation light based on the determination that the target area is illuminated by extraneous visible light, and continues to perform extraneous light mitigation operations in operations 814 and 816. The system 200 can determine that the target area is also illuminated by extraneous fluorescent excitation light in any suitable manner. In some examples, the system 200 determines that the target area is illuminated by extraneous fluorescent excitation light in response to the determination that the target area is illuminated by extraneous visible light. In other examples, the system 200 determines that the target area is illuminated by extraneous fluorescent excitation light in response to the determination that the target area is illuminated by extraneous visible light and further in response to determining that the extraneous visible light exceeds a threshold amount and / or lasts for a threshold duration. In a further example, the system 200 determines that the target area is illuminated by extraneous fluorescent excitation light based on the identification of a source of the extraneous visible light. For example, if system 200 knows that the extraneous visible light source (e.g., the instrument shaft) does not reflect fluorescence excitation light (e.g., absorbs substantially all NIR light), or is a secondary light source of visible light only, system 200 will not determine that the target area is illuminated by extraneous fluorescence excitation light.

[0102] At operation 814, system 200 estimates the amount of extraneous fluorescent excitation light incident on the target area. In some examples, system 200 estimates the amount of extraneous fluorescent excitation light incident on the target area by estimating the amount of extraneous visible light incident on the target area. In some examples, system 200 determines the amount of extraneous visible light incident on the target area based on the visible light image (e.g., based on a comparison of signal levels of pixels depicting the target area in an adjusted visible light image with a background model, such as operation 308 of method 300). For example, in Figure 5B In the example of , system 200 may determine that the intensity of extraneous visible light (having a normalized signal level of 0.25) at the current time (frame 1600) is approximately 56% greater than the upper threshold level of background model 504 (signal level of 0.16).

[0103] In an alternative example, system 200 determines the amount of extraneous visible light based on a comparison of the current visible light image with one or more previously captured visible light images depicting the target area. Figure 5BFor example, the system 200 may determine that at the current time (frame 1600), the intensity of the extraneous visible light (having a normalized signal level of 0.25) is approximately 100% greater than the running average normalized signal level of the target area over frames 1 to 1400 (approximately 0.125). The system 200 may then determine the amount of extraneous fluorescent excitation light incident on the target area based on the amount of extraneous visible light incident on the target area.

[0104] At operation 816, the system 200 adjusts the fluorescence signal level of the fluorescence image corresponding to the target area based on the estimated amount of extraneous fluorescent excitation light incident on the target area. The fluorescence signal adjustment can be based on the correlation between the amount of extraneous fluorescent excitation light and the increased fluorescence signal produced thereby. The correlation can be, for example, a one-to-one ratio or some other correlation that can be determined theoretically or empirically. For example, if the system 200 determines that the intensity of the extraneous visible light is approximately 56% greater than the upper threshold level of the background model 504, the system 200 can reduce the fluorescence signal level of the target area by 56%. In some examples, the system 200 can adjust the fluorescence signal level of the target area to the level indicated by the background model.

[0105] At operation 818, system 200 provides the adjusted fluorescence image for display by a display device. In some examples, the adjusted fluorescence image is combined with the visible light image (provided at operation 314 of method 300) to present an enhanced image (e.g., a visible light image superimposed with the adjusted fluorescence signal). Processing of method 800 then returns to operation 302 to access subsequently captured fluorescence images and visible light images and repeat method 800 for the target area.

[0106] In the example of method 800 described above, the target region of the object is depicted by a subset of pixels of the visible light image, and the system 200 processes the target region as a whole (e.g., based on an average pixel value of all pixels corresponding to the target region or some other combined statistical value). In an alternative example, the target region is a region of the object depicted by a single pixel of the visible light image, and the system 200 processes multiple target regions on a single pixel basis for each pixel within the visible light image or within a selected portion of the visible light image (e.g., within block 406 or within other portions of the visible light image selected as described above). For example, the system 200 may perform method 800 for each pixel of the entire fluorescence image or the selected portion of the fluorescence image to determine for each pixel whether the corresponding target region is illuminated by extraneous fluorescence. For each target region determined to be illuminated by extraneous fluorescence excitation light, the system 200 may adjust the corresponding fluorescence signal level as described in operations 814 and 816.

[0107] Fig. 9An illustrative method 900 for detecting and mitigating extraneous light is shown. Fig. 9 Operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Fig. 9 Any of the actions shown in . Fig. 9 One or more operations shown in may be performed by system 200, any components included therein, and / or any implementation thereof. The operations of method 900 may be performed in any suitable manner, including any manner described herein.

[0108] At operation 902, system 200 obtains a first light image of a scene illuminated with a first light. The first light image is captured over a time period and depicts an object located at the scene. In some examples, a signal level of the first light image is adjusted to compensate for an uneven distribution of the first light. In some examples, the first light image is a visible light image of a scene illuminated with visible light (e.g., blue light).

[0109] At operation 904 , the system 200 tracks a target area of ​​the object in the first light image within the time period.

[0110] At operation 906 , the system 200 determines that the target area is illuminated by the extraneous first light based on the first light image and a background model representing a reflectivity of the target area.

[0111] At operation 908 , the system 200 performs an extraneous light mitigation operation based on determining that the target area is illuminated by the extraneous first light.

[0112] Fig.10 An illustrative method 1000 for detecting and mitigating extraneous light is shown. Fig.10 Operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Fig.10 Any of the actions shown in . Fig.10 One or more operations shown in may be performed by system 200, any components included therein, and / or any implementation thereof. The operations of method 1000 may be performed in any suitable manner, including any manner described herein.

[0113] At operation 1002, the system 200 obtains a first light image of a scene illuminated with a first light and a second light image of the scene illuminated with a second light. In some examples, the first light image and the second light image are captured substantially simultaneously so that they substantially represent the same state of the scene. In some examples, the signal level of the first light image and / or the second light image is adjusted to compensate for the uneven distribution of the first light and the second light. In some examples, the first light image is a visible light image of a scene illuminated with visible light, and the second light image is a fluorescent image of a scene illuminated with fluorescent excitation light.

[0114] At operation 1004 , the system 200 determines that the target area is illuminated by extraneous second light based on the first light image and a background model representing reflectivity (of the first light) of the target area of ​​an object at the scene.

[0115] At operation 1006 , the system 200 adjusts a signal level in the second light image corresponding to the target area based on a determination that the target area is illuminated by the extraneous second light.

[0116] At operation 1008, the system 200 provides the adjusted second light image for display by the display device. The process then returns to operation 1002 to repeat the process for the next or subsequently acquired first and second light images.

[0117] The systems and methods described herein may be used in conjunction with a computer-assisted surgical system. Fig.11 An illustrative computer-assisted surgical system 1100 ("surgical system 1100") is shown that can be used in conjunction with the systems and methods described herein. As described herein, system 200 can be implemented by, connected to, and / or otherwise used in conjunction with surgical system 1100.

[0118] As shown, surgical system 1100 includes a manipulation system 1102, a user control system 1104, and an auxiliary system 1106 that are communicatively coupled to each other. A surgical team can utilize surgical system 1100 to perform computer-assisted surgical procedures on an object 1108. As shown, the surgical team can include a surgeon 1110-1, an assistant 1110-2, a nurse 1110-3, and an anesthesiologist 1110-4, all of whom can be collectively referred to as "surgical team members 1110." During a surgical session, additional or alternative surgical team members can be present to serve a particular embodiment.

[0119] Although Fig.11 100 is shown performing a minimally invasive surgical procedure, but it should be understood that the surgical system 1100 may similarly be used to perform open surgical procedures or may similarly be used for other types of surgical procedures that may similarly benefit from the accuracy and convenience of the surgical system 1100. Furthermore, it should be understood that the surgical procedure session during which the surgical system 1100 may be used may include not only open surgical procedures, but also other types of surgical procedures that may similarly benefit from the accuracy and convenience of the surgical system 1100. Fig.11The surgical phases of the surgical procedures shown may also include preoperative, postoperative, and / or other suitable phases of the surgical procedures. A surgical procedure may include any procedure that uses manual and / or instrumental techniques on a subject to investigate, diagnose, and / or treat a physical condition of the subject. In addition, a surgical procedure may include any non-clinical procedure, for example, a procedure that is not performed on a living subject, such as a calibration or testing procedure, a training procedure, and an experimental or research procedure.

[0120] like Fig.11 As shown, the manipulation system 1102 includes a plurality of manipulator arms 1112 (e.g., manipulator arms 1112-1 to 1112-4), to which a plurality of surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool with tissue interaction functionality), medical tool, imaging device (e.g., an endoscope), sensing instrument (e.g., a force sensing surgical instrument), diagnostic instrument, etc. that may be used for a computer-assisted surgical procedure performed on an object 1108 (e.g., by being at least partially inserted into the object 1108 and manipulated to perform a computer-assisted surgical procedure on the object 1108). Although the manipulation system 1102 is depicted and described herein as including four manipulator arms 1112, the manipulation system 1102 may include only a single manipulator arm 1112 or any other number of manipulator arms that may serve a particular embodiment.

[0121] The manipulator arm 1112 and / or a surgical instrument attached to the manipulator arm 1112 may include one or more displacement sensors, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 1100 may be configured to use the kinematic information to track (e.g., determine the position and orientation of the surgical instrument) and / or control the surgical instrument.

[0122] The user control system 1104 is configured to facilitate the surgeon 1110-1 to control the manipulator arm 1112 and the surgical instruments attached to the manipulator arm 1112. For example, the surgeon 1110-1 can interact with the user control system 1104 to remotely move or manipulate the manipulator arm 1112 and the surgical instruments. To this end, the user control system 1104 provides the surgeon 1110-1 with an image (e.g., a high-definition 3D image, a synthetic medical image, and / or a fluorescent image) of the surgical field associated with the object 1108 captured by an imaging system (e.g., the imaging system 100). In some examples, the user control system 1104 includes a stereoscopic viewer with two displays, wherein the surgeon 1110-1 can view a stereoscopic image of the surgical field associated with the object 1108 and generated by the stereoscopic imaging system. The surgeon 1110-1 can use the image to perform one or more procedures with one or more surgical instruments attached to the manipulator arm 1112.

[0123] To facilitate control of the surgical instruments, the user control system 1104 includes a set of master controls. The master controls can be manipulated by the surgeon 1110-1 to control the movement of the surgical instruments (e.g., by utilizing robotics and / or teleoperation techniques). The master controls can be configured to detect various hand, wrist, and finger movements of the surgeon 1110-1. In this way, the surgeon 1110-1 can intuitively perform procedures using one or more surgical instruments.

[0124] The auxiliary system 1106 includes one or more computing devices configured to perform the primary processing operations of the surgical system 1100. In such a configuration, the one or more computing devices included in the auxiliary system 1106 may control and / or coordinate operations performed by various other components of the surgical system 1100 (e.g., the manipulator system 1102 and the user control system 1104). For example, a computing device included in the user control system 1104 may transmit instructions to the manipulator system 1102 via one or more computing devices included in the auxiliary system 1106. As another example, the auxiliary system 1106 may receive and process image data (e.g., fluorescence image data 218 and / or processed fluorescence image data 226) from the manipulator system 1102 representing an image captured by an imaging device (e.g., imaging device 102) attached to one of the manipulator arms 1112.

[0125] In some examples, the assistance system 1106 is configured to present visual content to a surgical team member 1110 who may not have access to the images provided to the surgeon 1110-1 at the user control system 1104. To this end, the assistance system 1106 may include a display monitor 1114 that is configured to display one or more user interfaces, such as an image of the surgical field (e.g., a 2D image, a synthetic medical image, and / or a fluorescent image), information related to the object 1108 and / or the surgical procedure, and / or any other visual content that may serve a particular embodiment. For example, the display monitor 1114 may display an image of the surgical field and additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 1114 is implemented by a touch screen display with which the surgical team member 1110 can interact (e.g., via touch gestures) to provide user input to the surgical system 1100.

[0126] The control system 1102, the user control system 1104, and the auxiliary system 1106 can be communicatively coupled to each other in any suitable manner. Fig.11 As shown, the operating system 1102, the user control system 1104, and the auxiliary system 1106 are communicatively coupled via a control line 1116, which can represent any wired or wireless communication link that can serve a particular implementation. To this end, the operating system 1102, the user control system 1104, and the auxiliary system 1106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0127] The devices, systems, and methods described herein have been described with reference to fluorescence. However, it should be understood that the systems and methods described herein are not limited to fluorescence, but can be applied to any other type of luminescence, including but not limited to photoluminescence (e.g., phosphorescence, etc.), electroluminescence, chemiluminescence, mechanical luminescence, radioluminescence, etc. In addition, the scene can be illuminated with NIR light for purposes other than fluorescence imaging, such as spatial frequency domain imaging (SFDI) and other optical imaging techniques. The systems and methods described herein can be used to detect extraneous NIR light at a target area of ​​a scene and perform mitigation operations, including adjusting the signal level of a captured image based on the detected extraneous NIR light. In addition, the devices, systems, and methods described herein can be used to detect and mitigate any extraneous electromagnetic energy (e.g., ultraviolet light and infrared light) incident on a target area of ​​an object at a scene.

[0128] In some examples, a non-transient computer-readable medium storing computer-readable instructions can be provided according to the principles described herein. When executed by a processor of a computing device, the instructions can direct the processor and / or computing device to perform one or more operations, including one or more operations described herein. Such instructions can be stored and / or transmitted using any of a variety of known computer-readable media.

[0129] The non-transitory computer-readable media described herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile memory media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (RAM), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0130] Fig.12 A functional diagram of an illustrative computing device 1200 is shown, which may be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by the computing device 1200.

[0131] like Fig.12 As shown, computing device 1200 may include a communication interface 1202, a processor 1204, a storage device 1206, and an input / output (I / O) module 1208 that are communicatively connected to each other via a communication infrastructure 1210. Fig.12 An exemplary computing device 1200 is shown in FIG. Fig.12 The components shown in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig.12 Components of computing device 1200 are shown in .

[0132] The communication interface 1202 may be configured to communicate with one or more computing devices. Examples of the communication interface 1202 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (such as a wireless network interface card, etc.), a modem, an audio / video connection, and any other suitable interface.

[0133] The processor 1204 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more instructions, processes, and / or operations described herein. The processor 1204 may perform operations by executing computer-executable instructions 1212 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1206.

[0134] Storage device 1206 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1206 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 1206. For example, data representing computer executable instructions 1212 configured to direct processor 1204 to perform any of the operations described herein may be stored within storage device 1206. In some examples, the data may be arranged in one or more databases residing within storage device 1206.

[0135] The I / O module 1208 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1208 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 1208 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0136] The I / O module 1208 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1208 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0137] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it is apparent that various modifications and changes may be made to the exemplary embodiments and additional embodiments may be implemented without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined or replaced with features of another embodiment described herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: a memory for storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: obtaining a first light image of a scene illuminated with first light in a first wavelength band and captured over a time period, the first light image depicting an object located at the scene; tracking pixel values ​​of a target area of ​​the object in the first light image during the time period; and Based on a comparison of signal levels of pixels depicting the target area of ​​the object with a background model representing a reflectivity of the target area, it is determined whether the target area is illuminated by extraneous first light.

2. The system of claim 1, wherein: The processor is further configured to execute the instructions to adjust the signal level of the pixel depicting the target area in the first light image based on an incident first light model representing a three-dimensional spatial distribution of first light from a first light source; and The comparison is performed using adjusted signal levels of the pixels delineating the target area.

3. The system of claim 1 or 2, wherein the first light comprises narrowband light, and the first light image is captured by an imaging system in a narrowband light channel of the imaging system corresponding to the narrowband light.

4. A system according to any one of claims 1 to 3, wherein: During the time period, the scene is further illuminated by a second light; and The processor is further configured to execute the instructions to: obtaining a second light image of the scene captured during the time period; estimating an amount of second light incident on the target area based on an incident second light model representing a three-dimensional spatial distribution of second light from a second light source; and A signal level of the second light image is adjusted based on the estimated amount of second light incident on the target area.

5. The system of any one of claims 1-4, wherein the processor is further configured to execute the instructions to determine that the target area is illuminated by an external second light based on the determination that the target area is illuminated by the external first light.

6. The system of claim 5, wherein the processor is further configured to execute the instructions to: estimating an amount of extraneous second light incident on the target area based on the background model and the first light image; and The signal level of the second light image is adjusted further based on an estimated amount of extraneous second light incident on the target area.

7. The system according to claim 5, wherein: In response to a determination that the target area is illuminated by extraneous first light, determining that the target area is illuminated by extraneous second light is performed.

8. The system according to claim 7, wherein: Further in response to determining that the exogenous first light exceeds a threshold amount and / or persists for a threshold duration, determining that the target area is illuminated by exogenous second light is performed.

9. The system of claim 7, wherein: The processor is further configured to execute the instructions to identify an object at the scene that is a possible cause of the extraneous first light; and Determining that the target area is illuminated by extraneous second light is performed further based on the object at the scene being the possible cause of the extraneous first light.

10. The system according to any one of claims 4 to 9, wherein: The first light is visible light; and The second light is NIR light.

11. The system of any one of claims 1-10, wherein the processor is further configured to execute the instructions to perform an extraneous light mitigation operation based on determining that the target area is illuminated by extraneous first light.

12. The system of claim 11, wherein the extraneous light mitigation operation comprises providing a notification that the target area is illuminated by extraneous light.

13. The system of claim 11, wherein the extraneous light mitigation operation comprises: identifying objects at the scene that are likely causes of the extraneous light; and Indicate the object.

14. The system of any one of claims 1-13, wherein the processor is further configured to execute the instructions to generate the background model based on the signal levels of the pixels depicting the target area during the time period.

15. The system of claim 14, wherein the background model is generated further based on signal levels of pixels depicting one or more additional target regions in additional first light images captured during one or more previous time periods.

16. The system according to any one of claims 1 to 13, wherein: The object comprises a tissue; and The processor is further configured to execute the instructions to: determining the type of the tissue at the target area; as well as The background model is selected based on the determined type of the tissue at the target region.

17. A system according to any one of claims 1-16, wherein the processor is configured to execute the instructions to determine that the target area is illuminated by external first light based on a determination that the signal level of the pixels depicting the target area exceeds a threshold signal level of the background model within a threshold number of frames or a threshold time period.

18. A system according to any one of claims 1-16, wherein the processor is configured to execute the instructions to determine that the target area is illuminated by an external first light based on a determination that the signal level of the pixels depicting the target area exceeds a threshold signal level of the background model by at least a threshold amount.

19. The system of any one of claims 1-18, wherein the processor is further configured to execute the instructions to select the target area based on user input.

20. The system of any one of claims 1-18, wherein the processor is further configured to execute the instructions to automatically select the target region based on image segmentation.

21. The system of any one of claims 1-20, wherein the pixels depicting the target area include a subset of all pixels of each frame of the first light image.

22. The system of claim 1, wherein the processor is further configured to execute the instructions to determine a reflectivity of the target area based on the signal levels of the pixels depicting the target area in the first light image.

23. A method comprising: obtaining a first light image of a scene illuminated with first light in a first wavelength band and captured over a time period, the first light image depicting an object located at the scene; tracking pixel values ​​of a target area of ​​the object in the first light image during the time period; and Based on a comparison of signal levels of pixels depicting the target area with a background model representing a reflectivity of the target area, it is determined whether the target area is illuminated by extraneous first light.

24. The method according to claim 23, further comprising: adjusting the signal levels of the pixels depicting the target area in the first light image based on an incident first light model representing a three-dimensional spatial distribution of first light from a first light source; Wherein said comparing is performed using adjusted signal levels of said pixels delineating said target area.

25. The method of claim 23 or 24, wherein the first light comprises narrowband light, and the first light image is captured by an imaging system in a narrowband light channel of the imaging system corresponding to the narrowband light.

26. The method according to any one of claims 23 to 25, wherein: During the time period, the scene is further illuminated by a second light; and The method further comprises: obtaining a second light image of the scene captured during the time period; estimating an amount of second light incident on the target area based on an incident second light model representing a three-dimensional spatial distribution of second light from a second light source; and A signal level of the second light image is adjusted based on the estimated amount of second light incident on the target area.

27. The method of any one of claims 23-26, further comprising determining that the target area is illuminated by extraneous second light based on the determination that the target area is illuminated by extraneous first light.

28. The method according to claim 27, further comprising: estimating an amount of extraneous second light incident on the target area based on the background model and the first light image; and The signal level of the second light image is adjusted further based on an estimated amount of extraneous second light incident on the target area.

29. The method according to claim 27, wherein: In response to a determination that the target area is illuminated by extraneous first light, determining that the target area is illuminated by extraneous second light is performed.

30. The method of claim 29, wherein: Further in response to determining that the exogenous first light exceeds a threshold amount and / or persists for a threshold duration, determining that the target area is illuminated by exogenous second light is performed.

31. The method of claim 29, further comprising: identifying an object at the scene that is a possible cause of the extraneous first light; Wherein determining that the target area is illuminated by the extraneous second light is performed further based on the object at the scene as the possible cause of the extraneous first light.

32. The method according to any one of claims 26 to 31, wherein: The first light is visible light; and The second light is NIR light.

33. The method of any one of claims 23-32, further comprising performing an extraneous light mitigation operation based on determining that the target area is illuminated by extraneous first light.

34. The method of claim 33, wherein the extraneous light mitigation operation comprises providing a notification that the target area is illuminated by extraneous light.

35. The method of claim 33, wherein the extraneous light mitigation operation comprises: identifying an object at the scene that is a possible cause of the extraneous first light; and Indicate the object.

36. The method of any one of claims 23-35, further comprising generating the background model based on the signal levels of the pixels delineating the target area during the time period.

37. The method of claim 36, wherein the background model is generated further based on signal levels of pixels depicting one or more additional target areas in additional first light images captured during one or more previous time periods.

38. The method according to any one of claims 23 to 35, wherein: The objects include organizations; and The method further comprises: determining the type of the tissue at the target area; as well as The background model is selected based on the determined type of the tissue at the target region.

39. The method according to any one of claims 23-38, wherein determining that the target area is illuminated by the external first light is based on determining that the signal level of the pixels depicting the target area exceeds a threshold signal level of the background model within a threshold number of frames or a threshold time period.

40. The method of any one of claims 23-38, wherein determining that the target area is illuminated by extraneous first light is based on a determination that the signal level of the pixels depicting the target area exceeds a threshold signal level of the background model by at least a threshold amount.

41. The method of any one of claims 23-40, further comprising selecting the target area based on user input.

42. The method of any one of claims 23-40, further comprising automatically selecting the target region based on image segmentation.

43. The method of any one of claims 23-42, wherein the pixels depicting the target area include a subset of all pixels of each frame of the first light image.

44. The method of claim 23, further comprising determining a reflectivity of the target area based on the signal levels of the pixels depicting the target area in the first light image.

45. A system comprising: a memory for storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: obtaining a visible light image of a scene illuminated with visible light and fluorescent excitation light and captured over a period of time, the visible light image depicting an object located at the scene; tracking pixel values ​​of a target area of ​​the object in the visible light image within the time period; adjusting a signal level of a pixel in the visible light image depicting the target area based on an incident visible light model representing a three-dimensional spatial distribution of visible light from a light source; determining that the target region is illuminated by extraneous fluorescent excitation light based on a comparison of the adjusted signal levels of the pixels depicting the target region with a background model that models the signal levels of the pixels depicting the target region illuminated with ideal light; and Based on determining that the target area is illuminated by extraneous fluorescent excitation light, an extraneous light mitigation operation is performed.

46. ​​The system of claim 45, wherein the processor is further configured to execute the instructions to: obtaining a fluorescent image of the scene captured during the time period, the fluorescent image depicting fluorescent light emitted from the scene; estimating the amount of fluorescent excitation light incident on the target area based on an incident fluorescent excitation light model representing a three-dimensional spatial distribution of fluorescent excitation light from a fluorescent excitation light source; and The signal level of the fluorescent image is adjusted based on an estimated amount of fluorescent excitation light incident on the target area.

47. The system of claim 45 or 46, wherein the extraneous light mitigation operation comprises: estimating the amount of extraneous fluorescent excitation light incident on the target area based on the background model and the visible light image; and The signal level of the fluorescent image is adjusted based on an estimated amount of extraneous fluorescent excitation light incident on the target area.

48. The system of any one of claims 45-47, wherein the extraneous light mitigation operation comprises providing a notification that the target area is illuminated by extraneous fluorescent excitation light.

49. The system of any one of claims 45-48, wherein the extraneous light mitigation operation comprises: identifying objects at the scene that are likely causes of the extraneous fluorescent excitation light; and Indicate the object.

50. A system comprising: a memory for storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: obtaining an image of a scene illuminated with light, the image depicting an object at the scene; determining, based on the image and a background model representing a reflectivity of a target area of ​​the object, that the target area is illuminated by extraneous light; and Based on determining that the target area is illuminated by extraneous light, an extraneous light mitigation operation is performed.

51. The system of claim 50, wherein the processor is further configured to execute the instructions to: adjusting signal levels of pixels in the image depicting the target area based on an incident light model representing a three-dimensional spatial distribution of light from a light source; Wherein the determination that the target area is illuminated by extraneous light is performed based on the image having adjusted signal levels of the pixels depicting the target area.

52. A system comprising: a memory for storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: obtaining a first light image of a scene illuminated with a first light and a second light image of the scene illuminated with a second light; determining, based on the first light image and a background model representing a reflectivity of a target area of ​​an object at the scene, that the target area is illuminated by an external second light; and Based on a determination that the target area is illuminated by the extraneous second light, a signal level in the second light image corresponding to the target area is adjusted.

Images