Apparatus, system and method for discounting objects while managing the automatic exposure of image frames depicting the objects

Abstract

An exemplary apparatus may identify an object region in an image frame captured by an image capture system that corresponds to a depiction of an object depicted in the image frame. The apparatus may determine a frame auto-exposure value for the image frame by discounting the object region in the image frame. Based on the frame auto-exposure value, the apparatus may update one or more auto-exposure parameters for use by the image capture system to capture additional image frames. Corresponding apparatuses, systems, and methods for managing the auto-exposure of image frames are also disclosed.

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Patent Application No. 63 / 050,598, filed July 10, 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Auto-exposure algorithms operate by analyzing image frames to determine how much light is present in the scene depicted by the image frame and by updating the auto-exposure parameters of the image capture device that captures the image frame based on this analysis. In this way, the auto-exposure parameters may be continuously updated to cause the image capture device to provide the desired amount of exposure for the image frame being captured. Without good auto-exposure management, detail may be lost during the image capture process, either by overexposing (e.g., detail is lost due to saturation and the image appears too bright) or by underexposing (e.g., detail is lost due to noise and the image appears too dark).

[0003] While conventional auto-exposure algorithms work well for many types of images, they can be challenged by images that depict objects that have a different luminance than other depicted objects in the image, especially when the objects occupy a relatively large portion of the image and the objects are not of particular interest to the viewer of the image. When conventional auto-exposure algorithms process image frames that depict such objects, the algorithms are likely to overexpose or underexpose the image frames and / or encounter other undesirable problems (e.g., brightness inconsistencies as the objects move in and out of the frame). Summary of the Invention

[0004] The following description presents a simplified summary of one or more aspects of the devices, systems, and methods described herein. This summary is not an extensive overview of all contemplated aspects, and is not intended to identify key or critical elements of all aspects, nor to delineate the scope of each and every aspect. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed description presented below.

[0005] An exemplary apparatus for managing auto-exposure of image frames includes one or more processors and a memory storing executable instructions that, when executed by the one or more processors, cause the apparatus to perform various operations described herein. For example, the apparatus may identify an object region corresponding to a depiction of an object depicted in an image frame captured by an image capture system. The apparatus may determine a frame auto-exposure value for the image frame by discounting the object region in the image frame. Based on the frame auto-exposure value, the apparatus may update one or more auto-exposure parameters for use by the image capture system to capture additional image frames.

[0006] An exemplary system for managing automatic exposure of image frames includes an illumination source, an image capture device, and one or more processors. The illumination source may be configured to illuminate a scene including an internal view of a body during a medical procedure. The image capture device may be configured to capture a sequence of image frames during the medical procedure. The sequence of image frames may include image frames depicting the scene during the medical procedure. The one or more processors may be configured to determine a color gamut for an environmental image of the scene depicted in the image frames. For example, the color gamut may include a range of red colors corresponding to blood and tissue visible in the internal view of the body. The one or more processors may identify object regions within the image frames that correspond to depictions of objects depicted in the image frames. The identification may be performed based on chrominance characteristics of pixel units included in the image frames, for example, by determining whether the chrominance characteristics of the pixel units are within the identified color gamut for the environmental image of the scene. The one or more processors may determine a frame auto-exposure value for the image frame by discounting object areas in the image frame, and may update one or more auto-exposure parameters for use by the image capture device or illumination source to capture additional image frames based on the frame auto-exposure value.

[0007] An exemplary non-transitory computer-readable medium may store instructions that, when executed, cause one or more processors of a computing device to perform various operations (acts) described herein. For example, the one or more processors may identify an object region corresponding to a depiction of an object depicted in an image frame captured by an image capture system. The one or more processors may determine a frame auto-exposure target for the image frame by discounting the object region in the image frame. Based on the frame auto-exposure target, the one or more processors may update one or more auto-exposure parameters for use by the image capture system to capture additional image frames.

[0008] An exemplary method for managing the auto-exposure of image frames may include various operations described herein, each of which is performed by a computing device such as the auto-exposure management apparatus described herein. For example, the method may include identifying an object region in an image frame captured by an image capture system that corresponds to a depiction of an object depicted in the image frame. The method may further include determining a frame auto-exposure value and a frame auto-exposure target for the image frame by discounting the object region in the image frame. Based on the frame auto-exposure value and the frame auto-exposure target, the computing device performing the method may update one or more auto-exposure parameters for use by the image capture system to capture additional image frames.

[0009] The accompanying drawings illustrate various embodiments and are a part of this specification. The illustrated embodiments are merely examples and are not intended to limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. [Brief explanation of the drawings]

[0010] [Figure 1] 1 illustrates an exemplary auto-exposure management device for managing the auto-exposure of image frames according to principles described herein.

[0011] [Figure 2] 1 illustrates an exemplary auto-exposure management method for managing auto-exposure of image frames according to principles described herein.

[0012] [Figure 3] 1 illustrates an exemplary auto-exposure management system for managing the auto-exposure of image frames according to principles described herein.

[0013] [Figure 4A] 1 illustrates how local features may be utilized to identify object regions corresponding to depictions of exemplary objects depicted in exemplary image frames in accordance with the principles described herein.

[0014] [Figure 4B] 1 illustrates how global features may be utilized to identify object regions corresponding to depictions of exemplary objects depicted in exemplary image frames according to principles described herein.

[0015] [Figure 4C] 1 illustrates how pixel units associated with identified object regions may be discounted as part of determining frame auto-exposure data points for an exemplary image frame in accordance with the principles described herein.

[0016] [Figure 5] 1 shows an exemplary flow diagram for managing the auto-exposure of an image frame according to principles described herein.

[0017] [Figure 6] 1 shows an example flow diagram for identifying object regions in an example image frame according to principles described herein.

[0018] [Figure 7A] 1 illustrates exemplary geometric features that may be analyzed in color space to facilitate object region identification within an exemplary image frame in accordance with principles described herein. [Figure 7B] 1 illustrates exemplary geometric features that may be analyzed in color space to facilitate object region identification within an exemplary image frame in accordance with principles described herein.

[0019] [Figure 8] 1 illustrates an exemplary range of weight values ​​that may be assigned to a pixel unit to indicate the confidence level that the pixel unit corresponds to a representation of an object depicted in an image frame in accordance with the principles described herein.

[0020] [Figure 9] 1 shows an exemplary flow diagram for determining a frame auto-exposure value and a frame auto-exposure target based on weighted pixel units according to principles described herein.

[0021] [Figure 10] 1 illustrates an exemplary technique for updating auto-exposure parameters in accordance with principles described herein.

[0022] [Figure 11] 1 illustrates an exemplary computer-assisted medical system according to principles described herein.

[0023] [Figure 12] 1 illustrates an exemplary computing system in accordance with principles described herein. DETAILED DESCRIPTION OF THE INVENTION

[0024] Apparatuses, systems, and methods for managing auto-exposure of image frames are described herein. As mentioned above, auto-exposure management of image frames depicting certain types of objects may be associated with unique challenges. For example, if one particular object has a different luminance than other content depicted in the image frame (e.g., if the object is significantly darker or brighter than the other content), the object may significantly affect the average auto-exposure value or auto-exposure target for the image frame (e.g., significantly raising or lowering the average). The larger the object relative to the image frame, the more pronounced this effect may be. If the object is a subjectively important part of what the image frame depicts (e.g., something that a viewer of the image frame is likely to want to look at in more detail), it may be desirable for the object to influence auto-exposure management in this way, and conventional auto-exposure algorithms may work well. However, if the object is an extraneous object that is necessarily depicted in the scene but where a viewer is unlikely to want to focus on or view other content in detail (e.g., an object that is separate and distinct from the scene's environmental imagery), the object's effect on the average auto-exposure value and / or target of the image frame may be undesirable. For example, when auto-exposure management attempts to provide proper exposure for the object, it may compromise the exposure of other content depicted in the image frame (e.g., the scene's environmental imagery that a user may want to view in more detail). Specifically, other objects and / or scene content depicted in the image frame may be at least somewhat over- or under-exposed due to the extraneous object's undesirable effect on the image frame's auto-exposure characteristics.

[0025] As an example of where this type of problem arises, consider an endoscopic image capture device that captures an internal view of a body during a medical procedure (e.g., a surgical procedure) on the body. In this situation, a viewer of the endoscopic image (e.g., a surgeon or other person assisting the medical procedure) may desire to see details of anatomical structures (e.g., tissue) present in the internal view. However, one or more other objects necessarily present in the scene may meet one or more of the following criteria for a foreign object: (1) significantly different in appearance from other images in the scene (e.g., significantly darker or lighter than the environmental image of the scene) or (2) unlikely to be a significant focal area for the viewer of the image. For example, image frames captured by an endoscope in a computer-aided medical system (e.g., a single-port computer-aided medical system) may depict one or more foreign objects used to accomplish the medical procedure along with images of the body's internal anatomical structures. One example of a foreign object may be the dark shaft of an instrument used to manipulate tissue as part of the medical procedure. For example, the shaft of an instrument may be covered by a dark sheath and may be visible in the image frame when the instrument is used to perform tissue manipulation actions during a procedure. Similarly, objects associated with the ultrasound probe, the head of the instrument, tools supported by the instrument, and / or other instruments may fit the criteria for foreign objects that are likely to undesirably affect automatic exposure management in an endoscopic scene. Another example of a foreign object may be brightly colored (e.g., white) gauze or other such material used as part of a medical procedure (e.g., mesh material for treating hernias, etc.).

[0026] A viewer of an image frame captured in this scenario (e.g., medical staff assisting in the performance of a medical procedure) may wish to see details of anatomical content (e.g., body tissue, blood, etc.) rather than details of large and / or dark instrument shafts or other such foreign objects that are likely to be present in the captured image. Accordingly, the automatic exposure management apparatus, systems, and methods described herein may operate to identify regions of an image frame that are likely to correspond to (depict or constitute part of) depictions of foreign objects such as instrument shafts, so that these object regions can be discounted (e.g., ignored or downplayed) as a factor underlying the automatic exposure management. In this way, foreign objects may be less likely to undesirably raise or lower the average brightness of the scene, thereby mitigating or solving the problems discussed above with respect to undesirable over- or under-exposure of tissue content. Furthermore, the automatic exposure management described herein may help stabilize the auto-exposure characteristics (e.g., average brightness, etc.) of an image frame sequence that may otherwise vary widely as fixtures and other extraneous objects enter and exit the frame, thereby causing flickering and inconsistent auto-exposure of the scene. Thus, the automatic exposure management described herein may mitigate or resolve the problem of brightness fluctuations caused by moving fixtures and other related problems (e.g., viewer distraction, viewer-induced eye strain, etc.).

[0027] Examples of medical procedures involving endoscopic views in which instruments and other foreign objects are depicted along with body tissues and anatomical structures are referenced throughout this specification to illustrate various aspects of the claimed subject matter. However, it will be understood that such endoscopic images are intended as examples only, and that the principles described herein may be applied, in various implementations, to any suitable type of content that may be useful for a particular application or use case. As some additional examples, for example, the automatic exposure management described herein may find use in photography applications in which objects are likely to be present in the scene, such as dark or brightly colored tripods holding cameras, boom microphones, mispositioned thumbs or fingers blocking portions of the camera lens, and / or other such foreign objects that are likely to undesirably affect the automatic exposure management of image frame sequences.

[0028] Various specific embodiments will now be described in detail with reference to the drawings. It will be understood that the specific embodiments described below are provided as non-limiting examples of how various novel and inventive principles may be applied in various contexts. In addition, it will be understood that other examples not expressly described herein may be captured by the scope of the claims set forth below. The automatic exposure management apparatus, systems, and methods described herein may provide any of the advantages described above, as well as various additional and / or alternative advantages that will be described and / or made apparent below.

[0029] 1 illustrates an exemplary automatic exposure management apparatus 100 (apparatus 100) for managing the automatic exposure of image frames in accordance with the principles described herein. Apparatus 100 may be implemented by computer resources (e.g., a server, processor, memory device, storage device, etc.) included within an image capture system (e.g., an endoscopic image capture system, etc.), by computer resources of a computing system associated with the image capture system (e.g., communicatively coupled to the image capture system), and / or by any other suitable computing resources that may be useful in a particular implementation.

[0030] As shown, device 100 may include, but is not limited to, memory 102 and processor 104 selectively and communicatively coupled to each other. Memory 102 and processor 104 may each include or be implemented by computer hardware configured to store and / or process computer software. Various other components of computer hardware and / or software not explicitly shown in FIG. 1 may also be included within device 100. In some examples, memory 102 and processor 104 may be distributed among multiple devices and / or multiple locations as may be useful for a particular implementation.

[0031] The memory 102 may store and / or otherwise maintain executable data used by the processor 104 to perform any of the functionality described herein. For example, the memory 102 may store instructions 106 that may be executed by the processor 104. The memory 102 may be implemented by one or more memory or storage devices, including any memory or storage device described herein configured to temporarily or non-temporarily store data. The instructions 106 may be executed by the processor 104 to cause the apparatus 100 to perform any of the functionality described herein. The instructions 106 may be implemented by any suitable application, software, code, and / or other executable data instance. In addition, the memory 102 may maintain any other data that is accessed, managed, used, and / or transmitted by the processor 104 in a particular implementation.

[0032] Processor 104 may be implemented by one or more computer processing devices, including general-purpose processors (e.g., central processing units (CPUs), graphics processing units (GPUs), microprocessors, etc.), special-purpose processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), image signal processors, or the like. Using processor 104 (e.g., when processor 104 is directed to perform operations represented by instructions 106 stored in memory 102), apparatus 100 may perform various functions related to managing the auto-exposure of image frames depicting objects (e.g., foreign objects) whose effect on the auto-exposure management of the image frames is to be discounted (e.g., objects such as dark instrument shafts present in a scene inside a body during an endoscopic medical procedure).

[0033] FIG. 2 illustrates an exemplary automatic exposure management method 200 (method 200) that apparatus 100 may perform to manage the automatic exposure of image frames in accordance with the principles described herein. While FIG. 2 illustrates exemplary operations according to one embodiment, other embodiments may omit, add, reorder, and / or modify any of the operations shown in FIG. 2. In some examples, multiple operations shown in or described in connection with FIG. 2 may be performed simultaneously with one another (e.g., in parallel) rather than sequentially as shown and / or described. One or more operations illustrated in FIG. 2 may be performed by an automatic exposure management apparatus (e.g., apparatus 100), an automatic exposure management system (e.g., an implementation of an automatic exposure management system described below), and / or any implementation thereof.

[0034] At operation 202, apparatus 100 may identify an object region within an image frame captured by the image capture system. The object region may correspond to a depiction of an object depicted in the image frame. For example, the object to which the object region corresponds may be a foreign object, such as any of those described above (e.g., a low-brightness instrument shaft or ultrasound probe, a high-brightness gauze or mesh material, etc.), or another object that meets some or all of the criteria described above for a foreign object (e.g., being a relatively large object, being unlikely to be of interest to the viewer, having auto-exposure characteristics that are significantly different from other depicted content, etc.). As described in more detail below, apparatus 100 may identify the object region based on local features associated with pixel units (e.g., individual pixels or groups of pixels) included in the image frame, based on global features associated with the image frame, and / or based on any other factors that may be useful in a particular implementation.

[0035] At operation 204, apparatus 100 may determine one or more frame auto-exposure data points for the image frame by discounting object regions within the image frame. For example, by discounting (e.g., completely ignoring or otherwise downplaying) pixel units identified as being within object regions, apparatus 100 may determine an auto-exposure value for the image frame (frame auto-exposure value), an auto-exposure target for the image frame (frame auto-exposure target), and / or any other frame auto-exposure data points that may be useful in a particular implementation.

[0036] An auto-exposure value will be understood to represent a particular auto-exposure-related characteristic (e.g., luminance, signal strength, chrominance, etc.) of a particular image frame or portion thereof (e.g., region, pixel unit, etc.). For example, device 100 may detect such characteristics by analyzing image frames captured by an image capture system. A unit auto-exposure value may refer to a luminance determined for a pixel unit. For example, a unit auto-exposure value may be determined as the luminance of an individual pixel, or as the average luminance of a group of pixels in an implementation in which pixels are grouped into pixel cells in a grid, or the like. As another example, a frame auto-exposure value may refer to the average luminance of some or all of the pixel units included in an image frame, such that a frame auto-exposure value corresponds to an image frame in a similar way that a frame auto-exposure value corresponds to a particular pixel unit.

[0037] In these examples, it will be understood that the average luminance (and / or one or more other average exposure-related features in particular examples) referred to by the auto-exposure value may be determined as any type of average that may be useful for a particular implementation. For example, the average auto-exposure value for an image frame may refer to the average luminance of the pixel units in the image frame, determined by summing the respective luminance values ​​for each pixel unit in the image frame and then dividing the sum by the total number of values. As another example, the average auto-exposure value for an image frame may refer to the median luminance of the pixel units in the image frame, determined as the central luminance value when the respective luminance values ​​for each pixel unit are ordered by value. As yet another example, the average auto-exposure value for an image frame may refer to the mode luminance of the pixel units in the image frame, determined as whichever of the respective luminance values ​​for each pixel unit is most relevant or most frequently repeated. In other examples, other types of averages (other than mean, median, or mode) and / or other types of exposure-related features (other than luminance) may be used to determine the auto-exposure value in any manner that may be useful for a particular implementation.

[0038] An auto-exposure target will be understood to refer to a target (e.g., a goal, desired value, ideal value, optimal value, etc.) for the auto-exposure value of a particular image frame or a portion thereof (e.g., a region, a pixel unit, etc.). Apparatus 100 may determine the auto-exposure target based on a particular environment and any appropriate criteria, and the auto-exposure target may be associated with the same auto-exposure-related characteristic (e.g., luminance, signal strength, chrominance, etc.) as represented by the auto-exposure value. For example, the auto-exposure target may be determined at a desired level of luminance (or other exposure-related characteristic), such as a luminance level associated with a neutral gray color or an equivalent color. Thus, a unit auto-exposure target may refer to a desired target luminance determined for a pixel unit (e.g., a desired target luminance for an individual pixel or an average desired target luminance determined for a group of pixels in an implementation in which pixels are grouped into pixel cells in a grid). As another example, a frame auto-exposure target may refer to an average desired target luminance for some or all of the pixel units included in an image frame, and thus may represent an auto-exposure target that corresponds to an image frame in a manner similar to how a unit auto-exposure target corresponds to a particular pixel unit. Similar to what was described above regarding how the frame auto-exposure value may be determined, the frame auto-exposure target in such an example may be determined by averaging the individual unit auto-exposure targets using a mean, median, mode, or other suitable type of averaging technique.

[0039] The determination of frame auto-exposure data points, such as frame auto-exposure values ​​and / or frame auto-exposure targets, in act 204 may discount object regions identified in act 202 in any manner that may be useful for a particular implementation. For example, in implementations that utilize a weighted average of unit auto-exposure data points to determine frame auto-exposure data points, as described in more detail below, the weight values ​​assigned to pixel units that correspond to object regions may be set lower than the weight values ​​of pixel units that do not correspond to object regions, or may be zeroed out entirely. In these ways, different implementations may be configured to discount object regions by completely ignoring object regions for purposes of auto-exposure management (e.g., completely eliminating the influence of object regions on auto-exposure management), by reducing the influence of object regions to a more limited extent (e.g., downplaying, but not completely eliminating, the influence of object regions on auto-exposure management), or by doing both according to a confidence level associated with each pixel unit.

[0040] At operation 206, apparatus 100 may update (e.g., adjust or maintain) one or more auto-exposure parameters for use by the image capture system to capture one or more additional image frames. In some examples, apparatus 100 may update one or more auto-exposure parameters based on auto-exposure values, auto-exposure targets, and / or other auto-exposure data points for pixels of the image frame as those data points are determined (e.g., at operation 204). For example, assuming apparatus 100 has determined a frame auto-exposure value and / or a frame auto-exposure target, apparatus 100 may update one or more auto-exposure parameters at operation 206 based on the frame auto-exposure value and / or the frame auto-exposure target. For example, apparatus 100 may determine an auto-exposure gain for the image frame (frame auto-exposure gain) based on the frame auto-exposure value and the frame auto-exposure target, and may perform updating of one or more auto-exposure parameters based on the frame auto-exposure gain.

[0041] Apparatus 100 may update the auto-exposure parameters by either adjusting the parameters appropriately or maintaining the parameters based on the auto-exposure gain. In this manner, the image capture system may capture one or more additional image frames (e.g., subsequent image frames in the image frame sequence being captured) using auto-exposure parameters (e.g., exposure time parameters, shutter aperture parameters, illumination intensity parameters, image signal analog and / or digital gain, etc.) that may reduce the difference between the detected auto-exposure values ​​for those additional image frames and the desired auto-exposure target for those additional image frames. Thus, the additional image frames may be captured with more desirable exposure characteristics than those that would have been captured without such adjustments, and a user of apparatus 100 may experience a better image (e.g., an image showing content details other than extraneous objects at a desired brightness level, etc.).

[0042] Apparatus 100 may be implemented by one or more computing devices or computing resources of a general-purpose or special-purpose computing system, as described in more detail below. In particular embodiments, one or more computing devices or computing resources implementing apparatus 100 may be communicatively coupled to other components, such as an image capture system used to capture image frames that apparatus 100 is configured to process. In other embodiments, apparatus 100 may be included within (e.g., implemented as part of) an automatic exposure management system. Such an automatic exposure management system may be configured to perform all of the same functions described herein as performed by apparatus 100 (including, for example, the operations of method 200 described above), but may further incorporate additional components, such as an image capture system, such that it may perform functionality associated with those additional components.

[0043] 3 illustrates an exemplary automatic exposure management system 300 (system 300) for managing the automatic exposure of image frames. As shown, system 300 may include an implementation of apparatus 100 along with an image capture system 302 including an illumination source 304 and an image capture device 306 incorporating a shutter 308, an image sensor 310, and a processor 312 (e.g., one or more image signal processors implementing an image signal processing pipeline). Within system 300, apparatus 100 and image capture system 302 may be communicatively coupled to enable apparatus 100 to instruct image capture system 302 in accordance with operations described herein and to enable image capture system 302 to capture and provide image frame sequences 314 and / or other suitable captured image data to apparatus 100. Each of the components of image capture system 302 is described in more detail below.

[0044] As noted above, the principles described herein may apply to a wide variety of imaging scenarios, although many of the examples explicitly described herein relate to medical procedures that may be implemented using a computer-assisted medical system, such as that described in further detail below in connection with FIG. 11 . In such examples, the scene from which the image is captured may include an internal view of a body on which the medical procedure is performed (e.g., a living animal body, a human or animal cadaver, a portion of a human or animal anatomy, tissue removed from a human or animal anatomy, a non-tissue workpiece, a training model, etc.). Thus, system 300 or certain components thereof (e.g., image capture system 302) may be integrated with a computer-assisted medical system (e.g., implemented by the imaging and computational resources of the computer-assisted medical system), and objects to be discounted in the automatic exposure management may include objects associated with the computer-assisted medical system or the medical procedure (e.g., equipment included in the computer-assisted medical system, gauze or mesh material used for the medical procedure, etc.).

[0045] In certain such examples, device 100 may be configured to identify object regions based on differences in color of foreign objects and tissues characterized in an internal view of the body. For example, device 100 may determine, for environmental imagery of a scene depicted in an image frame (e.g., for images of elements of the scene other than foreign objects, such as instruments, gauze, or other objects foreign to the internal body), a color gamut that encompasses a range of red colors corresponding to blood and tissue visible in the internal view of the body. Then, based on one or more chrominance features of pixel units included in the image frame, device 100 may identify object regions within the image frame that correspond to depictions of objects depicted in the image frame by performing operations including, for example, determining whether the chrominance features of the pixel units are within the color gamut for the environmental imagery of the scene.

[0046] Illumination source 304 may be implemented to provide any type of illumination (e.g., visible light, infrared or near-infrared light, fluorescent excitation light, etc.) and may be configured to interact with image capture device 306 in image capture system 302. For example, illumination source 304 may provide a particular amount of illumination to a scene to facilitate image capture device 306 in capturing an optimally illuminated image of the scene.

[0047] Image capture device 306 may be implemented by any suitable camera or other device configured to capture images of a scene. For example, in the medical procedure example, image capture device 306 may be implemented by an endoscopic image capture device configured to capture a sequence of image frames 314, which may include image frames depicting a view (e.g., an internal view) of a body undergoing the medical procedure. As shown, image capture device 306 may include components such as a shutter 308, an image sensor 310, and a processor 312.

[0048] Image sensor 310 may be implemented by any suitable image sensor, such as a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or the like.

[0049] The shutter 308 may interact with the image sensor 310 to assist in capturing and detecting light from a scene. For example, the shutter 308 may be configured to expose the image sensor 310 to a particular amount of light for each image frame captured. The shutter 308 may include an electronic shutter and / or a mechanical shutter. The shutter 308 may control how much light the image sensor 310 is exposed to by opening to a particular aperture size defined by a shutter aperture parameter and / or for a particular time period defined by an exposure time parameter. As described in more detail below, these shutter-related parameters may be included among the auto-exposure parameters that the apparatus 100 is configured to update.

[0050] The processor 312 may be implemented by one or more image signal processors configured to implement at least a portion of an image signal processing pipeline. The processor 312 may process auto-exposure statistics input (e.g., by tapping the signal midway through the pipeline to detect and process various auto-exposure data points and / or other statistics), perform optical artifact correction on data captured by the image sensor 310 (e.g., by reducing fixed pattern noise, correcting defective pixels, correcting lens shading issues, etc.), perform signal reconstruction operations (e.g., white balance operations, demosaic and color correction operations, etc.), apply image signal analog and / or digital gains, and / or perform any other functions that may be useful in a particular implementation. Various auto-exposure parameters may dictate how the functionality of the processor 312 should be performed. For example, as described in more detail below, the auto-exposure parameters may be set to determine the analog and / or digital gains that the processor 312 applies.

[0051] In some examples, an endoscopic implementation of image capture device 306 may include a stereoscopic endoscope that includes two full sets of image capture components (e.g., two shutters 308, two image sensors 310, etc.) to accommodate stereoscopic differences presented to the two eyes (e.g., left and right eyes) of an observer of the captured image frames. Conversely, in other examples, an endoscopic implementation of image capture device 306 may include a monoscopic endoscope with a single shutter 308, a single image sensor 310, etc.

[0052] Apparatus 100 may be configured to control various auto-exposure parameters of image capture system 302 and may adjust such auto-exposure parameters in real time based on incoming image data captured by image capture system 302. As described above, certain auto-exposure parameters of image capture system 302 may be associated with shutter 308 and / or image sensor 310. For example, apparatus 100 may direct shutter 308 according to an exposure time parameter corresponding to how long the shutter should allow image sensor 310 to be exposed to the scene, a shutter aperture parameter corresponding to the aperture size of shutter 308, or any other suitable auto-exposure parameter associated with shutter 308. Other auto-exposure parameters may be associated with aspects of the image capture process unrelated to image capture system 302 or shutter 308 and / or sensor 310. For example, the apparatus 100 may adjust an illumination intensity parameter of the illumination source 304 corresponding to the intensity of the illumination provided by the illumination source 304, an illumination duration parameter corresponding to the time period during which illumination is provided by the illumination source 304, or an equivalent parameter. As yet another example, the apparatus 100 may adjust a gain parameter corresponding to one or more analog and / or digital gains (e.g., analog gain, Bayer gain, RGB gain, etc.) applied by the processor 312 to the image data (e.g., luminance data) generated by the image sensor 310.

[0053] Any of these or other suitable parameters, or any combination thereof, may be updated and / or otherwise adjusted by apparatus 100 for subsequent image frames based on an analysis of the current image frame. For example, in one example where the frame auto-exposure gain (e.g., frame auto-exposure target divided by frame auto-exposure value) is determined to be 6.0, various auto-exposure parameters may be set as follows: 1) current illumination intensity parameter may be set to 100% (e.g., maximum output), 2) exposure time parameter may be set to 1 / 60 seconds (e.g., 60 fps), 3) analog gain may be set to 5.0 (with a cap of 10.0), 4) Bayer gain may be set to 1.0 (with a cap of 3.0), and 5) RGB gain may be set to 2.0 (with a cap of 2.0). With these settings, the gain is distributed across analog gain (10.0 / 5.0=2.0), Bayer gain (3.0 / 1.0=3.0), and RGB gain (2.0 / 2.0=1.0) to establish a desired total auto exposure gain of 6.0 for the frame (3.0 x 2.0 x 1.0=6.0).

[0054] 4A-4C illustrate various aspects of an exemplary image frame 402 depicting an exemplary object whose effect on automatic exposure management of an image frame sequence may be discounted by apparatus 100. For example, image frame 402 may be an image frame captured by system 300 and included as one of the image frames of image frame sequence 314. Image frame 402 is shown to depict various objects 404 in front of a background that may be of interest to a viewer of image frame 402, as well as a particular object 406 that may be a foreign object to be discounted (e.g., because it is unlikely to be of interest to the viewer, has a significantly different luminance from object 404 and the background, etc.). For example, in a medical procedure example, each of objects 404 may represent an anatomical object featured on an anatomical background (e.g., a background featuring blood, tissue, etc. that is similar in luminance to the anatomical object), while object 406 may represent an instrument shaft, a piece of gauze, or other such foreign object that may be desirable for automatic exposure management to discount. 4A and 4B and described in many examples herein, it will be understood that multiple foreign objects may be discounted from the image frame by discounting each object in the same manner as described herein for object 406 and / or other foreign objects described herein. For example, image frame 402 may depict multiple instrument shafts (e.g., from two different instruments, three different instruments, etc.), may depict an instrument shaft and a piece of gauze, etc.

[0055] 4A illustrates how local features may be utilized to identify object regions corresponding to depictions of object 406 depicted in image frame 402. For example, local features may include chrominance features (e.g., color characteristics) or luminance features (e.g., brightness characteristics) associated with individual pixel units contained within image frame 402, as well as corresponding characteristics expected for other content of image frame 402 and / or object 406.

[0056] While FIG. 4A is depicted in black and white, the various Arabic numerals 1-9 used to fill the various objects and background regions depicted in image frame 402 will be understood to represent different colors, and alternatively, different brightness values. In this Arabic numeral-based chrominance / luminance notation, Arabic numerals that are close to each other (e.g., 1 and 2, 8 and 9, etc.) will be understood to represent similar colors (e.g., red and red-orange, green and green-yellow, etc.) or similar brightness levels, while Arabic numerals that are farther apart from each other (e.g., 1 and 8, 2 and 9, etc.) will be understood to represent more distinct colors (e.g., red and green, blue and orange, etc.) or different brightness levels. In some instances, this Arabic numeral-based notation may be interpreted as enveloping, such that Arabic numeral 1 is considered adjacent to Arabic numeral 9 and the color or brightness represented by Arabic numeral 1 is similar to the color or brightness represented by Arabic numeral 9.

[0057] 4A in terms of chrominance features, the color of object 406 (shown to be the color indicated by Arabic numeral 9) may differ significantly from the color of object 404 (shown to be a similar color within the range of Arabic numerals 3-5) and the background. One local feature used to identify an object region corresponding to a depiction of object 406 may be whether the color of object 406 is close to a predetermined color that might be expected for known foreign objects (e.g., a neutral color such as metallic gray or black for an instrument shaft or sheath that is commonly expected to be present in a medical procedure scene). If Arabic numeral 9 represents a neutral color (e.g., black, gray, etc.), while Arabic numerals 3-5 represent different shades of red (e.g., shades commonly associated with tissue, blood, and / or blood-like tissue present in an internal scene within the body), then device 100 may identify the object region based, at least in part, on how similar the color of object 406 is to the expected neutral color of the instrument.

[0058] As another example of local features used to identify object regions, device 100 may determine whether the color of object 406 differs significantly from the average color of the scene (e.g., the average color of the entire scene including the object, the average color of the scene's environmental image, etc.) or from the expected color for the scene's environmental image to be accounted for in automatic exposure management (e.g., the expected color for tissue and anatomical objects, such as the red associated with tissue or blood). For example, if it is again assumed that the numeral 9 represents a neutral color while the numerals 3 through 5 represent reddish hues, device 100 may identify object regions based, at least in part, on how the color of object 406 differs from the expected red of object 404 and the background. If the numeral-based notation is interpreted to represent luminance features rather than chrominance features, a similar estimation may be made with respect to luminance to further facilitate object region identification.

[0059] 4B illustrates how global features may be used to identify an object region corresponding to a depiction of object 406 depicted in image frame 402. For example, band features may be associated with object tracking performed on object 406. Computer vision techniques, kinematic tracking techniques (e.g., in examples where object 406 is a robotically controlled tool, etc.), and / or other methods of tracking an object within a sequence of image frames may be used by device 100 or by a system independent of device 100. In either case, device 100 may obtain data generated based on the object tracking to identify the location of object 406. For example, the object tracking data may represent a bounding box 408 or the like that indicates where object tracking determines that object 406 is currently located.

[0060] 4A and 4B, the apparatus 100 may successfully and efficiently identify an object region corresponding to the depiction of the object 406 in the image frame 402. Analysis including such local and / or band features is described in further detail below. Once an object region has been identified, the apparatus 100 may be configured to discount the identified object region in determining frame auto-exposure data points (e.g., frame auto-exposure values ​​and / or frame auto-exposure targets used as a basis for auto-exposure management of subsequent image frames).

[0061] For illustrative purposes, FIG. 4C shows how pixel units associated with identified object region 410 may be discounted as part of determining frame auto-exposure data points for image frame 402. As discussed above, a pixel unit may refer to either an individual pixel or a group of pixels, as may be useful for a particular implementation or image frame. For example, an image frame such as image frame 402 may be divided into grid cells, each cell may include one or more individual pixels, such that each pixel unit references a grid cell with its respective pixel. In one case, each cell of the grid into which the image frame is divided may include only one individual pixel, resulting in the aforementioned pixel unit having only individual pixels. In other cases, each cell of the grid may include several groups of pixels (e.g., 4 pixels, 16 pixels, 512 pixels, etc.), resulting in the aforementioned pixel unit including a group of pixels.

[0062] As shown in Figure 4C, the image frame 402 includes a grid of many small squares, each representing a pixel unit (e.g., an individual pixel or group of pixels) of the image frame 402. Each pixel unit in Figure 4C is shaded according to a weight value assigned to the pixel unit, as described in more detail below. For example, a pixel unit without any shading (e.g., a majority of the pixel unit that is still white) may be understood to have a weight value assigned that allows the pixel unit to be fully accounted for by the automatic exposure management, while a pixel unit that includes at least some shading (e.g., a pixel unit within the object region 410) may be understood to have a weight value assigned that allows the pixel unit to be discounted by the automatic exposure management.

[0063] 4C may represent different weight values, which may correspond to the confidence level with which device 100 should identify pixels within object region 410, as described in more detail below. For example, as shown, lighter shading may be assigned to peripheral pixel units around object 406 because these pixel units may partially represent object 406 and partially represent other images (e.g., background) and / or because it may be difficult for device 100 to determine with 100% confidence whether a given pixel unit is associated with object 406 based on local and global features and / or other factors that may be used. As a result of the different weight values, device 100 may discount different portions of object region 410 to different extents when performing auto-exposure management (e.g., when determining frame auto-exposure data points and when updating auto-exposure parameters based on the frame auto-exposure data points). For example, for interior portions of the object area 410 that are shaded black, the device 100 may completely ignore these pixel units in the automatic exposure management, whereas for peripheral portions of the object area 410 that are cross-hatched (shaded with fine parallel lines) or shaded with dots, the device 100 may discount the influence of these pixel units to a different extent (e.g., to a greater extent for pixel units shaded with cross-hatching and to a lesser extent for pixel units shaded with dots).

[0064] 5 shows an example flow diagram 500 for managing the auto-exposure of image frames, for example, using an implementation of apparatus 100, method 200, and / or system 300. As shown, flow diagram 500 illustrates various operations 502-512, each of which is described in more detail below. It will be understood that operations 502-512 represent one embodiment, and that other embodiments may omit, add, reorder, and / or modify any of these operations. As described, the various operations 502-512 of flow diagram 500 may be performed on one image frame or multiple image frames (e.g., each image frame) in an image frame sequence. It will be understood that, depending on various conditions, not every operation may be performed on every frame, and the combination and / or order of operations performed among frames in an image frame sequence may vary.

[0065] In operation 502, an image frame captured by an image capture system may be obtained (e.g., accessed, loaded, captured, generated, etc.). As previously described, in particular examples, the image frame may be an image frame depicting one or more objects, including an extraneous object, whose effect on the automatic exposure management of the image frame sequence is to be discounted. For example, the obtained image frame may be similar to image frame 402 described above, and the extraneous object may be object 406. Operation 502 may be performed in any suitable manner, such as by accessing the image frame from an image capture system (e.g., if operation 502 is performed by an implementation of device 100 that is communicatively coupled to an image capture system) or by using the integrated image capture system to capture the image frame (e.g., if operation 502 is performed by an implementation of system 300 that includes integrated image capture system 302).

[0066] At operation 504, apparatus 100 may identify an object region within the image frame acquired at operation 502 based on any suitable factors that may be useful in a particular implementation. For example, the identification of the object region at operation 504 may be based on one or more local features associated with pixel units included in the image frame, based on one or more global features associated with the image frame, or based on a combination of both (e.g., a combination of local features associated with pixel units included in the image frame and global features associated with the image frame). To this end, as shown, operation 504 may include one or both of operation 506 in which apparatus 100 analyzes local features of pixel units of the image frame and operation 508 in which apparatus 100 analyzes global features of the image frame. As shown in FIG. 5, FIGS. 6-8 further illustrate various aspects of how object regions may be identified at operation 504.

[0067] 6 illustrates an example flow diagram 600 for identifying an object region within an example image frame, such as the image frame acquired in operation 502. As shown, flow diagram 600 includes multiple operations 602-616 that may occur between when apparatus 100 begins performing operation 504 (marked start) and when flow diagram 600 is completed (marked end) and the object region has been identified.

[0068] In operation 602, the apparatus 100 may iterate over each pixel unit of an image frame or portion of an image frame. For each pixel unit P i For , the pixel units may be analyzed in operation 604 (which may include, for example, performing one or more of operations 606-610) to determine a weight value (W i ) may be assigned in operation 612. Additionally (e.g., before, after, or simultaneously with the execution of operations 604 and 612), each pixel unit P i Unit auto exposure value (V i ) and / or Unit Auto Exposure Target (T i) may be determined in operation 614. Next, in operation 616, a weighted pixel unit is calculated for each pixel unit P based on the results of operations 604-614 performed. i may be determined. As shown, apparatus 100 may continue to process each pixel unit in this manner as long as there are still pixel units of the image frame that have not yet been processed (unfinished), or may terminate when all of the pixel units of the image frame have been iterated through in operation 602 (completed). In particular examples, rather than iterating through all of the pixel units of the image frame, a particular region of the image frame may be considered (e.g., a central region of the image frame, such as the central 50% of the image frame, the central 80% of the image frame, etc.), while another region of the image frame (e.g., a peripheral region of the image frame, such as the outer 50% of the image frame, the outer 20% of the image frame, etc.) may be ignored for purposes of automatic exposure management. In such examples, operation 602 may terminate its iteration (completed) when all pixels in the region to be considered (e.g., the central region) have been iterated through.

[0069] In operation 604, the apparatus 100 selects the current pixel unit P i may be analyzed to determine whether the pixel unit corresponds to a representation of a particular object (e.g., a foreign object) within the image frame, or, in particular implementations, whether the pixel unit corresponds to a confidence level (e.g., on a scale from 0% confidence to 100% confidence or another suitable scale such as a high-medium-low confidence scale) that it corresponds to a representation of a particular object. As shown, to accomplish this analysis in operation 604, apparatus 100 may perform any or all of operations 606-610 or other suitable operations not expressly shown to help achieve the same purpose.

[0070] Operations 606 and 608 are each shown in FIG. 6 as falling into the local category because these operations determine a confidence level for a pixel unit based largely or entirely on one or more local features associated with the pixel unit being analyzed. The local features may include various types of pixel unit features, as described above in connection with FIG. 4A . For example, the one or more local features associated with the pixel unit may include a luminance feature of the pixel unit (e.g., an average brightness of the pixel unit, etc.). As another example, the one or more local features associated with the pixel unit may include a chrominance feature of the pixel unit (e.g., an average color of the pixel unit, etc.). In some implementations, the one or more local features associated with the pixel unit may include both a chrominance feature and a luminance feature of the pixel unit. For example, the confidence that a pixel unit depicts a foreign object may be determined based on a comparison of both a chrominance feature and a luminance feature in a manner described below.

[0071] In operation 606, one or more local features of the pixel unit may be compared to corresponding features associated with the object for which the object region is being identified (e.g., in one particular example, an instrument shaft having low luminance and a neutral metallic color). In such an example, the identification of the object region may be based on a comparison of the luminance features of the pixel unit with the luminance features associated with the object (e.g., to determine whether the pixel unit is similarly dark as expected for an instrument shaft object) and / or a comparison of the chrominance features of the pixel unit with the chrominance features associated with the object (e.g., to determine whether the pixel unit is similarly neutral in color as expected for an instrument shaft object).

[0072] 7A shows exemplary geometric features that may be analyzed in an exemplary color space 700 to help identify object regions within an image frame. As part of operation 606 (or in a separate operation not explicitly shown), apparatus 100 may calculate the current pixel unit P i The color data associated with each pixel unit may be normalized and decomposed to distinguish the chrominance features of the color data from the luminance features of the color data. For example, the normalized color data for each pixel unit may be decomposed from a red-green-blue (RGB) color space (in which the chrominance and luminance features for each pixel are jointly represented by red, green, and blue values) into a different color space that describes the primary, secondary, and / or tertiary colors and separately describes the luminance features. As an example shown in FIG. 7A , the decomposition of the color data may include converting the color data from the RGB color space to a YUV color space. It will be understood that in other implementations, the decomposition of the color data may include converting the color data to a cyan-magenta-yellow-black-red-green-blue (CMYKRGB) color space, a CIELAB color space, or another suitable color space that allows the chrominance features to be conveniently analyzed independently from the luminance features.

[0073] In Figure 7A, several different points 702 (e.g., points 702-1 through 702-4) are plotted in a UV coordinate space associated with a YUV color space 700 to represent the respective colors of several example pixel units (or, more generally, to represent the respective chrominance characteristics of the example pixel units). Although not shown in Figure 7A, it will be understood that for each example pixel unit represented by a point 702 in UV space, a separate luminance value Y may be associated with the pixel unit (e.g., to represent the luminance characteristic of the pixel unit).

[0074] To compare the chrominance features of a pixel unit with those of a particular object (e.g., a foreign object that is discounted in automatic exposure management), apparatus 100 may apply geometric principles within the UV coordinate plane. For example, if the chrominance features of a particular object are represented by points 704 in UV coordinate space, apparatus 100 may calculate the distance (e.g., Euclidean distance) between particular points 702 and 704 to determine an objective and quantitative measure of how similar or dissimilar the color of the pixel unit is to the color of the particular object. For example, because point 702-1 is relatively close to point 704, a comparison of point 702-1 and point 704 may indicate that the chrominance features represented by these points are very similar (as indicated by the relatively close proximity of the points), whereas a comparison of point 702-4 and point 704 may indicate that the chrominance features represented by these points are quite different (as indicated by the relatively large distance between the points).

[0075] In some examples, the object comparison of operation 606 may include determining whether the distance between points exceeds or does not exceed a particular threshold. Such a determination may be used in assigning a weight value for a given pixel unit, as described in more detail below. For illustrative purposes, a threshold 706 is drawn around point 704 having a radius 708. Any point 702 sufficiently close to point 704 that is within the circle of threshold 706 may be considered to exceed or meet threshold 706. For example, the chrominance feature represented by point 702-1 is sufficiently similar to the chrominance feature represented by point 704 to meet threshold 706. Conversely, a point 702 that is sufficiently far from point 704 that is outside the circle of threshold 706 may be considered to not exceed or meet threshold 706. For example, the chrominance features represented by points 702-2 through 702-4 are not sufficiently similar to the chrominance feature represented by point 704, respectively, so as not to meet threshold 706.

[0076] 6 , one or more local features of the pixel unit may be compared to corresponding features associated with scene content other than the object (e.g., an environmental image, such as bright red blood cells, tissue, and / or other scene content present in an interior view of a body along with a dark, neutral colored instrument shaft). Thus, in these examples, identification of the object region may be based on comparing the luminance features of the pixel unit to luminance features associated with the environmental image of the scene (e.g., to determine whether the pixel unit is similarly bright as expected for the blood and tissue depicted in the scene) and / or based on comparing the chrominance features of the pixel unit to chrominance features associated with the environmental image of the scene (e.g., to determine whether the pixel unit is similarly red as expected for the blood and tissue depicted in the scene).

[0077] To further illustrate operation 608, FIG. 7B shows additional example geometric features that may be analyzed within the example color space 700 described above in connection with FIG. 7A. FIG. 7B shows the same point 702 in the UV coordinate space of YUV color space 700. In addition, FIG. 7B shows a color gamut 710 that device 100 may determine to be associated with an environmental image of a scene (e.g., a color gamut encompassing the range of red colors corresponding to blood and tissue visible from an internal view of the body for an example medical procedure). For example, color gamut 710 may be predefined or accessed by device 100 (e.g., loaded from memory as a profile for a particular scenario, such as a surgical procedure), or may be determined based on the average color gamut of the scene or its environmental image as depicted by the image frame or previous image frames analyzed. While color gamut 710 is shown as an irregular shape, it will be understood that color gamut 710 may, in other examples, be implemented as a circle, a polygon, or any other suitable shape that may be useful in a particular implementation.

[0078] To compare the chrominance features of the pixel unit with those of other scene content (e.g., those of an environment image in a scene that includes objects and content other than the extraneous object to be discounted), the device 100 may again apply geometric principles within the UV coordinate plane. For example, if the chrominance features of the environment image of a scene are represented by a color gamut 710 in the UV coordinate space, the device 100 may determine whether a particular point 702 is included within the color gamut 710. For example, because points 702-3 and 702-4 are located within the boundary of the color gamut 710, these points may be determined to be likely to represent the environment image of the scene (e.g., so as not to depict an extraneous object). Conversely, because points 702-1 and 702-2 are located sufficiently outside the boundary of the color gamut 710, these points may be determined to be likely not to represent the environment image of the scene (e.g., so as to be more likely to depict an extraneous object).

[0079] The luminance features for different pixel units may be analyzed and compared to known luminance features for extraneous objects (e.g., instrument shafts) or environmental images (e.g., blood and tissue) present in the scene in a manner similar to that described for chrominance features in connection with Figures 7A and 7B. However, rather than being represented on a two-dimensional coordinate plane, as is the case for chrominance features, luminance features, distances, thresholds, ranges (similar to color gamuts), etc., all may be determined and represented on a one-dimensional number line.

[0080] Returning to FIG. 6 , operation 610 may be performed in addition to or as an alternative to operation 606 and / or operation 608. At operation 610, apparatus 100 may perform object tracking to determine the location of an object (e.g., a foreign object to be discounted) within a scene depicted in an image frame. Operation 610 is shown as falling into the global category because operation 610 is configured to help determine a confidence for each pixel unit based generally or entirely on one or more global features associated with the image frame being analyzed. As discussed above in connection with FIG. 4B , the one or more global features may include object position features determined based on object tracking data received from an object tracking system that tracks the position of a particular object within a scene depicted in the image frame.

[0081] The object tracking system may be implemented by any suitable system and may operate in any manner that may be useful for a particular implementation. For example, in an example where the object to be tracked is an instrument controlled by a robotic arm in a computer-assisted medical system, the object tracking system may be integrated into the computer-assisted medical system (e.g., with an implementation of device 100 and / or system 300) and may track the position of the object based on kinematic data associated with the movements of the robotic arms. The kinematic data may be continuously generated and tracked by the computer-assisted medical system based on sensors that indicate the movements each robotic arm is commanded to make and how the robotic arms are positioned. Thus, such data may be translated to indicate, for example, where an instrument controlled by one robotic arm is positioned in space relative to an imaging device (e.g., an endoscope) controlled by another robotic arm (or, in certain implementations, by the same robotic arm).

[0082] In the same or other examples, the object tracking system may track the location of the object based on computer vision techniques applied to the image frames of the image frame sequence that includes the image frame. For example, object recognition techniques (including, for example, techniques that utilize machine learning or other types of artificial intelligence) performed to recognize instruments in a medical procedure or to recognize other types of objects in another context may also be used to help track where the object is located in the scene.

[0083] Object tracking data determined using kinematics, computer vision, or other suitable techniques may be used instead of, or in addition to, data derived from the locality-based techniques described above as a basis for apparatus 100 to determine a confidence level for whether each pixel unit corresponds to a representation of an object in an image frame. Thus, the object tracking data may be determined and represented in any suitable manner. As one example, the object tracking system may output coordinates of a bounding box surrounding a representation of an object in an image frame (e.g., such as bounding box 408 in FIG. 4B ). As another example, the object tracking system may output a semantic segmentation map of various objects depicted in a scene (e.g., instruments, anatomical objects, etc.), which includes semantic segmentation data for objects to be discounted in automatic exposure management.

[0084] In operation 612, each of a plurality of pixels of the image frame or region thereof (e.g., each pixel unit P i ) are calculated based on the pixel unit analysis performed in operation 604 and / or other suitable weighting factors (e.g., the spatial location of the pixel unit within the image frame, etc.). i) may be assigned a weight value. Each weight value assigned in operation 612 may indicate a respective confidence level (e.g., the confidence level that each particular pixel unit corresponds or does not correspond to a representation of a foreign object) that the pixel unit is included in the representation of a foreign object, as indicated by any local-based pixel comparison operation, such as operations 606 and 608, by any global-based object tracking operation, such as operation 610, or by any other confidence analysis, such as may be performed as part of operation 604 in a particular implementation. For example, the probability that a pixel unit depicts a foreign object may be estimated using the Bayer formula or by taking into account local and global features in another suitable manner.

[0085] In some examples, the weight value may be determined in a manner that takes into account how likely each pixel unit is to be within the viewer's focal area of ​​the image frame, and thus how relatively important each pixel unit is considered to be relative to other pixel units in the image frame. For example, in certain implementations, it may be assumed that the viewer is likely to focus their attention near the center of the image frame, and thus the weight value assigned to each pixel unit may be based, at least in part, on the pixel unit's proximity to the center of the image frame (e.g., a higher weight value indicates closer proximity to the center, and a lower weight value indicates a farther distance from the center). As another example, an implementation may include an eye tracking configuration for determining in real time which portion of the image frame the viewer is focusing on, and the weight value assigned to each pixel unit may be based, at least in part, on the pixel unit's proximity to a detected real-time focal area (e.g., rather than or in addition to the center of the image frame). In still other examples, the weight value assigned to a pixel unit may be influenced by other spatial position-based criteria (e.g., proximity to other assumed focal areas in the image frame other than the center) or non-spatial position-based criteria. Alternatively, each pixel unit may be treated as equally important regardless of its spatial location in a particular instance, such that the weight value is based entirely on a reliability analysis rather than on the spatial location of the pixel unit.

[0086] 8 illustrates an exemplary range of weight values ​​that may be assigned to a pixel unit in operation 612 to indicate a confidence level that the pixel unit is included in the depiction of a foreign object within an image frame. Specifically, weight value 802 is shown as sliding on a confidence scale from 100% confidence to 0% confidence based on the pixel unit analysis performed in operation 604. It will be understood that weight value 802 represents only the confidence-based aspect of the overall weight value that may be assigned to a particular pixel unit, and that one or more other aspects (e.g., spatial location-based aspects as described above) may also be considered in assigning an overall weight value to a particular pixel unit.

[0087] If the analysis performed in operation 604 (e.g., an analysis associated with any of operations 606-610) yields a very high level of confidence, the weight value 802 may exceed an upper threshold 804 and be assigned a first weight value. For example, the first weight value may be a minimum weight value (e.g., 0%), referred to herein as a null weight value. The first weight value may cause the automatic exposure management to discount (e.g., completely ignore) this pixel unit due to a high level of confidence that the pixel unit depicts an object that should be discounted. Conversely, if the analysis performed in operation 604 yields a very low level of confidence, the weight value 802 may exceed a lower threshold 806 and be assigned a second weight value. For example, the second weight value may be a maximum weight value (e.g., 100%), referred to herein as a full weight value. The second weight value may cause the automatic exposure management to assign significant or full weight to this pixel unit due to a high level of confidence that the pixel unit does not depict an object that should be discounted. If the confidence level determined by operation 604 is between these thresholds (e.g., not very high or very low), weight value 802 may be assigned an action weight value between the first and second weight values ​​(e.g., greater than 0% and less than 100%). For example, an action weight value may cause a pixel unit to account to a limited extent for a pixel unit due to the possibility that the pixel unit partially depicts an object that should be discounted (e.g., depicting part of the edge of the object, etc.), or due to a lack of certainty about whether the pixel unit depicts the object.

[0088] Each analysis associated with operation 604 may contribute to an overall weight value assigned to the pixel unit. For example, the overall weight value may be assigned based on an analysis in operation 606 indicating how similar in chrominance the pixel unit is to the expected chrominance of the object, based on an analysis in operation 608 indicating how similar in chrominance the pixel unit is to the expected chrominance of an ambient image of the scene, and / or based on additional analyses of local or global features.

[0089] In particular implementations, the overall weight value may be assigned based on the local feature analysis associated with operation 606 or 608, another analysis of local features of the pixel unit (e.g., features related to the chrominance or luminance of the pixel unit), or a combination of these local feature analyses (e.g., based on a combination of the object comparison in operation 606 and the scene comparison in operation 608). As an example of how the local reliability analysis may be translated into a weight value (or one aspect of an overall weight value that combines several such aspects), consider again the chrominance threshold value discussed above in connection with FIG. 7A (and associated with operation 606). In this example, as shown in FIG. 7A, device 100 may compare the chrominance features of a particular pixel unit with the chrominance features associated with the foreign object by: 1) determining the distance (e.g., in color space) between a first point (e.g., one of points 702) representing the chrominance features of the particular pixel unit and a second point (e.g., point 704) representing the chrominance features associated with the object, and 2) assigning a weight value to the particular pixel unit based on the distance in color space between the first point and the second point.

[0090] In this example, when the distance between the first point and the second point is greater than a first distance threshold (e.g., when the first point is located outside the outer radius from the second point), the weight value 802 may be determined to not exceed the lower threshold 806 (and therefore be assigned the second weight value) because this large distance indicates that the pixel unit is very unlikely to depict an object. Conversely, when the distance between the first point and the second point is less than a second distance threshold (e.g., when the first point is located within the inner radius from the second point), the weight value 802 may be determined to exceed the upper threshold 804 (and therefore be assigned the first weight value) because this small distance indicates that the pixel unit is very likely to depict an object. In another example, when the distance between the first point and the second point is between a first distance threshold and a second distance threshold (e.g., when the first point is between an inner radius and an outer radius from the second point), the weight value 802 may be determined to exceed the lower threshold 806 but not exceed the upper threshold 804 (and thus be assigned a particular action weight value), because a moderate distance indicates that the pixel unit is likely to partially depict the object (e.g., the pixel unit is at the edge of the object, resulting in some individual pixels of the pixel unit depicting the object and other pixels not depicting the object), or indicates that it is undetermined whether the pixel unit depicts the object.

[0091] Returning to FIG. 6 , in this particular example, the weight value 802 assigned based on operation 606 may serve as the overall weight value for the pixel unit, or may serve as one factor or aspect that is considered (e.g., combined, averaged) along with other factors or aspects in determining the overall weight value. For example, the weight value 802 assigned based on operation 606 as described above may be combined with one or more separate weight values ​​802 (e.g., other first, second, or operational weight values) assigned based on a similar analysis related to operation 608 or based on other local characteristics of the pixel unit. Different weight values ​​assigned based on different analyses may be combined in any suitable manner. For example, the apparatus 100 may be configured to use as the overall weight value the maximum weight value returned from any analysis, the minimum weight value returned from any analysis, or the median or mode of all weight values ​​returned from the analysis. Alternatively, the apparatus 100 may be configured to combine the different weight values ​​into the overall weight value by calculating the average of the weight values. For example, if a second weight value (e.g., an overall weight value of 100%) is returned from operation 606 and a first weight value (e.g., a null weight value of 0%) is returned from operation 608, the device 100 may average these two weight values ​​into an operating weight value (e.g., 50%).

[0092] Additionally, a global analysis, such as the object tracking of operation 610, may be used in addition to or as an alternative to the local analysis described above. For example, pixel units that are entirely contained within a bounding box due to a foreign object tracked by operation 610 may be assigned a first weight value 802 (e.g., a null weight value), pixel units that are entirely outside such a bounding box may be assigned a second weight value 802 (e.g., a full weight value), and pixel units that are determined to be on (or near) the boundary of the bounding box may be assigned a motion weight value 802 (e.g., a weight value greater than the first weight value and less than the second weight value). Similar to the weight values ​​802 described above in connection with local features, the weight values ​​802 assigned based on the global feature analysis in this manner may be used as overall weight values ​​in certain instances or may include one aspect of the overall weight value as determined by combining multiple such aspects. For example, the overall weight value determined based on operation 610 may be combined with one or more other global weight values ​​or one or more local weight values ​​in any manner described herein (e.g., using the maximum weight value, using the minimum weight value, calculating an average weight value).

[0093] In operation 614, one or more unit auto exposure data points (e.g., a unit auto exposure value (Vi), a unit auto exposure target (Ti) for the current pixel unit (Pi), etc.) may be determined. For example, regardless of whether operation 604 reveals that the pixel unit is part of a foreign object, is not part of an object, or is undetermined, operation 614 may analyze characteristics of the pixel unit, such as the luminance of the pixel unit, to determine how bright the pixel unit is (e.g., a unit auto exposure value Vi) and / or what brightness value is desired for the pixel unit (e.g., a unit auto exposure target Ti). In an implementation in which each pixel unit i is implemented by an individual pixel, the unit auto exposure value and unit auto exposure target determined in operation 614 may be implemented as a pixel auto exposure value and pixel auto exposure target for the individual pixel. Conversely, in implementations where each pixel unit i is implemented as a group of pixels within a region of the image frame, the unit auto-exposure value and unit auto-exposure target determined in operation 614 may be determined as an average (e.g., mean, median, mode, etc.) of the pixel auto-exposure values ​​and / or pixel auto-exposure targets of the individual pixels included within the pixel unit. As described, operation 614 may be independent of operations 604-612, and thus operation 614 may be performed prior to, subsequent to, or concurrently with operations 604-612.

[0094] In operation 616, the outputs of operations 612 and 614 are combined to form the pixel unit P i As shown, each weighted pixel unit may be a pixel unit P i Data relating to the unit auto exposure value for pixel unit P i Data relating to the unit auto exposure target, and pixel unit P iAs described in more detail below, the weighted pixel units for each pixel unit of an image frame may be used to determine frame auto-exposure data points (e.g., frame auto-exposure values ​​and frame auto-exposure targets) in a manner that discounts object regions associated with extraneous objects. Once the pixel units P of an image frame (e.g., a portion thereof) i Once all of the above have been repeated in operation 602, the flow proceeds to the end of flow diagram 600 (Complete), at which point operation 504 may be completed and object regions may be identified based on the respective weight values ​​assigned to the pixel units.

[0095] Returning to FIG. 5, flow proceeds from operation 504 to operation 510, where the apparatus 100 calculates a frame auto-exposure value (V F ) and the frame autoexposure target (T F ) In operation 510, the frame auto-exposure data points may be determined in a manner that discounts the object regions identified in operation 504. For example, the identified object regions may be encoded in weighted pixel units determined during operation 504 (e.g., in operation 616 of FIG. 6 ), and these weighted pixel units may be used to determine the frame auto-exposure data points.

[0096] 9 shows an example flow diagram 900 implementing one way in which operation 510 may be performed. Flow diagram 900 illustrates how apparatus 100 determines a frame auto-exposure value and a frame auto-exposure target based on weighted pixel units determined as part of identifying object regions in operation 504. Specifically, as shown, different weighted pixel units (e.g., each pixel unit P analyzed in operation 504) may be weighted based on the weighted pixel units. i) may provide input data for the various operations 902-910 of flow diagram 900 to be performed, such that the frame auto-exposure value and frame auto-exposure target are ultimately determined by discounting the object area.

[0097] In operation 902, the apparatus 100 calculates the corresponding weight value W i Each unit auto exposure value V from each of the weighted pixel units by i may be scaled, and these scaled unit auto-exposure values ​​may be combined (e.g., summed) together to form a single value. Similarly, in operation 904, the apparatus 100 may calculate the corresponding weight value W i Each unit from each of the pixel units is weighted by the auto exposure target T i may be scaled, and these scaled unit auto-exposure targets may be combined (e.g., summed, etc.) to form another single value. In operation 906, apparatus 100 may combine each of the weight values ​​in a similar manner (e.g., by summing the weight values ​​together, etc.).

[0098] In operation 908, the apparatus 100 may determine a frame auto-exposure value based on the respective weight values ​​assigned to the pixel units. For example, the frame auto-exposure value may be determined as a weighted average of the unit auto-exposure values ​​of the respective pixel units. The apparatus 100 may determine the weighted average in operation 908 based on the outputs from operations 902 and 906 (e.g., by dividing the output of operation 902 by the output of operation 906) to form the frame auto-exposure value. In this manner, the frame auto-exposure value VF may be determined according to Equation 1 (where i is an index used to iterate through each weighted pixel unit).

number

[0099] In operation 910, the apparatus 100 may determine a frame auto-exposure target based on the respective weight values ​​assigned to the pixel units. For example, the frame auto-exposure target may be determined as a weighted average of the unit auto-exposure targets of the pixel units. The apparatus 100 may determine the weighted average in operation 910 based on outputs from operations 904 and 906 (e.g., dividing the output of operation 904 by the output of operation 906) to form the frame auto-exposure target. In this manner, the frame auto-exposure value TF may be determined according to Equation 2 (where i is an index used to iterate through each weighted pixel unit).

number

[0100] In other embodiments, a weighted average incorporating weight values ​​and unit auto-exposure data points for various pixel units may be calculated in other ways to similarly discount identified object regions based on a weighted method that removes or reduces the influence on auto-exposure management of pixel units that are determined to correspond at least in part to the depiction of extraneous objects within the image frame.

[0101] 5 may be completed, and flow may proceed to operation 512 in flow diagram 500, where apparatus 100 may update auto-exposure parameters for the image capture system based on the frame auto-exposure value and / or frame auto-exposure target determined by discounting the object area. At operation 512, apparatus 100 may update (e.g., adjust or maintain) auto-exposure parameters for the image capture system in preparation for the image capture system to capture a subsequent image frame in the image frame sequence.

[0102] FIG. 10 illustrates an exemplary technique 1000 for updating the auto-exposure parameters in operation 512. As illustrated, a previously determined frame auto-exposure value and frame auto-exposure target are used as inputs for the operations shown in FIG. 10 . For example, operation 1002 may receive the frame auto-exposure value and frame auto-exposure target as inputs and use them as a basis for determining a frame auto-exposure gain. The frame auto-exposure gain may be determined to correspond to the ratio of the frame auto-exposure target to the frame auto-exposure value. In this manner, if the frame auto-exposure value already equals the frame auto-exposure target (e.g., so that no further adjustments are required to align with the target), the frame auto-exposure gain may be set to a gain of 1, so that the system does not boost or attenuate the auto-exposure value for subsequent frames captured by the image capture system. Conversely, if the frame auto-exposure target differs from the frame auto-exposure value, the frame auto-exposure gain may be set to correspond to a value less than or greater than 1 to cause the system to boost or attenuate the auto-exposure value for subsequent frames in an attempt to bring the auto-exposure value closer to the desired auto-exposure target.

[0103] In operation 1004, the frame auto exposure gain may be taken as input along with other data (e.g., other frame auto exposure gains) determined for previous image frames in the image frame sequence. Based on these inputs, operation 1004 applies filtering to ensure that the auto exposure gain does not change more quickly than desired, thereby ensuring that the image frames presented to the user change gradually while maintaining consistent brightness. The filtering performed in operation 1004 may be performed using a smoothing filter, such as a temporal infinite impulse response (IIR) filter, or other digital or analog filter as may be useful in a particular implementation.

[0104] In operation 1006, the auto-exposure gain of the filtered frame may be used as a basis for adjusting one or more auto-exposure parameters of the image capture system (e.g., for use by the image capture device or illumination source to capture additional image frames). For example, as described above, the adjusted auto-exposure parameters may include an exposure time parameter, a shutter aperture parameter, a brightness gain parameter, or equivalent parameters. For image capture systems in which the illumination of the scene is largely or completely controlled by the image capture system (e.g., image capture systems including the endoscopic image capture devices described above, image capture systems including flash or other illumination sources, etc.), the adjusted auto-exposure parameters may further include an illumination intensity parameter, an illumination duration parameter, or equivalent parameters.

[0105] Adjustments to the auto-exposure parameters of an image capture system may cause the image capture system to expose subsequent image frames in a variety of different ways. For example, by adjusting the exposure time parameter, the shutter speed may be adjusted for a shutter included in the image capture system. For example, the shutter may be left open for a longer period of time (to thereby increase the amount of exposure time of the image sensor) or for a shorter period of time (to thereby decrease the amount of exposure time of the image sensor). As another example, by adjusting the shutter aperture parameter, the shutter aperture may be adjusted to open wider (to thereby increase the amount of light exposed to the image sensor) or less wide (to thereby decrease the amount of light exposed to the image sensor). As yet another example, by adjusting the brightness gain parameter, the sensitivity (e.g., ISO sensitivity) may be increased or decreased to amplify or attenuate the illuminance as captured by the image capture system. For implementations in which the image capture system controls the lighting of the scene, the lighting intensity and / or lighting duration parameters may be adjusted to increase the intensity and duration of the light used to illuminate the scene being captured, thereby affecting how much light the image sensor is exposed to.

[0106] Returning to FIG. 5 , after the operations of flow diagram 500 are performed, the current image frame is considered fully processed by apparatus 100, and the flow may return to operation 502, where a subsequent image frame in the image frame sequence may be acquired. The process may be repeated for the subsequent image frame and / or other subsequent image frames. It will be appreciated that in certain examples, every image frame may be analyzed according to flow diagram 500 to keep auto-exposure data points (e.g., frame auto-exposure values ​​and frame auto-exposure targets, etc.) and auto-exposure parameters as up-to-date as possible. In other examples, only certain image frames (e.g., every other image frame, every third image frame, etc.) may be analyzed to conserve processing bandwidth in scenarios where more periodic auto-exposure processing still allows design specifications and targets to be achieved. It will also be appreciated that auto-exposure effects may tend to lag several frames behind brightness changes in the scene, because auto-exposure parameter adjustments made based on one particular frame do not affect the exposure of that frame, but rather affect subsequent frames.

[0107] Based on the adjustments that device 100 makes to the auto-exposure parameters (and / or based on maintaining the auto-exposure parameters at their current levels when appropriate), device 100 may successfully manage the auto-exposure for image frames being captured by the image capture system, and subsequent image frames may be captured with desirable auto-exposure characteristics so as to have an appealing and informative appearance when presented to a user.

[0108] As described, apparatus 100, method 200, and / or system 300 may each, in certain embodiments, be associated with a computer-assisted medical system used to perform medical procedures on a body (e.g., surgical procedures, diagnostic procedures, exploratory procedures, etc.) For illustrative purposes, FIG. 11 shows an exemplary computer-assisted medical system 1100 that may be used to perform various types of medical procedures, including surgical and / or non-surgical procedures.

[0109] As shown, the computer-assisted medical system 1100 may include a manipulator assembly 1102 (a manipulator cart is shown in FIG. 11 ), a user control device 1104, and an auxiliary device 1106, all of which are communicatively coupled to one another. The computer-assisted medical system 1100 may be utilized by a medical team to perform computer-assisted medical procedures or other similar operations on the body of a patient 1108 or any other body that may be useful in a particular implementation. As shown, the medical team may include a first user 1110-1 (such as a surgeon for a surgical procedure), a second user 1110-2 (such as a patient-side assistant), a third user 1110-3 (such as another assistant, nurse, trainee, etc.), and a fourth user 1110-4 (such as an anesthesiologist for a surgical procedure), all of which may be collectively referred to as users 1110, and each of which may control, interact with, or otherwise be a user of the computer-assisted medical system 1100. There may be more, fewer, or alternative users during a medical procedure, which may be useful in a particular implementation. For example, team compositions for different medical or non-medical procedures may differ and may include users with different roles.

[0110] 11 illustrates a minimally invasive medical procedure, such as a minimally invasive surgical procedure, in progress, it will be understood that the computer-assisted medical system 1100 may likewise be used to perform open medical procedures or other types of procedures, such as, for example, exploratory imaging procedures, simulated medical procedures used for training purposes, and / or other procedures.

[0111] As shown in FIG. 11 , the manipulator assembly 1102 may include one or more manipulator arms 1112 (e.g., manipulator arms 1112-1 through 1112-4) to which one or more instruments may be coupled. The instruments may be used for computer-assisted medical procedures on a patient 1108 (e.g., in surgical cases, by being at least partially inserted into and manipulated within the patient 1108). While the manipulator assembly 1102 is shown and described herein as including four manipulator arms 1112, it will be appreciated that the manipulator assembly 1102 may include a single manipulator arm 1112 or any other number of manipulator arms as may be useful in a particular implementation. While the example of FIG. 11 shows the manipulator arms 1112 to be robotic manipulator arms, it will be understood that in some examples, one or more instruments may be partially or fully manually controlled, for example, by being hand-held and manually controlled by a person. For example, these partially or wholly manually controlled instruments may be used in conjunction with, or as an alternative to, the computer-assisted instruments coupled to the manipulator arm 1112 shown in FIG.

[0112] During a medical procedure, the user controls 1104 may be configured to facilitate remote control by the user 1110-1 of the manipulator arm 1112 and instruments attached to the manipulator arm 1112. To this end, the user controls 1104 may provide the user 1110-1 with an image of the field of operation associated with the patient 1108 as captured by an imaging device. To facilitate control of the instruments, the user controls 1104 may include a set of master controls. These master controls may be operated by the user 1110-1 to control the movement of the manipulator arm 1112 or any instruments coupled to the manipulator arm 1112.

[0113] The auxiliary device 1106 may include one or more computing devices configured to perform auxiliary functions supporting a medical procedure, such as imaging devices, image processing, or providing air insufflation, electrocautery energy, lighting, or other energy for cooperating components of the computer-assisted medical system 1100. In some examples, the auxiliary device 1106 may be configured with a display monitor 1114 configured to display one or more user interfaces or graphical or textual information supporting a medical procedure. In some examples, the display monitor 1114 may be implemented with a touchscreen display to provide user input functionality.

[0114] As described in more detail below, the device 100 may be implemented within or operate in conjunction with a computer-aided medical system 1100. For example, in a particular implementation, the device 100 may be implemented by computing resources contained within an instrument (e.g., an endoscope or other imaging instrument) attached to one of the manipulator arms 1112, or by computing resources associated with the manipulator assembly 1102, the user controls 1104, the auxiliary instruments 1106, or other system components not explicitly shown in FIG.

[0115] The manipulator assembly 1102, the user control device 1104, and the auxiliary device 1106 may be communicatively coupled to one another in any suitable manner. For example, as shown in FIG. 11 , the manipulator assembly 1102, the user control device 1104, and the auxiliary device 1106 may be communicatively coupled via control lines 1116, which may represent any wired or wireless communication link that may be useful in a particular implementation. To this end, the manipulator assembly 1102, the user control device 1104, and the auxiliary device 1106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0116] In particular implementations, one or more processes described herein may be implemented, at least in part, as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. Generally, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to thereby perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0117] Computer-readable media (also referred to as processor-readable media) include any non-transitory medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a computer processor). Such media may take many forms, including, but not limited to, non-volatile media and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random access memory (DRAM), which typically constitutes main memory. Common forms of computer-readable media include, for example, disks, hard disks, magnetic tapes, any other magnetic media, compact disc read-only memory (CD-ROM), digital video disc (DVD), any other optical media, random access memory (RAM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EPROM), FLASH®-EEPROM, any other memory chip or cartridge, or any other tangible media from which a computer can read.

[0118] 12 illustrates an exemplary computing system 1200 that may be specifically configured to perform one or more of the processes described herein. For example, computing system 1200 may include or implement (partially implement) an automatic exposure management apparatus such as apparatus 100, an automatic exposure management system such as system 300, or any other computing system or device described herein.

[0119] As shown in Figure 12, computing system 1200 may include a communication interface 1202, a processor 1204, a storage device 1206, and an input / output ("I / O") module 1208 communicatively connected via a communication infrastructure 1210. While an exemplary computing system 1200 is shown in Figure 12, the components illustrated in Figure 12 are not intended to be limiting. Additional or alternative components may be used in other embodiments. The components of computing system 1200 shown in Figure 12 will now be described in additional detail.

[0120] Communications interface 1202 may be configured to communicate with one or more computing devices. Examples of communications interface 1202 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0121] Processor 1204 generally represents any type or form of processing unit that can process data or that can interpret, execute, and / or direct the execution of one or more of the instructions, processes, and / or operations described herein. Processor 1204 may direct the execution of operations in accordance with one or more applications 1212 or other computer-executable instructions, such as those that may be stored on storage device 1206 or other computer-readable media.

[0122] Storage device(s) 1206 may include one or more data storage media, devices, or configurations and may utilize any type, form, and combination of data storage media and / or devices. For example, storage device(s) 1206 may include, but are not limited to, a hard drive, a network drive, a flash drive, a magnetic disk, an optical disk, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or any combination or subcombination thereof. Electronic data, including the data described herein, may be temporarily and / or permanently stored on storage device(s) 1206. For example, data representing one or more executable applications 1212 configured to instruct processor 1204 to perform any of the operations described herein may be stored on storage device(s) 1206. In some examples, data may be located in one or more databases residing within storage device(s) 1206.

[0123] I / O module 1208 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual experience. I / O module 1208 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, I / O module 1208 may include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0124] I / O module 1208 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In particular embodiments, I / O module 1208 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be useful in a particular implementation.

[0125] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing system 1200. For example, one or more applications 1212 resident in storage device 1206 may be configured to direct processor 1204 to perform one or more processes or functions associated with processor 104 of apparatus 100. Similarly, memory 102 of apparatus 100 may be implemented by or within storage device 1206.

[0126] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the following claims. For example, specific features of one embodiment described herein may be combined with or substituted for features of other embodiments described herein. Accordingly, the description and drawings should be considered in an illustrative sense, and not in a limiting sense.

Claims

1. one or more processors; a memory for storing executable instructions; 1. An apparatus comprising: The instructions, when executed by the one or more processors, cause the apparatus to: identifying, within an image frame captured by an image capture system, a plurality of pixel units corresponding to a depiction of an object depicted within said image frame; determining, for each of the plurality of pixel units, a confidence level as to whether the respective pixel unit corresponds to the representation of the object, the determined confidence levels comprising different confidence level values; determining a frame auto-exposure value for the image frame by ignoring the plurality of pixel units corresponding to the representation of the object to different degrees based on the different confidence level values ​​of the pixel units corresponding to the representation of the object; updating one or more auto-exposure parameters for use by the image capture system to capture additional image frames based on the frame auto-exposure values; Device.

2. The apparatus of claim 1 , wherein identifying the plurality of pixel units corresponding to the depiction of the object is based on different colors of the object and colors of tissue depicted in the image frame.

3. identifying the plurality of pixel units corresponding to the representation of the object based on one or more local features associated with pixel units included in the image frame; the one or more local features associated with the pixel units included in the image frame include chrominance features of the pixel units included in the image frame; identifying the plurality of pixel units corresponding to the depiction of the object includes comparing the chrominance features of the pixel units included in the image frame with chrominance features associated with the object; 3. The device according to claim 1 or 2.

4. Comparing the chrominance features of the pixel units included in the image frame with the chrominance features associated with the object includes, for each individual pixel unit included in the image frame: determining a distance in a color space between a first point representing a chrominance feature of the individual pixel unit and a second point representing the chrominance feature associated with the object; and assigning a weight value to the individual pixel unit based on the distance between the first point and the second point, the weight value being: If the distance exceeds an upper threshold, assign it as a first weight value; if the distance does not exceed a lower threshold, assign a second weight value, the second weight value being greater than the first weight value; if the distance exceeds the lower threshold but does not exceed the upper threshold, assigning an operating weight value between the first weight value and the second weight value; identifying the plurality of pixel units corresponding to the representation of the object based on the weight values ​​assigned to the individual pixel units; 4. The apparatus of claim 3.

5. identifying the plurality of pixel units corresponding to the representation of the object based on one or more local features associated with pixel units included in the image frame; the one or more local features associated with the pixel units included in the image frame include chrominance features and luminance features of the pixel units included in the image frame; Identifying the plurality of pixel units corresponding to the representation of the object comprises: comparing the chrominance features of the pixel units included in the image frame with chrominance features associated with the object; comparing the luminance characteristics of the pixel units included in the image frame with luminance characteristics associated with the object.

3. The device according to claim 1 or 2.

6. The instructions, when executed by the one or more processors, cause the apparatus to determine a color gamut for a background of a scene depicted in the image frame; identifying the plurality of pixel units corresponding to the representation of the object based on one or more local features associated with pixel units included in the image frame; the one or more local features associated with the pixel units included in the image frame include chrominance features of the pixel units included in the image frame; identifying the plurality of pixel units corresponding to the representation of the object includes determining whether the chrominance characteristics of the pixel units included in the image frame are within the color gamut for the background of the scene; 3. The device according to claim 1 or 2.

7. the image capture system includes an endoscopic image capture device configured to capture the image frames as part of a sequence of image frames captured during performance of a medical procedure on a body; the scene depicted in the image frames is an internal view of the body; the color gamut for the background of the scene includes a range of red colors corresponding to blood and tissue visible in the internal view of the body; 7. The apparatus of claim 6.

8. 8. The apparatus of claim 1, wherein identifying the plurality of pixel units corresponding to the depiction of the object is based on coordinates determined based on object tracking data received from an object tracking system that tracks the position of the object within a scene depicted in the image frame.

9. the object includes an instrument controlled by a robotic arm; the object tracking system tracks the position of the object based on kinematic data associated with movement of the robotic arm.

9. The apparatus of claim 8.

10. The apparatus of claim 8 , wherein the object tracking system tracks the position of the object based on computer vision techniques applied to image frames of a sequence of image frames that includes the image frame.

11. Identifying the plurality of pixel units corresponding to the representation of the object comprises: chrominance or luminance features associated with pixel units included in the image frame; and a position feature of the object, An apparatus according to any one of claims 1 to 10.

12. identifying, by a computing device, in an image frame captured by an image capture system, a plurality of pixel units corresponding to a depiction of an object depicted in the image frame; determining, by the computing device, for each of the plurality of pixel units, a confidence level as to whether the respective pixel unit corresponds to the representation of the object, the determined confidence level comprising different confidence level values; determining, by the computing device, a frame auto-exposure value and a frame auto-exposure target for the image frame by disregarding the plurality of pixel units corresponding to the representation of the object to different degrees based on the different confidence level values ​​of the pixel units corresponding to the representation of the object; updating, by the computing device, one or more auto-exposure parameters for use by the image capture system to capture additional image frames based on the frame auto-exposure value and the frame auto-exposure target. method.

13. identifying the plurality of pixel units corresponding to the representation of the object based on one or more local features associated with pixel units included in the image frame; the one or more local features associated with the pixel units included in the image frame include chrominance features of the pixel units included in the image frame; identifying the plurality of pixel units corresponding to the depiction of the object includes comparing the chrominance features of the pixel units included in the image frame with chrominance features associated with the object; The method of claim 12.

14. Comparing the chrominance features of the pixel units included in the image frame with the chrominance features associated with the object includes, for each individual pixel unit included in the image frame: determining a distance in a color space between a first point representing a chrominance characteristic of the individual pixel unit and a second point representing the chrominance characteristic associated with the object; and assigning a weight value to the individual pixel unit based on the distance between the first point and the second point, the weight value being: If the distance exceeds an upper threshold, assign it as a first weight value; if the distance does not exceed a lower threshold, assign a second weight value, the second weight value being greater than the first weight value; if the distance exceeds the lower threshold but does not exceed the upper threshold, assigning an operating weight value between the first weight value and the second weight value; identifying the plurality of pixel units corresponding to the representation of the object based on the weight values ​​assigned to the individual pixel units; The method of claim 13.

15. Identifying the plurality of pixel units corresponding to the representation of the object comprises: chrominance features associated with pixel units included in the image frame; and a position feature of the object, The method according to any one of claims 12 to 14.

Images