Apparatus, system, and method for managing auto exposure of image frames depicting color biased content

Abstract

An illustrative device can determine a color cast metric for an image frame captured by an image capture system. The color cast metric can indicate an extent to which the image frame is cast to a particular color. Based on the color cast metric and an adaptive target control function, the device can determine a frame auto exposure target. Based on the frame auto exposure target, the device can update one or more auto exposure parameters for use by the image capture system in capturing additional image frames. Corresponding devices, systems, and methods for managing auto exposure of image frames are also disclosed.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 050,583, filed July 10, 2020, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Automatic exposure algorithms operate by analyzing image frames to determine how much light is present in the scene depicted by the frame and updating the automatic exposure parameters of the image capture device based on this analysis. In this way, the automatic exposure parameters are continuously updated to ensure the image capture device provides the necessary exposure for the image frame being captured. Without proper automatic exposure management, detail may be lost during image capture due to overexposure (e.g., loss of detail due to saturation and the image appearing too bright) or underexposure (e.g., loss of detail due to noise and the image appearing too dark).

[0004] While conventional automatic exposure algorithms are sufficient to serve a wide range of image types, images depicting color-biased content (e.g., content that is biased towards a portion of the color spectrum rather than a balanced representation of multiple colors) can present unique challenges. For example, some colors naturally appear darker than others due to their lower brightness. When conventional automatic exposure algorithms process image frames depicting such color-biased content, the algorithm may overexpose or underexpose the image frame and / or encounter other undesirable problems. Summary of the Invention

[0005] The following description presents a simplified overview of one or more aspects of the apparatus, system, and method described herein. This overview is not a comprehensive summary of all anticipated aspects, and is neither intended to identify key or decisive elements of all aspects, nor to indicate the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed descriptions that follow.

[0006] An illustrative apparatus for managing automatic exposure of image frames may include one or more processors and a memory storing executable instructions that, when executed by the one or more processors, cause the apparatus to perform various operations described herein. For example, the apparatus may determine a color-skew metric of an image frame captured by an image capture system. The color-skew metric indicates the degree to which an image frame is offset to a particular color. Based on the color-skew metric and an adaptive target control function, the apparatus may determine an automatic exposure target for the frame. Based on the automatic exposure target, the apparatus may update one or more automatic exposure parameters for the image capture system to use in capturing additional image frames.

[0007] An illustrative system for managing automatic exposure of image frames may include an illumination source, an image capture device, and one or more processors. The illumination source may be configured to illuminate tissue within a body during a medical procedure. The image capture device may be configured to capture a sequence of image frames during the medical procedure. The sequence of image frames may include image frames depicting an internal view of the body, representing tissue illuminated by the illumination source. One or more processors may be configured to determine a color cast metric for the image frames. The color cast metric may indicate the degree to which the image frame is shifted towards red. Based on the color cast metric and an adaptive target control function, one or more processors may determine an automatic exposure target for the frames. Based on the automatic exposure target, one or more processors may update one or more automatic exposure parameters for use by the image capture device or the illumination source when capturing additional image frames in the sequence of image frames.

[0008] This document contains illustrative, non-transitory computer-readable medium-storeable instructions that, when executed, cause one or more processors of a computing device to perform various operations described herein. For example, one or more processors may determine a color cast metric of an image frame captured by an image capture system. The color cast metric may indicate the degree to which the image frame is skewed to a particular color. One or more processors may also determine the output of an adaptive target control function given an input of the color cast metric. Based on the output of the adaptive target control function, one or more processors may update one or more automatic exposure parameters for the image capture system to use in capturing additional image frames.

[0009] Illustrative methods for managing automatic exposure of image frames may include various operations described herein, each of which may be performed by a computing device, such as the automatic exposure management apparatus described herein. For example, the method may include determining a color cast metric of an image frame captured by an image capture system. The color cast metric may indicate the degree to which the image frame is offset to a particular color. The method may further include determining an automatic exposure target for the frame based on the color cast metric and determining an automatic exposure value for the frame. The method may further include updating one or more automatic exposure parameters based on the automatic exposure target and the automatic exposure value for use by the image capture system to capture additional image frames. Attached Figure Description

[0010] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals denote the same or similar elements.

[0011] Figure 1 An illustrative automatic exposure management device is shown for managing the automatic exposure of image frames according to the principles described herein.

[0012] Figure 2An illustrative automatic exposure management method is shown, which manages the automatic exposure of image frames based on the principles described herein.

[0013] Figure 3 An illustrative automatic exposure management system for managing the automatic exposure of image frames, based on the principles described herein, is shown.

[0014] Figure 4 Illustrative image frames and color cast characteristics of image frames are shown according to the principles described herein.

[0015] Figure 5 An illustrative flowchart is shown for managing the automatic exposure of image frames based on the principles described herein.

[0016] Figure 6 An illustrative technique for determining the color cast metric of an image frame, based on the principles described herein, is shown.

[0017] Figure 7 An illustrative diagram is shown depicting the chromaticity characteristics of the decomposed color data according to the principles described herein.

[0018] Figure 8 An illustrative flowchart is shown, based on the principles described herein, for quantifying color shift as part of the determination of a color cast metric.

[0019] Figure 9 An illustrative technique for determining the automatic exposure target of a frame based on a color cast metric, according to the principles described herein, is presented.

[0020] Figures 10A-10D Illustrative implementations of various adaptive target control functions based on the principles described herein are shown.

[0021] Figure 11 An illustrative technique for updating automatic exposure parameters based on the principles described herein is shown.

[0022] Figure 12 An illustrative computer-aided medical system based on the principles described herein is shown.

[0023] Figure 13 An illustrative computational system based on the principles described herein is shown. Detailed Implementation

[0024] This paper describes apparatus, systems, and methods for managing automatic exposure of image frames. As mentioned above, automatic exposure management of image frames depicting color-biased content can present unique challenges when the content depicted in the image frame is more balanced across the color spectrum. For example, if an image frame shifts towards a specific color associated with lower brightness compared to other colors, and automatic exposure management does not properly account for this shift, an excessively high automatic exposure target can be identified. This can lead to overexposure in subsequent image frames. For instance, an automatic exposure algorithm might attempt to brighten images that appear dark because their chromaticity has shifted towards colors that appear naturally dark, rather than because the scene is poorly lit. When this happens, the automatic exposure algorithm may saturate the channel associated with the specific color to which the image frame has shifted, resulting in inaccurate colors in subsequent image frames and loss of detail due to overexposure. For example, if the color to which the image frame has shifted is red, and the red channel is saturated as the automatic exposure algorithm overexposes subsequent image frames, the red in subsequent image frames may appear as an unrealistic orange hue for the captured image.

[0025] As an example of a potential problem, consider an endoscopic image capture device that captures views of the interior of the body during medical procedures (e.g., surgical procedures). Since blood and blood tissue are ubiquitous in the human body, in this case, the image capture device may capture image frames that are offset towards red (e.g., the color associated with blood depicted in the internal view). Because red is often associated with low brightness (e.g., red appears darker than other colors even when exposed to and / or reflecting the same amount of light as other colors), such image frames may be prone to the aforementioned overexposure and red saturation problems (e.g., in some examples, making the blood appear slightly orange).

[0026] The apparatus, systems, and methods described herein provide automatic exposure management to address these and other problems with images depicting color-biased content (e.g., images biased towards a single color, such as red). For example, the automatic exposure management described herein can evaluate, quantify, and otherwise determine the color shift of an image frame (e.g., the degree or extent to which an image frame is shifted towards a particular color), and can take this color shift into account when determining an automatic exposure target (e.g., the brightness of the automatic exposure algorithm target for subsequent image frames). In this way, automatic exposure management can avoid overcompensating potentially misleading brightness associated with certain colors (e.g., the fact that red appears darker than other colors), and avoid overexposure, underexposure, and / or channel saturation that may be a result of such overcompensation.

[0027] Throughout this specification, examples of medical procedures involving endoscopic views of the bloody interior of the body, such as those described above, will be referenced to illustrate various aspects of the claimed subject matter. However, it should be understood that endoscopic images of bloody (and therefore severely red-shifted) scenes are merely examples, and the principles described herein can be applied in various implementations to any suitable type of color-shifted content to serve a particular application or use case. For example, as several additional examples, the automatic exposure management described herein can be applied to capturing close-up images of red objects, capturing scenes in rooms painted red, and capturing certain landscapes (e.g., sunsets, autumn foliage, etc.). Furthermore, the principles described herein can be applied to various color-shifted images and scenes with colors other than red (e.g., blue, yellow, purple, etc.), whose chromaticity properties similarly cause these colors to naturally exhibit more or less luminosity than colors considered by conventional automatic exposure algorithms.

[0028] Various specific embodiments will now be described in detail with reference to the accompanying drawings. It will be understood that the specific embodiments described below are provided as non-limiting examples of how various novel and inventive principles can be applied to various situations. Furthermore, it should be understood that other examples not explicitly described herein may also be covered by the scope of the appended claims. The automatic exposure management apparatus, system, and method described herein can provide any of the benefits described above, as well as various additional and / or alternative benefits that will be described below and / or apparent.

[0029] Figure 1 An illustrative automatic exposure management apparatus 100 (apparatus 100) for managing automatic exposure of image frames according to the principles described herein is shown. Apparatus 100 may be implemented by computer resources (e.g., servers, processors, memory devices, storage devices, etc.) included in an image capture system (e.g., an endoscopic image capture system, etc.), computer resources of a computing system associated with the image capture system (e.g., communicatively coupled to the image capture system), and / or by any other suitable computing resources that may serve a particular implementation.

[0030] As shown in the figure, device 100 may include (but is not limited to) a memory 102 and a processor 104, which are selectively and communicatively coupled to each other. The memory 102 and the processor 104 may each include or be implemented by computer hardware configured to store and / or process computer software. Figure 1 Various other components of computer hardware and / or software not explicitly shown may also be included within device 100. In some examples, memory 102 and processor 104 may be distributed among multiple devices and / or multiple locations that may serve a particular implementation.

[0031] Memory 102 may store and / or otherwise maintain executable data used by processor 104 to perform any of the functions described herein. For example, memory 102 may store instructions 106 that can be executed by processor 104. Memory 102 may be implemented by one or more memory or storage devices (including any memory or storage device described herein) configured to store data in a transient or non-transient manner. Instructions 106 may be executed by processor 104 to cause device 100 to perform any of the functions described herein. Instructions 106 may be implemented by any suitable application, software, code, and / or other instance of executable data. Furthermore, memory 102 may also maintain any other data accessed, managed, used, and / or transferred by processor 104 in a particular implementation.

[0032] Processor 104 may be implemented by one or more computer processing devices, including general-purpose processors (e.g., central processing unit (CPU), graphics processing unit (GPU), microprocessor, etc.), special-purpose processors (e.g., application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc.), image signal processors, etc. Using processor 104 (e.g., when processor 104 is instructed to perform an operation represented by instructions 106 stored in memory 102), device 100 can perform various functions associated with managing the automatic exposure of image frames depicting color-biased content (e.g., content offset to a particular color such as the red of blood, to which endoscopic images of the inside of the body are typically offset).

[0033] Figure 2 An illustrative automatic exposure management method 200 (method 200) based on the principles described herein is shown, which apparatus 100 can execute to manage the automatic exposure of image frames. Although Figure 2 Illustrative operation according to one embodiment is shown, but other embodiments may omit, add, reorder, and / or modify it. Figure 2 Any of the operations shown. In some examples, Figure 2 The multiple operations shown can be executed concurrently (e.g., in parallel) with each other, rather than sequentially as shown in the figure. Figure 2 One or more of the operations shown may be performed by an automatic exposure management device (e.g., device 100), an automatic exposure management system (e.g., an implementation of the automatic exposure management system described below), and / or any implementation thereof.

[0034] At operation 202, device 100 can determine a color cast metric of an image frame captured by the image capture system. The color cast metric can indicate the degree to which the image frame is offset to a particular color. For example, in a medical procedure example where the image frame depicts an internal endoscopic view of a body undergoing a medical procedure, the color cast metric determined at operation 202 can be implemented as a red metric indicating the degree to which the image frame is offset to the red color of the blood and bloody tissue depicted in the image frame. The color cast metric determined at operation 202 can be determined in any manner described herein (e.g., including techniques described in more detail below) and can be represented in any manner that may serve a particular implementation. For example, the color cast metric can be expressed as a percentage varying between 0% (e.g., for a perfectly neutral image frame whose offset from any particular color does not exceed that of any other color) and 100% (e.g., for an image frame completely offset to a particular color). In this example, a negative percentage can be used to represent a color deviation offset to a color other than a particular color, or this can be interpreted in another way. In other implementations, other representations can be used to quantify the color cast measure, such as floating-point representation (e.g., a measure that varies between 0.00 and 1.00 or between -1.00 and 1.00), integer representation (e.g., a measure that varies between 0 and 5, between 0 and 500, between -10 and 10, etc.), or other suitable representations.

[0035] At operation 204, device 100 can determine the output of an adaptive target control function given an input of a color cast metric determined at operation 202. For example, as will be described and explained in more detail below, the adaptive target control function can define how to analyze image frames with different color cast metrics based on auto-exposure data points (e.g., auto-exposure target, auto-exposure value, etc.) determined for the image frame. For example, a particular adaptive target control function can define a threshold that, if not exceeded by the color cast metric of a particular image frame, causes the auto-exposure management of the image frame to be performed in a manner that does not specifically define the color deviation of the image frame's content. On the other hand, if the color cast metric exceeds the threshold, the auto-exposure management can specifically define the color deviation represented by the color cast metric.

[0036] At operation 206, device 100 can determine the frame auto-exposure target based on the output of the adaptive target control function determined at operation 204. In some examples, device 100 may also determine one or more other auto-exposure data points (e.g., frame auto-exposure values, etc.) at operation 206, or as... Figure 2This is part of an additional operation not explicitly shown. Since the output of the adaptive target control function is determined with the color cast metric of the image frame as the input of the function, it is also possible that at operation 206, the device 100 can be said to determine the automatic exposure target of the frame based on the color cast metric determined at operation 202.

[0037] The auto-exposure value will be understood as representing a specific auto-exposure-related characteristic (e.g., brightness, signal strength, etc.) of a particular image frame or a portion thereof (e.g., region, group of pixels, etc.). For example, apparatus 100 can detect these characteristics by analyzing image frames captured by an image capture system. A pixel auto-exposure value can refer to the brightness determined for a single pixel or, in an implementation where pixels are grouped together as pixel cells in a grid, the average brightness determined for a group of pixels. As another example, a frame auto-exposure value can refer to the average brightness of some or all pixels or groups of pixels included within an image frame, such that the correspondence between frame auto-exposure values ​​and image frames is similar to the correspondence between pixel auto-exposure values ​​and specific pixels or groups of pixels.

[0038] In these examples, it should be understood that the average brightness (and / or one or more other average exposure-related characteristics in some examples) referred to by the auto exposure value can be determined as any type of average that can serve a particular implementation. For example, the average auto exposure value of an image frame can refer to the average brightness of the pixels in the image frame, which is determined by summing the corresponding brightness values ​​of each pixel or group of pixels in the image frame and then dividing the sum by the total number of values. As another example, the average auto exposure value of an image frame can refer to the median brightness of the pixels in the image frame, which is determined as the center brightness value when all corresponding brightness values ​​of each pixel or group of pixels are sorted by value. As yet another example, the average auto exposure value of an image frame can refer to the pattern brightness of the pixels in the image frame, which is determined as the most common or most frequently repeated brightness value among all corresponding brightness values ​​of each pixel or group of pixels. In other examples, other types of averages (other than average, median, or pattern values) and / or other types of exposure-related characteristics (other than brightness) can also be used to determine the auto exposure value in any way that can serve a particular implementation.

[0039] An auto-exposure target is understood to refer to a target (e.g., purpose, desired value, ideal value, optimal value, etc.) for the auto-exposure value of a specific image frame or a portion thereof (e.g., region, pixel, group of pixels, etc.). Apparatus 100 can determine the auto-exposure target based on a specific environment and any suitable criteria, and the auto-exposure target can be associated with the same auto-exposure-related characteristics (e.g., brightness, signal strength, etc.) represented by the auto-exposure value. For example, the auto-exposure target can be determined at a desired brightness level (or other exposure-related characteristic), such as a brightness level associated with intermediate gray levels. Therefore, a pixel auto-exposure target can refer to the desired target brightness determined for a single pixel or the average desired target brightness determined for a group of pixels in an implementation where pixels are grouped together as pixel cells in a grid. As another example, a frame auto-exposure target can refer to the average desired target brightness of some or all pixels or groups of pixels included within an image frame; therefore, the auto-exposure target corresponding to an image frame can be represented in a manner similar to how a pixel auto-exposure target corresponds to a specific pixel or group of pixels. Similarly, as described above regarding how to determine frame auto exposure values, the frame auto exposure target in such examples can be determined by averaging individual pixel auto exposure targets using the average, median, mode value, or other suitable averaging techniques.

[0040] As described above, various problems (e.g., overexposure, channel saturation, etc.) can arise when an image frame is significantly deviated or shifted to a particular color (e.g., a low-brightness color, such as red) and the auto-exposure target is determined without considering this color shift. Therefore, the original frame auto-exposure target determined at operation 206 based on the aforementioned criteria (e.g., targeting mid-gray, etc.) can be adjusted (e.g., scaled) to account for the image frame's color deviation, as quantified by the color cast metric determined at operation 202 and defined by the adaptive target control function whose output is determined at operation 204. Specific examples of how to determine the frame auto-exposure target based on the color cast metric and the adaptive target control function will be provided in more detail below.

[0041] At operation 208, device 100 may update (e.g., adjust or maintain) one or more automatic exposure parameters for use by the image capture system to capture one or more additional image frames. In some examples, device 100 may update one or more automatic exposure parameters based on the automatic exposure value of the image frame pixels, the automatic exposure target, and / or other automatic exposure data points, since these data points have been determined (e.g., at operation 206). For example, assuming device 100 has already determined the automatic exposure value (frame automatic exposure value) and the automatic exposure target (frame automatic exposure target) of the image frame, device 100 may update one or more automatic exposure parameters at operation 208 based on the frame automatic exposure value and / or the frame automatic exposure target. For example, device 100 may determine the automatic exposure gain (frame automatic exposure gain) of the image frame based on the frame automatic exposure value and the frame automatic exposure target, and may perform an update of one or more automatic exposure parameters based on the frame automatic exposure gain.

[0042] Device 100 can update the automatic exposure parameters based on automatic exposure gain adjustment parameters or appropriate maintenance parameters. In this way, the image capture system can capture one or more additional image frames (e.g., subsequent image frames in a sequence of image frames being captured) using automatic exposure parameters (e.g., exposure time parameters, shutter aperture parameters, illumination intensity parameters, analog and / or digital gain of the image signal, etc.) that can reduce any difference between the automatic exposure value detected in these additional image frames and the desired automatic exposure target for these additional image frames. Therefore, additional image frames can be captured with more ideal exposure characteristics compared to capture without such adjustment, and the user of device 100 can experience superior images (e.g., images displaying details at the desired brightness level, etc.).

[0043] Apparatus 100 may be implemented by one or more computing devices or by computing resources of a general-purpose or special-purpose computing system, as will be described in more detail below. In some embodiments, one or more computing devices or computing resources implementing apparatus 100 may be communicatively coupled to other components, such as an image capture system for capturing image frames that apparatus 100 is configured to process. In other embodiments, apparatus 100 may be included within (e.g., implemented as part of) an automatic exposure management system. Such an automatic exposure management system may be configured to perform all the same functions described herein that will be performed by apparatus 100 (e.g., including the operation of method 200 described above), but may further incorporate additional components such as an image capture system so that functions associated with these additional components can also be performed.

[0044] Figure 3An illustrative automatic exposure management system 300 (system 300) for managing automatic exposure of image frames is shown. As shown, system 300 may include an implementation of device 100 and an image capture system 302, which includes an illumination source 304 and an image capture device 306 incorporating a shutter 308, an image sensor 310, and a processor 312 (e.g., one or more image signal processors implementing an image signal processing pipeline). Within system 300, device 100 and image capture system 302 are communicatively coupled to allow device 100 to guide image capture system 302 according to the operation described herein, and to allow image capture system 302 to capture and provide device 100 with a sequence of image frames 314 and / or other suitable captured image data. The components of image capture system 302 will now be described.

[0045] Illumination source 304 can be implemented to provide any type of illumination source (e.g., visible light, infrared or near-infrared light, fluorescence excitation light, etc.) and can be configured to interoperate with image capture device 306 within image capture system 302. For example, illumination source 304 can provide a certain amount of illumination to a scene so that image capture device 306 can capture an optimally illuminated image of the scene. As previously mentioned, while the principles described herein can be applied to a wide variety of imaging scenarios, many of the examples explicitly described herein relate to medical procedures that can be performed using a computer-assisted medical system, as will be described in more detail below in conjunction with Figure 10. In such examples, the scene in which the image is captured may include an internal view of a body in which a medical procedure is being performed (e.g., the body of a live animal, a human or animal carcass, a portion of a human or animal anatomy, tissue removed from a human or animal anatomy, non-tissue artifacts, training models, etc.), and system 300 or certain components thereof (e.g., image capture system 302) may be integrated with the computer-assisted medical system (e.g., through its imaging and computational resources). In these examples, the specific color to which the captured image frame is offset may include red (e.g., the color associated with blood, which is commonly seen in views of the inside of the body).

[0046] Image capture device 306 can be implemented by any suitable camera or other device configured to capture images of a scene. For example, in a medical procedure example, image capture device 306 can be implemented by an endoscopic image capture device configured to capture an image frame sequence 314, which may include image frames depicting views of the body undergoing the medical procedure (e.g., internal views). As shown, image capture device 306 may include components such as shutter 308, image sensor 310, and processor 312.

[0047] Image sensor 310 can be implemented by any suitable image sensor, such as a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.

[0048] Shutter 308 can interoperate with image sensor 310 to help capture and detect light from a scene. For example, shutter 308 can be configured to expose image sensor 310 to a certain amount of light for each captured image frame. Shutter 308 may include an electronic shutter and / or a mechanical shutter. Shutter 308 can control how much light image sensor 310 is exposed to by opening to a specific aperture size defined by shutter aperture parameters and / or opening for a specified amount of time defined by exposure time parameters. As will be described in more detail below, these shutter-related parameters may be included in the automatic exposure parameters configured to be updated by device 100.

[0049] Processor 312 may be implemented by one or more image signal processors configured to implement at least a portion of an image signal processing pipeline. Processor 312 may process automatic exposure statistical inputs (e.g., detecting and processing various automatic exposure data points and / or other statistical data by clicking signals in the middle of the pipeline), perform optical artifact correction on data captured by image sensor 310 (e.g., correcting defective pixels, correcting lens shading problems, etc. by reducing fixed pattern noise), perform signal reconstruction operations (e.g., white balance operations, de-mosaicing and color correction operations, etc.), apply analog and / or digital gains to the image signal, and / or perform any other functions that may serve a particular implementation. Various automatic exposure parameters may indicate how the functions of processor 312 are performed. For example, automatic exposure parameters may be set to define the analog and / or digital gains applied by processor 312, as will be described in more detail below.

[0050] In some examples, the endoscopic implementation of image capture device 306 may include a stereoscopic endoscope comprising two complete sets of image capture components (e.g., two shutters 308, two image sensors 310, etc.) to accommodate the stereoscopic differences between the viewer's two eyes (e.g., left and right eyes) when presenting the captured image frame. Conversely, in other examples, the endoscopic implementation of image capture device 306 may include a single-mirror endoscope with a single shutter 308 and a single image sensor 310, etc.

[0051] The device 100 can be configured to control various automatic exposure parameters of the image capture system 302, and can adjust such automatic exposure parameters in real time based on incoming image data captured by the image capture system 302. As described above, some automatic exposure parameters of the image capture system 302 can be associated with the shutter 308 and / or the image sensor 310. For example, the device 100 can guide the shutter 308 according to an exposure time parameter corresponding to the length of time the shutter allows the image sensor 310 to be exposed to the scene, a shutter aperture parameter corresponding to the aperture size of the shutter 308, or any other suitable automatic exposure parameter associated with the shutter 318. Other automatic exposure parameters can be associated with aspects of the image capture system 302 or with the image capture process independent of the shutter 308 and / or the sensor 310. For example, the device 100 can adjust an illumination intensity parameter of the illumination source 304 corresponding to the illumination intensity provided by the illumination source 304, an illumination duration parameter corresponding to the time period during which the illumination source 304 provides illumination, etc. As yet another example, device 100 may adjust gain parameters corresponding to one or more analog and / or digital gains (e.g., analog gain, Bayer gain, RGB gain, etc.) applied by processor 312 to the luminance data generated by image sensor 310.

[0052] Any one or any combination of these or other suitable parameters may be updated and / or otherwise adjusted by device 100 based on analysis of the current image frame for use in subsequent image frames. For example, in an example where the frame auto-exposure gain (e.g., frame auto-exposure target divided by frame auto-exposure value) is determined to be 6.0, various auto-exposure parameters may be set as follows: 1) the current illumination intensity parameter may be set to 100% (e.g., maximum output); 2) the exposure time parameter may be set to 1 / 60 second (e.g., 60 fps); 3) the analog gain may be set to 5.0 (upper limit 10.0); 4) the Bayer gain may be set to 1.0 (upper limit 3.0); and 5) the RGB gain may be set to 2.0 (upper limit 2.0). Using these settings, the gain is distributed between analog gain (10.0 / 5.0 = 2.0), Bayer gain (3.0 / 1.0 = 3.0), and RGB gain (2.0 / 2.0 = 1.0) to establish the total auto exposure gain of 6.0 (3.0 * 2.0 * 1.0 = 6.0) required for the frame.

[0053] Figure 4Illustrative image frames 402 (e.g., image frames 402-A to 402-C) and corresponding color cast characteristics 404 (e.g., color cast characteristics 404-A to 404-C) are shown. More specifically, as illustrated, a first image frame 402-A represents an illustrative image frame depicting content that is relatively neutral in terms of color offset (e.g., the content does not show a significant offset to any particular color compared to any other color). A second image frame 402-B represents an illustrative image frame depicting content that is offset to a particular color to a limited extent compared to other colors. A third image frame 402-C represents an illustrative image frame depicting content that is offset to a particular color to a greater extent. For example, image frames 402-B and / or 402-C could represent image frames depicting an internal view of a body, to which the particular color offset (albeit to different degrees) could be red, associated with blood depicted in the internal view of the body.

[0054] Although Figure 4 The various numbers 1-9, depicted as a black and white image but used to fill the various shapes and background areas depicted in image frame 402, will be interpreted as representing different colors. In this number-based color notation, numbers that are close to each other (e.g., 1 and 2, 8 and 9, etc.) will be interpreted as representing similar colors (e.g., red and reddish-orange, green and yellowish-green, etc.), while numbers that are far from each other (e.g., 1 and 8, 2 and 9, etc.) will be interpreted as representing more distinct colors (e.g., red and green, blue and orange, etc.). In some examples, this number-based notation can be interpreted as encircling, such that the number 1 is considered adjacent to the number 9, and the color represented by the number 1 is similar to the color represented by the number 9.

[0055] A graph illustrating the color cast characteristic 404 is plotted next to each image frame 402. In these graphs, the x-axis represents the different colors represented by the numbers 1-9, while the y-axis represents the degree to which the image frame is shifted towards these colors. Thus, as shown by the color cast characteristic 404-A corresponding to image frame 402-A, image frame 402-A is not particularly shifted to any color (e.g., as evidenced by the absence of any major peaks or valleys in the graph curves), but is rather neutral (e.g., composed of all colors along the spectrum, as evidenced by the relatively flat graph curves).

[0056] Conversely, as shown by color cast characteristic 404-B corresponding to image frame 402-B, image frame 402-B does indeed shift to colors in the upper part of the color spectrum (e.g., colors represented by numbers around 5-8), as evidenced by the valleys on the left and right sides of the curve and the peaks in the upper middle part of the curve. For example, if the number 5 represents red, then color cast characteristic 404-B could indicate that image frame 402-B includes a large amount of red and orange, and possibly yellow, but not necessarily a large amount of green and blue, etc.

[0057] More obviously, as shown by the color cast characteristic 404-C corresponding to image frame 402-C, image frame 402-C can be significantly shifted towards colors in the middle of the spectrum (e.g., colors represented by numbers 4-6, and especially number 5), as evidenced by the prominent peak in the middle of the graph curve and the valleys along other parts of the graph curve. In this example, if number 5 again represents red, then color cast characteristic 404-C indicates that image frame 402-C is highly shifted or deviated towards various shades of red (e.g., red, pink, reddish-orange, reddish-purple, etc.), which could be the case for an internal view of the body depicting blood and bloody tissue prevalent in a medical procedure scene.

[0058] The automatic exposure management described herein can be configured not to apply special color shifts to relatively neutral image frames (such as image frame 402-A). However, as will be described in more detail below, the automatic exposure management described herein can take into account the relatively mild color shift of image frame 402-B and the more significant color shift of image frame 402-C in a manner that gradually considers the color shift when determining the frame automatic exposure target for these image frames. Ultimately, considering the frame automatic exposure target in this way may result in more beneficial updates to the automatic exposure parameters of the image capture system that captures image frame sequences in these color shift scenarios.

[0059] Figure 5 An illustrative flowchart 500 is shown for managing automatic exposure of image frames using implementations such as apparatus 100, method 200, and / or system 300. As shown, flowchart 500 illustrates various operations 502-514, each of which will be described in more detail below. It should be understood that operations 502-514 represent one embodiment, and other embodiments may omit, add, reorder, and / or modify any of these operations. As will be described, the various operations 502-514 of flowchart 500 may be performed for one or more image frames (e.g., each image frame) in an image frame sequence. It should be understood that, depending on various conditions, not every operation can be performed for every frame, and the combination and / or order of operations performed from frame to frame in the image frame sequence may differ.

[0060] At operation 502, an image frame captured by the image capture system can be obtained (e.g., accessed, loaded, captured, generated, etc.). As previously described, in some examples, the image frame may be an image frame depicting image content with at least some degree of color deviation (e.g., a very small degree of deviation in the case of neutral content, and a relatively large degree of deviation in the case of content severely offset to a particular color). For example, the obtained image frame may resemble any of the image frames 402 described above. Operation 502 can be performed in any suitable manner, such as by accessing the image frame from the image capture system (e.g., in the case where operation 502 is performed by an implementation of means 100 communicatively coupled to the image capture system) or by capturing the image frame using an integrated image capture system (e.g., in the case where operation 502 is performed by an implementation of system 300 including an integrated image capture system 302).

[0061] At operation 504, device 100 can determine a color cast metric of the image frame acquired at operation 502. For example, as described above, the color cast metric can be a quantitative representation indicating the degree to which the image frame is offset towards a particular color. In some implementations, the automatic exposure management being performed can be configured to identify an offset towards a single specific color (e.g., red in an implementation used for endoscopic images typically depicting blood in a medical setting). Therefore, the color cast metric in these examples can indicate the degree to which the image frame is offset towards that specific color. In other implementations, the automatic exposure management being performed can be configured to identify an offset towards any color (rather than a specific color). Therefore, the color cast metric in these examples can indicate the degree to which the image frame is offset towards any one color (e.g., the opposite of a neutral degree of the image frame relative to many colors). Figure 5 As shown, Figures 6-8 The illustration further illustrates how color shift is analyzed and quantified when the color cast metric is determined at operation 504.

[0062] Figure 6 An illustrative technique 600 for determining a color cast metric for an image frame is shown, and further reference is made to the following description. Figure 7 and Figure 8The concept is illustrated. As shown, color data 602 can be used as input to technique 600. Color data 602 can be associated with an image frame obtained at operation 502 described above, and can be represented in any suitable color space (e.g., color data format), such as the red-green-blue (RGB) color space, used by the image capture system when the image frame is captured and provided to device 100. Color data 602 can then be processed in multiple stages (e.g., normalization color data stage 604, decomposition color data stage 606, and / or quantization offset stage 608) so that, ultimately, color data 602 can form the basis of frame color cast metric 610, which is the output of technique 600.

[0063] At stage 604, device 100 can normalize color data 602 to avoid bias caused by brightness scaling. Regardless of how color data 602 is represented, it can represent luminance data (associated with the brightness of each pixel, regardless of color) and chromaticity data (associated with the color of each pixel, regardless of brightness). In some color spaces and color data formats, luminance and chromaticity data are represented separately, while in other color spaces and color data formats, these concepts are represented in a way that combines them, which can provide convenience in image capture, image rendering, etc. An example of a color space that combines chromaticity and luminance concepts in its data representation is the RGB color space, which uses three values ​​to represent chromaticity and luminance characteristics: an "R" value associated with the red channel, a "G" value associated with the green channel, and a "B" value associated with the blue channel. In the RGB color space, if pixels have unequal brightness, two pixels that are both red in terms of chromaticity characteristics can have different "R" values.

[0064] By normalizing the color data at stage 604, the color data represented in a color space such as RGB can be effectively placed on a uniform field according to the luminance characteristics of each pixel, allowing analysis of the chromaticity characteristics of each pixel regardless of luminance characteristics. A color cast metric determined by technique 600 may be needed to determine the degree to which pixels and frames are offset to a particular chromaticity, regardless of luminance characteristics. For example, it may be necessary to treat bright white pixels as neutral pixels and dark red pixels as red-offset pixels, even though in a color space such as RGB, bright white pixels may be associated with a larger "R" value than dark red pixels (simply due to their different luminance characteristics). One way to normalize color data in the RGB color space is to determine the maximum values ​​of the "R", "G", and "B" values ​​and generate normalized values ​​based on these. For example, in the example of bright white and dark red pixels above, the maximum value in the dark red pixel may be red ("R" value) with a significant margin, but the bright white pixel may not have a maximum value (or at least no significant margin) because in this case, the "R", "G", and "B" values ​​may be very similar.

[0065] At stage 606, device 100 can decompose normalized color data to distinguish between the chromaticity attribute and the luminance attribute of the color data. For example, the normalized color data for each pixel can be decomposed from the aforementioned RGB color space into different color spaces that consider primary, secondary, and / or tertiary colors, and respectively consider luminance characteristics. As an example, the decomposition of color data at stage 606 may include converting the color data from the RGB color space to the Cyan-Magenta-Yellow-Black-Red-Green-Blue (CMYKRGB) color space. As another example, the decomposition of color data at stage 606 may include converting the color data to the YUV color space, the CIELAB color space, or another suitable color space that allows for convenient analysis of the color data in an exposure-independent manner (e.g., in a manner that separately represents luminance and chromaticity characteristics).

[0066] To further illustrate these concepts, Figure 7 An illustrative diagram 700 depicts the exposure-independent chromaticity characteristics of the decomposed color data. As shown, illustrative diagram 700 includes a barycentric plot of the color data converted to a color space (such as CMYKRGB). It should be understood that similar diagrams for other suitable color spaces (e.g., YUV color space, CIELAB color space, etc.) can illustrate similar principles as shown in the CMYKRGB diagram here, since each of these color spaces also allows for convenient analysis of chromaticity attributes independently of luminance attributes.

[0067] exist Figure 7The example illustrates simulated human tissue colors to illustrate the intended drawing of an endoscopic image frame that can be used to depict a view of the body's interior. In illustrative Figure 700, a neutral point 702 is shown at the center of the drawing to represent a neutral pixel (e.g., black, white, or any shade of gray), so that it is not more offset from a primary or secondary color (e.g., red, magenta, blue, cyan, green, or yellow) compared to any other color. Illustrative Figure 700 also shows a point 704 marked as a red vertex to represent a pure red pixel and not including any aspect of any other primary or secondary color. As shown, a large number of corresponding points 706 illustrate where each pixel of a particular image frame depicting a view of the body's interior can be drawn within illustrative Figure 700. In fact, each point in this example is shown as being offset to some extent from the red vertex (e.g., clustered along a neutral red hue line connecting points 702 and 704), rather than exhibiting a strong characteristic associated with other primary or secondary colors. Therefore, the image frame represented by the explanatory figure 700 can be represented as a very reddish image frame, such as the image that can be expected to be captured by the endoscope inside the body.

[0068] Back Figure 6 At stage 608, device 100 can quantify the color shift to determine frame color cast metric 610. For example, device 100 can determine the degree to which the chromaticity attribute of the color data shifts towards a specific color, and based on the determination of the degree to which the chromaticity attribute of the color data shifts towards the specific color, frame color cast metric 610 can be determined. As an example, based on Figure 7 The image frame shown in Figure 700 can be analyzed in stage 608 to analyze the chromaticity characteristics of each pixel of the image frame associated with each point 706, thereby quantifying the redness of the image frame in an objective manner (e.g., determining an objective measurement of the degree of redness shift in the image frame in this example).

[0069] To illustrate how this quantization determination is performed at stage 608 Figure 8 An illustrative flowchart 800 for quantifying color offset is shown as part of determining a color cast metric. As shown, flowchart 800 includes multiple operations 802-808 that can be performed between device 100 reaching stage 608 (labeled START) and flowchart 800 completing and determining frame color cast metric 610 (labeled END).

[0070] At operation 802, device 100 can iterate through each pixel or a portion of the image frame. For each pixel i, a pixel color cast metric (CSM) can be determined at operation 804. i And weight values ​​(W) can be assigned at operation 816. iAs shown in the figure, this process can continue as long as there are still unanalyzed pixels (incomplete) in the image frame, and can end when all pixels in the image frame have been traversed (completed). As mentioned earlier, in some examples, pixels can be processed in groups (e.g., pixels are divided into grid cells, etc.) instead of individually. Therefore, instead of determining pixel color cast metric and weight values ​​for each individual pixel, it should be understood that these implementations can determine pixel color cast metric and weight values ​​for each group of pixels. Furthermore, in some implementations, for the purpose of automatic exposure management, instead of traversing all pixels (or groups of pixels) in the image frame, a certain region of the image frame (e.g., the central region of the image frame, such as the central 50% of the image frame, the central 80% of the image frame, etc.) can be considered, while another region of the image frame (e.g., the peripheral region of the image frame, such as the outer 50% of the image frame, the outer 20% of the image frame, etc.) can be ignored. In these examples, operation 802 completes the traversal (completes) when all pixels in the region to be considered (e.g., the central region) have been traversed.

[0071] At operation 804, device 100 can determine a pixel color cast metric (e.g., one pixel color cast metric per pixel) for the pixels included in the image frame in any suitable manner. The pixel color cast metric determined at operation 804 can be an objective quantitative measure of the degree to which a pixel is offset towards a particular color (e.g., red, etc.). For example, each pixel (or group of pixels) can be associated with a specific point 706 as shown in illustrative figure 700, and the pixel color cast metric can be determined based on the linear distance from the specific point 706 to the neutral point 702 (e.g., the greater the distance, the greater the pixel color cast metric) to the red point 704 (e.g., the smaller the distance, the greater the pixel color cast metric), combinations thereof, etc. As previously stated, red is merely an example; therefore, in other implementations, the pixel color cast metric can be determined based on the proximity of point 706 to another color point, not the red point 704. Furthermore, as previously stated, Figure 7 The CMYKRGB color space shown should be understood as merely an example of a color space; the pixels of an image frame can be decomposed into the color space and plotted as points 706. Therefore, other suitable color spaces (e.g., YUV, CIELAB, etc.) that represent chromaticity characteristics in a similar manner to luminance separation can also be used to geometrically determine how close a pixel is to a specific color within the color space.

[0072] At operation 806, a weight value can be assigned to each pixel (e.g., each pixel i) of a plurality of pixels in an image frame or a region thereof based on the spatial location of the plurality of pixels within the image frame. The weight value assigned at operation 806 can reflect how likely each pixel is to be within a viewer's region of interest in the image frame, and therefore how important each pixel is considered relative to other pixels in the image frame. For example, in some implementations, it can be assumed that the viewer may focus their attention near the center of the image frame, so each pixel can be assigned a weight indicating its proximity to the center of the image frame (e.g., a higher weight value indicates closer to the center, and a lower weight value indicates farther from the center). As another example, one implementation may include eye-tracking functionality to determine in real time which part of the image frame the viewer is focusing on, and each pixel can be assigned a weight indicating its proximity to the detected real-time region of interest (e.g., not the center of the image frame or other than the center of the image frame). In other examples, pixels can be weighted according to other spatial location-based criteria (e.g., proximity to a moving object of interest within the image frame, proximity to another assumed region of interest in the image frame other than the center, etc.) or non-spatial location-based criteria. Alternatively, each pixel can be treated as equally important, regardless of its spatial location, and no weight value can be assigned (e.g., operation 806 can be omitted entirely).

[0073] Because operations 804 and 806 may not depend on each other, these operations can be performed independently, in any order, or in parallel with each other for each pixel (or group of pixels) i. In some examples, even if the pixel color cast metric can be dynamically re-evaluated and determined for each image frame at operation 804, the assignment of weight values ​​for each pixel at operation 806 may only be performed once (e.g., assigning static values ​​for each image frame or sequence of image frames). In other examples, operations 804 and 806 can be performed dynamically for each image frame.

[0074] Once all pixels i (or a portion thereof) of the image frame have been traversed at operation 802, the process can continue (complete) to operation 808. At operation 808, device 100 can determine the frame color cast metric 610 as a weighted average of the pixel color cast metrics. For example, as shown in the input of operation 808, it can be based on the pixel color cast metric (CSM). i ) and the assigned weight values ​​(W) iThe frame color cast metric 610 is determined using a weighted average. Just as each pixel color cast metric can correspond to the degree to which a pixel (or group of pixels) is shifted toward a particular color, the frame color cast metric 610 can correspond to the degree to which an image frame is determined to be shifted toward a particular color. Because a weighted average is used, the frame color cast metric can indicate a greater degree of color shift toward a particular color than that of lower-weighted pixels (e.g., peripheral pixels). The weighted average can be calculated using any suitable averaging method of any type (e.g., arithmetic mean, median mean, pattern mean, combinations thereof, etc.) and can be combined with weight values ​​to give higher-weighted pixels a greater weight than lower-weighted pixels in any way that serves a particular implementation. As mentioned earlier, weight values ​​may not be used, or pixels in an image frame (or a portion of an image frame) may use the same weight value (e.g., 1). In these scenarios, the frame color cast metric 610 can depend on the pixel color cast metric (CSM). i However, it may not depend on the weight value (W). i ).

[0075] return Figure 6 The frame color cast metric 610 is shown as the output of technique 600 used to determine the color cast metric based on the quantization of the color offset performed at stage 608. Therefore, the return... Figure 5 Once the frame color deviation metric 610 is determined, the device 100 can proceed from operation 504 to operation 506.

[0076] At operation 506, device 100 can apply an adaptive target control function to the frame auto-exposure target based on the frame color cast metric 610 determined at operation 504. For example, if the frame color cast metric 610 indicates that the image frame is offset to a particular color to a relatively low degree (low offset), applying the adaptive target control function at operation 506 allows device 100 to determine the frame auto-exposure target at operation 508 without making any special allowances for color deviations in the image frame. Conversely, if the frame color cast metric 610 indicates that the image frame is offset to a particular color to a relatively high degree (high offset), applying the adaptive target control function at operation 506 allows device 100 to determine the frame auto-exposure target at operation 510, where color deviations in the image frame can be taken into account and restoration measures can be taken. In some examples, a frame color cast metric 610 below a threshold color cast metric may indicate a low offset, and / or a frame color cast metric 610 above a threshold color cast metric may indicate a high offset.

[0077] To explain the functions of operating 506-510 in more detail, Figure 9An illustrative technique 900 for determining a frame auto-exposure target based on a frame color cast metric is shown. Specifically, as shown, operations 902-906 can be performed to generate a frame auto-exposure target 908 based on a frame color cast metric 610, which helps to take into account the degree of color deviation of the frame in order to provide the benefits of auto-exposure and avoid the auto-exposure problems described.

[0078] At operation 902, device 100 may determine a first (e.g., original) frame auto-exposure target based on image frames captured by the image capture system. For example, the original frame auto-exposure target may be determined based on the average or weighted average of pixel auto-exposure targets, which are determined to be targets for auto-exposure values ​​associated with mid-grayscale, etc.

[0079] At operation 904, device 100 can determine a scaling value as the output of an adaptive target control function (e.g., a predetermined adaptive target control function) given an input of a color cast metric (e.g., frame color cast metric 610). The adaptive target control function can refer to a function designed to map various potential color cast metrics to various desired auto-exposure targets. For example, in some implementations, the adaptive target control function can take a frame color cast metric as input and output a scaling value that can be used to scale the original frame auto-exposure target to produce a second (e.g., desired) frame auto-exposure target. The second frame auto-exposure target can take color deviation into account, thereby avoiding the aforementioned exposure problem.

[0080] Figures 10A-10D Exemplary implementations of various adaptive target control functions are shown, which can be used to consider color deviation content in various ways. In each illustrated adaptive target control function 1002 (e.g., Figure 10A The adaptive target control function 1002-A to Figure 10D In the adaptive target control function 1002-D, the input color cast metric (color cast metric) is represented along the x-axis, where lower color cast metric values ​​are represented towards the left of the axis (low), and higher color cast metric values ​​are represented towards the right of the axis (high). The output scaling value of the adaptive target control function is then represented along the y-axis. Therefore, the frame color cast metric 610 determined in the above manner can be plotted along the x-axis, allowing the output of the adaptive target control function 1002 to be determined based on the y-values ​​returned by the function of the plotted x-values.

[0081] As shown in each adaptive target control function 1002, if the input color cast metric is sufficiently low (e.g., below a threshold color cast metric), the returned scaling value can be a large scaling value, referred to herein as a null scaling value. A null scaling value may have no effect (e.g., no effect) when used to scale the original frame auto-exposure target. For example, if scaling the original frame auto-exposure target is performed by multiplying the original frame auto-exposure target by the scaling value, the null scaling value can have a value of 1 so that multiplying with that value does not change the original frame auto-exposure target. However, as shown in each adaptive target control function 1002, if the input color cast metric is sufficiently high (e.g., above a threshold color cast metric), the returned scaling value can be a scaling value, referred to herein as a reductive scaling value. A reductive scaling value can have a reductive effect when used to scale the original frame auto-exposure target. For example, if scaling the original frame auto-exposure target is performed by multiplying the original frame auto-exposure target by the scaling value, the reductive scaling value can have a value less than 1 (e.g., between 0 and 1) so that multiplying with that value restores the original frame auto-exposure target. For an image frame with the largest color cast metric (e.g., a pure red image frame if the specific color is red), it might be desirable to autoexpose the original frame to the maximum amount needed to restore the target image frame. Figures 10A-10D In this context, the maximum restore scaling value is marked as Full and is indicated as the highest color cast metric to be applied. Depending on the implementation, the Full restore value can be defined as any suitable level between zero and an empty scaling value (e.g., between 0 and 1).

[0082] like Figure 10A and Figure 10B Examples of the adaptive target control functions 1002-A and 1002-B are shown below. Different parts of the adaptive target control function can return an empty scaling value (the minimum scaling amount to produce the original frame auto-exposure target) or a full scaling value (the maximum scaling amount to produce the original frame auto-exposure target). Specifically, as shown in the figure, if the color deviation of the content depicted in the image frame is insufficient for special processing, the color cast metric given as input may not exceed the color cast threshold 1004-1, and the output returned by the adaptive target control function 1002 can be an empty scaling value. Conversely, if the color deviation of the content depicted in the image frame is severe, the color cast metric given as input may exceed the color cast threshold 1004-2, and the output returned by the adaptive target control function 1002 can be a full scaling value.

[0083] Between these extreme values, adaptive target control functions 1002-A and 1002-B represent the middle portion of a gradually restored scaling value that can be returned for a color cast metric that exceeds threshold 1004-1 but does not exceed threshold 1004-2. For example, if device 100 determines that the color cast metric given as input exceeds color cast threshold 1004-1 but does not exceed color cast threshold 1004-2, the output of adaptive target control function 1002-A or 1002-B can be a restored scaling value that is less than the empty scaling value and greater than the fully restored scaling value. For example, in these scenarios, the output of the adaptive target control function can be determined based on a decreasing function (such as a monotonically decreasing function). Figure 10A The adaptive target control function 1002-A in the figure is shown as a monotonically decreasing nonlinear function (e.g., a monotonically decreasing power function), while Figure 10B The adaptive target control function 1002-B in the example is shown as a monotonically decreasing linear function. In other examples, other types of monotonically decreasing functions (or non-monotonic functions in some implementations) can be used to provide results similar to those provided by the adaptive target control functions 1002-A and 1002-B.

[0084] In some examples, device 100 can determine a color bias metric corresponding to a value selected from a set of discrete values. For example, instead of a color bias metric represented as a fraction or floating-point number, a color bias metric can be implemented as a discrete value, such as a binary value (e.g., 0 representing a negligible color bias, 1 representing a significant color bias), or a value selected from a set of discrete values ​​(e.g., values ​​0, 1, 2, 3, and 4 representing a range from no color bias (0) to a very high color bias (4), etc.). In these examples, an adaptive target control function can map each discrete color bias metric possibility to a corresponding scaling value in the adaptive target control function's output for the color bias metric.

[0085] To illustrate, Figure 10C The adaptive target control function 1002-C illustrates a binary example where image frames with any amount of color offset less than the color cast threshold 1004-3 are assigned a first binary color cast metric (e.g., a low or zero color cast metric), and image frames with any amount of color offset greater than the color cast threshold 1004-3 are assigned a second binary color cast metric (e.g., a high or 1 color cast metric). In this example, the adaptive target control function 1002-C returns an empty scaling value for the low color cast metric and a full scaling value for the high color cast metric. Similarly, Figure 10DThe Adaptive Target Control Function 1002-D illustrates another discrete-value implementation, where the color cast metric is assigned one of five distinct values ​​based on where the color offset of the image frame falls relative to several color cast thresholds 1004-4 to 1004-7. In this example, the Adaptive Target Control Function 1002-D returns the empty scaling value for the lowest color cast metric, the full scaling value for the highest color cast metric, and varies other reconstructive scaling values ​​between the empty and full scaling values ​​for the other potential color cast metrics.

[0086] return Figure 9 After determining the scaling value at operation 904 based on any adaptive target control function 1002 that can be implemented in a particular embodiment, device 100 can perform operation 906. At operation 906, device 100 can determine frame auto-exposure target 908 by scaling the original frame auto-exposure target determined at operation 902 to the scaling value determined at operation 904. For example, if the scaling value determined at operation 904 is an empty scaling value, the frame auto-exposure target can be determined at operation 906 to be equal to the original frame auto-exposure target determined at operation 902. Conversely, if the scaling value determined at operation 904 is a reverted scaling value (e.g., reaching and including the full scaling value), the frame auto-exposure target can be determined at operation 906 to be equal to the original frame auto-exposure target reverted based on the reverted scaling value determined at operation 902. In this way, frame auto-exposure target 908 can be reverted to account for color deviations when the image frame depicts content that is offset to some extent.

[0087] While the example above describes a scenario where the frame auto-exposure target 908 is determined based on an unweighted raw frame auto-exposure target and a weighted frame color cast metric, it should be understood that the same or similar frame auto-exposure can be determined in alternative ways that may be employed in some implementations. As an illustrative alternative, instead of using the frame color cast metric 610 (CSM... f The weighted average is determined, and then used to determine the scaling value to be mapped to the unweighted original frame auto-exposure target. The device 100 can determine the scaling value at the pixel level and determine the weighted average of the pixel auto-exposure target that has been scaled to account for color shift. For example, this can be done for each pixel (or group of pixels) I and the pixel color shift metric CSM. i Determine the pixel auto-exposure target. Then, a scaling value can be determined for each pixel (or group of pixels) based on a pixel color cast metric and applied to the corresponding pixel auto-exposure target. The frame auto-exposure target can then be determined based on a weighted average of these scaled pixel auto-exposure targets (which already accounts for the color shift at each pixel). In other implementations, the frame auto-exposure target, which considers color shift and pixel weights, can be determined using alternative methods that may be suitable for a particular implementation.

[0088] return Figure 5 Operation 512 can be performed independently of and in parallel with operations 504-510. At operation 512, apparatus 100 can determine the frame auto-exposure value in any manner suitable for a particular implementation. For example, apparatus 100 can determine the pixel auto-exposure value for each pixel (or a group of pixels in some implementations), and then determine the frame auto-exposure value by averaging the pixel auto-exposure values ​​according to any suitable averaging technique (e.g., average, median, mode, etc.). In some examples, a weighted average of the pixel auto-exposure values ​​can be calculated for the frame auto-exposure value, or the average of pixels included only in a specific region (e.g., the central region excluding the peripheral portion of the image frame) can be used.

[0089] Once the frame auto-exposure value is determined at operation 512 and the frame auto-exposure target is determined at operation 508 or 510, the process can proceed to operation 514, where device 100 can update the auto-exposure parameters of the image capture system based on the frame auto-exposure value and the frame auto-exposure target. At operation 514, device 100 can update (e.g., adjust or maintain) the auto-exposure parameters of the image capture system to prepare for the image capture system to capture subsequent image frames in the image frame sequence.

[0090] Figure 11 An illustrative technique 1100 for updating the auto exposure parameters at operation 514 is shown. As shown, the frame auto exposure target and frame auto exposure value are previously determined and used as... Figure 11 The inputs to the operation are shown. For example, operation 1102 can receive a frame auto-exposure value and a frame auto-exposure target as inputs and use them as the basis for determining the frame auto-exposure gain. The frame auto-exposure gain can be determined as the ratio corresponding to the frame auto-exposure target to the frame auto-exposure value. In this way, if the frame auto-exposure value is already equal to the frame auto-exposure target (e.g., so that no further adjustments are needed to align with the target), the frame auto-exposure gain can be set to gain 1 so that the system does not attempt to increase or decrease the auto-exposure value of subsequent frames captured by the image capture system. Conversely, if the frame auto-exposure target is different from the frame auto-exposure value, the frame auto-exposure gain can be set to a value corresponding to less than or greater than 1 so that the system increases or decreases the auto-exposure value of subsequent frames in an attempt to bring the auto-exposure value closer to the desired auto-exposure target.

[0091] At operation 1104, the frame auto-exposure gain can be used as input along with other data determined for previous image frames in the image frame sequence (e.g., other frame auto-exposure gains). Based on these inputs, operation 1104 applies filtering to ensure that the auto-exposure gain does not change faster than expected, thereby ensuring that the image frames presented to the user maintain consistent brightness and change gradually (e.g., by not changing faster than a threshold rate). The filtering performed at operation 1104 can be performed using a smoothing filter such as a time-infinite impulse response (IIR) filter or another such digital or analog filter that may serve a particular implementation.

[0092] At operation 1106, the filtered automatic exposure gain can be used as a basis for adjusting one or more automatic exposure parameters of the image capture system (e.g., for the image capture device or illumination source to capture additional image frames). For example, as described above, the adjusted automatic exposure parameters may include exposure time parameters, shutter aperture parameters, brightness gain parameters, etc. For image capture systems where the scene illumination is mainly or entirely controlled by the image capture system (e.g., image capture systems including the aforementioned endoscopic image capture device, image capture systems including flash lamps or other illumination sources, etc.), the adjusted automatic exposure parameters may further include illumination intensity parameters, illumination duration parameters, etc.

[0093] Adjusting the automatic exposure parameters of an image capture system allows it to expose subsequent image frames in various ways. For example, by adjusting the exposure time parameter, the shutter speed of the shutter included in the image capture system can be adjusted. For instance, the shutter can be kept open for a longer period (e.g., increasing the exposure time of the image sensor) or a shorter period (e.g., decreasing the exposure time of the image sensor). As another example, by adjusting the shutter aperture parameter, the shutter aperture can be adjusted to open wider (e.g., increasing the amount of light exposed to the image sensor) or smaller (e.g., decreasing the amount of light exposed to the image sensor). As yet another example, by adjusting the brightness gain parameter, sensitivity (e.g., ISO sensitivity) can be increased or decreased to amplify or attenuate the illumination captured by the image capture system. Regarding the implementation of scene lighting control in the image capture system, illumination intensity and / or illumination duration parameters can be adjusted to increase the intensity and duration of light used to illuminate the captured scene, thereby also affecting the amount of light exposed to the image sensor.

[0094] return Figure 5After executing the operations in flowchart 500, the current image frame can be considered to have been fully processed by device 100, and the process can return to operation 502, where subsequent image frames of the image frame sequence can be obtained. This process can be repeated for subsequent image frames and / or other subsequent image frames. It should be understood that in some examples, each image frame can be analyzed according to flowchart 500 to keep the auto-exposure data points (e.g., frame auto-exposure values ​​and frame auto-exposure targets, etc.) and auto-exposure parameters as up-to-date as possible. In other examples, such analysis can only be performed on certain image frames (e.g., every other image frame, every three image frames, etc.) to save processing bandwidth while still allowing design specifications and objectives to be achieved in more periodic auto-exposure processing. It should also be understood that the auto-exposure effect may tend to lag behind brightness changes in the scene by several frames because adjustments to auto-exposure parameters based on a specific frame do not affect the exposure of that frame but rather subsequent frames.

[0095] Based on any adjustments made by device 100 to the automatic exposure parameters (and / or based on maintaining the automatic exposure parameters at their current level when appropriate), device 100 can successfully manage the automatic exposure of image frames captured by the image capture system, and subsequent image frames can be captured with the desired automatic exposure characteristics so as to have an attractive and beneficial appearance when presented to the user.

[0096] As previously described, in some examples, device 100, method 200, and / or system 300 may each be associated with a computer-aided medical system for performing medical procedures (e.g., surgical procedures, diagnostic procedures, exploratory procedures, etc.) on the body. For illustration, Figure 12 An illustrative computer-aided medical system 1200 is shown that can be used to perform various types of medical procedures, including surgical and / or non-surgical procedures.

[0097] As shown in the figure, the computer-assisted medical system 1200 may include a manipulator component 1202. Figure 12The diagram shows a manipulator cart, user control unit 1204, and auxiliary device 1206, all of which are communicatively coupled to each other. A medical team can use the computer-assisted medical system 1200 to perform computer-assisted medical procedures or other similar operations on the body of the patient 1208 or on any other body that can serve a particular implementation. As shown, a medical team may include a first user 1210-1 (such as a surgeon for a surgical procedure), a second user 1210-2 (such as a patient-side assistant), a third user 1210-3 (such as another assistant, nurse, trainee, etc.), and a fourth user 1210-4 (such as an anesthesiologist for a surgical procedure), all of whom can be collectively referred to as users 1210, and each of them can control, interact with, or otherwise become a user of the computer-assisted medical system 1200. During a medical procedure, there may be more, fewer, or alternative users serving a particular implementation. For example, the team composition may differ for different medical or non-medical procedures and may include users with different roles.

[0098] Although Figure 12 The illustration depicts a minimally invasive medical procedure in progress, such as a minimally invasive surgical procedure; however, it should be understood that the computer-assisted medical system 1200 can be similarly used to perform open medical procedures or other types of operations. For example, it can also perform operations such as exploratory imaging, simulated medical procedures for training purposes, and / or other operations.

[0099] like Figure 12 As shown, the manipulator assembly 1202 may include one or more manipulator arms 1212 (e.g., manipulator arms 1212-1 to 1212-4), to which one or more instruments may be coupled. The instruments may be used in computer-aided medical procedures on a patient 1208 (e.g., in a surgical example, by insertion into and manipulation within the patient 1208 at least partially). Although the manipulator assembly 1202 is depicted and described herein as comprising four manipulator arms 1212, it should be recognized that the manipulator assembly 1212 may include a single manipulator arm 1202 or any other number of manipulator arms that may serve a particular implementation. Figure 12 The example shows manipulator arm 1212 as a robotic manipulator arm, but it should be understood that in some examples, one or more instruments may be partially or fully manually controlled, such as by hand and by human manual control. For example, these partially or fully manually controlled instruments may be coupled to... Figure 12 The computer-aided device shown in the diagram 1212 is used in conjunction with or as a substitute for the manipulator arm 1212.

[0100] During medical procedures, the user control device 1204 can be configured to facilitate remote operation and control of the manipulator arm 1212 and instruments attached to the manipulator arm 1212 by the user 1210-1. To this end, the user control device 1204 can provide the user 1210-1 with an image of the operating area associated with the patient 1208, captured by an imaging device. For ease of instrument control, the user control device 1204 may include a set of master controllers. These master controllers can be operated by the user 1210-1 to control the movement of the manipulator arm 1212 or any instruments coupled to the manipulator arm 1212.

[0101] The assistive device 1206 may include one or more computing devices configured to perform assistive functions to support medical procedures, such as providing air, electrocautery, lighting, or other energy to the imaging equipment, image processing, or coordination components of the computer-assisted medical system 1200. In some examples, the assistive device 1206 may be configured with a display monitor 1214 configured to display one or more user interfaces, or graphical or textual information supporting the medical procedure. In some cases, the display monitor 1214 may be implemented by a touchscreen display and provide user input functionality.

[0102] As will be described in more detail below, device 100 may be implemented within or operate with computer-assisted medical system 1200. For example, in some implementations, device 100 may be implemented using computing resources included in an instrument (e.g., an endoscope or other imaging instrument) attached to one of the manipulator arms 1212, or by means of manipulator assembly 1202, user control device 1204, auxiliary device 1206, or Figure 12 It is implemented using computing resources associated with another system component not explicitly shown in the document.

[0103] The manipulator assembly 1202, the user control device 1204, and the auxiliary device 1206 can be communicatively coupled to each other in any suitable manner. For example, such as Figure 12 As shown, the controller assembly 1202, user control device 1204, and auxiliary device 1206 can be communicatively coupled via control line 1216, which can represent any wired or wireless communication link that can serve a particular implementation. Therefore, the controller assembly 1202, user control device 1204, and auxiliary device 1206 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0104] In some embodiments, one or more processes described herein may be implemented at least in part as instructions contained in a non-transitory computer-readable medium and executable by one or more computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., memory, etc.) and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0105] Computer-readable media (also known as processor-readable media) include any non-transitory medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and / or volatile media. Non-volatile media can include, for example, optical discs or magnetic disks, and other persistent storage. Volatile media can include, for example, dynamic random access memory (DRAM), which typically constitutes main memory. Common forms of computer-readable media include, for example, magnetic disks, hard disks, magnetic tape, any other magnetic media, compact disc read-only memory (CD-ROM), digital video disc (DVD), any other optical media, random access memory (RAM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EPROM), FLASH-EEPROM, any other memory magnetic chip or cassette tape, or any other tangible medium that a computer can read.

[0106] Figure 13 The figure illustrates an illustrative computing system 1300, which may be specifically configured to perform one or more processes described herein. For example, computing system 1300 may include or implement (or partially implement) an automatic exposure management device such as device 100, an automatic exposure management system such as system 300, or any other computing system or device described herein.

[0107] like Figure 13 As shown, the computing system 1300 may include a communication interface 1302, a processor 1304, a storage device 1306, and an input / output (“I / O”) module 1308, all communicatively connected via a communication infrastructure 1310. Although Figure 13 The illustration shows a computing system 1300, but... Figure 13 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 13 The components of the computing system 1300 shown.

[0108] Communication interface 1302 can be configured to communicate with one or more computing devices. Examples of communication interface 1302 include, but are not limited to, wired network interfaces (such as network interface cards), wireless network interfaces (such as wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0109] Processor 1304 generally represents any type or form of processing unit capable of processing data or interpreting, executing, and / or directing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 1304 may direct the execution of surgery according to one or more applications 1312 or other computer-executable instructions, such as those that may be stored in storage device 1306 or another computer-readable medium.

[0110] Storage device 1306 may include one or more data storage media, devices, or configurations and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1306 may include, but is not limited to, hard disk drives, network drives, flash drives, magnetic disks, optical disks, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or combinations or sub-combinations thereof. Electronic data, including the data described herein, may be stored temporarily and / or permanently in storage device 1306. For example, data representing one or more executable applications 1312 configured to boot processor 1304 to perform any of the procedures described herein may be stored in storage device 1306. In some examples, data may be arranged in one or more databases residing within storage device 1306.

[0111] I / O module 1308 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input from a single virtual experience. I / O module 1308 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 1308 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0112] I / O module 1308 may include one or more means for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 1308 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be used in a particular implementation.

[0113] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing system 1300. For example, one or more applications 1312 residing in storage device 1306 may be configured to bootstrap processor 1304 to execute one or more processes or functions associated with processor 104 of device 100. Similarly, memory 102 of device 100 may be implemented by or within storage device 1306.

[0114] In the foregoing description, various illustrative embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and alterations can be made therein, and additional embodiments can be implemented without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive.

Claims

1. An apparatus for managing the automatic exposure of image frames, comprising: One or more processors; as well as A memory storing executable instructions, which, when executed by the one or more processors, cause the device to: Determine a color cast metric for an image frame captured by an image capture system, the color cast metric indicating the degree to which the image frame is offset to a specific color; Based on the color cast metric and the adaptive target control function, the automatic exposure target for the frame is determined; as well as Based on the frame automatic exposure target, one or more of the exposure time parameter, shutter aperture parameter, or illumination intensity parameter are updated for the image capture system to use in capturing additional image frames. The determination of the color cast metric for the image frame includes: Determine a pixel color cast metric for a plurality of pixels included in the image frame, the pixel color cast metric indicating the degree to which the plurality of pixels are offset to the specific color; and The color cast metric of the image frame is determined based on the pixel color cast metric and the weight values ​​associated with the plurality of pixels.

2. The apparatus according to claim 1, wherein, Determining the color cast metric of the image frame includes: Normalized represents the color data of the image frame; Decompose the color data to distinguish between the chromaticity attribute and the brightness attribute of the color data; Determine the degree to which the chromaticity attribute of the color data is shifted towards the specific color; and The color cast metric is determined based on the degree to which the chromaticity attribute of the color data is offset from the specific color.

3. The apparatus according to claim 2, wherein, Decomposing the color data includes converting the color data from the red-green-blue color space (RGB color space) to the cyan-magenta-yellow-black-red-green-blue color space (CMYKRGB color space).

4. The apparatus according to claim 2, wherein, Decomposing the color data includes converting the color data into the YUV color space or the CIELAB color space.

5. The apparatus according to claim 1, wherein, Determining the color cast metric of the image frame further includes: Based on the spatial location of the plurality of pixels within the image frame, weight values ​​are assigned to the plurality of pixels within the image frame; and Based on the pixel color cast metric and the assigned weight values, the color cast metric of the image frame is determined as the weighted average of the pixel color cast metric.

6. The apparatus according to any one of claims 1 to 5, wherein, Determining the frame auto-exposure target includes: Based on the image frames captured by the image capture system, determine the automatic exposure target of the original frame; Given the input of the color cast metric, the scaling value is determined as the output of the adaptive target control function; and The automatic exposure target of the frame is determined by scaling the original frame automatic exposure target by the scaling value.

7. The apparatus according to claim 6, wherein: The color cast metric given as input does not exceed the color cast threshold; The output of the adaptive target control function includes an empty scaling value; as well as Determining the frame auto exposure target includes determining the frame auto exposure target to be equal to the original frame auto exposure target.

8. The apparatus according to claim 6, wherein: The color cast metric given as input exceeds the color cast threshold; The output of the adaptive target control function includes the restored scaling value; as well as Determining the frame auto exposure target includes determining the frame auto exposure target to be equal to the original frame auto exposure target restored based on the restoration scaling value.

9. The apparatus according to claim 8, wherein, The output of the adaptive target control function is determined based on a monotonically decreasing nonlinear function.

10. The apparatus according to claim 6, wherein, Determining the color bias metric includes determining the color bias metric to correspond to a value selected from a set of discrete values, each of which corresponds to a different scaling value of the output of the adaptive target control function.

11. The apparatus according to any one of claims 1 to 5, wherein, The image capture system includes an endoscopic image capture device configured to capture the image frames as part of a sequence of image frames captured during a medical procedure performed on the body.

12. The apparatus according to claim 11, wherein: The image frame depicts an internal view of the body; and The specific color includes red.

13. The apparatus according to any one of claims 1 to 5, wherein: When executed by the one or more processors, the instructions cause the device to: Determine the automatic exposure value of the image frame; as well as Based on the frame auto-exposure value and the frame auto-exposure target, determine the frame auto-exposure gain; and The update of one or more of the exposure time parameter, the shutter aperture parameter, or the illumination intensity parameter is based on the frame auto exposure gain.

14. The apparatus according to claim 13, wherein: When executed by the one or more processors, the instructions cause the device to use a smoothing filter and filter the frame auto exposure gain based on one or more frame auto exposure parameter gains associated with one or more image frames in an image frame sequence that includes the image frame; as well as The update of one or more of the exposure time parameter, the shutter aperture parameter, or the illumination intensity parameter is based on the filtered frame auto-exposure gain.

15. The apparatus according to any one of claims 1 to 5, wherein, When executed by the one or more processors, the instructions cause the device to update the luminance gain parameters based on the frame auto-exposure target.

16. A system for managing automatic exposure of image frames, comprising: A light source configured to illuminate tissues within the body during a medical procedure; An image capturing device including an image sensor and a shutter, the image capturing device being configured to capture a sequence of image frames during the medical procedure, the sequence of image frames including image frames depicting an internal view of the body, the image frames characterizing the tissue illuminated by the illumination source; as well as One or more processors, said one or more processors being configured to: Determine the color cast metric of the image frame, the color cast metric indicating the degree to which the image frame shifts towards red; Based on the color cast metric and the adaptive target control function, the automatic exposure target for the frame is determined; as well as Based on the frame auto-exposure target, one or more auto-exposure parameters are updated for use by the image capture device or the illumination source when capturing additional image frames of the image frame sequence, wherein the one or more auto-exposure parameters include one or more of the following: The exposure time parameter corresponds to how long the shutter allows the image sensor to be exposed to the illuminated tissue. Shutter aperture parameters corresponding to the aperture size of the shutter, or The lighting intensity parameter corresponds to the lighting intensity provided by the lighting source. The determination of the color cast metric for the image frame includes: Determine a pixel color cast metric for a plurality of pixels included in the image frame, the pixel color cast metric indicating the degree to which the plurality of pixels are offset toward the red color; and The color cast metric of the image frame is determined based on the pixel color cast metric and the weight values ​​associated with the plurality of pixels.

17. The system according to claim 16, wherein, Determining the color cast metric of the image frame includes: Normalized represents the color data of the image frame; Decompose the color data to distinguish between the chromaticity attribute and the brightness attribute of the color data; Determine the degree to which the chromaticity attribute of the color data shifts towards red; and The color cast metric is determined based on the degree to which the chromaticity attribute of the color data is shifted toward the red color.

18. The system according to claim 16, wherein, Determining the color cast metric of the image frame further includes: Based on the spatial location of the plurality of pixels within the image frame, weight values ​​are assigned to the plurality of pixels within the image frame; and Based on the pixel color cast metric and the assigned weight values, the color cast metric of the image frame is determined as the weighted average of the pixel color cast metric.

19. The system according to any one of claims 16 to 18, wherein, Determining the frame auto-exposure target includes: Based on the image frames captured by the image capture device, determine the automatic exposure target of the original frame; Given the input of the color cast metric, the scaling value is determined as the output of the adaptive target control function; and The automatic exposure target of the frame is determined by scaling the original frame automatic exposure target by the scaling value.

20. The system according to any one of claims 16 to 18, wherein, The one or more processors are further configured to update a luminance gain parameter based on the frame auto-exposure target, the luminance gain parameter corresponding to the gain applied to the luminance data generated by the image sensor.

21. A non-transitory computer-readable medium storing instructions, which, when executed, cause one or more processors of a computing device to: Determine a color cast metric for an image frame captured by an image capture system, the color cast metric indicating the degree to which the image frame is offset to a specific color; Given the input of the color cast metric, determine the output of the adaptive objective control function; as well as Based on the output of the adaptive target control function, one or more of the exposure time parameter, shutter aperture parameter, or illumination intensity parameter are updated for the image capture system to use in capturing additional image frames. The determination of the color cast metric for the image frame includes: Determine a pixel color cast metric for a plurality of pixels included in the image frame, the pixel color cast metric indicating the degree to which the plurality of pixels are offset to the specific color; and The color cast metric of the image frame is determined based on the pixel color cast metric and the weight values ​​associated with the plurality of pixels.

22. The non-transitory computer-readable medium according to claim 21, wherein, Determining the color cast metric of the image frame includes: Normalized represents the color data of the image frame; Decompose the color data to distinguish between the chromaticity attribute and the brightness attribute of the color data; Determine the degree to which the chromaticity attribute of the color data is shifted towards the specific color; and The color cast metric is determined based on the degree to which the chromaticity attribute of the color data is offset from the specific color.

23. The non-transitory computer-readable medium according to claim 21, wherein, Determining the color cast metric of the image frame further includes: Based on the spatial location of the plurality of pixels within the image frame, weight values ​​are assigned to the plurality of pixels within the image frame; and Based on the pixel color cast metric and the assigned weight values, the color cast metric of the image frame is determined as the weighted average of the pixel color cast metric.

24. The non-transitory computer-readable medium according to any one of claims 21 to 23, wherein, When the instruction is executed, it causes the one or more processors to: Based on the image frames captured by the image capture system, determine the automatic exposure target of the original frame; as well as The frame auto-exposure target is determined by scaling the output of the adaptive target control function given the input of the color cast metric.

25. The non-transitory computer-readable medium according to any one of claims 21 to 23, wherein: The image capture system includes an endoscopic image capture device configured to capture the image frames as part of a sequence of image frames captured during a medical procedure performed on the body; The image frame depicts an internal view of the body; and The specific color includes red.

26. A method for managing automatic exposure of image frames, comprising: A computing device determines a color cast metric for an image frame captured by an image capture system, the color cast metric indicating the degree to which the image frame is offset to a specific color; The automatic exposure target for the frame is determined by the computing device and based on the color cast metric; The automatic exposure value of the frame is determined by the computing device; as well as The computing device updates one or more of the exposure time parameter, shutter aperture parameter, or illumination intensity parameter based on the frame auto-exposure target and the frame auto-exposure value, for the image capture system to use in capturing additional image frames. The determination of the color cast metric for the image frame includes: Determine a pixel color cast metric for a plurality of pixels included in the image frame, the pixel color cast metric indicating the degree to which the plurality of pixels are offset to the specific color; and The color cast metric of the image frame is determined based on the pixel color cast metric and the weight values ​​associated with the plurality of pixels.

27. The method according to claim 26, wherein, Determining the color cast metric of the image frame includes: Normalized represents the color data of the image frame; Decompose the color data to distinguish between the chromaticity attribute and the brightness attribute of the color data; Determine the degree to which the chromaticity attribute of the color data is shifted towards the specific color; and The color cast metric is determined based on the degree to which the chromaticity attribute of the color data is offset from the specific color.

28. The method according to claim 26, wherein, Determining the color cast metric of the image frame further includes: Based on the spatial location of the plurality of pixels within the image frame, weight values ​​are assigned to the plurality of pixels within the image frame; and Based on the pixel color cast metric and the assigned weight values, the color cast metric of the image frame is determined as the weighted average of the pixel color cast metric.

29. The method according to any one of claims 26 to 28, wherein, Determining the frame auto-exposure target includes: Based on the image frames captured by the image capture system, determine the automatic exposure target of the original frame; Given the input of the color cast metric, the scaling value is determined as the output of the adaptive objective control function; and The automatic exposure target of the frame is determined by scaling the original frame automatic exposure target by the scaling value.

30. The method according to claim 29, wherein: The color cast metric given as input does not exceed the color cast threshold; The output of the adaptive target control function includes an empty scaling value; as well as Determining the frame auto exposure target includes determining the frame auto exposure target to be equal to the original frame auto exposure target.

31. The method according to claim 29, wherein: The color cast metric given as input exceeds the color cast threshold; The output of the adaptive target control function includes the restored scaling value; as well as Determining the frame auto exposure target includes determining the frame auto exposure target to be equal to the original frame auto exposure target restored based on the restoration scaling value.

32. The method according to any one of claims 26 to 28, wherein: The image capture system includes an endoscopic image capture device configured to capture the image frames as part of a sequence of image frames captured during a medical procedure performed on the body; The image frame depicts an internal view of the body; and The specific color includes red.

33. The method according to any one of claims 26 to 28, further comprising determining a frame auto-exposure gain by the computing device based on the frame auto-exposure value and the frame auto-exposure target; in, The update of one or more of the exposure time parameter, the shutter aperture parameter, or the illumination intensity parameter is based on the frame auto exposure gain.

Images