Approaches to characterizing an unknown ambient illumination spectrum for an unknown illuminant by means of a known illuminant

Abstract

Introduced here are approaches to accurately capturing spectral information for a target object that is illuminated with a known illumination spectrum in the presence of an unknown illumination spectrum. The unknown illumination spectrum could include light corresponding to more than one illuminant (e.g., the sun through different windows, a combination of different types of illuminants, or the sun in combination with another type of illuminant such as an incandescent bulb, fluorescent tube, or LEDs). These approaches enable the unknown illumination spectrum to be calibrated and characterized, thereby enabling the unknown illumination spectrum to be redefined as any desired illumination spectrum, for example, one that is ideal for illuminating the objects in a scene.

Description

PATENTAttorney Docket No. 133121 .8009. WO02APPROACHES TO CHARACTERIZING AN UNKNOWN AMBIENT ILLUMINATION SPECTRUM FOR AN UNKNOWN ILLUMINANT BY MEANS OF A KNOWN ILLUMINANTTECHNICAL FIELD

[0001] Various embodiments concern approaches to characterizing the illumination spectra of uncontrolled illuminants that illuminate an object of interest.BACKGROUND

[0002] Illuminants can be characterized by color temperature as measured in degrees Kelvin (K). The color temperature of an illuminant is the temperature at which the color of light emitted from a heated blackbody is matched by the color of the illuminant. For illuminants that do not substantially emulate blackbodies, such as fluorescent bulbs and light-emitting diodes (LEDs), the temperature at which the color of light emitted from a heated blackbody approximates the color of the illuminant is called the correlated color temperature (CCT).

[0003] Ideally, the scene in a digital image will be lit by one or more illuminants with the same color temperature. However, this rarely happens. Instead, scenes are normally lit by multiple illuminants with different color temperatures - a scenario commonly referred to as “mixed lighting.” As an example, consider that an indoor scene could be lit by overhead lighting elements, indirect sunlight coming through a window, etc. Such a scenario may also occur due to different lighting conditions that have different color temperatures. For instance, while a pair of digital images of an outdoor scene may be created only moments apart, the color temperature will be different if the sun becomes more or less obscured in the meantime.

[0004] White balancing is the process by which a digital image is balanced in an attempt to bring the color temperature back to neutral. At a high level, white balancing removes unrealistic color casts to emulate what the human eye does when viewing white objects. This is done so that objects in digital images that appear white to the human eye are rendered white in those digital images.PATENTAttorney Docket No. 133121 .8009. WO02BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 illustrates an example of a computing device with an illuminant for illuminating an object of interest and an image sensor for generating digital images in conjunction with illumination events performed by the illuminant.

[0006] Figure 2 includes a schematic diagram of the approach set forth above for establishing the ambient illumination spectrum of one or more unknown illuminants.

[0007] Figure 3 illustrates a process for acquiring color information may be useful in establishing a reflectance spectrum for an object.

[0008] Figure 4 illustrates a network environment that includes a characterization module executing on a computing device.

[0009] Figure 5 is a block diagram illustrating an example of a processing system that is capable of implementing the operations described herein.

[0010] Various features of the technology described herein will become more apparent to those skilled in the art upon studying the Detailed Description in conjunction with the drawings. Embodiments are shown in the drawings for the purpose of illustration. Those skilled in the art will recognize that alternative embodiments could be employed without departing from the teachings of this disclosure. Accordingly, while certain embodiments are shown in the drawings, the technology is amenable to various modifications.PATENTAttorney Docket No. 133121 .8009. WO02DETAILED DESCRIPTION

[0011] The precise spectrum of ambient light that is striking a given object is very difficult to accurately determine through analysis of a digital image. Not only are multiple illuminants generally contributing to illumination of the given object, but disentangling the respective contributions can be extraordinarily difficult. Moreover, changes in the ambient light can distort the color rendering of the digital image itself, necessitating white balancing or a separate ambient capture of a calibration target that is then used to estimate the ambient illumination while the digital image is being processed.

[0012] White balancing aims to ensure that colors in a digital image appear natural under different lighting conditions by adjusting the intensities of the values of the pixels - for example, the red, green, and blue values of the pixels in a digital image output by an RGB image sensor - so that neutral colors - like white and gray - appear truly neutral. In practice, however, white balancing tends to be a guess of the ambient light in a scene, typically under the assumption that the colors in the scene should roughly average a particular color point. Where that assumption is not valid, or where an absolute measure of the color or reflectivity spectrum of a given object is sought, the corrections made by white balancing tend to be insufficient. Said another way, white balancing may cause a digital image to be overly bright or overly dim to compensate for an abundance or scarcity of ambient light, respectively. Properly correcting the colors in a digital image is more complicated - and in some cases, nearly impossible - where multiple illuminants are present.

[0013] Simply put, when a scene does not include a known reflectance spectrum, algorithms that are designed for automatic white balancing can (and often do) produce digital images that are visibly incorrect in terms of color. These algorithms are commonly called “AWB algorithms.” For example, if a scene is predominantly red, AWB algorithms may mistake this for a color cast induced by a warm illuminant and then try to compensate by making the average color of pixels in a digital image of the scene closer to neutral. But this will introduce a bluish color cast that may be quite noticeable. Similarly, if a scene is predominantly blue, AWB algorithms may mistake this for a colorPATENTAttorney Docket No. 133121 .8009. WO02 cast induced by a cool illuminant and then try to compensate by making the average color of pixels in a digital image of the scene closer to neutral. This will introduce a reddish color cast that may be quite noticeable.

[0014] At a high level, an AWB algorithm assesses the scene included in a digital image that is provided as input by either (i) identifying the brightest part as white and then adjusting the red, green, and blue channels to balance the color across the digital image based on that reference or (ii) assuming that the average color in the digital image should be neutral gray and then adjusting the red, green, and blue channels to make whites and grays appear colorless. AWB algorithms typically rely on statistical methods, like the Gray World assumption or White Patch assumption, to perform this correction. With the Gray World assumption, the AWB algorithm is designed to assume that the average color of a scene is a neutral gray, so the AWB algorithm adjusts the red, green, and blue channels so their averages are equal, in an effort to correct color casts caused by the ambient lighting. With the White Patch assumption, the AWB algorithm assumes that the brightest pixel in the digital image represents a white surface under the scene’s ambient lighting, so the AWB algorithm scales the color channels so that this pixel becomes neutral white, in an effort to correct the overall color balance.

[0015] In addition to the aforementioned drawbacks of white balancing, AWB algorithms are computationally intensive because they must perform, on a per-pixel basis for at least large portions of the digital image, operations (e.g., summing RGB values), statistical analysis (e.g., histograms, averages, percentiles), and matrix operations or optimization - all while maintaining real-time performance and adapting to varying lighting conditions and scene content. As an example, for a 1 -megapixel digital image, a Gray World AWB algorithm would require roughly 3 million additions (i.e. , 1 per channel per pixel) and roughly 3 million divisions to normalize. On a modern central processing unit (CPU) or digital signal processor (DSP), this would take roughly 1-10 milliseconds (ms) and consume roughly 3-4 megabytes (MB) of random-access memory (RAM). For the same digital image, a White Patch AWB algorithm would complete roughly 3 million comparisons to establish the maximum value per channelPATENTAttorney Docket No. 133121 .8009. WO02 and then complete a scaling step afterwards. The White Patch AWB algorithm would have a similar time and memory profile as the Gray World AWB algorithm. More advanced AWB algorithms - like Gray Edge and CNN-based ones - may require I Q- 100 times more operations, leading to larger time and memory profiles. These computational costs will be even higher if the AWB algorithm is tasked with processing multiple digital images in rapid succession, for example, in the event that the digital images are representative of frames of a video.

[0016] Introduced here are approaches to accurately capturing spectral information for a target object that is illuminated with a known illumination spectrum in the presence of one or more unknown illuminants with an unknown illumination spectrum (which may also be called an “unknown ambient light spectrum”). As mentioned above, the unknown illumination spectrum could include light corresponding to more than one illuminant (e.g., the sun through different windows, a combination of different types of illuminants, or the sun in combination with another type of illuminant such as an incandescent bulb, fluorescent tube, or LEDs). The approaches enable the unknown illumination spectrum to be calibrated and characterized by a computer program (also called a “calibration module” or “characterization module”), thereby enabling the unknown illumination spectrum to be redefined as any desired light spectrum, for example, one that is ideal for illuminating the objects in a scene.

[0017] For the purpose of illustration, the approaches are described in the context of a “reference illuminant” or “reference illumination” that corresponds to a known illumination spectrum. The reference illumination is used to probe a target object with unknown reflectivity that is in a scene in order to then establish the spectral properties of the unknown illumination spectrum. After the spectral properties of the unknown illumination spectrum are known (e.g., measured, computed, or otherwise determined in accordance with the approaches discussed below), the digital image that is obtained in conjunction with the ambient light spectrum can be appropriately calibrated.

[0018] The observed color in a digital image is the product of the illumination - which can be represented by a spectral power distribution - and the reflectance spectrums ofPATENTAttorney Docket No. 133121 .8009. WO02 surfaces visible in the digital image. Capturing the spectral reflectivity - which can be presented in part by a color spectrum - of a target object in the presence of ambient light can pose a significant issue due to spatial variations in the spectral, specular, angular, and intensity variations of known ambient illuminants (e.g., the high-intensity white LEDs commonly used by mobile phones to create flashes for photography), as well as the unknown ambient illuminants (e.g., the sun). Note that the spectral power distribution represents the power as measured over a range of wavelengths, generally across the visible spectrum (typically ~380 to 780 nanometers) but preferably across the widest range of wavelengths permissible. When this range of wavelengths is limited to the visible spectrum, the spectral power distribution can be presented using color, for example, in the Red-Green-Blue (RGB) color space or CIELAB color space (commonly referred to as the “L*a*b* color space” or “Lab* color space”). Hence, color can be a subset of the spectral power distribution. Below, it is explained how a secondary ilium inant with known intensity (and known spectral power distribution) can be used as a means of determining and then calibrating or redefining the spectral power distribution and intensity of illumination contributed by one or more unknown ambient illuminants. While this secondary illuminant generally emits visible light, this secondary illuminant could instead emit non-visible light, for example, in the ultraviolet or infrared regions of the electromagnetic spectrum, in which case the image sensor would also be able to detect non-visible light that is generated by this secondary illuminant and is reflected back to the image sensor by objects in the scene being imaged. Accordingly, the approaches described here may be generally applicable to any type of illuminant that emits electromagnetic radiation and that can serve as the known illuminant and a corresponding type of image sensor.Terminology

[0019] References to “an embodiment” or “some embodiments” mean that the feature being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are these occurrences necessarily referring to alternative embodiments that are mutually exclusive of each other.PATENTAttorney Docket No. 133121 .8009. WO02

[0020] The term “based on” is to be construed in an inclusive sense rather than an exclusive sense. That is, in the sense of “including but not limited to.” Accordingly, the term “based on” is meant to be interpreted as “based at least in part on” unless otherwise noted.

[0021] The terms “connected,” “coupled,” and variants thereof are intended to cover any connection or coupling between two or more components, either direct or indirect. For example, components may be electrically or communicatively coupled to each other despite not sharing a physical connection.

[0022] The term “module” may be used to refer to a self-contained unit of functionality that is implemented in software, firmware, hardware, or any combination thereof. A module typically operates by receiving one or more inputs, processing those inputs through defined operations or logic, and then producing one or more outputs. Modules are often designed with standardized interfaces to enable integration, reuse, and independent development or maintenance within complex systems. However, modules can also be designed with specialized interfaces that allow them to occupy a specific position within a processing pipeline, where each module takes, as input, the output of a preceding module and supplies its own output as input to a succeeding module, enabling a structured and sequential flow of data or operations.

[0023] The term “illuminant” may be used to refer to any source of light that emits or reflects electromagnetic radiation, generally in the visible spectrum. Illuminants can be natural (e.g., sunlight and moonlight) or artificial (e.g., incandescent bulbs, fluorescent lamps, LEDs, organic LEDs, quantum dots, and lasers), and may have well-defined spectral characteristics or vary unpredictably.

[0024] The terms “known ambient illuminant” and “known illuminant” may be used to refer to a light source whose spectral illumination profile is predetermined and known to the characterization module, allowing its contribution to a scene’s overall lighting to be quantified and accounted for. Because this known profile can be controlled, a known illuminant could also be called a “controlled illuminant” or “controllable illuminant.” This known profile - such as the spectral power distribution and intensity pattern of an LEDPATENTAttorney Docket No. 133121 .8009. WO02 located near a mobile phone’s rear-facing camera - allows the characterization module to distinguish and isolate the effects of the known illuminant from those of unknown ambient illuminants.

[0025] The terms “unknown ambient illuminant” and “unknown illuminant” may be used to refer to a light source whose spectral illumination profile is not known by the characterization module and / or cannot be controlled by the characterization module. Accordingly, an unknown illuminant could also be called an “uncontrolled illuminant” or “uncontrollable illuminant.” Because its profile - such as the contributions of the sun or moon in illuminating a scene - are not known, the characterization module may attempt to isolate the effects of any known illuminants in order to glean insights into the effects of the unknown illuminant.

[0026] The terms “reflectance spectrum” and “reflectivity spectrum” may be used to refer to a precise, quantitative measurement of how a surface reflects light at different wavelengths across a spectrum. Importantly, a reflectance spectrum is a full profile across the spectrum - not a single perceived color. Color refers to how a surface appears to the human eye, which is based on how the surface reflects or emits light across visible wavelengths. Consider, for example, a gray card that is commonly used in photography. The gray card doesn’t just “look gray.” It reflects about 18 percent of light equally across all visible wavelengths, which makes it useful for calibration. This known and consistent reflectance curve is what allows for accurate white balancing and color correction.

[0027] A "spectral power distribution" describes how much power - or intensity - an illuminant emits at each wavelength across a range of the electromagnetic spectrum, providing a detailed profile of the illuminant’s color characteristics. At a high level, the spectral power distribution is a series of values - usually presented as a graph - that shows how “strong” each wavelength is in illumination from a given illuminant, helping to define its color quality and rendering behavior. The term “illumination spectrum” is commonly used interchangeably with the term “spectral power distribution,” as bothPATENTAttorney Docket No. 133121 .8009. WO02 indicate the color “ingredients” of an illuminant by showing how strong each wavelength component is.Overview of Problem

[0028] Enabling cameras (and more specifically, image sensors) to produce digital images with accurate color, shade, or intensity is important for various applications. Accurate representation is especially true for digital images with photography applications (e.g., three-dimensional capture), health applications (e.g., telemedicine), cosmetic applications, and commerce applications - especially electronic commerce (also called “e-commerce”) applications where the actual objects may not be readily viewable to consumers. Typically, color and shade are estimated by putting a spectral sensing tool in close proximity to the object to block the ambient light and provide a well-characterized illumination at a fixed distance. Examples of spectral sensing tools include spectroradiometers that measure the absolute spectral power distribution of an illuminant; spectrophotometers that measure how much light is absorbed, transmitted, or reflected by a material at different wavelengths; monochromators that isolate a narrow band of wavelengths (i.e. , a nearly single color or “monochromatic” light) from a broader spectrum and characterizes how a material reacts to those wavelengths; and hyperspectral and multispectral cameras that capture digital images in dozens or hundreds of narrow wavelength bands.

[0029] With regard to these spectral sensing tools, there are several matters of practical concern, namely:• Limited sensor performance, particularly signal-to-noise ratio (also called “SNR”) and linearity;• Potential interference from ambient light;• Implications of single exposure versus multi-exposure and associated motion sensitivity;• The assumption of Lambertian scattering by the target object; and• Illuminant emission stability over different currents and temperatures.PATENTAttorney Docket No. 133121 .8009. WO02Several approaches to correcting the aforementioned concerns, so as to arrive at a geometrically corrected spectrum from the target object with unknown reflectance, are provided in International Application No. PCT / US2020 / 022639 that is titled “White Balance with Reference llluminants” and International Application No. PCT / US2022 / 070485 that is titled “Nonlinearity Correction and Range Fitting for Stereoscopy Through Illumination and Approaches to Using the Same for Noncontact Color Determination,” each of which is incorporated herein by reference in its entirety.Overview of Approach

[0030] We begin with a known illumination from an illuminant that has a known illumination spectrum defined in a suitable color space. Consider, for example, the computing device 100 shown in Figure 1. The computing device 100 includes a display 102 and a front-facing camera 104 that are situated within a housing 106. Here, the computing device 100 is a mobile phone. However, those skilled in the art will recognize that the approaches discussed below could be readily adapted for other types of computing devices, such as wearable electronic devices (e.g., watches, fitness trackers, or head-mounted devices), tablet computers, laptop computers, Internet of Things devices (e.g., surveillance cameras, surveillance doorbells), semi- or fully-autonomous electronic devices (e.g., robotic systems, drones), and the like. As shown in Figure 1 , the front-facing camera 104 is usually set within an aperture or notch in the display 102 but could be offset from the display 102 entirely. The front-facing camera 104 is typically one of multiple cameras included in the computing device 100. For example, the computing device 100 may also include one or more rear-facing cameras 108 and an illuminant 110 that are also situated within the housing 106. The illuminant 110 may be designed to produce a single color - such as a white LED that emits broad-spectrum light for the purpose of flooding a scene with a “flash” - or multiple colors, as in systems that use red, green, and blue LEDs in combination to create a wider range of hues through controlled blending.

[0031] While the front- and rear-facing cameras 104, 108 generally include the same type of image sensor - for example, a complementary metal-oxide semiconductorPATENTAttorney Docket No. 133121 .8009. WO02(CMOS) image sensor - the image sensors need not be identical to each other, In fact, the image sensors generally have different capabilities. For example, the image sensors in front-facing cameras may be optimized for compact size, low power consumption, and decent low-light performance, while the image sensors in rear-facing cameras tend to be larger, more advanced, and may include technologies light Backside Illumination (BSI), Quad Bayer or Tetracell pixel layouts for binning, or stacked sensor designs, allowing for higher resolution and better dynamic range.

[0032] The display 102 and illuminant 110 can serve as known illuminants for the front-facing camera 104 and rear-facing cameras 108, respectively. If the front-facing camera 104 generates a digital image in conjunction with light being emitted by the display 102, the illumination spectrum of the display 102 may be known to, and controllable by, a characterization module that is executing on the computing device 100. Similarly, if one or more of the rear-facing cameras 108 generates a digital image in conjunction with light being emitted by the illuminant 110, the illumination spectrum of the illuminant 110 may be known to, and controllable by, the characterization module.

[0033] For the purpose of illustration, the known illumination spectrum is labeled without loss of generality as Rki, Gki and Bki. Note that, in some embodiments, the known illuminant is a discrete source of light that is separate from the existing illuminants (e.g., the display 102 and illuminant 110) of the computing device 100. Consider, for example, a mobile phone that includes an existing illuminant situated along its rear side to provide illumination in the form of a “flash” while capturing digital images, like the one shown in Figure 1. The known illuminant could be located proximate to this existing illuminant, or the known illuminant could replace this existing illuminant - in which case the known illuminant may be configurable to produce white light. For example, the known illuminant might be electrically and / or communicatively connected to the computing device 100. In other embodiments, the known illuminant is representative of an existing illuminant. As discussed above, the display 102 of the computing device 100 could be used as the known illuminant, so long as each emitted color is independently controllable. Similarly, the illuminant 110 could be used as the known illuminant, so long as each emitted color is independent controllable.PATENTAttorney Docket No. 133121 .8009. WO02

[0034] The target object may be any object whose reflectance is sought yet unknown, including, for example, an area of skin. Following the convention mentioned above, the unknown reflectance spectrum can be defined as Rus, Gus and Bus.

[0035] As mentioned above, the target object is located in an ambient environment under an unknown ambient lighting condition. This ambient environment could be an outdoor space or an indoor space (e.g., a room within a structure). In outdoor spaces, the unknown ambient lighting condition is typically caused - exclusively or primarily - by natural sources, such as the sun or moon, or artificial sources like streetlights or vehicle headlights. In indoor spaces, the unknown ambient lighting condition is generally due to artificial sources, such as ceiling- or wall-mounted fixtures, electronic displays, or fireplaces, though natural light may also contribute via windows, doorways, and similar openings. Notably, the specific source or combination of sources contributing to the unknown ambient lighting condition is not critical to the implementation of the approaches described here. Following the convention mentioned above, the unknown illumination spectrum can be defined as Rua, Gua and Bua.

[0036] A calibrated image sensor can be located in a position such that it does not receive any significant direct photon flux, from either the known ilium inant Rki, Gki and Bki or the unknown ambient illuminant Rua, Gua and Bua. Rather, the calibrated image sensor can receive reflected light from the scene being imaged, including the target object.

[0037] A characterization module can acquire a series of digital images that are captured in conjunction with different amounts of illumination from the known illuminant. Said another way, the characterization module can obtain a series of digital images in which amounts of illumination from the known illuminant are varied. In some embodiments, the characterization module may prompt the generation of the series of digital images and emittance of the different amounts of illumination (e.g., by sending appropriate instructions to the display 102 and front-facing camera 104, or by sending appropriate instructions to the illuminant 110 and rear-facing camera 108). In other embodiments, the characterization module may retrieve the series of digital imagesPATENTAttorney Docket No. 133121 .8009. WO02 from a memory (e.g., a local memory on the computing device 100, or a remote memory that is accessible to the computing device) if, for example, the series of digital images was already generated.

[0038] Regardless of its approach to acquiring the series of digital images, the characterization module can record a series of digital images in which the amounts of Rki, Gki and Bki are varied. While the scene being imaged is almost certainly illuminated by one or more unknown illuminants, an assumption could be made that the scene is being primarily illuminated by the known illuminant. (Note, however, that this assumption need not necessarily be made. The approaches disclosed here could be used to measure the reflectivity of an object or illumination spectrum of an unknown illuminant by using a known illuminant that is providing as little as 25, 10, 5, or even 1 percent of overall illumination. Accordingly, the known illuminant does not necessarily need to provide a flash that overwhelms the contribution of the unknown illuminant.)

[0039] By implementing this assumption, the change in measured R, G, and B values for each digital image can be presumed to be due to the known illuminant only. For example, the calibrated image sensor can generate - and the characterization module can record - a first series of digital images with different R values, a second series of digital images with different G values, and a third series of digital images with different B values, one at a time, followed by all colors on and all colors off. Other suitable modulations could be employed. Given the differences in color from the changes in the known illuminant, the characterization module can estimate the reflectance for each R, G, and B spectral element. Said another way, the characterization module can estimate the unknown reflectance spectrum Rus, Gus and Bus on a per-pixel basis based on an analysis of the series of digital images, by presuming that changes are due entirely to the variation in the amounts of Rki, Gki and Bki. Comparing these estimated values with the known reflectance for the known illuminant results in the derivation of deltas, which can subtract out the ambient illumination. With the ambient illumination eliminated, the known illuminant can be used to define the exact reflectance spectrum of the target object.PATENTAttorney Docket No. 133121 .8009. WO02

[0040] At a high level, instead of placing a calibration card (or some other reference item) in the scene, the known ilium inant can be strobed (e.g., across different colors, intensities, etc.) to determine the reflectivity spectrum of the target object. Then, observing the color with the known illuminants off, the characterization module can obtain the ambient colors that are the product of the estimated reflectance spectrum in combination with the unknown ambient illumination. This will ensure that the scene is “color calibrated” properly.

[0041] Ideally, the known illuminant should provide a change in the target object color that is significant compared to the noise level of the image sensor. Depending on the intensity of the ambient illumination, the known illuminant’s distance from the target object may need to be reduced until the illuminant provides a suitably observable level of illumination change. If the computing device is a mobile phone, for example, that uses the display as the known illuminant, then the distance range may be on the order of several inches to several feet (e.g., roughly the length of the arm). When an individual indicates (e.g., through an interface presented on the display) an interest in initiating an image capture process for the purpose of color calibration, the individual may be instructed whether to locate the display nearer or further from the object (e.g., a part of a human body, such as the face) being imaged. When the display is determined to be located within the distance range, the calibration module may indicate as much through the interface. The calibration module may determine whether the display is located within the distance range based on an analysis of an image generated by the frontfacing camera (e.g., do the individual pixel values indicate that there is a sufficient level of illumination for change to be detected, is the face - or certain facial structures - a size that indicates the mobile phone is appropriately placed) or an output produced by another sensor, such as an infrared sensor, Light Detection and Ranging (LiDAR) scanner, or camera system (e.g., the TrueDepth camera system in the case of mobile phones developed by Apple Inc.).

[0042] After the reflectivity of the target object is known, the ambient illumination spectrum can be determined by the characterization module. Additionally, the ambient illumination spectrum can be corrected and the true color - represented by thePATENTAttorney Docket No. 133121 .8009. WO02 reflectance spectrum - can be established for not just the target object but also any other objects whose measured color is affected by the ambient illumination spectrum. These steps may be performed by the characterization module in the form of a processing pipeline, in which the output of one step serves as the input for the following step. Designing and implementing a pipeline may be useful, as it allows the characterization module to more easily process digital images in rapid succession (e.g., as necessary for video processing, editing, etc.). Thus, having determined the reflectivity of the target object and having defined the ambient lighting condition, the characterization module can both correct for a suboptimal ambient lighting condition and render how the target object (or the scene as a whole) would look under any arbitrary illumination. The latter process may be called “relighting.”

[0043] Note that, in some embodiments, the distance between the image sensor and target object and / or the angular position of the target object with respect to the image sensor are measured and used within this calibration routine. To establish the distance, an output produced by the computing device (e.g., its infrared sensor) could be used to establish the distance and / or the angular position. Additionally or alternatively, the distance and / or the angular position could be estimated based on an analysis of one of the digital images captured using the image sensor and documented by the characterization module. The distance and / or the angular position could be used to perform geometric corrections necessitated if, for example, the surface of the target object is not orthogonal to the direction of the light - whether visible or non-visible - emitted theretoward by the known illuminant.

[0044] Furthermore, this approach allows the reflectance in known locations (e.g., a larger area of skin that was not characterized as the target object) to be determined more simply using the ambient correction determined for the target object.

[0045] This approach is particularly useful when measuring a target object with low reflectivity. The ambient illumination may be much brighter than the illumination emitted by the known illuminant, such that the increased signal level of the “corrected ambient illumination” can provide improved SNR over the known illuminant. The approach alsoPATENTAttorney Docket No. 133121 .8009. WO02 provides value in allowing a target object to be spectrally measured farther from the known illuminant.

[0046] While the approach introduced here is applicable to different situations, there are a few conditions that generally hold true, namely:• The change in ambient illumination over time (e.g., as the series of digital images are generated) is either known - and therefore, can be accounted for - or negligible; and• The change in distance between a near measurement and a far measurement where the known illuminant is either not used or has no meaningful impact, does not result in a significant change in the ambient illumination spectrum. For example, the characterization module may assume that a mobile phone being located proximate to a face will not change the illumination on the face. This may be the case with quasi-diffuse illumination or direct illumination not coming from behind the mobile phone. Note that the term “near measurement” may be used to refer to a measurement where the known illuminant is close enough to cause modulation of color of the target object, while the term “far measurement” may be used to refer to a measurement where the known illuminant has no significant effect on color of the target object.

[0047] Figure 2 includes a schematic diagram of the approach set forth above for establishing the ambient illumination spectrum of one or more unknown illuminants. At a high level, a characterization module can initially record digital images of an object 208 that are generated by an image sensor 204 of a computing device 202 while the object 208 is differentially illuminated by an illuminant 206 of the computing device. Each of the digital images can be generated in conjunction with a different color and / or intensity of illumination by the illuminant 206. For example, the illuminant 206 may be a singlechannel illuminant that is able to emit a single color of light (e.g., white), in which case each of the digital images may be associated with a different intensity of illumination. As another example, the illuminant 206 could be a multi-channel illuminant - or the illuminant 206 could be one of multiple single-channel illuminants - that is able multiplePATENTAttorney Docket No. 133121 .8009. WO02 colors of light (e.g., one or more LEDs for red light, one or more LEDs for green light, one or more LEDs for blue light), in which case the digital images could include (i) a first subset of digital images that are captured in conjunction with different intensities of a first colored light (e.g., red), (ii) a second subset of digital images that are captured in conjunction with different intensities of a second colored light (e.g., green), (iii) a third subset of digital images that are captured in conjunction with different intensities of a third colored light (e.g., blue), etc. In Figure 2, the notation for this series of digital images is where I denotes an image (e.g., a matrix of RGB pixel values), C G {R, G.B} denotes the channel whose intensity is varied in the corresponding subset (i.e. , R for red-light-modulated digital images, G for green-light-modulated digital images, and B for blue-light-modulated digital images), and i G {1,2, ...,nc] indexes each digital image within the corresponding color’s subset (e.g., different levels of red light, green light, or blue light).

[0048] Thereafter, the characterization module can estimate, for the object 208, a reflectance spectrum based on an analysis of the red, green, and blue values of the digital images and a known illumination spectrum that is associated with the ilium inant 206. For example, the characterization module may establish, based on the red, green, and blue values of the digital images, appropriate red, green, and blue values for the object on a per-pixel basis. Because the illumination spectrum of the illuminant 206 is known to the characterization module, the unknown reflectance spectrum of the object 208 can be calculated. The digital images are - at a high level - representative of the product of the known illumination spectrum Rki, Gki and Bki and the unknown reflectance spectrum Rus, Gus and Bus, allowing the characterization module to isolate the unknown reflectance spectrum Rus, Gus and Bus. Further information regarding how reflectance can be computed on a per-pixel basis based on a known illumination spectrum and a known response profile of the image sensor can be found in International Application No. PCT / US2018 / 041328, titled “Techniques for Creating Characterization Matrices for Reflectance, Lighting or Sensor Response” and incorporated by reference herein in its entirety.PATENTAttorney Docket No. 133121 .8009. WO02

[0049] Moreover, the characterization module can record a digital image of the object 208 that is generated by the image sensor 204 while the illuminant 206 does not emit any light. The characterization module can estimate the unknown illumination spectrum of any unknown illuminants that illuminate the object based on an analysis of red, green, and blue values of the digital image and the reflectance spectrum estimated for the object. Because the characterization module has already estimated the reflectance spectrum of the object, the unknown illumination spectrum can be calculated. Said another way, since the illuminant 206 is not contributing to illumination of the object 208, the digital image is - at a high level - representative of the product of the unknown reflectance spectrum Rus, Gus and Bus and the unknown illumination spectrum Rua, Gua and Bua, allowing the characterization module to isolate the unknown illumination spectrum Rua, Gua and Bua.

[0050] Note that the characterization module could implement a similar process if the image sensor 204 were to generate an image of the object 208 while the illuminant 206 emitted white light instead of, or in addition to, the “no light” digital image.

[0051] The aforementioned approach has several benefits.

[0052] For example, the characterization module - or another module - could post, to an interface, another version of the digital image that is modified to account for the unknown illumination spectrum of the unknown illuminant(s) contributing to illumination of the object. This version of the digital image may be a “color corrected” one, which could be dynamically generated by the characterization module or in response to receiving input indicative of a request to do so.

[0053] As another example, the characterization module may create another version of the digital image that is modified so as to mimic being captured under a different ambient lighting condition. Specifically, the characterization module could receive, through an interface, input that is indicative of a request to adjust the digital image to mimic a given ambient lighting condition. In response to receiving the input, the characterization module can adjust at least some of the red, green, and blue values ofPATENTAttorney Docket No. 133121 .8009. WO02 the digital image, so as to create another version of the digital image, based on how the given ambient lighting condition compared to an actual ambient lighting condition as represented by the unknown illumination spectrum. This version of the digital image could be stored and / or posted to the interface for review.

[0054] In embodiments where the display of the computing device 202 is used as the illuminant 206 and the image sensor 204 is included in a front-facing camera of the computing device 202, information to facilitate the aforementioned approach could be presented either visually (e.g., via the display) or audibly (e.g., via an audio output mechanism). For example, the characterization module may indicate, through an interface, that the computing device should be positioned and oriented such that a given object (e.g., a face) is viewable by the front-facing camera. Moreover, the characterization module may indicate, through the interface, whether the computing device needs to be positioned nearer to, or further from, the given object. As discussed above, the image sensor 206 should ideally be within a certain proximity of the given object to ensure that changes in color and / or intensity of light emitted by the illuminant 206 are suitably observable by the image sensor 204.Processes for Acquiring Digital Images and Establishing Spectral Information

[0055] Figure 3 illustrates a process 300 for acquiring color information may be useful in establishing a reflectance spectrum for an object. An illuminant - or multiple illuminants - can initially strobe through multiple color channels. Assume, for example, that the display of a computing device will serve as the source of illumination as shown in Figure 2. In such an embodiment, the display can perform a series of discrete illumination events by illuminating the object with a first colored light (e.g., red) by activating at least some of its illuminants (e.g., LEDs) with varying intensities (step 301 ), illuminating the object with a second colored light (e.g., green) by activating at least some of its illuminants with varying intensities (step 302), and illuminating the object with a third colored light (e.g., blue) by activating at least some of its illuminants (step 303). As mentioned above, while the process 300 is described in the context ofPATENTAttorney Docket No. 133121 .8009. WO02 illuminants that emit visible light, some or all of the illuminants could instead emit non- visible light. For example, the illumination events could involve a first colored light, a second colored light, and a non-visible light (e.g., ultraviolet or infrared), or the illumination events could involve a colored light, a first non-visible light (e.g., ultraviolet), and a second non-visible light (e.g., infrared). As another example, the illumination events could involve exclusive non-visible light (e.g., corresponding to different non- visible portions of the electromagnetic spectrum).

[0056] In some embodiments, the entire display is illuminated for each illumination event, but the intensity of each illumination event performed for a given colored light might be different. For example, if the display includes an array of LEDs, those LEDs could emit red light at 90 percent, 80 percent, 70 percent, 60 percent, 50 percent, and 40 percent intensity. Similar intensities could be used for green light and blue light, so that the same intensities are used across the different color channels. Those skilled in the art will recognize that a greater or lesser number of illumination events could be performed for each color channel and that the difference in intensity between consecutive illumination events could be greater than ten or less than ten. Generally, the intensity does not fall below a threshold (e.g., 30 or 40 percent) to ensure that the display remains primarily responsible for illuminating the object. In other embodiments, a percentage of the LEDs in the display is used as a proxy for intensity. For example, rather than adjust the intensity of the individual LEDs, the total number of LEDs being illuminated could serve as a reference for intensity (e.g., for 80 percent intensity, 80 percent of the LEDs could be illuminated).

[0057] Each illumination event can vary in length (also called its “duration”). Illumination events produced in the context of photography tend to range from 15 - 100 ms. However, some illumination events could have sub-millisecond durations, while other illumination events could have multi-second durations.

[0058] A multi-channel image sensor may capture a series of digital images in conjunction with the series of illumination events (step 304). Specifically, the imagePATENTAttorney Docket No. 133121 .8009. WO02 sensor may capture at least one digital image under the colored light produced for each illumination event. Following step 304, several subsets of digital images will be available to the characterization module, namely, a first subset of digital images that are associated with the first color channel, a second subset of digital images that are associated with the second color channel, and a third subset of digital images that are associated with the third color channel. Each digital image in the first subset may correspond to the first color channel at a different intensity. Similarly, each digital image in the second and third subsets may correspond to the second and third color channels at a different intensity, respectively. Moreover, a reference digital image can be generated that does not coincide with light being emitted by the first, second, or third color channel (step 305). Said another way, the multi-channel image sensor may also capture a reference image without any illumination events being performed.

[0059] A characterization module can then establish a reflectance profile for the object based on an analysis of the series of digital images and an illumination spectrum that is known to be associated with the ilium inant. For example, the characterization module may calculate reflectance of the object on a per-pixel basis based on the module can then establish a reflectance profile for the object based on the series of digital images and the illumination spectrum.Overview of Characterization Module

[0060] Figure 4 illustrates a network environment 400 that includes a characterization module 404 executing on a computing device 402. The computing device 402 could be, for example, computing device 100 of Figure 1 or computing device 200 of Figure 2. In some embodiments, the characterization module 404 is an integral part of the computing device 402 on which it resides. For example, the characterization module 404 may be part of, or accessible to, the operating system of the computing device. In other embodiments, the characterization module 404 is representative of a computer program that can be selectively executed by the computing device, for example, upon receiving input indicative of a request to determinePATENTAttorney Docket No. 133121 .8009. WO02 the color and shade of a target object of interest.

[0061] Individuals may be able to interface with the characterization module 404 via an interface 406. As discussed above, the characterization module 404 may be responsible for driving illuminants to illuminate a target object and then examining digital images of the target object generated by an image sensor to establish characteristics thereof. As discussed above, the illuminants and image sensor could be part of the computing device 402. The characterization module 404 may also be responsible for creating or supporting interfaces through which an individual can view the digital images, initiate post-processing operations, manage preferences, etc.

[0062] The characterization module 404 may reside in a network environment 400 as shown in Figure 4. Thus, the computing device 402 that executes the characterization module 404 may be connected to one or more networks 408a-b. The networks 408a-b can include personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), cellular networks, the Internet, etc.

[0063] Generally, the characterization module 404 resides on the same computing device 402 as the image sensor and known illuminant. For example, the characterization module 404 may be part of a mobile application through which an image sensor of a mobile phone can be operated. In other embodiments, the characterization module 404 is communicatively coupled to the image sensor and / or the known illuminant across a network. For example, the characterization module 404 may be executed by a platform (also referred to as a “cloud platform”) that resides on a computer server and that is accessible via a network (e.g., the Internet), and the platform may acquire digital images from another computing device (e.g., a mobile phone) via the network.

[0064] In some embodiments, the characterization module 404 is executed by a cloud computing service operated by Amazon Web Services®, Google Cloud Platform™, Microsoft Azure®, or a similar technology. In such embodiments, thePATENTAttorney Docket No. 133121 .8009. WO02 characterization module 404 may reside on a computer server that is communicatively coupled to one or more other computer servers 410. The other computer servers 410 can include color mixing models, items necessary for post-processing such as heuristics and algorithms, and other assets.Overview of Processing System

[0065] Figure 5 is a block diagram illustrating an example of a processing system 500 that is capable of implementing the operations described herein. For example, components of the processing system 500 may be hosted on a computing device that includes the known illuminant, image sensor, and a characterization module.

[0066] The processing system 500 may include a processing unit 502, main memory 506, non-volatile memory 510, network adapter 512, display 518, input / output device 520, control device 522 (e.g., a keyboard or pointing device), drive unit 524 including a storage medium 526, and signal generation device 530 that are communicatively connected to a bus 516. The bus 516 is illustrated as an abstraction that represents one or more physical buses or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 516, therefore, can be a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), inter-integrated circuit (l2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as “Firewire”).

[0067] The processing unit 502 can have generic characteristics similar to general- purpose CPUs or graphical processing units (GPUs), or the processor may be an application-specific integrated circuit (ASIC) that provides control functions to the computing device of which the processing system 500 is a part.

[0068] While the main memory 506, non-volatile memory 510, and storage medium 526 are shown to be a single medium, the terms “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store onePATENTAttorney Docket No. 133121 .8009. WO02 or more sets of instructions 504, 508, 528. The terms “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying instructions for execution by the processing system 500.

[0069] In general, the routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a computer program. Computer programs typically comprise one or more instructions (e.g., instructions 504, 508, 528) set at various times in various memory and storage devices in a computing device. When read and executed by the processing unit 502, the instructions cause the processing system 500 to perform operations to execute elements involving the various aspects of the present disclosure.

[0070] Further examples of machine- and computer-readable media include recordable-type media, such as volatile memory and non-volatile memory 510, removable disks, hard disk drives, and optical disks (e.g., Compact Disk Read-Only Memory (CD-ROMS) and Digital Versatile Disks (DVDs)), and transmission-type media, such as digital and analog communication links.

[0071] The network adapter 512 enables the processing system 500 to mediate data in a network 514 with an entity that is external to the processing system 500 through any communication protocol supported by the processing system 500 and the external entity. The network adapter 512 can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, bridge router, a hub, a digital media receiver, a repeater, or any combination thereof.Remarks

[0072] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of thePATENTAttorney Docket No. 133121 .8009. WO02 invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0073] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments can vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.

[0074] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.

Claims

1. PATENT Attorney Docket No. 133121 .8009. WO02CLAIMSWhat is claimed is:1 . A non-transitory medium with instructions stored thereon that, when executed by a processing unit of a computing device that also includes an illuminant and an image sensor, cause the processing unit to perform operations comprising: recording digital images of an object that are generated by the image sensor while the object is differentially illuminated by the illuminant, wherein each of the digital images is generated in conjunction with a different color or intensity of illumination by the illuminant; estimating, for the object, a reflectance spectrum based on an analysis of red, green, and blue values of the digital images and a known illumination spectrum that is associated with the illuminant; recording a digital image of the object that is generated by the image sensor while the illuminant does not emit any light; and estimating an unknown illumination spectrum that is associated with one or more unknown ilium inants that illuminate the object based on an analysis of red, green, and blue values of the digital image and the reflectance spectrum estimated for the object.

2. The non-transitory medium of claim 1 , wherein the illuminant is a multi-channel illuminant that is able to emit red light, green light, and blue light, and wherein the digital images includes (i) a first subset of digital images that are captured in conjunction with different intensities of red light, (ii) a second subset of digital images that are captured in conjunction with different intensities of green light, and (iii) a third subset of digital images that are captured in conjunction with different intensities of blue light.

3. The non-transitory medium of claim 2, wherein the operations further comprise:PATENTAttorney Docket No. 133121 .8009. WO02 posting, to an interface, a second digital image that is representative of the digital image as modified to account for the unknown illumination spectrum of the one or more unknown illuminants.

4. The non-transitory medium of claim 1 , wherein said estimating the reflectance spectrum is accomplished by establishing, based on the red, green, and blue values of the digital images, appropriate red, green, and blue values for the surface on a per-pixel basis.

5. The non-transitory medium of claim 1 , wherein the operations further comprise: receiving, through an interface, input that is indicative of a request to adjust the digital image to mimic a given ambient lighting condition; adjusting at least some of the red, green, and blue values of the digital image, so as to create a second digital image, based on how the given ambient lighting condition compares to an actual ambient lighting condition as represented by the unknown illumination spectrum; and posting the second digital image to the interface for review.

6. The non-transitory medium of claim 1 , wherein the object is a face of an individual, wherein the image sensor is included in a front-facing camera of the computing device, and wherein the instructions further comprise: indicating, through an interface, that the individual is to position and orient the computing device such that the face is viewable by the frontfacing camera; and indicating, through the interface, whether the individual needs to position the computing device nearer to, or further from, the face.PATENTAttorney Docket No. 133121 .8009. WO027. A method performed by a computer program executing on a computing device, the method comprising: recording digital images of an object that are generated by an image sensor while the object is differentially illuminated by a light source, wherein each of the digital images is generated in conjunction with a different color or intensity of illumination by the light source; estimating, for the object, a reflectance spectrum based on an analysis of the digital images and a known illumination spectrum that is associated with the light source; recording a digital image of the object that is generated by the image sensor while the light source does not emit any light; and estimating an unknown illumination spectrum that is associated with one or more unknown ilium inants that illuminate the object based on an analysis of the digital image and the reflectance spectrum estimated for the object.

8. The method of claim 7, wherein the image sensor and the light source are part of the computing device.

9. The method of claim 7, wherein the computing device is a mobile phone or a tablet computer, wherein the light source is a display, and wherein the image sensor is included in a front-facing camera, such that each of the digital images is representative of a photograph that a user of the computing device has taken of himself or herself.

10. The method of claim 7, further comprising: prompting performance of illumination events by the illuminant by transmitting an instruction, or a series of instructions, to the illuminant; andPATENTAttorney Docket No. 133121 .8009. WO02 prompting generation of the digital images by transmitting an instruction, or a series of instructions, to the image sensor that causes the digital images to be generated in conjunction with the illumination events.11 . The method of claim 10, wherein the illumination events include (i) a first subset of illumination events, each of which corresponds to a first color but a different intensity of the first color, (ii) a second subset of illumination events, each of which corresponds to a second color but a different intensity of the second color, and (iii) a third subset of illumination events, each of which corresponds to a third color but a different intensity of the third color.

12. The method of claim 7, wherein the object is a part of a human body.

13. The method of claim 7, wherein the light source is configured to emit electromagnetic radiation within a non-visible portion of the electromagnetic spectrum.

14. The method of claim 7, wherein the digital images are representative of frames of a video.

Images