Detecting light conditions in images

Abstract

Implementations described herein relate to methods, devices, and computer-readable media to determine light conditions during capture of an image. In some implementations, the method includes obtaining an image and determining if image brightness of the image is above a brightness threshold. If the image brightness is above the brightness threshold, the method further includes outputting a label that indicates that the image depicts non-overcast conditions, if the image brightness is not above the brightness threshold, the method further includes determining if a cumulative distribution function (CDF) for the image has a jump. If the CDF has a jump, the method includes comparing a bin number of the jump with a threshold bin number and outputting a label based on the comparison; else, the method includes determining a divergence value, comparing the divergence value to a divergence threshold, and outputting a label based on the comparison.

Description

DETECTING LIGHT CONDITIONS IN IMAGESBACKGROUND

[0001] When capturing a photo or video with a digital camera, the output photo or video may have certain unsatisfactory attributes. For example, depending on whether the sky is overcast or not, exposure and white balance adjustments applied to the raw image detected by the digital camera sensor (e.g., a CMOS sensor) may produce an overexposed output image and / or an image that has incorrect white balance.

[0002] Since sunlight can vary significantly depending on the cloud coverage, tuning the image if it is captured in outdoor conditions is not adequate to cover the spectrum of outdoor lighting conditions. Some cameras use center metering, e.g., measuring light in the middle of the captured image) to determine exposure parameters, independent of whether the image is captured indoor or outdoor. Based on center metering, metering weights are generated to improve auto exposure.

[0003] Some techniques to determine whether the sky is overcast (e.g., cloud coverage classification) utilize machine learning (ML) models, e.g., neural network models, k-nearest neighbor classifiers, etc. However, machine learning techniques are computationally expensive, require large training datasets, and require pre-training that takes computational resources and time. Also, when the ML model output has an error, e.g., an image taken under cloudy conditions being identified as sunny or vice-versa, it is difficult to determine the reason for the error, since such machine learning models are black box. Fixing failures usually requires retraining the model, e.g., performing additional model training with more examples of test cases for which failure occurs.

[0004] In some techniques, individual pixels of the image may be labeled as sky or cloud based on color features in the image. However, per-pixel cloud classification is computationally expensive and may not be feasible on digital cameras with relatively low computational capacity and / or running on battery power. Thus, it is difficult to detect whether the sky is overcast or not from the raw image captured by a digital image sensor.

[0005] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.SUMMARY

[0006] Implementations described herein related to systems, methods, and non-transitory computer-readable media to automatically determine light conditions in an image. Tn some implementations, a computer-implemented method to determine light conditions during capture of an image includes obtaining an image from a camera and determining if image brightness of the image is above a brightness threshold. If the image brightness is above the brightness threshold, the method further includes outputting a label that indicates that the image depicts non-overcast conditions.

[0007] If the image brightness is not above the brightness threshold, the method further includes determining if a cumulative distribution function (CDF) for the image has a jump. If the CDF has a jump, the method includes determining if a bin corresponding to the jump is above a threshold bin number. If the bin corresponding to the jump is above the threshold bin number, the method includes outputting the label that indicates that the image depicts nonovercast conditions. If the bin corresponding to the jump is not above the threshold bin number, outputting a label that indicates that the image depicts overcast conditions.

[0008] If the CDF does not have a jump, the method further includes determining a divergence value based on blue-red (B / R) and blue-green (B / G) channels of the image. If the divergence value is above a divergence threshold, the method includes outputting the label that indicates that the image depicts overcast conditions. If the divergence value is not above the divergence threshold, the method includes outputting the label that indicates that the image depicts non-overcast conditions.

[0009] Tn some implementations, the label includes a numerical value between zero and one, wherein zero is associated with non-overcast conditions and wherein values between zero and one indicate overcast conditions. In some implementations, higher values in the label correspond to greater intensity of clouds. In some implementations, the image is a frame of a video, and the method further includes smoothing the numerical value using temporal filtering. In some implementations, the method is performed for a subset of frames of the video. In some implementations, the video has a framerate greater than 5 frames per second, and the subset of frames includes 5 frames per second. In some implementations, the method may further include providing the numerical value to an image processing technique that utilizes the label to perform exposure adjustment, white balance adjustment, or exposure and white balance adjustment on the image.

[0010] In some implementations, determining if image brightness of the image is above the brightness threshold includes determining that overall brightness of the image exceeds a threshold light value.

[0011] In some implementations, the method further includes obtaining a segmentation mask for the image that identifies each pixel of the image as a sky pixel or a non-sky pixel. In these implementations, determining if the image brightness of the image is above the brightness threshold includes transforming the sky pixels into hue saturation value (HSV) colorspace; after the transforming, setting a color value (V) of the transformed pixels to 100%; after setting the color value, transforming the sky pixels to red-green-blue (RGB) colorspace; and determining a proportion of pixels of the transformed image in the RGB colorspace that are classified as blue pixels. If the proportion of pixels that are classified as blue pixels meets a threshold proportion, the method includes determining that the image brightness is above the brightness threshold. If the proportion of pixels that are classified as blue pixels does not meet the threshold proportion, the method includes determining that the image brightness is not above the brightness threshold.

[0012] In some implementations, the method further includes obtaining a segmentation mask for the image, wherein the segmentation mask identifies each pixel of the image as a sky pixel or a non-sky pixel. In these implementations, determining if the image brightness of theimage is above the brightness threshold includes transforming the sky pixels into hue saturation value (HSV) colorspace; and determining a proportion of pixels of the transformed image that are classified as gray based on a comparison of a saturation component (S) of pixel value of the sky pixels in the HSV colorspace with a grayness threshold. If the proportion of pixels that are classified as gray meets a threshold proportion, the method includes determining that the image brightness is not above the brightness threshold. If the proportion of pixels does not meet the threshold proportion, the method includes determining that the image brightness is above the brightness threshold.

[0013] Tn some implementations, the method further includes determining the threshold bin number based on overall brightness of the image. In some implementations, determining the threshold bin number is based on a time of day associated with capture of the image.

[0014] In some implementations, the method further includes obtaining a segmentation mask for the image that identifies each pixel of the image as a sky pixel or a non-sky pixel, wherein the image is a low-resolution image that includes auto exposure (AE) statistics. In these implementations, determining if a cumulative distribution function (CDF) for the image has a jump includes generating a blue-red (B / R) histogram based on pixel values of the sky pixels; normalizing the B / R histogram to represent a discrete probability distribution; calculating the CDF for the image based on the normalized B / R histogram; determining that the CDF for the image has the jump if slope of the CDF exceeds a slope threshold; and determining that the CDF for the image does not have the jump if the slope of the CDF does not exceed the slope threshold. In some implementations, if it is determined that the CDF for the image does not have the jump, the method further includes generating a blue-green (B / G) histogram based on pixel values of the sky pixels; and determining pairwise distances between the B / R histogram and the B / G histogram. In these implementations, the divergence value is an aggregate value determined from the pairwise distances.

[0015] Some implementations include a computing device comprising a hardware image processor, an image sensor coupled to the hardware image processor, and a memory coupled to the hardware image processor, with instructions stored thereon that, when executed by thehardware image processor, cause the hardware image processor to perform any of the methods described herein.

[0016] In some implementations, the computing device further includes a display screen coupled to the hardware image processor. In these implementations, the memory has further instructions stored thereon that cause the hardware image processor to perform operations that include performing one or more exposure adjustment, white balance adjustment, or exposure and white balance adjustment on the image using the numerical value to obtain a preview image; and displaying the preview image on the display screen.

[0017] Some implementations include a non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Fig. 1 is a block diagram of an example network environment which may be used for one or more implementations described herein.

[0019] Fig. 2 illustrates an example image, a corresponding segmentation map, and a corresponding segmentation mask.

[0020] Fig. 3 is a block diagram illustrating an example method to determine light conditions during capture of an image, according to some implementations.

[0021] Figures 4A-4C each illustrate an example image, a corresponding B / R histogram, and a cumulative distribution function.

[0022] Figures 5A-5C each illustrate an example image, a corresponding B / R histogram, and a cumulative distribution function.

[0023] Fig. 6 is a block diagram of an example device which may be used for one or more implementations described herein.DETAILED DESCRIPTION

[0024] The light conditions under which an image is captured, e.g., indoors or outdoors, under sunshine or under overcast conditions, in the morning or in the evening, etc. can affect the quality of the image. While image processing techniques can be utilized to post-process a captured image to adjust one or more parameters (e g., white balance, exposure, saturation, etc.), such processing may be cumbersome to perform manually, and may produce an unsatisfactory output image. Further, such post-processing is computationally expensive. Also, such image processing techniques cannot be utilized to provide a preview of an image being captured (e.g., in real time) via a screen of a device (e.g., a smartphone or other digital camera) used for image capture.

[0025] Implementations described herein related to systems, methods, and non-transitory computer-readable media to automatically determine light conditions in an image. For example, a label may be output that indicates whether the image is captured under non-overcast conditions or overcast conditions. A label may, for example, indicate that an image is captured under non-overcast conditions by indicating that the image is captured under sunny conditions, or by indicating that the image is captured under clear conditions. In some examples, overcast conditions may correspond to 70%-100% cloud coverage. In some examples, non-overcast conditions may correspond to less than 70% cloud coverage. Tn various examples, the amount of cloud coverage that corresponds to overcast or non-overcast conditions may be different based on various factors, e.g., location (latitude and / or longitude), time of day, cloud density (that indicates the amount of water in a cloud), etc.

[0026] In some implementations, a raw image (e.g., auto exposure statistics) captured by an image sensor of the device may be utilized to determine the light conditions in the image. In some implementations, the label may indicate an intensity of clouds at the time of image capture. The label can be utilized as an input to techniques that modify the image, e.g., to perform exposure adjustment, white balance adjustment, etc. The detection of light conditions can be performed in near real time and the detected light conditions can be utilized to generate a label that can be used to adjust one or more properties of the image prior to providing a livepreview image. Some implementations also include detecting light conditions during video capture.

[0027] A client device such as a smartphone, a smartwatch, a tablet, a standalone digital camera, or any other client device that includes a camera can implement light condition detection to improve the preview image as well as the quality of an output image captured by the camera.

[0028] Fig. 1 illustrates a block diagram of an example network environment 100, which may be used in some implementations described herein. In some implementations, network environment 100 includes one or more server systems, e.g., server system 102 in the example of Fig. 1. Server system 102 can communicate with a network 130, for example. Server system 102 can include a server device 104 and a database 106 or other storage device. In some implementations, server device 104 may provide an image application 158.

[0029] Network environment 100 also can include one or more client devices, e.g., client devices 120, 122, 124, and 126, which may communicate with each other and / or with server system 102 via network 130. Network 130 can be any type of communication network, including one or more of the Internet, local area networks (LAN), wireless networks, switch or hub connections, etc. In some implementations, network 130 can include peer-to-peer communication between devices, e.g., using peer-to-peer wireless protocols (e.g., Bluetooth®, Wi-Fi Direct, etc.), etc. One example of peer-to-peer communications between two client devices 120 and 122 is shown by arrow 132.

[0030] For ease of illustration, Fig. 1 shows one block for server system 102, server device 104 and database 106, and four blocks for client devices 120, 122, 124, and 126. Server blocks 102, 104, and 106 may represent multiple systems, server devices, and network databases, and the blocks can be provided in different configurations than shown. For example, server system 102 can represent multiple server systems that can communicate with other server systems via the network 130. In some implementations, server system 102 can include cloud hosting servers, for example. In some examples, database 106 and / or other storage devices can be provided in server system block(s) that are separate from server device 104 and can communicate with server device 104 and other server systems via network 130.

[0031] Also, there may be any number of client devices Each client device can be any type of electronic device, e.g., desktop computer, laptop computer, portable or mobile device, cell phone, smartphone, standalone camera, tablet computer, television, TV set top box or entertainment device, wearable devices (e.g., display glasses or goggles, wristwatch, headset, armband ewelry, etc.), personal digital assistant (PDA), media player, game device, etc. Some client devices may also have a local database similar to database 106 or other storage. In some implementations, network environment 100 may not have all of the components shown and / or may have other elements including other types of elements instead of, or in addition to, those described herein.

[0032] In various implementations, end-users Ul, U2, U3, and U4 may communicate with server system 102 and / or each other using respective client devices 120, 122, 124, and 126. In some examples, users Ul, U2, U3, and U4 may interact with each other via applications running on respective client devices and / or server system 102, and / or via a network service, e.g., a social network service or other type of network service, implemented on server system 102. For example, respective client devices 120, 122, 124, and 126 may communicate data to and from one or more server systems (e g., system 102).

[0033] In some implementations, the server system 102 may provide appropriate data to the client devices such that each client device can receive communicated content or shared content uploaded to the server system 102 and / or network service. In some examples, users U1-U4 can interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0034] A network service implemented by server system 102 can include a system allowing users to perform a variety of communications, form links and associations, upload and post shared content such as images, text, video, audio, and other types of content, and / or perform other functions. For example, a client device can display received data such as content posts sent or streamed to the client device and originating from a different client device via a server and / or network service (or from the different client device directly), or originating from a server system and / or network service In some implementations, client devices can communicate directly with each other, e.g., using peer-to-peer communications between client devices asdescribed above. Tn some implementations, a “user” can include one or more programs or virtual entities, as well as persons that interface with the system or network.

[0035] In some implementations, any of client devices 120, 122, 124, and / or 126 can provide one or more applications. For example, as shown in Fig. 1, client device 120 may provide a camera application 156. Client devices 122-126 may also provide similar applications.

[0036] In some implementations, a client device may include applications that provide various types of functionality, e.g., calendar, address book, e-mail, web browser, shopping, transportation (e.g., taxi, train, airline reservations, etc.), entertainment (e.g., a music player, a video player, a gaming application, etc.), social networking (e.g., messaging or chat, audio / video calling, sharing images / video, etc.) and so on. In some implementations, one or more of the applications may be standalone applications that execute on client device 120. In some implementations, one or more of the applications may access a server system, e g., server system 102, that provides data and / or functionality of the applications.

[0037] A user interface on a client device 120, 122, 124, and / or 126 can enable the display of user content and other content, including images, video, data, and other content as well as communications, privacy settings, notifications, and other data. Such a user interface can be displayed using software on the client device, software on the server device, and / or a combination of client software and server software executing on server device 104, e.g., application software or client software in communication with server system 102. The user interface can be displayed by a display device of a client device or server device, e.g., a touchscreen or other display screen, projector, etc. In some implementations, application programs running on a server system can communicate with a client device to receive user input at the client device and to output data such as visual data, audio data, etc. at the client device.

[0038] In various implementations, any of client devices 120-126 may be used by a user to capture an image, e g., a still image and / or a video. In some implementations, database 106 may store content items, e.g., images and / or videos captured by any of client devices 120-126.

[0039] Tn some implementations, client devices 120-126 may include a camera. The camera may include an image sensor (e.g., a CMOS sensor, or other digital sensor). The image sensor may capture raw images, including auto exposure (AE) statistics. In some implementations, the client device 120 may include a camera pipeline (also referred to as an image capture pipeline) that includes a hardware image processor (separate from a main processor of the device, or part of the main processor). In some implementations, the raw image may be a low-resolution image. In some implementations, client devices 120-126 may include a screen. In these implementations, the hardware image processor may perform exposure adjustment, white balance adjustment, etc. on the raw image to obtain a preview image and display the preview image on the screen.

[0040] An image as referred to herein can include a digital image having pixels with one or more pixel values (e.g., color values, brightness values, etc.). An image can be a still image (e.g., still photos, images with a single frame, etc.), a dynamic image (e.g., animations, animated GIFs, cinemagraphs where a portion of the image includes motion while other portions are static, etc.), or a video (e.g., a sequence of images or image frames that may include audio). While the remainder of this document refers to an image as a static image, it may be understood that the techniques described herein are applicable for dynamic images, video, etc. For example, implementations described herein can be used with still images (e.g., a photograph, or other image), videos, or dynamic images.

[0041] Fig. 2 illustrates an example image, a corresponding segmentation map, and a corresponding segmentation mask. An example image 202 is captured by a camera, e.g., a digital camera using a complementary metal-oxide semiconductor (CMOS) sensor. In this example, image 202 is a low resolution image and includes auto exposure (AE) stats for the image. In some implementations, the low resolution image may have a resolution substantially lower, e.g., one, two, or more orders of magnitude lower than the image captured by the sensor.

[0042] Fig. 2 also illustrates a segmentation map 204 for the image 202. The segmentation map is a binary map that separates sky pixels (in white) in the image from other pixels (in black). Any suitable image segmentation technique can be utilized to obtain the segmentation map. In some implementations, machine learning techniques may be utilized to perform imagesegmentation. Tn some implementations, the image segmentation technique may be implemented in hardware, e.g., in the camera pipeline, and may process a raw image captured by the sensor to generate the segmentation map.

[0043] In some implementations, the segmentation map and segmentation mask may be determined in the camera hardware itself, e.g., by a signal processor that is adj acent to the sensor and that can process raw images (including auto exposure statistics) as detected by the sensor. In some implementations, the segmentation map and segmentation mask may be generated using a trained machine learning model (e.g., a segmentation model). The segmentation model may be provided an image from a buffer in the camera hardware. For example, the image provided to the segmentation model may be a YUV image (where Y corresponds to brightness, U corresponds to blue projection, and V corresponds to red projection). In some implementations, the image and the segmentation mask may be utilized to determine light conditions under which the image was captured.

[0044] Generating a segmentation mask can enable downstream processing, e.g., determining the light conditions under which the image was captured, and utilizing such conditions to perform exposure adjustment, white balance adjustment, etc. In some implementations, performing such adjustments prior to further processing the image, e.g., encoding the image in a compressed format such as JPEG, can enable higher quality output images to be produced by the camera. Further, in some implementations, the image 202 may be a low resolution image, e.g., several orders of magnitude smaller than the actual image captured by the camera. Using such an image to determine a segmentation mask can enable computation of segmentation mask with low computational requirements in terms of processing hardware as well as power. Also, use of the low resolution image and corresponding segmentation mask can enable detection of light conditions to be performed at a high frequency, e.g., at the rate of two, three, five, or more frames per second. By determining light conditions at such high frequency, a video captured by the camera can also be subjected to exposure adjustment, white balance adjustment, etc. based on the determined instantaneous light conditions under which the video is captured. For example, as the camera moves (e g., pans, zooms, rotates, etc.) during videocapture, the light conditions can change over a set of frames (e.g., if the camera pans from an outdoor scene to an indoor scene).

[0045] Fig. 2 further illustrates a segmentation mask 206 for the image 202 based on segmentation map 204. In the example of Fig. 2, segmentation mask 206 retains pixel values for the sky pixels, and all other pixels are set to a uniform value (e.g., black). In various implementations, segmentation mask 206 may be utilized to determine light conditions under which the image 202 was captured.

[0046] While Fig. 2 illustrates an outdoor image with a sky portion near the top of the image, the segmentation map and segmentation mask may be generated for any image. For example, if the image is captured with a camera angle that is sideways, the sky portion may be on the left or right side of the image. In another example, if the image is captured indoors, there may be no sky portion and the segmentation mask is blank. In further examples, the sky may be partially occluded in some portions, e.g., if a tree or other structure is present, and the segmentation map includes segmentation that corresponds to the image.

[0047] Fig. 3 is a block diagram illustrating an example method to determine light conditions during capture of an image, according to some implementations.

[0048] In some implementations, some or all of the method 300 can be implemented on one or more client devices 120, 122, 124, or 126 as shown in Fig. 1, one or more server devices, and / or on both server device(s) and client device(s). In described examples, the implementing system includes one or more digital processors or processing circuitry ("processors"), and one or more storage devices (e.g., a database 106 or other storage). In some implementations, different components of one or more servers and / or clients can perform different blocks or other parts of the method 300. In some examples, a first device is described as performing blocks of method 300. Some implementations can have one or more blocks of method 300 performed by one or more other devices (e.g., other client devices or server devices) that can send results or data to the first device.

[0049] Tn some implementations, the method 300, or portions of the method, can be initiated automatically by a system. In some implementations, the implementing system is a first device. For example, the method (or portions thereof) can be periodically performed, or performed based on one or more particular events or conditions, e.g., a client device capturing an image or video using a camera of the client device. In some implementations, method 300 may be implemented entirely in circuitry that is part of an image capture pipeline (camera pipeline) of the client device that captures the image. For example, dedicated circuitry such as an application-specific integrated circuit (ASIC), a co-processor, an image processor, etc. that is part of the image capture pipeline (e.g., processes a raw image captured by an image sensor of the client device) may implement method 300.

[0050] Method 300 may begin at block 302. At block 302, an image is obtained. For example, the image may be obtained from a digital camera. For example, the image may be captured using an image sensor such as a CMOS sensor. In some implementations, the image may be a raw image captured by the sensor before any adjustments. In some implementations, the image may include auto exposure statistics captured by the image sensor. In some implementations, the image may be a low resolution image. Block 302 may be followed by block 304.

[0051] At block 304, it is determined if the image brightness of the image is above a brightness threshold. For example, an image that has a brightness above the brightness threshold may be captured outdoors in daylight. In some implementations, the overall brightness of the image (over all pixels of the image) exceeds a threshold light value (e.g., 80,000 lux, which is an example brightness threshold), it is determined that the image brightness is above the brightness threshold.

[0052] In some implementations, determining whether the image brightness is above the brightness threshold may utilize a segmentation mask. The segmentation mask may be obtained for the image, e.g., from an image segmenter that is implemented as part of the camera pipeline. The segmentation mask may identify individual pixels of the image as a sky pixel (depicting the sky) or a non-sky pixel (not depicting the sky). In some implementations, if it is determined that the segmentation mask identifies less than a threshold percentage of the image pixels assky pixels (e g , 15%, 20%, etc ), method 300 may be terminated without performing the subsequent blocks (blocks 406-320).

[0053] In some implementations, determining if the image brightness of the image is above the brightness threshold may be performed using the segmentation mask. Brightness of the image may be determined from aperture, exposure time, and ISO sensitivity. The brightness of the image corresponds to a lux value (light) in the scene captured in the image. The image may be in red-green-blue (RGB) colorspace. The sky pixels are transformed from the RGB space into hue saturation value (HSV) colorspace. After the transformation, the V component (color value) of the transformed pixels may be set to 100%.

[0054] The image may be transformed back into RGB colorspace and a proportion of pixels of the transformed image that are classified as blue pixels may be determined. To identify blue pixels, the R component of individual pixels may be determined and compared to a threshold value. In some implementations, the threshold value for the R component may be 166 such that pixels with R component below 166 are classified as blue pixels (since the R component of such pixels may be far from the G and B components).

[0055] In some implementations, the blue pixels may be weighted based on a blueness strength (e.g., which may be determined based on the comparison of the R component and the threshold value. In some implementations, the weight may be inverse to a distance between the R component of the pixel and the threshold value. For example, pixel values that are farther from the threshold value may be assigned a higher weight, e.g., such a pixel may increment the aggregate count by more than 1, and pixel values that are closer to the threshold (barely blue) may be assigned a lower weight, e.g., such a pixel may increment the aggregate count less than 1. In implementations where pixels are weighted based on the blueness strength, really blue pixels have a larger impact on the aggregate count compared to barely blue pixels.

[0056] In some implementations, if a proportion of pixels that are classified as blue pixels meets a threshold proportion (e.g., at least 5%), it may be determined that the image brightness is above the brightness threshold, else it may be determined that the image brightness is not above the brightness threshold.

[0057] Tn some implementations, the saturation component (S) of the transformed image (after converting to HSV colorspace) may be utilized to determine if the image brightness of the image is above the brightness threshold. If the saturation component for a pixel is below a grayness threshold, the pixel may be classified as gray. In some implementations, only a subset of pixels that have a saturation component below the grayness threshold, and that have a color value (V) component above a value threshold may be classified as gray. In some implementations, the grayness threshold may include a gray saturation threshold (e.g., set at 0.2) and a gray value threshold (e g., set at 0.3). The use of a gray value threshold can be valuable in distinguishing between overcast and non-overcast conditions under low light conditions (e.g., when the image is captured at dusk) since under low light conditions, sky pixels may be detected as a dark gray blue color.

[0058] In some implementations, if a proportion of pixels that are classified as gray meets a threshold proportion, it may be determined that the image brightness is not above the brightness threshold, else it may be determined that the image brightness is above the brightness threshold.

[0059] In some implementations, a combination of criteria may be utilized to determine if the image brightness of the image is above the brightness threshold. For example, in some implementations, the proportion of blue pixels and the proportion of gray pixels may be compared to respective threshold proportions, and the image brightness is determined to be above the brightness threshold only if both thresholds are met. In some implementations, either criterion (proportion of blue pixels or proportion of gray pixels) being met may suffice for the image brightness to be determined as being above the brightness threshold. In some implementations, e.g., if the overall brightness of the image is above the threshold light value, determination of blue pixels and / or gray pixels may not be performed. In some implementations, the determination of blue pixels and / or gray pixels may be performed if the overall brightness of the image is in a particular range.

[0060] If it is determined at block 304 that the image brightness is above the brightness threshold, block 304 is followed by block 306. Else, block 304 is followed by block 320.

[0061] At block 306, it is determined if a cumulative distribution function (CDF) for the image has a jump. In some implementations, it may be determined that the CDF has a jump at a particular point (e.g., bin number) if the slope of the CDF at that point (e.g., bin number) exceeds a slope threshold. The slope threshold corresponds to the rate of change between histogram bins.

[0062] In some implementations, the image may be a low-resolution image that includes auto exposure statistics. The sky pixels of the image are identified using the segmentation mask for the image. Determining if the CDF for the image has a jump may include generating a blue- red (B / R) histogram based on pixel values of the sky pixels of the image. For each sky pixel, a ratio of the blue component (B) of the pixel value to the value of the red component (R) of the pixel value may be determined. The B / R histogram may be normalized to represent a discrete probability distribution. In some implementations, the B / R ratio for each pixel may be constrained to a value between 0 and 1 using the sigmoid function. In these implementations, the pixel values may be distributed between a plurality of bins (e.g., 100 bins with a bin width of 0.01) where each bin corresponds to a particular range of B / R ratio values. In some implementations, normalizing the B / R histogram may include, after generating the histogram (distributing pixel values between the plurality of bins), dividing each histogram bin (which contain respective pixel counts) by the total number of pixels in the histogram.

[0063] In some implementations, the cumulative distribution function (CDF) for the image may be calculated based on the normalized B / R histogram. If the CDF for the image has a slope (e.g., a maximum slope for a segment of the CDF) that exceeds a slope threshold, it is determined that the CDF for the image has a jump, else it is determined that the CDF for image does not have a jump. In some implementations, the slope threshold may be selected as a relatively high value, e.g., the slope threshold may be selected such that it corresponds to a jump in CDF values within a single bin (e.g., B / R ratio values within 0.01 of each other) or a small number of bins. If it is determined that the CDF for the image has a jump, block 306 is followed by block 308. Else, block 306 is followed by block 310.

[0064] At block 308, the bin number at which the jump is detected (slope of the CDF exceeding the slope threshold) is compared to a threshold bin number. The jump may beindicative of a clearly overcast or clearly sunny image, based on the bin at which the sharp jump is seen.

[0065] Figures 4A-4C each illustrate an example image, a corresponding B / R histogram, and a cumulative distribution function. An example is illustrated in Fig. 4A. In Fig. 4A, an image 402 is depicted along with a corresponding cumulative distribution function 404, and a B / R histogram 406. Image 402 is captured in the late afternoon under overcast skies. In Fig. 4A, the threshold bin (405) is bin number 40. In contrast, a higher threshold bin number may be selected for images that are captured in bright conditions (e.g., in the morning). An example is illustrated in Fig. 4B. In Fig. 4B, an image 422 is depicted along with a corresponding cumulative distribution function 424, and a B / R histogram 426. In Fig. 4B, the threshold bin (425) is bin number 40. Image 422 is captured on a clear (non-overcast) morning.

[0066] In Fig. 4C, an image 442 is depicted along with a corresponding cumulative distribution function 444, and a B / R histogram 446. The image 442 of Fig. 4C is captured in the morning with overcast skies (with some clouds present). In Fig. 4C, there is no jump in the CDF since the slope of the CDF 444 does not meet the slope threshold. In some implementations, a combination of both the time of day and the overall brightness of the image may be used to select the threshold bin number.

[0067] In some implementations, the threshold bin number may be determined based on the overall brightness of the image (e.g., including sky pixels and non-sky pixels). In some implementations, the threshold bin number may be determined based on a time of day associated with capture of the image. For example, for images that are captured in the evening, sunny skies may be observed in a CDF jump at darker values, and accordingly, the threshold bin number may be selected as a low bin number. The use of an adaptive threshold enables adjusting for darker scenes (e.g., sunset) where the overall brightness of the image is lower. In some implementations, the adaptive threshold may be obtained based on interpolation between two thresholds using a brightness value. The adaptive threshold moves closer to the higher threshold if the overall brightness is high (bright scene) and moves closer to the lower threshold if the overall brightness is low (dark scene). In some implementations, the interpolation may be linear interpolation.

[0068] Figures 5A-5C illustrate examples of different threshold bin numbers under different conditions. In Fig. 5A, an image 502 is depicted along with a corresponding cumulative distribution function 504, and a B / R histogram 506. The image 502 is taken near sunset and under clear skies. The threshold 505 corresponds to bin number 20.

[0069] In Fig. 5B, an image 522 is depicted along with a corresponding cumulative distribution function 524, and a B / R histogram 526. The image 522 is taken in the morning under clear skies. The threshold 525 corresponds to bin number 40. In Fig. 5C, an image 542 is depicted along with a corresponding cumulative distribution function 544, and a B / R histogram 546. The image 542 is taken in the evening under overcast skies. The threshold 545 is at bin number 37 (not shown).

[0070] If the bin number at which the jump is detected is above a threshold bin number, block 308 is followed by block 320. Else, block 308 is followed by block 312.

[0071] If it is determined at block 306 that the CDF does not have a jump, block 306 is followed by block 310. At block 310, a divergence value is determined based on B / R and bluegreen (B / G) channels of the image. A blue-green (B / G) histogram is generated based on pixel values of the sky pixels. In some implementations, values of the G channel of the image (sky pixels of the image) are obtained based on Gr and Gb channels. For example, in some implementations, the Gr and Gb channels may be averaged to obtain the G channel - G = (Gr + Gb) / 2. In some implementations, the B / G ratio may be normalized, similar to B / R ratio as described above. In some implementations, the sigmoid function may be used to constrain the B / R ratio between 0 and 1, and a plurality of bins determined for the B / G channel (e.g., 100 bins with a bin width of 0.01).

[0072] In some implementations, a pairwise distance may be determined between the B / R and B / G channels of the image based on the B / R histogram and the B / G histogram. A divergence value may be determined based on aggregating the pairwise distances. In some implementations, the pairwise distance may be calculated using Kullback-Liebler (KL) divergence. KL divergence may indicate the entropy from one probability distribution (Q) to another probability distribution (P) and can be calculated as:where P and Q may be B / R and B / G distributions. The DKL value is the divergence value, aggregated (e.g., using the summation function) over pairwise distances.

[0073] The KL divergence (or other information distance metric that is used) may indicate whether there is a clear difference in shape between the B / R and B / G histograms. A clear difference in shape is indicative of cloudy conditions which typically lead to some cloudy (e.g., gray) patches and some sky (e.g., blue) patches in the image.

[0074] In some implementations, the divergence value may be compared to a divergence threshold. If it is determined that the divergence value is above the divergence threshold, block 310 may be followed by block 312. Else, block 310 may be followed by block 320.

[0075] At block 312, a label is output that indicates that the image depicts overcast conditions.

[0076] At block 320, a label is output that indicates that the image depicts sunny conditions.

[0077] In some implementations, the label that is output may include a numerical value between zero and one, e.g., a floating point number. In these implementations, a numerical value of zero is associated with sunny conditions, and values between zero and one indicate overcast conditions. In some implementations, higher values in the label may correspond to greater intensity of clouds. In some implementations, a label value > 0.5 is indicative of overcast conditions.

[0078] In some implementations, the numerical value in the label may be provided as input to an image processing technique. The image processing technique may utilize the label to perform exposure adjustment, white balance adjustment, or exposure and white balance adjustment on the image, or other operations. The use of the numerical value that indicates the light conditions under which the image was captured (e.g., sunny, mildly overcast, cloudy, etc.) during such adjustments enables the image processing algorithm to perform the operations taking into account the different light conditions under which different images may be capturedand produce an output image that more accurately reflects the subjects depicted in the image. For example, if an image is captured in overcast conditions, there may be more shadows and correspondingly, the subject brightness may be increased and / or the dark tone boost can be adjusted.

[0079] In some implementations, the image may be a frame of a video, and method 300 may be performed multiple times, once per frame of the video. In some implementations, the video may have a particular framerate (e.g., 25 frames per second, 30 fps, 60 fps, 90 fps, etc.) and method 300 may be performed for a subset of frames of the video. For example, method 300 may be performed five times per second such that 5 captured frames are analyzed each second to determine the light conditions under which the video is captured. In various implementations, the subset of frames may include any suitable number of frames per second. Sampling the frames for which the determination of light condition is performed can reduce the computational load, since method 300 may only be performed for the subset of frames, and the determination of the light condition can be utilized for the entire set of frames (e.g., light conditions for neighboring frames of the evaluated frames may be set at the same value as determined for one or more frames in the set of frames).

[0080] In some implementations, after determining the output label including the numerical value (e g , at block 312 or block 320), the numerical value may be smoothed using temporal fdtering. Temporal filtering may include ensuring that there are no sudden changes determined in the light conditions between consecutive frames in the subset of frames of the video (e.g., that depict similar subject matter) so that any downstream image adjustments such as exposure adjustment, white balance adjustment, etc. produce consistent results across a plurality of frames. The light conditions may change over a longer portion (e.g., more than 2, 3, 5, or more frames), e.g., when the camera pans, zooms, rotates, etc. or changes occur in the captured scene such that the light conditions in the scene change.

[0081] Various blocks of method 300 may be combined, split into multiple blocks, or be performed in parallel. Method 300, or portions thereof, may be repeated any number of times using additional inputs. For example, method 300 may be performed for each image capturedby the camera. Tn some implementations, method 300 may be performed for a plurality of frames of a video captured by the camera to determine respective light conditions for the frames.

[0082] In some implementations, method 300 may be performed before a user activates image capture. For example, the user may start the camera, e.g., via a camera application on a smartphone, by starting a standalone digital camera, etc. and to obtain a preview of the image on the device screen or another screen. The user may use the device screen to frame the image capture per their preference. In this case, the image sensor is active and capturing image data from the scene. The image that is captured to display the preview (e.g., a raw image that includes auto exposure statistics) may be processed using method 300 to determine the light conditions. The image may be modified, e.g., by performing exposure adjustment, white balance adjustment, etc. to obtain a preview image that is displayed to the user on the device screen or another screen. This provides the technical benefit that the preview image has higher fidelity to the scene, thus allowing the user to determine whether the image is satisfactory. Further, the preview image may be closer to the output image due to such adjustments being performed. The output image itself may be of overall higher image quality since the camera pipeline can utilize the light conditions in the image during image processing.

[0083] Unlike light condition detection techniques that rely on machine learning models that are computationally expensive, method 300 requires less computational resources and energy. Since method 300 utilizes the raw image from the image sensor and can be implemented with a low resolution input image, the total computational load to detect light conditions is significantly lower than techniques that utilize high resolution input images that are in a file format such as JPEG or other format, rather than being raw images directly obtained from the image sensor.

[0084] Further, while the foregoing description recites the generation and use of B / R and / or B / G histograms, histograms in any colorspace can be utilized. For example, YUV colorspace may be utilized to calculate the CDF and to determine whether the CDF has a jump.

[0085] Further, method 300 may be performed in the camera pipeline without utilizing a main processor of the device that captures the image. Also, unlike machine learning techniques,method 300 provides reliable estimates of light conditions for arbitrary images and any erroneous labels (e.g., labeling a sunny image as overcast) can be examined easily. Still further, because method 300 has low computational demands, it can be used during video capture to automatically determine light conditions. Method 300 and portions thereof are parallelizable and can be utilized with high performance languages for image processing. Parallelizing the execution of one or more blocks of method 300 can reduce latency and overall computational cost (e.g., since multiple available processing units may be utilized to perform different blocks of method 300 in parallel).

[0086] Method 300 provides computationally efficient detection of light conditions and enables tuning for the detected light conditions, thus enabling high image quality in a variety of outdoor settings, due to improved auto white balance, tuning for greater color contrast between overcast skies and foreground objects, etc.

[0087] Fig. 6 is a block diagram of an example device 600 which may be used to implement one or more features described herein. In some implementations, device 600 can implement a server device, e.g., server system 102 or server device 104. In some implementations, device 600 may be used to implement a client device, a server device, or both client and server devices. Device 600 can be any suitable computer system, server, or other electronic or hardware device as described above.

[0088] One or more methods described herein can be run in a standalone program that can be executed on any type of computing device, a program run on a web browser, a mobile application (“app”) run on a mobile computing device (e.g., cell phone, smart phone, tablet computer, wearable device (wristwatch, armbandjewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head mounted display, etc.), laptop computer, etc.). In one example, a client / server architecture can be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives from the server the final output data for output (e.g., for display). Tn another example, all computations can be performed within the mobile app (and / or other apps) on the mobile computing device. In another example, computations can be split between the mobile computing device and one or more server devices.

[0089] Tn some implementations, device 600 includes one or more hardware processor 602 (also referred to as processor 602), a memory 604, input / output (I / O) interface 606, a display device 620, and a camera 630. Processor 602 can be one or more processors and / or processing circuits to execute program code and control basic operations of the device 600. A “processor” includes any suitable hardware system, mechanism or component that processes data, signals or other information. A processor may include a system with a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a hardware image processor coupled directly to camera 630, e.g., via a hardware link between the camera and the hardware image processor, or indirectly to camera 630, e.g., via VO interface 606, a field-programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for achieving functionality, a special-purpose processor to implement neural network model -based processing, neural circuits, processors optimized for matrix computations (e.g., matrix multiplication), or other systems.

[0090] In some implementations, processor 602 may include one or more co-processors for image processing. Processing need not be limited to a particular geographic location or have temporal limitations. For example, a processor may perform its functions in “real-time,” “offline,” in a “batch mode,” etc. Portions of processing may be performed at different times and at different locations, by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0091] Memory 604 is typically provided in device 600 for access by the processor 602 and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), Electrical Erasable Read-only Memory (EEPROM), Flash memory, etc., suitable for storing instructions for execution by the processor, and located separate from processor 602 and / or integrated therewith. Memory 604 can store software operating on the server device 600 by the processor 602, including an operating system 608, applications 612, and application data 614. Applications 612 may include applications such as an image processing application, a camera application, a video application, an image displayengine, etc. Tn some implementations, applications 612 can include instructions that enable processor 602 to perform functions described herein, e.g., some or all of the method of Fig. 3.

[0092] Applications 612 can include, e.g., camera applications (to capture images using camera 630), image editing applications, etc. One or more methods disclosed herein can operate in several environments and platforms, e.g., on a dedicated hardware image processor that implements the method, a programmable processor that implements the method, a device CPU, as a stand-alone computer program that can run on any type of computing device, as a mobile application ("app") run on a mobile computing device, etc.

[0093] Any of software in memory 604 can alternatively be stored on any other suitable storage location or computer-readable medium (e.g., in stored instructions in dedicated storage of a hardware image processor). In addition, memory 604 (and / or other connected storage device(s)) can store one or more images, user preferences, and / or other instructions and data used in the features described herein. Memory 604 and any other type of storage (magnetic disk, optical disk, magnetic tape, or other tangible media) can be considered "storage" or "storage devices."

[0094] VO interface 606 can provide functions to enable interfacing the server device 600 with other systems and devices. Interfaced devices can be included as part of the device 600 or can be separate and communicate with the device 600. For example, network communication devices, storage devices (e.g., memory and / or database 106), and input / output devices can communicate via VO interface 606. In some implementations, the VO interface can connect to interface devices such as input devices (camera 630, keyboard, pointing device, touchscreen, microphone, camera, scanner, sensors, etc.) and / or output devices (display device 620, speaker devices, printers, motors, etc.).

[0095] Some examples of interfaced devices that can connect to VO interface 606 can include one or more display devices 620 that can be used to display content, e.g., images, video, and / or a user interface of an application as described herein. Display device 620 can be connected to device 600 via local connections (e.g., display bus) and / or via networked connections and can be any suitable display device. Display device 620 can include any suitable display device such as an LCD, LED, or plasma display screen, CRT, television, monitor,touchscreen, 3-D display screen, or other visual display device. For example, display device 620 can be a flat display screen provided on a mobile device, multiple display screens provided in a goggles or headset device, or a monitor screen for a computer device.

[0096] The I / O interface 606 can interface to other input and output devices, e.g., network cards, hardware security modules provided as cards or in other form factor to connect to processor 602, etc. Additional examples of input and output devices include one or more cameras which can capture images. Some implementations can provide a microphone for capturing sound (e.g., as a part of captured images, voice commands, etc.), audio speaker devices for outputting sound, or other input and output devices.

[0097] For ease of illustration, Fig. 6 shows one block for each of processor 602, memory 604, I / O interface 606, display device 620, camera 630, software blocks 608 and 612, and application data 614. These blocks may represent one or more processors or processing circuitries, operating systems, memories, I / O interfaces, display devices (e.g., a front screen, a rear screen, an outer screen and inner screen of a foldable device, etc.), cameras (e.g., front and / or rear cameras, other cameras in the device, etc.), applications, and / or software modules. In other implementations, device 600 may not have all of the components shown and / or may have other elements including other types of elements instead of, or in addition to, those shown herein. While some components are described as performing blocks and operations as described in some implementations herein, any suitable component or combination of components of environment 100, device 600, similar systems, or any suitable processor or processors associated with such a system, may perform the blocks and operations described.

[0098] Methods described herein can be implemented by computer program instructions or code, which can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), flash memory, a rigid magnetic disk, an optical disk, a solid-state memory drive, etc. The program instructions canalso be contained in, and provided as, an electronic signal, for example in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.), or in a combination of hardware and software. Example hardware can be programmable processors (e.g., Field-Programmable Gate Array (FPGA), Complex Programmable Logic Device), general purpose processors, graphics processors, Application Specific Integrated Circuits (ASICs), and the like. One or more methods can be performed as part of or component of an application running on the system, or as an application or software running in conjunction with other applications and operating system.

[0099] Although the description has been described with respect to particular implementations thereof, these particular implementations are merely illustrative, and not restrictive. Concepts illustrated in the examples may be applied to other examples and implementations.

[0100] Note that the functional blocks, operations, features, methods, devices, and systems described in the present disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks as would be known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of particular implementations. Different programming techniques may be employed, e.g., procedural, or object-oriented. The routines may execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown as sequential in this specification may be performed at the same time.

Claims

CLAIMS1. A computer-implemented method to determine light conditions during capture of an image, the method comprising: obtaining an image from a camera; determining if image brightness of the image is above a brightness threshold; if the image brightness is above the brightness threshold, outputting a label that indicates that the image depicts non-overcast conditions; and if the image brightness is not above the brightness threshold, determining if a cumulative distribution function (CDF) for the image has a jump; if the CDF has a jump, determining if a bin corresponding to the jump is above a threshold bin number, and if the bin corresponding to the jump is above the threshold bin number, outputting the label that indicates that the image depicts nonovercast conditions and if the bin corresponding to the jump is not above the threshold bin number, outputting a label that indicates that the image depicts overcast conditions; and if the CDF does not have a jump, determining a divergence value based on blue-red (B / R) and blue-green (B / G) channels of the image; if the divergence value is above a divergence threshold, outputting the label that indicates that the image depicts overcast conditions; and if the divergence value is not above the divergence threshold, outputting the label that indicates that the image depicts nonovercast conditions.

2. The computer-implemented method of claim 1, wherein the label includes a numerical value between zero and one, wherein zero is associated with non-overcast conditionsand wherein values between zero and one indicate overcast conditions, wherein higher values correspond to greater intensity of clouds.

3. The computer-implemented method of claim 2, wherein the image is a frame of a video, further comprising smoothing the numerical value using temporal filtering, and wherein the method is performed for a subset of frames of the video.

4. The computer-implemented method of claim 3, wherein the video has a framerate greater than 5 frames per second, and wherein the subset of frames includes 5 frames per second.

5. The computer-implemented method of any one of claims 2 to 4, further comprising providing the numerical value to an image processing technique, wherein the image processing technique utilizes the label to perform exposure adjustment, white balance adjustment, or exposure and white balance adjustment on the image.

6. The computer-implemented method of any one of the preceding claims, wherein determining if image brightness of the image is above the brightness threshold comprises determining that overall brightness of the image exceeds a threshold light value.

7. The computer-implemented method of any one of the preceding claims, further comprising obtaining a segmentation mask for the image that identifies each pixel of the image as a sky pixel or a non-sky pixel, and wherein determining if the image brightness of the image is above the brightness threshold comprises: transforming the sky pixels into hue saturation value (HSV) colorspace; after the transforming, setting a color value (V) of the transformed pixels to 100%; after setting the color value, transforming the sky pixels to red-green-blue (RGB) colorspace; determining a proportion of pixels of the transformed image in the RGB colorspace that are classified as blue pixels;if the proportion of pixels that are classified as blue pixels meets a threshold proportion, determining that the image brightness is above the brightness threshold; and if the proportion of pixels that are classified as blue pixels does not meet the threshold proportion, determining that the image brightness is not above the brightness threshold.

8. The computer-implemented method of any one of the preceding claims, further comprising obtaining a segmentation mask for the image, wherein the segmentation mask identifies each pixel of the image as a sky pixel or a non-sky pixel, and wherein determining if the image brightness of the image is above the brightness threshold comprises: transforming the sky pixels into hue saturation value (HSV) colorspace; determining a proportion of pixels of the transformed image that are classified as gray based on a comparison of a saturation component (S) of pixel value of the sky pixels in the HSV colorspace with a grayness threshold; if the proportion of pixels that are classified as gray meets a threshold proportion, determining that the image brightness is not above the brightness threshold; and if the proportion of pixels does not meet the threshold proportion, determining that the image brightness is above the brightness threshold.

9. The computer-implemented method of any one of the preceding claims, further comprising determining the threshold bin number based on overall brightness of the image.

10. The computer-implemented method of any one of the preceding claims, further comprising determining the threshold bin number based on a time of day associated with capture of the image.

11. The computer-implemented method of any one of the preceding claims, further comprising:obtaining a segmentation mask for the image that identifies each pixel of the image as a sky pixel or a non-sky pixel, wherein the image is a low-resolution image that includes auto exposure (AE) statistics, and wherein determining if a cumulative distribution function (CDF) for the image has a jump comprises: generating a blue-red (B / R) histogram based on pixel values of the sky pixels; normalizing the B / R histogram to represent a discrete probability distribution; calculating the CDF for the image based on the normalized B / R histogram; determining that the CDF for the image has the jump if slope of the CDF exceeds a slope threshold; and determining that the CDF for the image does not have the jump if the slope of the CDF does not exceed the slope threshold.

12. The computer-implemented method of claim 11, wherein if it is determined that the CDF for the image does not have the jump, the method further comprises: generating a blue-green (B / G) histogram based on pixel values of the sky pixels; and determining pairwise distances between the B / R histogram and the B / G histogram, wherein the divergence value is an aggregate value determined from the pairwise distances.

13. A computing device comprising: a hardware image processor; an image sensor coupled to the hardware image processor; and a memory coupled to the hardware image processor, with instructions stored thereon that, when executed by the hardware image processor, cause the hardware image processor to perform operations comprising: obtaining an image from the image sensor; determining if image brightness of the image is above a brightness threshold; if the image brightness is above the brightness threshold, outputting a label that indicates that the image depicts non-overcast conditions; and if the image brightness is not above the brightness threshold,determining if a cumulative distribution function (CDF) for the image has a jump; if the CDF has a jump, determining if a bin corresponding to the jump is above a threshold bin number, and if the bin corresponding to the jump is above the threshold bin number, outputting the label that indicates that the image depicts nonovercast conditions and if the bin corresponding to the jump is not above the threshold bin number, outputting a label that indicates that the image depicts overcast conditions; and if the CDF does not have a jump, determining a divergence value based on blue-red (B / R) and blue-green (B / G) channels of the image; if the divergence value is above a divergence threshold, outputting the label that indicates that the image depicts overcast conditions; and if the divergence value is not above the divergence threshold, outputting the label that indicates that the image depicts nonovercast conditions.

14. The computing device of claim 13, wherein the label includes a numerical value between zero and one, wherein zero is associated with non-overcast conditions and wherein values between zero and one indicate overcast conditions, wherein higher values correspond to greater intensity of clouds.

15. The computing device of claim 13 or 14, further comprising a display screen coupled to the hardware image processor, and wherein the memory has further instructions stored thereon that cause the hardware image processor to perform operations comprising:performing one or more exposure adjustment, white balance adjustment, or exposure and white balance adjustment on the image using the numerical value to obtain a preview image; and displaying the preview image on the display screen.

16. The computing device of any one of claims 12 to 15, wherein the memory has further instructions stored thereon that cause the hardware image processor to perform operations comprising: obtaining a segmentation mask for the image that identifies each pixel of the image as a sky pixel or a non-sky pixel, wherein the image is a low-resolution image that includes auto exposure (AE) statistics, and wherein determining if a cumulative distribution function (CDF) for the image has a jump comprises: generating a blue-red (B / R) histogram based on pixel values of the sky pixels; normalizing the B / R histogram to represent a discrete probability distribution; calculating the CDF for the image based on the normalized B / R histogram; determining that the CDF for the image has the jump if slope of the CDF exceeds a slope threshold; and determining that the CDF for the image does not have the jump if the slope of the CDF does not exceed the slope threshold.

17. The computing device of claim 16, wherein if it is determined that the CDF for the image does not have the jump, the operations further comprise: generating a blue-green (B / G) histogram based on pixel values of the sky pixels; and determining pairwise distances between the B / R histogram and the B / G histogram, wherein the divergence value is an aggregate value determined from the pairwise distances.

18. A non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising: obtaining an image from a camera; determining if image brightness of the image is above a brightness threshold;if the image brightness is above the brightness threshold, outputting a label that indicates that the image depicts non-overcast conditions; and if the image brightness is not above the brightness threshold, determining if a cumulative distribution function (CDF) for the image has a jump; if the CDF has a jump, determining if a bin corresponding to the jump is above a threshold bin number, and if the bin corresponding to the jump is above the threshold bin number, outputting the label that indicates that the image depicts nonovercast conditions and if the bin corresponding to the jump is not above the threshold bin number, outputting a label that indicates that the image depicts overcast conditions; and if the CDF does not have a jump, determining a divergence value based on blue-red (B / R) and blue-green (B / G) channels of the image; if the divergence value is above a divergence threshold, outputting the label that indicates that the image depicts overcast conditions; and if the divergence value is not above the divergence threshold, outputting the label that indicates that the image depicts nonovercast conditions.

19. The non-transitory computer-readable medium of claim 18, with further instructions stored thereon that cause the processor to perform further operations comprising: obtaining a segmentation mask for the image that identifies each pixel of the image as a sky pixel or a non-sky pixel, wherein the image is a low-resolution image that includes auto exposure (AE) statistics, and wherein determining if a cumulative distribution function (CDF) for the image has a jump comprises: generating a blue-red (B / R) histogram based on pixel values of the sky pixels;normalizing the B / R histogram to represent a discrete probability distribution; calculating the CDF for the image based on the normalized B / R histogram; determining that the CDF for the image has the jump if slope of the CDF exceeds a slope threshold; and determining that the CDF for the image does not have the jump if the slope of the CDF does not exceed the slope threshold.

20. The non-transitory computer-readable medium of claim 19, wherein if it is determined that the CDF for the image does not have the jump, the method further comprises: generating a blue-green (B / G) histogram based on pixel values of the sky pixels; and determining pairwise distances between the B / R histogram and the B / G histogram, wherein the divergence value is an aggregate value determined from the pairwise distances.

Images