Computational photography under low light conditions

Abstract

To provide a method and apparatus for computational photography under low light conditions for an image capture device on a mobile computing device.SOLUTION: The method includes receiving, by a mobile computing device, sensor data regarding ambient conditions of a scene during low light conditions of the scene, selecting, based on the received sensor data regarding the ambient conditions of the scene, to capture, using one or more image capture devices of the mobile computing device, a plurality of images of the scene without using a flash, in response to capturing the plurality of images of the scene without using a flash, generating a computed image using the plurality of images of the scene, and providing the generated computed image.SELECTED DRAWING: Figure 8

Description

[Background technology]

[0001] background Mobile computing devices often include an image capture device, such as a camera, to capture an image of a scene. Conventional image capture devices that operate under low light conditions have many challenges. For example, some image capture devices produce poor quality images of a scene under low light conditions. While some solutions, such as flash photography, may be used to improve image quality in a single aspect, these solutions often create additional image quality problems.

[0002] This background discussion is provided for the purpose of generally presenting the context of the disclosure. Unless otherwise stated herein, the material described in this section is not admitted, expressly or impliedly, to be prior art to the present disclosure or the appended claims. Summary of the Invention

[0003] overview This document describes techniques and apparatus for computational photography in low light conditions. The techniques and apparatus utilize a mobile computing device having an image capture device and a sensor to receive sensor data. Based on the sensor data, the techniques select to capture multiple images of a scene without using a flash and then generate a computational image based on the multiple captured images.

[0004] In an aspect, a method of computational photography under low light conditions is disclosed, including a mobile computing device that receives sensor data related to ambient conditions of a scene during low light conditions for the scene. The mobile computing device selects to capture multiple images of the scene without using a flash based on the received sensor data related to the ambient conditions of the scene. One or more image capture devices of the mobile computing device may be used to capture the scene. The mobile computing device generates and provides post-computational images.

[0005] In other aspects, a system, a computer readable medium, and a method for performing computational photography under low light conditions are disclosed.

[0006] This Summary is provided to introduce simplified concepts of techniques and apparatus for computational photography under low light conditions, concepts that are further described in the Detailed Description and Figures that follow.

[0007] Details of one or more aspects of computational photography under low light conditions are described below. The same reference numbers are used in different examples in the description and figures to indicate similar elements. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example implementation of a mobile computing device that performs computational photography under low light conditions. [Figure 2] FIG. 1 illustrates an exemplary operating environment in which computational photography under low light conditions may be performed. [Figure 3A] FIG. 1 illustrates an example of sensors and data used in performing computational photography under low light conditions. [Figure 3B] FIG. 1 illustrates an example of sensors and data used in performing computational photography under low light conditions. [Figure 4] FIG. 1 illustrates an example of non-imaging data used to perform computational photography under low light conditions. [Figure 5] FIG. 1 illustrates an example implementation of a mobile computing device that collects distance data for performing computational photography under low light conditions. [Figure 6] FIG. 1 illustrates an exemplary weighted sum equation for computational photography under low light conditions. [Figure 7] FIG. 1 illustrates an exemplary machine learning model for computational photography under low light conditions. [Figure 8] FIG. 1 illustrates an exemplary method for computational photography under low light conditions. DETAILED DESCRIPTION OF THE INVENTION

[0009] While the features and concepts of the described techniques and apparatus for computational photography under low light conditions can be implemented in any number of different environments, aspects are described in the context of the following examples.

[0010] Detailed Description Overview This document describes techniques and apparatus for computational photography under low light conditions of an image capture device. The computational photography under low light conditions automatically determines whether to capture multiple images at different settings without using a flash from which a single, higher quality computational image can be produced. The automated determination balances multiple factors, such as the power constraints and image quality of the mobile computing device, in determining for a user of the mobile computing device, thereby assisting inexperienced users of the mobile computing device and saving time spent manually adjusting settings for experienced users of the mobile computing device.

[0011] The computational photography under low light conditions described herein may utilize sensor or device data, and the mobile computing device may select to perform the computational photography. Based on the sensor or device data, the mobile device selects to capture multiple images of the scene without using a flash, generates a computed image, and provides the computed image.

[0012] Users of mobile computing devices often struggle to take photos in low-light conditions. These users may manually adjust flash settings and other image quality settings on their mobile computing devices in the hopes of capturing high-quality images. However, even experienced photographers struggle to properly calibrate camera variables to produce high-quality images in low-light conditions. Low-light conditions are particularly challenging because photographers must choose between manually firing a flash or taking multiple images without a flash and combining the multiple images into a single image using computational photography methods. The described technique for computational photography in low-light conditions not only solves the photographer's problem by making such a decision, but also conserves computing resources, such as processing resources or battery power. To do so, the technique analyzes multiple factors as part of an automated decision, rather than requiring the user to either manually adjust settings on the image capture device using a trial-and-error approach or capture lower-quality images than would be captured using the technique.

[0013] The features and concepts of the described techniques and apparatus for computational photography under low light conditions for image capture devices can be implemented in any number of different environments, and aspects are described in the context of the following examples.

[0014] Exemplary Devices FIG. 1 illustrates a mobile computer that performs computational photography under low light conditions using the techniques described herein. 1 illustrates an example implementation 100 of a mobile computing device 102. The illustrated mobile computing device 102 may include one or more sensors 104, a first image capturing device 106, and a second image capturing device 108. As illustrated, the mobile computing device 102 is used to capture a scene 110. The scene 110 may be captured by one or more image capturing devices (e.g., the first image capturing device 106 and the second image capturing device 108), which may capture one or more images 112 of the scene 110 to generate a higher-quality post-computation image 114. In other words, the mobile computing device 102 captures multiple images 112 of the scene 110, each having a different level of quality. For example, one image may have a high level of object reflectivity, while another image may have distortion from object movement within the scene. The mobile computing device 102 can process these multiple images 112 to select the most desirable qualities (e.g., optimal scene brightness, optimal object motion, optimal scene type, optimal object range, and optimal object reflectivity) while removing undesirable characteristics (e.g., scene distortion, glare, washed-out appearance). Once selected, the mobile computing device 102 produces a single post-computation image 114 that is of higher quality than each of the multiple images 112 alone.

[0015] The mobile computing device 102 includes one or more sensors 104 that capture sensor data, which may be used to determine conditions in a captured scene 110. For example, the sensor data may include scene type data based on which the type of scene an image is intended to be captured in. This scene type data may be received, for example, from a spectroscopic sensor integrated with the mobile computing device 102. This scene type data and other sensor data are used by techniques to determine whether to capture an image with flash.

[0016] The sensors 104 may include an ambient light sensor that indicates the level of ambient light in each scene. The level of ambient light may be useful in determining the presence of low light conditions 116 as well as determining the lack of ambient light in a scene. Low light conditions 116 may apply to many different environments, including, but not limited to, fog, rain, smoke, snow, indoors, nighttime, etc. Low light conditions 116 may also apply to many different times of day, including dusk and dawn. For example, a scene may occur outdoors after sunset as natural light fades from the scene. In another example, a scene may occur indoors with a low amount of indoor lighting.

[0017] In another aspect, selecting to capture multiple images 112 of scene 110 without using a flash is based on received sensor data regarding the conditions of the scene and using one or more image capture devices (e.g., first image capture device 106 and second image capture device 108) of mobile computing device 102. As such, this determination can be made based on the captured sensor data using a decision tree, a weighted sum equation, or a machine learning model.

[0018] In yet another aspect, selecting to capture multiple images 112 of the scene 110 without using a flash is based on device data or sensor data regarding power consumption on the mobile computing device 102, as described above.

[0019] In the exemplary implementation 100, two image capture devices (e.g., a first image capture device 106 and a second image capture device 108) capture one or more images 112 of a captured scene 110. In addition, the sensor 104 may collect sensor data about the scene (e.g., scene brightness, object motion, scene type, object range, and object reflectivity). The second image capture device 108) and sensor 104 collect data (e.g., brightness data, object motion data, scene type data, object range data, and object reflectivity data) that is stored on the mobile computing device 102.

[0020] In one embodiment, the selection to capture the multiple images 112 of the scene 110 without a flash may be determined, at least in part, by machine learning based on human- or non-human-selected preferences for the quality of the computed images 114. In another embodiment, the selection to capture the multiple images 112 of the scene 110 without a flash is based on machine learning based on sensor data regarding the ambient conditions of the scene 110, the low-light conditions 116 of the scene 110, and machine-learned expectations of the image quality of the computed images or image quality captured using a flash. For example, the machine-learned expectations of image quality are based on a user of the mobile computing device 102 electing to delete an image after it is captured. In another example, the machine-learned expectations of image quality are based on an analysis of the image, such as individual pixel values. The pixel values ​​provide insight into whether the image is “washed out” or highly reflective, thereby indicating poor image quality, and the machine learning model suggests alternative settings for the computational photography.

[0021] Although not required, the technique may use a machine learning model trained using previous image captures made by the user or other users. For example, the use of machine learning may include supervised or unsupervised learning through the use of neural networks, including perceptrons, feedforward neural networks, convolutional neural networks, radial basis function neural networks, or recurrent neural networks. The likelihood of exposure-related defects with or without flash can also be determined through supervised machine learning. In supervised machine learning, a labeled set of previous image captures that identifies features associated with the images can be provided to construct a machine learning model, such as non-imaging data (e.g., accelerometer data, flicker sensor data, gyroscope data, radar data) and imaging data labeled based on their impact on images captured during low-light conditions 116 (e.g., blur defects, high-noise defects, or banding defects). Through this supervised machine learning, future image captures may be classified by exposure-related defects based on the associated features. Additionally, future image captures may be fed back into the dataset to further train the machine learning model. The machine learning model automatically determines whether to capture a single image with flash or multiple images under different settings without flash which will produce a higher quality single post-calculation image, thereby assisting inexperienced users of the mobile computing device 102 and saving time spent manually adjusting settings for experienced users of the mobile computing device 102.

[0022] 2 illustrates an exemplary operating environment 200 in which computational photography under low light conditions 116 for user device 102 may be implemented. While this document discloses some aspects of computational photography under low light conditions 116 for image capture devices (e.g., first image capture device 106 and second image capture device 108) performed on a mobile computing device 102 (e.g., a smartphone), it should be noted that computational photography under low light conditions 116 for image capture devices may be performed using any computing device, including, but not limited to, a mobile computing device 102-1, a tablet 102-2, a laptop or personal computer 102-3, a television 102-4, a wristwatch 102-5, imaging eyewear 102-6, etc.

[0023] The exemplary operating environment 200 shown in FIG. 2 includes one or more processors 202 and functions that, when executed by the one or more processors 202, cause the one or more processors 202 to perform methods for computational photography under low light conditions as described herein. and a computer-readable medium 204 including a memory 206, which may incorporate a machine learning component 208, device data 210, or an image merging component 212. The exemplary operating environment 200 also includes a user interface 214, one or more image capture devices 216 (e.g., a first image capture device 106 and a second image capture device 108), one or more sensors 104 capable of capturing sensor data, a display 218, and a flash generator 220. Multiple captured images 112 of a scene 110 under low light conditions 116 are utilized by the image merging component 212 to implement a method for computational photography under low light conditions. Each of the multiple captured images 112 of the scene 110 under low light conditions 116 can be captured with a different exposure or illumination setting (e.g., exposure stacking), and the image merging component 212 combines the multiple captured images 112 into a single, higher quality, post-computational image 114.

[0024] 3A illustrates examples of sensor(s) 104 that can be used to collect sensor data. For example, a mobile computing device 102 may include a non-imaging sensor 302 capable of producing non-imaging data and an imaging sensor 304 capable of producing imaging data in addition to other types of data. The non-imaging sensor 302 includes an acceleration sensor 306, a flicker sensor 308, a gyroscope 310, and a radar system 312. The accelerometer 306 and gyroscope 310 may be able to determine movement within the captured scene 110 due to movement of the mobile computing device 102. In other words, if there is movement and the mobile computing device lacks stability, computational photography in low light conditions 116 may be more likely or less likely to be preferred over flash photography. In other aspects, the flicker sensor 308 may measure rapid changes in luminance. If luminance is highly variable, computational photography for multiple images 112 may be preferred over flash photography for a single image. In additional aspects, the radar system 312 can determine that movement within the captured scene may be performed by the mobile computing device 102. In still additional aspects, any other sensors capable of providing sensor data may be utilized to determine the feasibility of performing computational photography under low light conditions 116.

[0025] The sensor data may be imaging data captured by an imaging sensor 304 including one or more image capture devices (e.g., the first image capture device 106 and the second image capture device 108) of the mobile computing device 102. The imaging sensor 304 of the image capture device may include a complementary metal-oxide-semiconductor (CMOS) sensor 314 or the like. For example, the CMOS sensor 314 may provide data indicative of object reflectivity of the scene 110. In cases where object reflectivity is high, performing computational photography under low light conditions 116 for multiple images 112 may be preferred over flash photography for capturing a single image.

[0026] 3B shows example device data 210 that can be used for computational photography in low light conditions. Examples include power consumption associated with generating a flash 316, power consumption associated with adjusting the shutters of one or more image capture devices 318, power consumption associated with adjusting the lenses of one or more image capture devices 320, and power consumption associated with capturing multiple images and post-processing those images 322. For example, the technique can determine and balance the power consumption cost of each of the device data 210, e.g., determine that generating a flash is more power-intensive than capturing multiple images with a flash and then post-processing those images (e.g., comparing 316 with 322).

[0027] FIG. 4 illustrates a mobile computing device 104 performing computational photography under low light conditions 116 based on non-imaging data 402 (shown received by sensor 104). 4 illustrates an example 400 of a scene capture system. In one example, non-imaging data is provided by an accelerometer 306 or gyroscope 310 (not shown), which can determine movement within a captured scene due to movement of the mobile computing device 102. Movement of the mobile computing device 102 is indicated by arrows 404, 406, 408, and 410, respectively. Data collected from the accelerometer 306 or gyroscope 310 indicates whether the image capture devices (e.g., the first image capture device 106 and the second image capture device 108) maintain the stability necessary to select to capture multiple images 112 of the scene 110 without using a flash 412, providing a higher quality post-computation image 114 than a flash image. When the image capture devices 106 and 108 are accelerating (e.g., having a sudden jerk or rapid movement change), more weight may be given to using a flash than to capturing multiple images to process into a single image.

[0028] 5 illustrates an environment 500 in which a distance (e.g., distance data) from a mobile computing device 102 to an object 502 is calculated. Selecting to capture multiple images 112 of a scene 110 may be based, in part, on the distance 504. Here, the mobile computing device 102 determines the distance 504 using a laser sensor 506, although other sensors, such as those based on data from the first image capture device 106 and the second image capture device 108, may also be used.

[0029] More specifically, a laser sensor 506 integrated with the mobile computing device 102 can emit infrared light 508 onto an object 502 in a scene and then receive infrared light 510 reflected from the object 502. The mobile computing device 102 then calculates the distance 504 based on the time difference between emitting and receiving the infrared light.

[0030] 6 illustrates an exemplary weighted sum equation 600 utilized in computational photography under low light conditions 116 for image capture devices (e.g., first image capture device 106 and second image capture device 108). Weighted sum equation 600 includes feature values ​​602 multiplied by weight values ​​604 added to other weight values ​​multiplied by their corresponding feature values. If the final sum (decision 606) exceeds a threshold, the technique selects to capture multiple images 112 of scene 110 without using a flash.

[0031] In additional aspects, each of the feature values ​​602 may include sensor data such as scene brightness, object motion, scene type, distance data, or object reflectivity (shown at 600 along with multiple other features and weights). In other aspects, each of the feature values ​​602 may include device data such as the power consumption required to generate a flash, adjust a shutter, adjust the lens of one or more image capture devices, or capture multiple images and perform post-processing on those images.

[0032] For example, power consumption 316 associated with generating a flash on a mobile computing device 102 operating at low power may favor performing computational photography under low light conditions 116 for multiple images 112 over flash photography for a single image. In another example, power consumption 318 associated with adjusting the shutters of one or more image capture devices on a mobile computing device 102 operating at low power may favor performing computational photography under low light conditions 116 for multiple images 112 over flash photography for a single image. In yet another example, power consumption 320 associated with adjusting the lenses of one or more image capture devices on a mobile computing device 102 operating at low power may favor performing computational photography under low light conditions 116 for multiple images 112 over flash photography for a single image. In yet another example, power consumption 320 associated with adjusting the lenses of one or more image capture devices on a mobile computing device 102 operating at low power may favor performing computational photography under low light conditions 116 for multiple images 112 over flash photography for a single image. The power consumption associated with generating computed images 114 of multiple image capture devices may favor performing computed photography under low light conditions 116 for multiple images 112 over flash photography of a single image.

[0033] 7 illustrates an exemplary convolutional neural network 700 for performing computational photography under low light conditions 116 using machine learning. In the depicted configuration, the convolutional neural network 700 performs computational photography under low light conditions 116. The general operation of the convolutional neural network 700 includes receiving sensor data 702 or device data 704, which is provided as an input layer 706 to neurons 708 in a hidden layer 710. Probabilities for different angle bins 712 are generated in an output layer 714.

[0034] In an embodiment, selecting to capture multiple images 112 of a scene 110 without using a flash utilizes training data including sensor data 702 (e.g., scene brightness, object motion, scene type, distance data, or object reflectivity) regarding ambient conditions, low light conditions 116, and human- or non-human-selected preferences for non-flash or flash-captured images. For example, a human user of a mobile computing device 102 may delete photos taken with a flash from the mobile computing device 102 due to the user's determination that the images were of poor quality. In another example, a non-human (e.g., software) accessing data on the mobile computing device 102 may analyze and determine that photos from the mobile computing device 102 taken with or without a flash are of poor quality, such as reduced clarity, resolution, white balance, color, or other measures of image quality. These image quality determinations can be used to build or refine a machine learning model, such as a convolutional neural network 700.

[0035] In particular, the hidden layer 710 includes a convolutional layer 716, a pooling layer 718, and a fully connected layer 720. In an embodiment, the convolutional layer 716 includes a first convolutional layer having geometric shapes identified by pixel values. The pixel values ​​may come from a previously captured image (e.g., a post-computation image of a previous scene) in addition to other provided sensor data (e.g., an accelerometer, a flicker sensor, a gyroscope) of the previously captured image. In one example, a filter may be applied to the sensor data 702 to select for geometric shapes (e.g., a square, a circle, a line, an ellipse). In an additional embodiment, the convolutional layer 716 may include a second convolutional layer that includes scene elements determined based on the geometric shape classification in the first convolutional layer. For example, the first convolutional layer may include a geometric shape that identifies two circles above a horizon line. The second convolutional layer can classify the two circles above a horizon line as a human face. In another example, the second convolutional layer may include elements from the scene 110, such as the moon, a tree, or a cliff edge. In yet another example, the second convolutional layer may identify scene elements, such as facial features, distances between objects, a stadium, or a mountain view, to name just a few. In yet a further aspect, the convolutional layer 716 may include a third convolutional layer with data regarding human or non-human preferences for deleting previous post-computation images captured for a previous scene. The human-selected preferences for deleting images captured for a previous scene may be used to train a machine learning model. A human may determine that a previous post-computation image or flash image lacks sufficient quality and subsequently delete the image. Alternatively, the machine learning model can be trained based on a human's active interactions with a previous image, such as choosing to send the image to another device, upload it to social media, store it, etc.

[0036] The convolution layer 716 uses learned filters (e.g., kernels) to perform convolution operations on the incoming data to extract features of the sensor data 702. The pooling layer 718 aggregates (e.g., combines) the outputs of multiple neurons 708 from the previous layer and presents the result as The pooling layer 718 may perform, for example, a weighted sum operation or a maximum operation, which is then passed to a single neuron in the next layer.

[0037] In an additional aspect, the training data is sensor data 702 including scene brightness, object motion, scene type, distance data, or object reflectance, and the selection to capture the multiple images 112 of the scene 110 without using a flash is based on a machine learning model constructed using the training data including the scene brightness, object motion, scene type, distance data, or object reflectance. Alternatively, the training data includes device data 704, such as data related to power consumption, including power for performing machine learning, and the selection to capture the multiple images 112 of the scene 110 without using a flash is based on a machine learning model constructed using the training data including data related to power consumption, including power for performing machine learning. This is just one way in which the technology not only determines which image is likely to be of higher quality, an image captured with a flash or multiple images captured without a flash and then processed into a single image, but instead determines to select based on power consumption when the machine learning confidence is low or when the weighted sum threshold is barely or nearly reached.

[0038] Exemplary Methods 8 illustrates an exemplary method 800 for computational photography in low light conditions. In this example, the method automatically determines whether to capture a single image using a flash or to capture multiple images without a flash and generate a single post-computational image therefrom. The automated determination balances multiple factors in making the determination, such as the power constraints of the mobile computing device 102 and image quality. This determination helps users of the mobile computing device 102 better capture images in low light conditions.

[0039] At 802, a mobile computing device receives sensor data regarding ambient conditions of a scene via one or more sensors during low light conditions in the scene. In one example, the sensor data may include scene type data, and the sensor data is received, at least in part, from a spectroscopic sensor integrated with the mobile computing device 102. In another example, the sensor data can be used to measure a distance 504 from an object 502 in the scene to the mobile computing device 102, such as by using an infrared laser. For example, the technique can determine the distance 504 from an object 502 in the captured scene by measuring the time it takes to emit infrared light 508 from a laser sensor 506 onto the object 502 in the scene, and then receive reflected infrared light 510 reflected from the object 502.

[0040] At 804, the mobile computing device selects to capture multiple images 112 of the scene 110 without a flash based on sensor data, such as reflectance or motion detection data. For example, the mobile computing device 102 may select to capture multiple images 112 of the scene 110 without a flash based on distance data, as described in connection with FIG. 5. In another example, the selection is determined by a decision tree, a weighted sum equation, as described in connection with FIG. 6, or a combination of these determination schemes. In yet another example, the mobile computing device 102 may select to capture multiple images 112 of the scene 110 without a flash based on sensor data, as analyzed by machine learning, as described in connection with FIG. 7.

[0041] At 806, the multiple images of the scene are combined to generate a post-computed image. An exemplary scheme for post-processing includes an image merging component 212 that, when executed by one or more processors 202, causes the one or more processors 202 to perform a method for computational photography under low light conditions as described herein. The multiple captured images 112 of the scene 110 under low light conditions 116 are utilized by the image merging component 212 to generate a computational photography under low light conditions. The system implements a method for computational photography, in which multiple captured images 112 of a scene 110 under low light conditions 116 can each be captured with the same or different exposure or lighting settings (e.g., exposure stacking), and an image merging component 212 combines the multiple captured images 112 into a high-quality single computed image 114.

[0042] At 808, the calculated image is provided, such as by display or storage on the mobile computing device 102.

[0043] However, as discussed above, the present technology may determine to capture an image using a flash based on the various criteria discussed above. In such a case, at 810, the mobile computing device selects to capture an image of the scene using a flash, such as by having the flash generator 220 provide it, and captures the image of the scene using the flash in conjunction with one or more image capture devices 216. At 812, the technology generates a flash-captured image, which is then provided by the technology at 814.

[0044] However, in some alternative or additional cases, the technique may choose to capture multiple images of a scene without using a flash and one image of the scene with a flash (e.g., capturing a non-flash image before and / or after the flash). In such cases, the technique chooses to provide either the post-calculation image at 808, the flash-captured image at 814, or a post-processed combination of both. In selecting one or the other, the technique determines which of the two images to provide is better, but both may be stored or presented for selection by the user.

[0045] However, to combine both, the technique selects to combine portions of the post-calculation image and the flash-captured image at 816. The technique may make the selection based on any defects in one or both images, such as blur or noise defects, or sensor data indicating the likelihood of noise or blur (e.g., prior to capture as described above). In one exemplary combination, portions of the post-calculation image taken without a flash are used to reduce noise in those portions, while portions of the flash-captured image that have motion and might otherwise appear blurred due to motion are then combined. Thus, at 818, the images are combined to provide a single image at 820 having portions of the scene captured with and without a flash.

[0046] In general, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of example methods may be described in the general context of executable instructions stored in computer-readable storage memory that is local and / or remote to a computer processing system, and implementations may include software applications, programs, functions, etc. Alternatively, or in addition, any of the functionality described herein can be performed, at least in part, by one or more hardware logic components, including, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SoCs), complex programmable logic devices (CPLDs), etc.

[0047] Some examples are described below. Example 1: A method including: a mobile computing device receiving sensor data regarding ambient conditions of a scene during low light conditions of the scene; selecting, based on the received sensor data regarding the ambient conditions of the scene, to capture multiple images of the scene without using a flash using one or more image capture devices of the mobile computing device; generating one calculated image using the multiple images of the scene in response to capturing the multiple images of the scene without using a flash; and providing the calculated image.

[0048] Example 2: The method described in Example 1, wherein receiving device data related to power consumption on the mobile computing device may determine a decision to select capturing multiple images of a scene without using a flash.

[0049] Example 3: The method of Example 2, wherein the power consumption includes power to generate a flash for one or more image capture devices, adjust a shutter of one or more image capture devices, adjust a lens of one or more image capture devices, or generate a post-computation image.

[0050] Example 4: The method of Example 1, wherein selecting to capture multiple images of a scene without using a flash performs machine learning, where the machine learning is based on sensor data regarding ambient conditions of the scene, low light conditions of the scene, and calculated machine learning expectations of image quality of the images or image quality captured using a flash.

[0051] Example 5: The method of example 4, wherein the machine learning includes device data related to power consumption, and the power consumption includes power for performing the machine learning.

[0052] Example 6: The method of Example 1, wherein selecting to capture multiple images of a scene without using a flash comprises performing machine learning using a machine learning model created using training data that includes sensor data regarding ambient conditions, low light conditions, and a human-selected preference for non-flash or flash-captured images.

[0053] Example 7: The method of Example 6, wherein the machine learning model includes a convolutional neural network, the convolutional neural network having a first convolutional layer including geometric shape classifications identified by pixel values.

[0054] Example 8: The method of Example 7, wherein the convolutional neural network includes a second convolutional layer, the second convolutional layer including scene elements determined based on the classification of geometric shapes in the first convolutional layer.

[0055] Example 9: The method of example 8, wherein the second convolutional layer includes scene elements including facial features, distance between objects, a stadium, or a mountain scene.

[0056] Example 10: The method of Example 8 or Example 9, wherein the convolutional neural network includes a third convolutional layer, the third convolutional layer including data regarding a human-selected preference for deleting a previous post-computation image captured for a previous scene.

[0057] Example 11: The method of Example 1, wherein selecting to capture multiple images of a scene without using a flash comprises performing machine learning and configuring the machine learning to account for ambient conditions, low light conditions, and non-human selection of non-flash or flash-captured images. This is done using a machine learning model created using training data that includes sensor data on preferences.

[0058] Example 12: The method described in Example 1, wherein the sensor data includes luminance data, the sensor data being received at least in part from a spectroscopic sensor integrated with the mobile computing device, and wherein selecting to capture multiple images of the scene without using a flash is based on the luminance data.

[0059] Example 13: The method described in Example 1, wherein the sensor data includes motion detection data, the sensor data is received at least in part from a spectroscopic sensor in a pre-flash setting, and selecting to capture multiple images of the scene without using a flash is based on the motion detection data.

[0060] Example 14: The method described in Example 1, wherein the sensor data includes scene type data, the sensor data being received at least in part from a spectroscopic sensor integrated with the mobile computing device, and wherein selecting to capture multiple images of the scene without using a flash is based on the scene type data.

[0061] Example 15: The method as described in Example 1, wherein the sensor data includes range data, and wherein selecting to capture multiple images of the scene without using a flash is based on the range data.

[0062] Example 16: The method of example 15, wherein the distance data is received, at least in part, from two image capture devices of the one or more image capture devices.

[0063] Example 17: The method described in Example 15, wherein the distance data is received, at least in part, from a laser sensor integrated with a mobile computing device, and the mobile computing device is configured to calculate distances from one or more image capture devices to objects in the scene.

[0064] Example 18: The method as described in Example 17, wherein the laser sensor projects infrared light onto an object in the scene, and the laser sensor receives infrared light reflected from the object.

[0065] Example 19: The method of Example 18, wherein a first time at which the laser sensor emits infrared light onto an object in the scene and a second time at which the laser sensor receives the reflected infrared light are multiplied by the speed of the infrared light to provide distance data.

[0066] Example 20: The method of example 1, wherein the sensor data includes object reflectance data, and wherein selecting to capture multiple images of the scene without using a flash is based on the object reflectance data.

[0067] Example 21: The method of Example 1, wherein selecting to capture multiple images of the scene without using a flash is based on a weighted sum equation that includes weights assigned to two or more of the sensor data, wherein the two or more of the sensor data include scene brightness, object motion, scene type, distance data, or object reflectivity.

[0068] Example 22: The method described in Example 21, wherein the weighted values ​​generate a sum that must exceed a threshold to allow selection of capturing multiple images of a scene without using a flash.

[0069] Example 23: The method of Example 1, wherein selecting to capture multiple images of the scene without using a flash is based on a weighted sum equation, the weighted sum equation including weights assigned to two or more device data, the two or more device data including power consumption for generating a flash for one or more image capture devices, adjusting a shutter of one or more image capture devices, adjusting a lens of one or more image capture devices, or generating the calculated image, the weights generating a sum, and selecting to capture multiple images of the scene without using a flash is based on the sum exceeding a threshold.

[0070] Example 24: The method described in Example 1, wherein the sensor data includes scene brightness, object motion, scene type, distance data, or object reflectance, and wherein selecting to capture multiple images of the scene without using a flash is based on a machine learning model constructed using training data including the scene brightness, object motion, scene type, distance data, or object reflectance.

[0071] Example 25: The method as described in example 1, wherein the sensor data includes non-imaging data. Example 26: The method of Example 25, wherein the non-imaging data includes data collected from an accelerometer, and the data collected from the accelerometer indicates whether the image capture device maintains the stability required to select to capture multiple images of the scene without using a flash.

[0072] Example 27: The method of example 1, wherein the sensor data is imaging data captured by one or more image capture devices of the mobile computing device.

[0073] Example 28: A mobile computing device comprising a processor, one or more sensors, image sensors, or flash generators, and a computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform any of the methods of Examples 1 to 27.

[0074] Conclusion Although aspects of computational photography under low light conditions for an image capture device have been described in language specific to features and / or methods, the subject matter of the appended claims is not necessarily limited to the particular features or methods described. Rather, the particular features and methods are disclosed as example implementations of the claimed computational photography under low light conditions for an image capture device, and other equivalent features and methods are intended to be within the scope of the appended claims. Furthermore, various aspects have been described, and it will be understood that each described aspect can be implemented independently or in conjunction with one or more other described aspects.

Claims

1. receiving, by a mobile computing device, sensor data relating to ambient conditions of a scene during low light conditions of the scene; selecting, based on the received sensor data regarding the ambient conditions of the scene, to capture a plurality of images of the scene without using a flash using one or more image capture devices of the mobile computing device; generating a post-computed image using the plurality of images of the scene in response to capturing the plurality of images of the scene without using the flash; providing said calculated image.

2. 10. The method of claim 1, further comprising receiving device data regarding power consumption on the mobile computing device, and wherein selecting to capture multiple images of the scene without using the flash is further based on the power consumption.

3. The power consumption is generating the flash of the one or more image capture devices; adjusting the shutters of the one or more image capture devices; adjusting the lenses of said one or more image capture devices; or The method of claim 2 including power for generating the post-computation image.

4. 2. The method of claim 1, wherein selecting to capture the plurality of images of the scene without using the flash comprises performing machine learning based on the sensor data regarding ambient conditions of the scene, the low light conditions of the scene, and machine learning expectations of image quality of the calculated images or image quality captured using the flash.

5. 2. The method of claim 1, wherein selecting to capture the plurality of images of the scene without using the flash is performed using machine learning, the machine learning being performed using a machine learning model created using training data that includes sensor data regarding ambient conditions, low light conditions, and human-selected preferences for non-flash or flash-captured images.

6. 6. The method of claim 5, wherein the machine learning model includes a convolutional neural network, the convolutional neural network having a first convolutional layer that includes geometric shape classifications identified by pixel values.

7. 7. The method of claim 6, wherein the convolutional neural network includes a second convolutional layer, the second convolutional layer including scene elements determined based on the geometric shape classification in the first convolutional layer.

8. 8. The method of claim 1, wherein the sensor data includes luminance data, the sensor data being received at least in part from a spectroscopic sensor integrated with the mobile computing device, and wherein selecting to capture multiple images of the scene without using a flash is based on the luminance data.

9. The sensor data includes motion detection data, the sensor data being received at least in part from a spectroscopic sensor in a pre-flash setting, and selecting to capture multiple images of the scene without using a flash is based on the motion detection data. The method according to any one of claims 1 to 8.

10. 10. The method of claim 1, wherein the sensor data includes scene type data, the sensor data being received at least in part from a spectroscopic sensor integrated with the mobile computing device, and wherein selecting to capture multiple images of the scene without using a flash is based on the scene type data.

11. The method of any preceding claim, wherein the sensor data includes distance data, and wherein selecting to capture multiple images of the scene without using a flash is based on the distance data.

12. 12. The method of claim 1, wherein the sensor data includes object reflectance data, and wherein selecting to capture multiple images of the scene without using a flash is based on the object reflectance data.

13. 13. The method of any one of claims 1 to 12, wherein the sensor data includes non-imaging data collected from an accelerometer, the data collected from the accelerometer indicating whether the image capture device maintains the stability required to select to capture multiple images of the scene without using the flash.

14. Selecting to capture multiple images of the scene without using the flash is based on a weighted sum equation, the weighted sum equation comprising: weights assigned to two or more device data, the two or more device data generating a flash for the one or more image capture devices; adjusting the shutters of the one or more image capture devices; adjusting the lenses of said one or more image capture devices; or including the power consumption for generating the post-computation image; 14. The method of any one of claims 1 to 13, wherein the weighted values ​​form a sum and the selecting to capture multiple images of the scene without using a flash is based on the sum exceeding a threshold.

15. a processor; one or more sensors, image sensors, or flash generators; a computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform the method of any of claims 1 to 14.

Images