Preview of one-shot autofocus in backlit scenes

Abstract

The method includes performing a process of fine-tuning the automatic exposure setting of an image sensor. The method also includes determining autofocus confidence based on at least one image captured by the image sensor while performing the process of fine-tuning the automatic exposure setting of the image sensor. The method further includes interrupting the process of fine-tuning the automatic exposure setting to capture an image with a higher exposure than at least one image, based on the autofocus confidence. The method additionally includes determining the autofocus setting based on the image captured with a higher exposure than at least one image. The method further includes setting the image sensor based on the autofocus setting.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 522,022, filed on June 20, 2023, the entire content of which is incorporated herein by reference.

Background Art

[0002] Many modern computing devices, including mobile phones, personal computers, and tablets, include image capture devices. Some image capture devices are configured to have a telephoto function.

Summary of the Invention

[0003] In one embodiment, the method includes determining autofocus reliability based on at least one image captured by an image sensor while performing a process of fine - tuning an automatic exposure setting of the image sensor. The method additionally includes interrupting the process of fine - tuning the automatic exposure setting based on the autofocus reliability and capturing an image at an exposure higher than that of the at least one image. The method further includes determining an autofocus setting based on an image captured at an exposure higher than that of the at least one image. The method also includes causing the image sensor to capture one or more additional images based on the autofocus setting.

[0004] In other embodiments, the computing system includes a control system configured to determine autofocus confidence based on at least one image captured by the image sensor while performing a process of fine-tuning the auto exposure setting of the image sensor. The control system is further configured to interrupt the process of fine-tuning the auto exposure setting based on the autofocus confidence and capture an image with a higher exposure than at least one image. The control system is additionally configured to determine the autofocus setting based on the image captured with a higher exposure than at least one image. The control system is also configured to cause the image sensor to capture one or more additional images based on the autofocus setting.

[0005] In a further embodiment, a non-temporary computer-readable medium is executable by one or more processors and stores program instructions that cause the one or more processors to perform an operation. The operation includes determining autofocus confidence based on at least one image captured by the image sensor while performing a process of fine-tuning the auto exposure setting of the image sensor. The operation also includes, based on the autofocus confidence, interrupting the process of fine-tuning the auto exposure setting and capturing an image with a higher exposure than at least one image. The operation further includes determining the autofocus setting based on the image captured with a higher exposure than at least one image. The operation additionally includes causing the image sensor to capture one or more additional images based on the autofocus setting.

[0006] In other embodiments, a system is provided which includes means for determining autofocus confidence based on at least one image captured by an image sensor while performing a process of fine-tuning the automatic exposure setting of the image sensor. The system further includes means for interrupting the process of fine-tuning the automatic exposure setting based on the autofocus confidence to capture an image with a higher exposure than at least one image. The system further includes means for determining the autofocus setting based on the image captured with a higher exposure than at least one image. The system also includes means for causing the image sensor to capture one or more additional images based on the autofocus setting.

[0007] The above summary is illustrative and not intended to be limiting. Further embodiments, features, and characteristics beyond those described above will become apparent from the figures and the following detailed description, as well as from the accompanying drawings. [Brief explanation of the drawing]

[0008] [Figure 1] An exemplary computing device is shown according to an exemplary embodiment. [Figure 2] This is a simplified block diagram showing some of the components of an exemplary computing system. [Figure 3] This is a flowchart of the method according to an exemplary embodiment. [Figure 4] An image is shown according to an exemplary embodiment. [Figure 5] This represents an image associated with a confidence value, according to an exemplary embodiment. [Figure 6] A series of image captures by an exemplary embodiment are shown. [Figure 7] This represents an image captured for autofocus, according to an exemplary embodiment. [Figure 8]This image shows an image captured with updated autofocus settings, according to an exemplary embodiment. [Modes for carrying out the invention]

[0009] Exemplary methods, devices, and systems are described herein. The words “example” and “exemplary” are used herein to mean “serving as an example, case, or illustration.” Any embodiment or feature described herein as “example” or “exemplary” should not be construed as necessarily preferable or advantageous to other embodiments or features unless otherwise specified. Other embodiments may be utilized and other modifications made without departing from the scope of the subject matter presented herein.

[0010] Therefore, the exemplary embodiments described herein are not intended to be limiting. It will be readily apparent that the aspects of this disclosure generally described herein and shown in the drawings can be arranged, replaced, combined, separated, and designed in a wide variety of different configurations.

[0011] Throughout this specification, the articles “a” or “an” are used to introduce elements of exemplary embodiments. Unless otherwise specified or indicated more clearly by the context, all references to “a” or “an” refer to “at least one,” and all references to “the” refer to “at least one.” The intention of using the conjunctions “or” or “or” in lists of at least two terms is to indicate any of the listed terms, or any combination thereof.

[0012] The use of ordinal numbers such as "first," "second," and "third" is for distinguishing each element and does not indicate a specific order of these elements. For the purposes of this explanation, the terms "multiple" and "a plurality of" refer to "two or more" or "more than one."

[0013] Furthermore, unless otherwise indicated by the context, the features shown in each figure may be used in combination with each other. Therefore, the drawings should generally be viewed as representations of components of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment. In the drawings, similar symbols typically identify similar components unless otherwise indicated by the context. Furthermore, unless otherwise stated, the drawings are not drawn to scale and are for illustrative purposes only. Additionally, the drawings are representative only and do not show all components. For example, additional structural or restrictive components may not be shown.

[0014] Furthermore, any enumeration of elements, blocks, or steps in this specification or in the claims is for clarity only. Therefore, such enumeration should not be construed as requiring or implying that these elements, blocks, or steps follow a particular arrangement or are performed in a particular order.

[0015] I. Overview Image capture devices may be integrated into a computing system (e.g., a smartphone, laptop, etc.). Additionally and / or alternatively, an image capture device may be a remote image capture device that communicates with a computing system (e.g., a smartphone, laptop, server device, etc.). Regardless of whether the image capture device is integrated into or separate from the computing system, the computing system may display a preview of the images that the image capture device can capture. For example, if a park is included in the field of view of the image capture device, the image capture device may send a preview including the park to the computing system, and the computing system may display a preview in which the park is included in the field of view of the image capture device.

[0016] A challenge that may arise in this process is capturing or displaying a properly focused image preview. Capturing or displaying a properly focused scene, especially one with a high dynamic range (a wide range of brightness between the darkest and brightest parts of the scene), can be particularly difficult because such scenes may include both very bright and particularly dark areas. For example, if a user photographs trees facing a setting sun, the image preview and / or image may capture the detail of the sky while the trees in front of the sky are quite dark and lack detail. Other images or image previews of the same scene may have a higher exposure and depict the trees in detail, but the sky may appear very bright and lack detail. In some cases, detail can be recovered from underexposed areas of the image, but not from overexposed areas. Therefore, a computing system may intentionally underexpose an image or image preview so that detail can be recovered from the dark parts of the image, for example, during post-processing. However, due to the high dynamic range of such images and the large underexposed portions of the image, the computing system may not be able to accurately determine where to focus on the image.

[0017] This specification describes a technique for an image capture device to perform autofocus using a one-shot autofocus process that schedules a special frame with appropriate exposure to determine autofocus. The computing system may determine the autofocus setting by scheduling a single frame with appropriate exposure within an image stream used for automatic exposure determination and maximizing dynamic range. The computing system may discard the special frame from the preview and / or exclude the special frame from the image displayed by the computing system. The computing system may also schedule the capture of the special frame at a lower frequency than the image used for automatic exposure setting determination and / or maximizing dynamic range.

[0018] As described above, when attempting to maximize the dynamic range, the computing system may continuously adjust the auto exposure settings from each captured image and / or each image preview. In some examples, individual captured images may be combined using high dynamic range (HDR) imaging techniques to increase the dynamic range of the image of the scene captured as a photograph or video. By capturing multiple images of the same scene at different exposures and combining those multiple images, a combined image with a higher dynamic range than that of the individual captured images may be produced. For each image, the computing system may evaluate the image for autofocus reliability or determine whether an accurate autofocus setting can be determined from the image. If the computing system determines that the autofocus reliability does not exceed a confidence threshold, or that an accurate autofocus setting cannot be determined from the image, the computing system may interrupt the process of fine-tuning the auto exposure settings and capture the image at a higher exposure than the image.

[0019] In particular, the computing system can schedule the capture of an image with high exposure at a specific future time so that the capture of the image does not interrupt the image capture and / or image preview. Since the captured image may have a higher exposure, the area of the image may be overexposed and / or washed out, and as a result, the computing system may not be able to display the details of those areas. Therefore, the computing system may display only the captured image when adjusting for automatic exposure, and the computing system may discard the captured image to determine autofocus.

[0020] To facilitate determining autofocus using an image with a higher exposure, the computing system may schedule the capture of an image for autofocus at a higher exposure. Scheduling the capture of an image for autofocus can help ensure that the image capture and / or image preview is not interrupted by the capture of an image with a higher exposure. Specifically, the computing system may schedule the capture of an image in advance based on, for example, the frame rate at which the preview operates, the frame rate at which the sensor captures an image, and / or the exposure time of the image used for automatic exposure, and / or the exposure time of the image used for autofocus.

[0021] For example, a computing system may display a preview of a scene at a frame rate of 30 frames per second. To determine the image stream to be used for the preview, the computing system may capture the image stream for display such that each image has an exposure time of 1 / 2000 second, and the computing system may decide that an image for autofocus can be captured with an exposure time of 1 / 1000 second. Due to the frame rate at which the computing system displays the preview of the scene and the exposure times, the computing system may capture an image for autofocus between two images captured for display, and the computing system may discard an image captured for autofocus without causing a delay in the display.

[0022] In some examples, the computing system may extend the display of one or more images in the image stream for display to compensate for the time it takes for the computing system to capture an image for autofocus. For example, the computing system may display a preview of the scene at a frame rate of 30 frames per second. The computing system may capture the image stream for display such that each image has an exposure time of 1 / 30 second, and after capturing each of these images, the computing system may adjust the automatic exposure setting. If the signal-to-noise ratio of the images in the image stream is less than a threshold amount, or if it is determined that the image is insufficient to determine autofocus, the computing system may schedule the capture of an image for autofocus. The computing system may determine that an image can be captured with an exposure time of 1 / 15 second, and the computing system may send a request to schedule the capture of an image for autofocus to the automatic exposure process. Due to the frame rate being 30 frames per second and the exposure time of the image stream being 1 / 30 second, the computing system may extend the display of the images captured for automatic exposure by two frames while capturing an image with an exposure time of 1 / 15 second.

[0023] II. Exemplary Systems and Methods Figure 1 shows an exemplary computing device 100. In the examples described herein, computing device 100 may be an image capture device and / or a video capture device. Computing device 100 is shown in the form factor of a mobile phone. However, computing device 100 may alternatively be implemented as a laptop computer, a tablet computer, and / or a wearable computing device, etc. Computing device 100 may comprise various elements such as a body 102, a display 106, and buttons 108 and 110. Computing device 100 may further comprise one or more cameras, such as a front camera 104 and at least one rear camera 112. In an example with multiple rear cameras as shown in Figure 1, each of the rear cameras may have a different field of view. For example, the rear cameras may include a wide-angle camera, a main camera, and a telephoto camera. The wide-angle camera can capture a wider portion of the environment compared to the main camera and the telephoto camera, and the telephoto camera can capture a more detailed image of a smaller portion of the environment compared to the main camera and the wide-angle camera.

[0024] The front camera 104 may be located on the side of the main unit 102 that normally faces the user during operation (for example, the same side as the display 106). The rear camera 112 may be located on the side of the main unit 102 opposite to the front camera 104. It is optional to refer to the cameras as front and rear, and the computing device 100 may have multiple cameras located on various sides of the main unit 102.

[0025] The display 106 may represent a cathode ray tube (CRT) display, a light-emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light-emitting diode (OLED) display, or any other type of display known in the art. In some examples, the display 106 may display the current image captured by the front camera 104 and / or the rear camera 112, an image that may be captured by one or more of these cameras, a recently captured image by one or more of these cameras, and / or a digital representation of one or more modified versions of these images. Thus, the display 106 may function as a viewfinder for the cameras. The display 106 may also support touchscreen functionality that may allow adjustment of settings and / or configurations of one or more aspects of the computing device 100.

[0026] The front camera 104 may comprise an image sensor and associated optical elements such as a lens. The front camera 104 may provide a zoom function or may have a fixed focal length. In another example, interchangeable lenses can be used with the front camera 104. The front camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front camera 104 may also be configured to capture still images, video images, or both. Furthermore, the front camera 104 may represent, for example, a monocular camera, a stereoscopic camera, or a multi-lens camera. The rear camera 112 may be arranged similarly or differently. Furthermore, one or more of the front camera 104 and / or the rear camera 112 may be an array of one or more cameras.

[0027] One or more of the front camera 104 and / or rear camera 112 may include, or be associated with, an illumination component that provides a light field for illuminating an object. For example, the illumination component may provide flash or constant illumination of the object. The illumination component may also be configured to provide a light field that includes one or more structured light, polarized light, and light having specific spectral components. Other known and used types of light fields for recovering three-dimensional (3D) models from objects are possible within the context of the examples herein.

[0028] The computing device 100 may also include an ambient light sensor that can continuously or ad-hoc determine the ambient brightness of the scene that the cameras 104 and / or 112 can capture. In some embodiments, the ambient light sensor can be used to adjust the display brightness of the display 106. Furthermore, the ambient light sensor may be used to determine, or assist in determining, the exposure length of one or more of the cameras 104 or 112.

[0029] The computing device 100 can be configured to capture images of a target object using the display 106 and the front camera 104 and / or rear camera 112. The captured images may be multiple still images or a video stream. Image capture can be triggered by activating button 108, pressing a soft key on the display 106, or by some other mechanism. Depending on the embodiment, images can be captured automatically at specific time intervals, for example, when button 108 is pressed, when the lighting conditions of the target object are appropriate, when the computing device 100 is moved a predetermined distance, or according to a predetermined capture schedule.

[0030] Figure 2 is a simplified block diagram showing some of the components of an exemplary computing system 200, such as an image capture device and a video capture device. The computing system 200 may be, but not limited to, a cellular mobile phone (e.g., a smartphone), a computer (desktop, notebook, tablet, server, or handheld computer, etc.), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a game console, a robotic device, a vehicle, or any other type of device. The computing system 200 may, for example, represent an embodiment of computing device 100.

[0031] As shown in Figure 2, the computing system 200 may include a communication interface 202, a user interface 204, a processor 206, data storage 208, and a camera component 224, all of which may be linked to communicate with each other by a system bus, network, or other connection mechanism 210. The computing system 200 may include at least some image capture and / or image processing functions. It should be understood that the computing system 200 may represent a physical image processing system, a specific physical hardware platform on which image sensing and / or processing applications run in software, or any other combination of hardware and software configured to perform image capture and / or image processing functions.

[0032] The communication interface 202 may enable the computing system 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, the communication interface 202 may facilitate circuit-switched and / or packet-switched communications, such as conventional telephone service (POTS) communications and / or Internet Protocol (IP) or other packet communications. For example, the communication interface 202 may include a chipset and antenna positioned for wireless communication with a wireless access network or access point. The communication interface 202 may also take the form of, or include, wired interfaces such as Ethernet®, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI®) ports, among other possibilities. The communication interface 202 may also take the form of, or include, wireless interfaces such as Wi-Fi, Bluetooth®, Global Positioning System (GPS), or wide-area wireless interfaces (e.g., WiMAX or 3GPP® Long-Term Evolution (LTE)). However, other forms of physical layer interfaces, as well as other types of standard or proprietary communication protocols, may be used via the communication interface 202. Furthermore, the communication interface 202 may include multiple physical communication interfaces (e.g., a Wi-Fi interface, a Bluetooth® interface, and a wide-area wireless interface).

[0033] The user interface 204 may function to enable the computing system 200 to interact with a human or non-human user, such as by receiving input from the user and providing output to the user. Therefore, the user interface 204 may include input components such as a keypad, keyboard, touch panel, computer mouse, trackball, joystick, and microphone. The user interface 204 may also include one or more output components, such as a display screen combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, LED, and / or OLED technology, or other currently known or future-developed technologies. The user interface 204 may also be configured to generate audible output(s) via speakers, speaker jacks, audio output ports, audio output devices, earphones, and / or other similar devices. The user interface 204 may also be configured to receive and / or capture audible speech(s), noise(s), and / or signals(s) via microphones and / or other similar devices.

[0034] In some examples, the user interface 204 may include a display that functions as a viewfinder for still camera and / or video camera functions supported by the computing system 200. Furthermore, the user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of camera functions and the capture of images. Some or all of these buttons, switches, knobs, and / or dials may be implemented by touch-sensitive panels.

[0035] The processor 206 may include one or more general-purpose processors, such as microprocessors, and / or one or more dedicated processors, such as digital signal processors (DSPs), graphics processing units (GPUs), floating-point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some cases, the dedicated processors may perform image processing, image alignment, and image merging. The data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated whole or partially with the processor 206. The data storage 208 may include removable and / or non-removable components.

[0036] The processor 206 may be able to perform various functions described herein by executing program instructions 218 (e.g., compiled or uncompiled program logic and / or machine code) stored in the data storage 208. Thus, the data storage 208 may include a non-temporary computer-readable medium that, when executed by the computing system 200, stores program instructions causing the computing system 200 to perform any of the methods, processes, or operations disclosed herein and / or in the accompanying drawings. The execution of the program instructions 218 by the processor 206 may result in the processor 206 using the data 212.

[0037] For example, program instructions 218 may include an operating system 222 installed on the computing system 200 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more application programs 220 (e.g., camera functionality, address book, email, web browsing, social networking, audio-to-text functionality, text translation functionality, and / or game applications). Similarly, data 212 may include operating system data 216 and application data 214. The operating system data 216 may be primarily accessible to the operating system 222, and the application data 214 may be primarily accessible to one or more application programs 220. The application data 214 may be located in a file system that is visible to or hidden from the user of the computing system 200.

[0038] The application program 220 can communicate with the operating system 222 through one or more application programming interfaces (APIs). These APIs can facilitate, for example, the application program 220 reading and / or writing application data 214, sending and receiving information via the communication interface 202, and receiving and / or displaying information on the user interface 204.

[0039] In some cases, the application program 220 may be referred to simply as "the app." Furthermore, the application program 220 may be downloadable to the computing system 200 through one or more online application stores or application marketplaces. However, the application program can also be installed to the computing system 200 by other means, such as via a web browser or through the physical interface of the computing system 200 (e.g., a USB port).

[0040] The camera component 224 may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, a shutter button, an infrared projector, and / or a visible light projector. The camera component 224 may include a component configured to capture images of the visible light spectrum (e.g., electromagnetic radiation having wavelengths of 380 to 700 nanometers) and / or a component configured to capture images of the infrared light spectrum (e.g., electromagnetic radiation having wavelengths of 701 nanometers to 1 millimeter), and so on. The camera component 224 may be controlled at least partially by software executed by the processor 206.

[0041] In a further example, one or more remote cameras 230 may be controlled by a computing system 200. For example, the computing system 200 may transmit control signals to one or more remote cameras 230 via a wireless or wired connection. Such signals may be transmitted as part of the surrounding computing environment. In such an example, inputs received by the computing system 200 (e.g., physical movement of a wearable device) may be mapped to the movement or other functions of one or more remote cameras 230. Images captured by one or more remote cameras 230 may be transmitted to the computing system 200 for further processing. Such images may be treated as images captured by cameras physically located on the computing system 200.

[0042] Figure 3 is a flowchart of Method 300 according to an exemplary embodiment. Method 300 may be performed by one or more computing systems (e.g., computing system 200 in Figure 2) and / or one or more processors (e.g., processor 206 in Figure 2). Method 300 may be performed on a computing system such as computing system 100 in Figure 1.

[0043] In block 302, method 300 includes determining autofocus confidence based on at least one image captured by an image sensor while performing a process of fine-tuning the auto exposure setting of the image sensor. The computing system may continuously perform the process of determining the auto exposure setting. The auto exposure setting may include one or more of the aperture (lens width), shutter speed (shutter opening and closing speed), and / or International Organization for Standardization (ISO) sensitivity (image sensor's sensitivity to light). For example, the computing system may capture or determine an image and, based on that image, determine the auto exposure setting. The computing system may update the auto exposure setting based on the determined auto exposure setting. The computing system may take further images with the updated auto exposure setting and determine a further updated auto exposure setting based on the further images taken. The computing system can repeat this process to continuously fine-tune the auto exposure setting, and therefore the process of fine-tuning the auto exposure setting may be a closed-loop process.

[0044] Figure 4 shows images 402, 404, and 406 according to an exemplary embodiment. The computing system may capture 402, 404, and 406 during the automatic exposure process described above, and as a result, the updated automatic exposure settings are determined after each subsequent image. For example, the computing system may determine the updated automatic exposure settings after image 402 and use them to capture image 404. The computing system may determine further updated automatic exposure settings after image 404 and use them to capture image 406.

[0045] In some examples, an image may be a high dynamic range image. As described above, a high dynamic range image may include areas that are very bright compared to the rest of the image, and / or areas that are very dark compared to the rest of the image. As shown in images 402, 404, and 406, a high dynamic range image may include the sun or other light source illuminating an object from behind. In some examples, a high dynamic range image may include a face in front of a window, a window in a room, and a building or rock in front of the sun.

[0046] A computing system may, for example, determine an automatic exposure setting that underexposes the scene in response to a determination that the image is a high dynamic range image. An underexposed image may be an image captured at an illumination level where a large portion of the image (e.g., the main part of the image) is dark and / or darker than normally perceived. Because high dynamic range images may contain very bright areas, if the image is underexposed, the computing system may be able to extract more information, and the computing system may be able to post-process the underexposed image to include more detail. In contrast, if the computing system captures the image using an exposure setting that overexposes the image, the computing system may not be able to post-process the overexposed image to include certain details. For example, the computing system may have captured image 404 using an exposure setting that intentionally underexposes the image, and therefore image 404 may include the sun, sky, ground, and some plants. If the computing system had captured image 404 using an exposure setting that does not underexpose the image, the sun, sky, and ground may be blown out, making the sun, sky, and ground appear as one area.

[0047] In some examples, the computing system may determine the automatic exposure setting based on sensor data collected by various sensors, such as phase detection sensors, light detection sensors, and / or image sensors. For example, the computing system may receive phase detection data from a phase detection sensor that can collect data with left and right views. The computing system may determine the exposure setting based on the phase detection data. Additionally and / or alternatively, the computing system may collect illumination data from one or more light detection sensors, and the computing system may determine the automatic exposure setting based on the collected illumination data. Further additionally and / or alternatively, the computing system may determine the automatic exposure setting based on images 402, 404, and / or 406. The computing system may analyze images, for example, image 402, to determine one or more illumination metrics. The computing system may determine the automatic exposure setting based on the determined illumination metrics. The automatic exposure setting may also be determined based on data collected by other sensors, such as a depth sensor.

[0048] The computing system may sequentially display images 402, 404, and 406 to the user, possibly as an image preview process or image capture process, while the computing system adjusts the automatic exposure settings. In some examples, the computing system may process images 402, 404, and / or 406 before displaying the image to the user. For example, the computing system may increase the brightness and / or saturation of one or more parts of the image (e.g., underexposed areas), or extract details from images 402, 404, and / or 406 before displaying the image to the user.

[0049] The computing system may determine the autofocus confidence based on the images captured with each automatic exposure setting. For example, Figure 5 shows images 402, 404, and 406 associated with confidence values ​​502, 504, and 506, respectively, according to an exemplary embodiment. The computing system may determine the confidence values ​​based on one or more metrics, such as the signal-to-noise ratio of the images captured by the image sensor.

[0050] Determining confidence values ​​502, 504, and 506 from images 402, 404, and 406 may be part of an autofocus process that the computing system may perform in parallel with the auto exposure process. For example, the computing system may capture image 402 and adjust the auto exposure settings based on the settings in which image 402 was captured. As part of the autofocus process, the computing system may use image 402 or other sensor data based on the settings in which image 402 was captured to determine a signal-to-noise ratio, or confidence value 502, to be 0.9. The computing system may compare the confidence value 502 of 0.9 to a confidence threshold (e.g., a threshold for the signal-to-noise ratio), and the computing system may determine that the confidence value 502 is above the confidence threshold.

[0051] The computing system may continue capturing images with the updated exposure settings to capture image 404, and then determine further updated exposure settings based on image 404 or the settings in which image 404 was captured. The computing system may capture image 406 based on these further updated settings. For each subsequent image, the computing system may concurrently run an autofocus method to determine the confidence level of each image.

[0052] After capturing image 406 and determining its confidence value 506, the computing system may determine that the confidence value 506 of image 406 does not exceed the confidence threshold, which likely indicates that accurate autofocus cannot be determined based on the sensor data collected under the exposure settings used to capture image 406. Based on this determination, the computing system may take one or more actions, as further described below. Based on one or more decisions and / or metrics, and in conjunction with other autofocus accuracy metrics determined based on sensor data received under the auto exposure settings under which the image was captured, for example, based on the confidence value of the image, the computing system may trigger one or more of these actions.

[0053] Returning to method 300, in block 304, method 300 includes interrupting the process of fine-tuning the auto exposure settings based on autofocus confidence to capture an image with a higher exposure than at least one of the above images. In some examples, capturing an image with a higher exposure than at least one of the above images may include scheduling the capture of an image with a higher exposure than at least one of the above images, for example, by adjusting the exposure time. In particular, the higher-exposure image may be captured with a longer exposure time than the lower-exposure image.

[0054] Figure 6 shows a series of image captures according to an exemplary embodiment. Figure 6 may include a series of images 600 captured over time 602, with each block representing a single image capture. In block 610, the computing system may determine that the auto exposure settings used to capture the image (e.g., image 406 in Figures 4 and 5) are insufficient to determine the correct autofocus settings. The computing system may then schedule the capture of an image with a higher exposure than the image captured in block 610 at a specific point in the future. For example, the computing system may schedule the capture of an image in block 612, two frames later.

[0055] In some examples, scheduling the capture of higher-exposure images is based on factors such as the display's frame rate, the exposure time of images captured during the auto-exposure process, and the exposure time required for images captured for autofocus. For example, if the display's frame rate is 30 frames per second and the exposure time of the image stream is 1 / 2000 second, the computing system may schedule the capture of an image for autofocus with an exposure time of 1 / 1000 second. Depending on the frame rate at which the computing system displays the images, the image for autofocus may be scheduled to be captured directly between the two images captured for display, while the two images captured for display may be displayed as usual (e.g., for 1 / 30 second, set based on the display's frame rate).

[0056] The computing system may also schedule the capture of higher exposure images so as to adequately compensate for the delay caused by capturing higher exposure images. For example, the computing system may decide that the display refreshes at a frame rate of 30 frames per second. The computing system may capture a stream of images to be displayed at approximately 1 / 30 second, and the computing system may decide that it may use an exposure time of 1 / 15 second to determine the image or other sensor data to be used for autofocus determination. Thus, the computing system may schedule the capture of an image with an exposure time of 1 / 15 second, based on the fact that the image capture is at least 2 frames ahead, and the computing system may allocate 1 / 15 seconds each to the next two frames to be displayed to compensate for the 1 / 15 second exposure time of the image captured for autofocus.

[0057] Other factors (e.g., software latency, hardware latency, etc.) may also be involved in determining the future time to capture the image.

[0058] Capturing an image for autofocus may interrupt the auto-exposure process. Specifically, the computing system may pause or stop the process of fine-tuning the auto-exposure settings of the image sensor and instead capture an image for autofocus. In some examples, the computing system may collect data for the auto-exposure process using the same sensor as the autofocus process and use the collected data for autofocus instead of auto-exposure. In further examples, the computing system may set the exposure settings to produce a higher-exposed image during the autofocus process and then reset the exposure settings to the same as before the autofocus process.

[0059] In some examples, the auto exposure module of a computing device may perform the task of fine-tuning the auto exposure settings, and interrupting the process of fine-tuning the auto exposure settings may involve the computing system sending a request to the auto exposure module. The request may include an instruction to reconfigure the exposure settings to increase the exposure of the next image captured by the image sensor.

[0060] As shown in the series of images 600, determining the auto exposure setting may be a closed-loop process, while determining the autofocus setting may be an open-loop process without iteration. Specifically, the computing system may determine and update the auto exposure setting for each block / image using data collected using the settings for each image in the series of images 600 excluding block 612, or for each image in the series of images 600 excluding block 612. The computing system may loop through the processes of capturing an image, displaying an image, determining an updated auto exposure setting based on the captured image or other sensor data using the image or the auto exposure setting of the image, and updating the auto exposure setting with the updated auto exposure setting. The computing system may run this loop indefinitely or until the auto exposure setting converges to a specific auto exposure setting. In contrast, the autofocus process may include capturing a single image (e.g., the image in block 612), and the computing system may use that image to determine the autofocus setting. Therefore, the determination of the autofocus setting may not be iterative in that the autofocus setting is not determined and updated for each subsequent image. Rather, the autofocus setting is determined based on a single image that is to be captured in advance. If the autofocus setting is insufficient, the computing system may schedule the capture of other images, and the computing system may use the images to further determine the autofocus setting. By separating the automatic exposure process and the autofocus process, the process described herein may be able to avoid the trade-off between high dynamic range and focus accuracy in challenging lighting conditions (e.g., backlit conditions).

[0061] Returning to method 300, block 306 includes determining an autofocus setting based on an image captured at a higher exposure than at least one of the above images. The autofocus setting may include focal length and / or focus distance. The autofocus setting may be determined based on data from various sensors, including, for example, a phase-detection sensor and / or a camera. An image captured at a higher exposure than the other images may be brighter than the other images.

[0062] Figure 7 shows an image 700 captured for autofocus according to an exemplary embodiment. Image 700 may be an image of the same scene as images 402, 404, and 406 in Figures 4-5. However, compared to images 402, 404, and 406, image 700 may have a higher exposure, which may help the computing system make a more accurate autofocus decision. As represented by image 700, the higher exposure setting used to capture image 700 may cause certain details in the image (e.g., the sun) to be overexposed. Based on image 700 and / or other sensor data captured at a higher exposure setting, the computing system may determine the autofocus image.

[0063] In particular, computing systems may be able to determine autofocus settings more accurately by using images captured at higher exposures, as these images may contain less noise. Furthermore, dark areas of the image may become brighter, allowing the computing system to determine which areas of the darker region to focus on. In post-processing, computing systems may be able to extract detail from dark areas because the image was underexposed and the image focused on specific parts of the dark areas.

[0064] Returning to method 300, block 308 includes causing the image sensor to capture one or more additional images based on the autofocus settings. After configuring the image sensor based on the autofocus settings, the computing system may proceed to perform a process for fine-tuning the auto exposure settings or a process for ensuring that data is captured based on the updated autofocus settings.

[0065] Figure 8 shows an image 800 captured with the updated autofocus settings according to an exemplary embodiment. The computing system can revert to the previously determined auto exposure settings, which helps maximize the dynamic range of the image and may allow the computing system to capture an image with maximum detail. The image 900 captured with the updated autofocus settings and the previously determined auto exposure settings may be displayed on a display, for example, after post-processing. In contrast, the image used to determine the autofocus (e.g., image 700 in Figure 7) may be excluded from display.

[0066] In some examples, the automatic exposure and autofocus processes may be performed on a camera or image sensor system located away from the computing system. The camera or image sensor system may transmit collected data to the computing system, which may determine an updated automatic exposure setting and then transmit it to the camera or image sensor system. If the computing system determines that the confidence level does not exceed a confidence threshold, it may schedule the capture of an image for autofocus. The computing system may transmit the exposure setting to the camera or image sensor system to capture an image for autofocus, and after capturing the image for autofocus, the computing system may determine an updated autofocus setting. The computing system may transmit the updated autofocus image to the remote camera or image sensor system to send an instruction to revert the exposure setting back to the automatic exposure setting. The computing system may include a display that shows the image used for automatic exposure and / or a processed image of the image used for automatic exposure.

[0067] In some examples, determining autofocus reliability involves determining the signal-to-noise ratio of at least one image captured by the image sensor while performing the process of fine-tuning the automatic exposure settings.

[0068] In some cases, determining autofocus confidence further involves determining that the signal-to-noise ratio is below a threshold, and interrupting the process of fine-tuning the auto exposure settings is triggered based on the determination that the signal-to-noise ratio is below a threshold.

[0069] In some examples, Method 300 further includes performing a process of fine-tuning the automatic exposure settings of the image sensor, while ensuring that at least one image captured by the image sensor is displayed, and excluding the display of images captured with a higher exposure than at least one image.

[0070] In some examples, method 300 further includes interrupting the process of fine-tuning the automatic exposure settings to allow at least one image captured by the image sensor to be displayed while performing the process of capturing an image that is more exposed than at least one image.

[0071] In some examples, interrupting the process of fine-tuning the auto exposure settings to capture an image with a higher exposure than at least one image includes (i) scheduling the capture of an image at a specific time in the future based on the frame rate of the display of at least one image captured by the image sensor; (ii) stopping the execution of the process of fine-tuning the auto exposure settings of the image sensor at that specific time; (iii) adjusting the auto exposure settings of the image sensor; and (iv) capturing an image with a higher exposure than at least one image.

[0072] In some examples, Method 300 includes a further process of fine-tuning the automatic exposure settings of the image sensor after configuring the image sensor based on the autofocus settings.

[0073] In some cases, the process of fine-tuning the auto exposure settings to capture an image with a higher exposure than at least one image is based on the determination that at least one image is a high dynamic range image.

[0074] In some examples, method 300 includes ensuring that at least one image is captured based on a first set of autofocus settings, and after capturing images with a higher exposure than at least one image, then capturing images based on a second set of autofocus settings different from the first set of autofocus settings.

[0075] In some cases, fine-tuning the auto exposure settings is based on multiple images, while determining the autofocus settings based on an image is done using a single image.

[0076] In some cases, the process of fine-tuning the automatic exposure settings of an image sensor is based on sensor data collected by one or more phase-detection sensors.

[0077] In some cases, the process of fine-tuning the automatic exposure settings of the image sensor is based on data from the depth sensor.

[0078] In some cases, determining the autofocus settings is not iterative.

[0079] In some cases, interrupting the process of fine-tuning the auto exposure settings to capture an image with a higher exposure than at least one image involves configuring the exposure settings to include an exposure time longer than the exposure time used to capture at least one image.

[0080] In some cases, fine-tuning the auto exposure settings is performed by the auto exposure module, and interrupting the process of fine-tuning the auto exposure settings to capture images with a higher exposure than at least one image involves sending a request to the auto exposure module to reconfigure the exposure settings.

[0081] In some cases, determining autofocus settings based on an image is an open-loop process.

[0082] In some cases, the process of fine-tuning the automatic exposure settings of an image sensor is a closed-loop process.

[0083] In some examples, the computing system includes a control system configured to perform the operations of method 300 and operations including the operations described above.

[0084] In some examples, the control system is further configured to determine the autofocus setting based on sensor data collected by one or more phase-detection sensors.

[0085] In some examples, a non-temporary computer-readable medium stores program instructions executable by one or more processors to cause one or more processors to perform the operations of method 300 and operations including the operations described above.

[0086] III. Conclusion This disclosure is not limited to the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the scope. In addition to those described herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art from the above description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0087] The above detailed description, with reference to the accompanying drawings, illustrates various features and operations of the disclosed systems, devices, and methods. In the drawings, similar reference numerals generally identify similar components unless otherwise indicated by the context. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized and other modifications may be made without departing from the scope of the subject matter presented herein. It will be readily apparent that the aspects of this disclosure generally described herein and shown in the drawings can be arranged, replaced, combined, separated, and designed in a wide variety of different configurations.

[0088] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures, and as described herein, each step, block, and / or communication may represent the processing and / or transmission of information according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, the actions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in an order different from that shown or described, including substantially simultaneous or reversed orders, depending on the functions involved. Furthermore, more or fewer blocks and / or actions may be used in any of the message flow diagrams, scenarios, and flowcharts described herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other in part or as a whole.

[0089] Steps or blocks representing the processing of information may correspond to circuits that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). Program code may include one or more instructions that can be executed by a processor to perform specific logical operations or actions in the method or technique. Program code and / or associated data may be stored in any type of computer-readable medium, such as a storage device including random access memory (RAM), disk drives, solid-state drives, or other storage media.

[0090] Computer-readable media may also include non-temporary computer-readable media such as register memory, processor cache, and RAM, which store data for short periods. Computer-readable media may also include non-temporary computer-readable media that store program code and / or data for long periods. Thus, computer-readable media may include secondary storage or persistent long-term storage such as read-only memory (ROM), optical or magnetic disks, solid-state drives, and compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage systems. Computer-readable media may be considered computer-readable storage media, e.g., tangible storage devices.

[0091] Furthermore, a step or block representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules within the same physical device. However, other information transmissions may be between software and / or hardware modules in different physical devices.

[0092] The specific arrangements shown in the drawings should not be considered limiting. It should be understood that other embodiments may more or less include each of the elements shown in the given drawings. Furthermore, some of the illustrated elements may be combined or omitted. Moreover, the illustrated embodiments may include elements not shown.

[0093] While various aspects and embodiments are disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to limit the scope, and the true scope is indicated by the following claims.

Claims

1. The process involves performing a process to fine-tune the automatic exposure settings of the image sensor, while determining the autofocus reliability based on at least one image captured by the image sensor. Interrupting the process of fine-tuning the automatic exposure setting of the image sensor in order to capture an image with a higher exposure than at least one image based on the autofocus reliability, The autofocus setting is determined based on the image captured with a higher exposure than at least one of the aforementioned images. A method comprising causing the image sensor to capture one or more additional images based on the autofocus setting.

2. Determining the autofocus reliability is The method according to claim 1, comprising determining the signal-to-noise ratio of at least one image captured by the image sensor while performing the process of fine-tuning the automatic exposure setting of the image sensor.

3. The method according to claim 2, further comprising determining the autofocus reliability by determining that the signal-to-noise ratio is less than a threshold for the signal-to-noise ratio.

4. While performing the process of fine-tuning the automatic exposure setting of the image sensor, the at least one image captured by the image sensor is displayed. The method according to claim 1, further comprising excluding the display of any image captured with a higher exposure than the at least one image.

5. The method according to claim 1, further comprising displaying the at least one image captured by the image sensor while interrupting the process of fine-tuning the auto exposure setting of the image sensor to capture the image at a higher exposure than the at least one image.

6. Interrupting the process of fine-tuning the automatic exposure setting of the image sensor in order to capture the image with a higher exposure than the at least one image is, Based on the frame rate of displaying the at least one image captured by the image sensor, the capture of an image with a higher exposure than the at least one image is scheduled for a specific point in the future. At the aforementioned specific point in time, the execution of the process of fine-tuning the automatic exposure setting of the image sensor is stopped, Adjusting the automatic exposure setting of the image sensor, The method according to claim 1, comprising capturing the image with a higher exposure than the at least one image.

7. The method according to claim 1, further comprising, after determining the autofocus setting, further performing the process of fine-tuning the auto exposure setting of the image sensor so that the one or more additional images are captured with different auto exposure settings.

8. The method according to claim 1, wherein interrupting the process of fine-tuning the auto exposure setting of the image sensor to capture the image at a higher exposure than the at least one image is based on the determination that the at least one image represents a high dynamic range scene.

9. The method according to claim 1, further comprising capturing at least one image based on a first set of autofocus settings, wherein the determined set of autofocus settings includes a second set of autofocus settings different from the first set of autofocus settings.

10. The method according to claim 1, wherein the fine-tuning of the automatic exposure setting of the image sensor is based on multiple images, and the determination of the autofocus setting is performed using a single image based on the image captured with a higher exposure than at least one of the images.

11. The method according to claim 1, wherein the process of fine-tuning the automatic exposure setting of the image sensor is based on sensor data collected by one or more phase detection sensors.

12. The method according to claim 1, wherein the process of fine-tuning the automatic exposure setting of the image sensor is based on data from a depth sensor.

13. The method according to claim 1, wherein determining the autofocus setting is not repetitive.

14. The method according to claim 1, wherein interrupting the process of fine-tuning the automatic exposure setting of the image sensor to capture the image at a higher exposure than the at least one image includes configuring the exposure setting to include an exposure time longer than the exposure time used to capture the at least one image.

15. The method according to claim 1, wherein the fine-tuning of the auto exposure setting of the image sensor is performed by an auto exposure module, and interrupting the process of fine-tuning the auto exposure setting of the image sensor to capture the image with an exposure higher than that of at least one image includes sending a request to the auto exposure module to reconfigure the exposure setting.

16. The method according to claim 1, wherein determining the autofocus setting based on the image captured at a higher exposure than the at least one image is an open-loop process.

17. The method according to claim 1, wherein fine-tuning the automatic exposure setting of the image sensor is a closed-loop process.

18. A computing system, Including a control system, the control system is The process involves performing a process to fine-tune the automatic exposure settings of the image sensor, while determining the autofocus reliability based on at least one image captured by the image sensor. Interrupting the process of fine-tuning the automatic exposure setting of the image sensor in order to capture an image with a higher exposure than at least one image based on the autofocus reliability, The autofocus setting is determined based on the image captured with a higher exposure than at least one of the aforementioned images. A computing system configured to cause the image sensor to capture one or more additional images based on the autofocus settings.

19. The computing system according to claim 18, further configured to determine the autofocus setting based on sensor data collected by one or more phase detection sensors.

20. A non-temporary computer-readable medium that stores program instructions executable by one or more processors to cause one or more processors to perform an operation, wherein the operation is: The process involves performing a process to fine-tune the automatic exposure settings of the image sensor, while determining the autofocus reliability based on at least one image captured by the image sensor. Interrupting the process of fine-tuning the automatic exposure setting of the image sensor in order to capture an image with a higher exposure than at least one image based on the autofocus reliability, The autofocus setting is determined based on the image captured with a higher exposure than at least one of the aforementioned images. A non-temporary computer-readable medium, comprising causing the image sensor to capture one or more additional images based on the autofocus setting.

Images