Method for capturing images for later refocusing

Abstract

A processor-implemented method for capturing images for later refocusing, comprising: estimating a depth map for a scene; determining a number of principal depths; capturing a set of images, each image focused on one of the principal depths; and processing captured images to produce an output image.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] The present application is based on and claims priority under 35 USC 119(e) of the applicant's own preliminary US patent application, serial number 61 / 525,102, entitled "Image Capture For Later Refocusing Or Focus-Manipulation," filed on August 18, 2011, which is hereby incorporated in its entirety by reference. The applicant's own preliminary pending US patent application, serial number 13 / 183,363, entitled "Method for Progressively Determining Depth from Defocused Images," filed on July 14, 2011, is also hereby incorporated in its entirety by reference. AREA OF INVENTION

[0002] The present patent application relates generally to the refocusing of an image after its acquisition and in particular to the creation of a depth map during image acquisition preparation in order to enable directed image acquisition for subsequent refocusing processing. BACKGROUND OF THE INVENTION

[0003] The issue of refocusing or focus manipulation after taking a picture is addressed. There are various scenarios where the need for refocusing is essential. The main scenario occurs when the camera focuses on a different object during the shot than the one the photographer intended. Even if the correct object was in focus, the photographer might still want to change the focus position afterward. Therefore, the existence of such an option gives the photographer the freedom to choose the sharpest object after the shot, even if the camera was initially focused on the intended subject.

[0004] Furthermore, focus manipulation can involve more than just refocusing. It provides the ability to create images that a conventional camera cannot capture. For example, it may be possible to allow focus on some objects while excluding others, even if the out-of-focus objects are located among the focused objects.

[0005] Another option that focus manipulation can provide is the ability to produce an image that is sharp everywhere, i.e., an image where all objects are in focus.

[0006] One possible solution to focus manipulation problems is the so-called plenoptic camera, which captures the entire light field of the scene and can be focused on any point after the image has been taken. However, plenoptic cameras use non-standard camera optics and require sensors with a very large number of pixels.

[0007] Therefore, a focus manipulation methodology using standard sensors and conventional camera optics is required. BRIEF DESCRIPTION OF THE EXECUTION FORMS

[0008] This document discloses and claims systems, methods, and computer program products for capturing images for subsequent refocusing and focus manipulation.

[0009] As described in more detail below, the device and processes of the disclosed embodiments permit automatic focus manipulation. Further aspects, tasks, desirable features, and advantages of the devices and methods disclosed herein will be better understood and obvious to a person skilled in the art in view of the detailed description and the drawings that follow, which illustrate various embodiments by way of example. However, it is expressly understood that the drawings serve only for illustrative purposes and are not intended as a definition of the limits of the claimed invention.

[0010] In describing the present invention as an embodiment of the method, a processor-implemented method for capturing images for subsequent refocusing may include: estimating a distance map for a scene, determining a number of principal depths, capturing a set of images, each image being focused on one of the principal depths, and processing captured images to produce an output image.

[0011] Estimation may involve calculating the depth-from-defocus, or it may involve taking images with different focus settings and finding the focus setting that yields the greatest spatial derivative for at least one section of the scene. Estimation may also involve using an external rangefinder.

[0012] The determination process can involve: dividing the scene into multiple rectangles, with the depth map representing the depths of the rectangles, where the depths correspond to a predetermined number of focus levels; counting depth map entries with a specific focus level value; sorting the counted depth map entries into a histogram; and selecting the depths with the most depth map entries as the principal depths. The number of principal depths can be a fraction of the number of focus levels, with the fraction being calculated according to a minimum number of focus levels from the histogram required to cover a predetermined percentage of the rectangles.

[0013] Alternatively, determining depth of field can involve identifying scene areas containing important objects and capturing various important object depths within the principal depths. Important objects can include faces, pets, vehicles, people, and / or recognized faces of specific individuals. The important object depths can be placed in the principal depths first. If the number of important objects is below a predetermined threshold, depths corresponding to the highest histogram counts but not representing important objects can be added to the principal depths. If the number of important objects is equal to or greater than the predetermined threshold, the number of important objects can determine the number of principal depths. The number of principal depths may be limited by available memory and / or the number of frames that can be captured essentially simultaneously.

[0014] The processing can involve selecting an image from the set according to user input or selection criteria. Alternatively, the processing can involve selecting multiple images from the set according to user input or selection criteria, aligning the selected images, and combining the selected images using blending functions. Each blending function is preferably equal to 1 in areas of a corresponding selected image that are in focus, zero in areas of other selected images that are in focus, and equal to values ​​between zero and one in other areas.

[0015] In one system execution mode, a processor executes instructions stored in memory to: estimate a distance map for a scene, determine a number of principal depths, capture a set of images, each image focused on one of the principal depths, and process captured images to produce an output image.

[0016] In a computer program product execution form, a machine-readable medium embodies non-temporary program instructions which, when executed by the machine, cause the machine to capture images for later refocusing by: estimating a distance map for a scene, determining a number of principal depths, capturing a set of images, each image focused on one of the principal depths, and processing captured images to produce an output image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Fig. Figure 1 shows a flowchart according to a first embodiment;

[0018] Fig. 2 shows a histogram according to the first embodiment;

[0019] Fig. Figure 3 shows a flowchart according to a second embodiment. DETAILED DESCRIPTION OF EXAMPLES OF EXECUTION

[0020] Embodiments of the present invention provide a new and patentable methodology for manipulating the focus of captured images.

[0021] One potential solution is to perform a focus bracketing sequence, that is, to capture images with all possible focus positions and, after the images are taken, to select a specific image or to combine several of the images. There are clear limitations to this solution. The speed of image capture can limit the number of images that can be taken, and each image requires a certain amount of storage capacity.

[0022] According to embodiments of the present invention, the camera, during the preview period or when the shutter button is half-pressed, estimates the distance map of the scene being photographed and identifies a small number of principal depths. Then, when the shutter button is fully pressed, a set of images focused on these principal depths is captured. The final processing to achieve refocusing or focus manipulation is based on this small number of captured images.

[0023] A key advantage of these designs is the use of a small number of images instead of the entire range of focus positions. For example, the full range of focus positions can include up to 40 or more focus points, whereas a typical scene might have four to six principal depths of field. The advantages of using a small number of images include: 1. Smaller storage space. 2. Smaller number of images to be processed. 3. Fewer processing artifacts, as the average delay between processed images is much smaller.

[0024] The delays between the different images are small, and therefore the various artifacts of image processing are less noticeable because new CMOS sensors can capture bundles of still images very quickly.

[0025] Now, with reference to Fig. 1. The embodiments of the present invention perform the illustrated procedure. 100 through: – Generating a depth map of the scene during the “half-press” or at a preview time in step 102 . – Based on the depth map, calculate a (small) number of main depths in step 104 . – In step 106Taking a bundle of images that focus on the step 104 The calculated main depths are focused on. – Processing the data in step 106 captured amount of images to proceed to step 108 to obtain the final, focus-manipulated image. – Outputting the final, focus-manipulated image in step 110 . Creating a depth map during the half-pushthrough.

[0026] A depth map can be created in several ways. One possible method is described in the U.S. patent application filed on July 14, 2011, with serial number 13 / 183,363, entitled “Method for Progressively Determining Depth from Defocused Images,” by M. Tzur and G. Rapaport, which is referenced above. In short, the depth map can be created by capturing multiple preview images at several predefined focus positions and then calculating a depth map for the entire image based on the “depth-from-defocus” concept. The advantage of such a method is that there is no need to sample the entire range of focus positions to produce a depth map for the whole image.

[0027] Another way to create a depth map (mentioned earlier) is to sample the entire range of focus positions, i.e., to capture preview images at all focus positions, and to use the best focus position for creating a depth map. This is called a "depth-from-focus" method. With such methods, the best focus position is usually determined by finding the position of the greatest spatial derivative across all focus positions.

[0028] Additional ways to create a depth map can use external rangefinder devices that are not necessarily part of the camera's focusing mechanism. Calculating the principal depths

[0029] Given a depth map of the scene, we want to calculate the principal depths within the scene. Principal depths are depths that belong to large or significant areas of the scene.

[0030] The following procedure can be used to find the principal depths based on large scene areas belonging to a specific depth. Assume the scene is divided into M × N rectangles, and the depth map represents the depth of each rectangle. Let the depth be denoted by D(x, y), where x = 1...M and y = 1...N. It is also assumed that the depth is represented by the corresponding focus level and that the number of focus levels is a finite number Q. Therefore, 1 ≤ D(x, y) ≤ Q.

[0031] Implementations can compute a histogram function H(q) for 1 ≤ g ≤ Q by counting the number of entries in D(x, y) that have a specific q-value. This can be achieved using the following pseudocode: Initialization H(q) = 0 for all 1 ≤ q ≤ Q Loop over all x = 1...M and y = 1...N H(D(x, y)) = H(D(x, y)) + 1 End

[0032] By sorting H(q) from highest to lowest count, the P principal depths are selected as the q-values ​​for which H(q) exhibits the P highest values. In practice, due to various factors, the peaks of the histogram function H(q) can be located, and the P principal depths can be selected as the q-values ​​for which the peaks exhibit the P highest values.

[0033] Now, with reference to Fig. Figure 2 provides an example histogram that gives information regarding each of 64 focus steps used to locate six principal depths. In this way, the principal depths corresponding to the largest areas in the image are selected. In this example, there are six principal depths, as shown in Fig. 2 are shown as points surrounded by circles. P can be defined a priori as a fraction of Q, or it can be calculated based on the histogram H(q). For example, embodiments can choose P as the smallest number such that the sum of the highest P counts is greater than or equal to half (or another fraction) of the total number of rectangles in the depth map. That is, the smallest P value such that where H(q i ) the sorted histogram is (highest value first).

[0034] Another consideration for finding principal depths involves taking significant areas in the image. Significant areas might be, for example, areas where faces or other important objects are detected during the preview. Other important objects could be pets, vehicles, people, or the faces of specific individuals identified during the preview. The depths of important objects, such as faces, could be included in the initial list of principal depths.

[0035] Different constraints can lead to different methods for calculating principal depths. For example, in the case of limited memory, the number of principal depths may be strictly limited by P0 (since it may not be possible to capture more than P0 images in a single bundle). In this case, the method for finding the principal depths should be modified according to this constraint so that the number of principal depths is less than or equal to P0 in all cases.

[0036] Now, with reference to Fig. 3 will be an exemplary procedure 300 This method has been shown to find major depths when the significant areas are faces. This procedure can be a subset of the previously described step. 104In this case, a number of depths P0 that can be processed is selected a priori by the user or according to certain heuristics based, for example, on device limitations. The final number of principal depths is determined according to the number of detected faces and the histogram count H(q).

[0037] In step 302 Depth maps of faces are linked to sets of equal depth. In step 304 A comparison is made between the number of such sets and the number of depths P0 that can be processed. If the number of sets of detected important object depths is less than P0, then depths corresponding to the highest histogram values ​​(and not corresponding to the important objects) can be added to the list of major depths after the depths of each set, as shown in the steps. 310 – 314described. Furthermore, if the number of detected important objects is greater than (or equal to) P0, then the number of principal depths can be determined by the number of important objects detected during a preview, as described in the steps. 306 – 308 described.

[0038] Processing a set of images to obtain a focus-manipulated image.

[0039] There are a number of examples of focus manipulation: 1. Refocusing on a different object. In this case, only one image is selected from the set of captured images. The user reviews all images and indicates which one, for example, is the desired one. Alternatively, an image can be automatically selected based on predefined criteria, such as which image has a detected face or other important object in focus, or which image has the highest depth map histogram count. 2. Refocusing on multiple objects. In this case, the user or a set of heuristics, as described above, selects multiple objects (more than one) to be in focus. The system will then combine the corresponding images into a single image such that all the selected objects will be in focus. Such a combination might be unrealistic in the sense that no real camera can produce such an image. The combination process might require image analysis to align the images. After alignment, the images can be combined using a blending function that is equal to one in the area where the image is in focus, equal to zero in the areas where other images are in focus, and equal to intermediate values ​​(greater than zero and less than one) in other areas.

[0040] For example, suppose the user selects two images, with a different object in focus in each image. Also assume that the two images I1(x, y) and I2(x, y) are already aligned. Then the final image I(x, y) = a1(x, y) I1(x, y) + a2(x, y) I2(x, y) Let a1(x, y) and a2(x, y) be mixed functions such that 0 ≤ a1(x, y) and a2(x, y) ≤ 1. Furthermore, a1(x, y) is equal to 1 in the region where the object is focused in I1(x, y) and equal to zero in the region where the object is focused in I2(x, y). In the other regions, the value of a1(x, y) gradually changes from 1 to 0 and vice versa. A similar logic applies to a2(x, y) with respect to I2(x, y). 3. All objects are in focus.

[0041] In this case, all images will be used and the final image will be be, where I i (x, y) are the aligned images and a i (x, y) are the mixing functions.

[0042] The resulting output can include an output image or an output video clip based on some or all of the processed images.

[0043] The terms “a” or “one / an,” as used herein, are to mean one or more than one. The expression “plural” is to mean two or more than two. The expression “another” is defined as a second or further one. The expressions “containing” and / or “with” are open expressions (e.g., encompassing). References in this document to “an embodiment,” “certain embodiments,” or similar expressions mean that a particular feature, structure, or characteristic described in connection with the embodiment is contained in at least one embodiment. Thus, occurrences of such phrases at various points in this specification do not necessarily all refer to the same embodiment.Furthermore, the specified features, structures, or characteristics can be combined in any suitable way on one or more embodiments without restriction. The expression "or," as used here, is to be interpreted as inclusive, or as encompassing any combination or combinations. Therefore, "A, B, or C" means "one of the following: A; B; C; A and B; A and C; B and C; A, B, and C." An exception to this definition appears only if a combination of elements, functions, steps, or actions is inherently mutually exclusive in any way.

[0044] According to the practices of those skilled in computer programming, embodiments are described by reference to operations performed by a computer system or equivalent electronic system. Such operations are sometimes referred to as being performed by a computer. It is understood that operations represented symbolically involve the manipulation by a processor, such as a central processing unit, of electrical signals representing data bits, and the maintenance of data bits at memory locations, such as in system memory, as well as any other processing of signals. The memory locations where data bits are maintained are physical locations with specific electrical, magnetic, optical, or organic properties corresponding to the data bits.Embodiments may also include an integrated circuit arrangement, including circuit elements that can perform special system operations.

[0045] When implemented in software, the elements of the embodiments are essentially the code segments for performing the required tasks. These non-volatile code segments can be stored on a processor-readable or computer-readable medium, which can include any medium capable of storing or transferring information. Examples of such media include an electronic circuit, a semiconductor memory device, a read-only memory (ROM), flash memory or other non-volatile memory, a floppy disk, a CD-ROM, an optical disc, a hard disk, a fiber optic medium, and so on. User input can include any combination of keyboard, mouse, touchscreen, voice command input, and so forth.User input can be used similarly to direct a browser application running on a user's computing facility to one or more network resources, such as web pages that can access computing resources.

[0046] While the invention has been described in connection with specific examples and various embodiments, the person skilled in the art will readily understand that many modifications and adaptations of the embodiment described herein are possible without deviating from the concept and scope of protection of the invention as claimed below. It is thus clearly understood that the present application is made only by way of example and not as a limitation with respect to the scope of protection of the invention claimed below. The description is intended to cover all variations, uses, or adaptations of the invention that generally follow the principles of the invention and such deviations from the present disclosure as fall within the known and customary practice in the technology to which the invention relates.

Claims

[1] Processor-implemented method for capturing images for later refocusing, comprising: Estimating a distance map for a scene; Determining a number of principal depths; Capturing a set of images, with each image focused on one of the principal depths; and Processing captured images to produce an output image. [2] Method according to claim 1, wherein the estimation includes the depth-from-defocus. [3] Method according to claim 1, wherein the estimation comprises taking pictures with different focus settings and finding the focus setting that yields a greatest spatial derivation for at least one section of the scene. [4] Method according to claim 1, wherein the estimation comprises using an external rangefinder. [5] Method according to claim 1, wherein the determining comprises: Dividing the scene into multiple rectangles, with the depth map representing the depths of the rectangles, where the depths correspond to a predetermined number of focus levels; and Select depths with the most depth map entries as the main depths. [6] Method according to claim 1, wherein the determining comprises: Dividing the scene into multiple rectangles, with the depth map representing the depths of the rectangles, where the depths correspond to a predetermined number of focus levels; Counting depth map entries with a specific focus level value; Sorting counted depth map entries into a histogram; and Select depths with the most depth map entries as the main depths. [7] Method according to claim 6, wherein the number of principal depths is a fraction of the number of focus levels, the fraction being calculated according to a minimum number of focus levels from the histogram required to cover a predetermined percentage of the rectangles. [8] Method according to claim 1, wherein the determining comprises: Identifying scene areas with important objects; and Recording various important object depths in the main depths. [9] Method according to claim 8, wherein the important objects include at least one of faces, pets, cars, people and recognized faces of specific persons. [10] Method according to claim 8, wherein the important object depths are placed first in the main depths. [11] Method according to claim 10, further comprising, if the number of important objects is less than a predetermined number, adding depths corresponding to the highest histogram counts but not corresponding to important objects to the main depths. [12] Method according to claim 8, further comprising, if the number of important objects is at least equal to a predetermined number, the number of important objects determines the number of principal depths. [13] Method according to claim 8, wherein the number of principal depths is limited at least by the available memory and / or the number of images that can be acquired substantially simultaneously. [14] Method according to claim 1, wherein the processing comprises selecting an image from the set according to a user input and / or automated selection criteria. [15] Method according to claim 14, wherein the selection criteria include which image has a recognized face in focus and / or which image has the highest depth map histogram count. [16] The method of claim 1, wherein the processing comprises: Selecting multiple images from the set according to user input and / or automated selection criteria; Aligning the selected images; and Linking the selected images using blending functions. [17] Method according to claim 16, wherein each mixing function is equal to one in areas of a corresponding selected image that are in focus, zero in areas of other selected images that are in focus, and equal to values ​​between zero and one in other areas. [18] Method according to claim 16, wherein the plural includes all captured images. [19] Method according to claim 1, further comprising generating an output video clip based on at least a part of the processed images. [20] Computer program product comprising a machine-readable medium on which non-temporary program instructions are tangibly embodied which, when executed by the machine, cause the machine to capture images for later refocusing by: Estimating a distance map for a scene; Determining a number of principal depths; Capturing a set of images, with each image focused on one of the principal depths; and Processing captured images to produce an output image. [21] System for capturing images for later refocusing, comprising: Means of estimating a distance map for a scene; Means for determining a number of principal depths; Means of capturing a set of images, with each image focused on one of the principal depths; and Means of processing captured images to produce an output image. [22] System for capturing images for later refocusing, comprising: a processor; and a memory that stores instructions, with the processor executing the instructions to: Estimating a distance map for a scene; Determining a number of principal depths; Capturing a set of images, with each image focused on one of the principal depths; and Processing captured images to produce an output image. [23] Integrated circuit for capturing images for later refocusing, comprising: a first circuit element for estimating a distance map for a scene; a second circuit element for determining a number of principal depths; a third circuit element for capturing a set of images, with each image focused on one of the principal depths; and a fourth circuit element for processing captured images to produce an output image.