Method and image capture device for minimizing errors in ambient light corrected images - Patents.com

Abstract

According to one aspect, a method for minimizing error in an ambient light corrected image due to motion is provided, the method comprising: capturing a plurality of primary images with controlled illumination intensity that varies over time; estimating an error value due to motion for each of a plurality of subsets of the plurality of primary images; and performing ambient light correction on the error-minimized subset of primary images having the smallest estimated error value to generate an error-minimized ambient light corrected (ALC) image. Estimating the error value for each subset of the plurality of primary images comprises: identifying a pair of primary images including a first primary image and a second primary image in the subset, the first primary image being captured in time before the second primary image; determining a displacement vector matrix between the pair of primary images; and estimating the error value for the subset of primary images based on the displacement vector matrix between the pair of primary images. The step of determining the displacement vector matrix includes the steps of: dividing the first primary image into a plurality of first grid elements and dividing the second primary image into a plurality of second grid elements; matching the first grid elements with second grid elements corresponding to the same positions on the object; and determining a displacement vector between each matched first grid element and second grid element to determine a displacement vector matrix describing the displacement vector of each grid element.

Description

[Technical field]

[0001] The present invention relates to a method for minimizing errors in ambient light corrected (ALC) images and an image capture device therefor. [Background technology]

[0002] Images of objects captured by a camera or image capture device are often illuminated by ambient light (environmental light) and other illumination sources. To be able to accurately compare images of the same object or to determine the color of the object, the ambient light must be corrected to produce an ambient light corrected image. However, such images are often produced by comparing pixels of the captured images, which introduces errors if there is any movement of the object between frames.

[0003] U.S. Patent Application Publication No. 2004 / 080623 discloses a system and method for minimizing motion clutter in an image producing device. Time-interleaved image subtraction reduces the magnitude of motion clutter without having a detrimental effect on the desired ambient light cancellation of still images. Summary of the Invention [Problem to be solved by the invention]

[0004] The object of the present invention is to solve the problems of the prior art. [Means for solving the problem]

[0005] According to a first particular aspect, a method for minimizing error in an ambient light corrected image due to motion is provided, the method comprising: capturing a plurality of primary images of controlled illumination intensity that varies over time; estimating an error value due to motion for each of a plurality of subsets of the plurality of primary images; and performing ambient light correction on the error-minimized subset of primary images having the smallest estimated error value to generate an error-minimized ambient light corrected (ALC) image. The step of estimating an error value for each subset of a plurality of primary images comprises: identifying pairs of primary images including a first primary image and a second primary image in the subset, where the first primary image is captured temporally before the second primary image; determining a displacement vector matrix between the pair of primary images; and estimating an error value for the subset of primary images based on the displacement vector matrix between the pair of primary images; wherein the step of determining the displacement vector matrix between the pair of primary images includes: dividing the first primary image into a plurality of first grid elements and dividing the second primary image into a plurality of second grid elements; matching the first grid elements with second grid elements corresponding to the same positions on the object; and determining a displacement vector between each matched first grid element and second grid element to determine a displacement vector matrix describing the displacement vector of each grid element.

[0006] Estimating an error value for a subset of primary images may comprise selecting a maximum displacement vector in a displacement vector matrix for the pair of primary images, and estimating an error value based on the maximum displacement vector.

[0007] The step of estimating an error value for each subset of primary images may include: identifying a plurality of pairs of primary images in the subset; determining a displacement vector matrix for each pair of primary images; and estimating an error value for the subset of the plurality of primary images based on the displacement vector matrix for each pair of primary images in the subset.

[0008] The step of estimating an error value for a subset of the multiple primary images may include the steps of selecting a largest absolute displacement vector from the maximum displacement vectors for each pair of primary images, and estimating an error value based on the largest absolute displacement vector in the subset of primary images.

[0009] Determining displacement vectors between a temporally adjacent pair of primary images may comprise matching a first grid element and a second grid element, and determining displacement vectors between non-temporally adjacent pairs of primary images may comprise adding displacement vectors of the temporally adjacent pairs of primary images to map motion trajectories between the non-temporally adjacent pairs of primary images.

[0010] Determining a displacement vector between each matched first grid element and second grid element may comprise performing a displacement estimation based on phase correlation.

[0011] Each subset of primary images may have multiple temporally adjacent primary images, and each subset of primary images may be offset by one temporally adjacent primary image from other subsets of primary images.

[0012] The step of estimating the motion error value for each of the plurality of subsets may be performed in real time for temporally successive subsets of the primary images. Each time an estimated error value for a subset of the primary images is smaller than any previously estimated error value, the method may include performing ambient light correction on the corresponding subset of the primary images to generate a minimized error ALC image.

[0013] According to a second aspect, there is provided a non-transitory computer readable storage medium having computer readable instructions which, when executed by a processor, cause the processor to perform a method according to the first aspect.

[0014] According to a third aspect there is provided a computer program which, when read by a computer, causes the computer to carry out the method according to the first aspect.

[0015] According to a fourth aspect, there is provided an image capture device having at least one processor; and at least one memory containing computer readable instructions; the at least one processor configured to read the computer readable instructions to cause the device to perform a method according to the first aspect.

[0016] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0017] Exemplary embodiments will now be described, by way of example only, with reference to the following drawings, in which: [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates a schematic of an image capture device that minimizes errors in ambient light corrected images. [Diagram 2] FIG. 2 is a flow chart illustrating steps of a first exemplary method for minimizing error in an ambient light corrected image. [Diagram 3] FIG. 3 illustrates diagrammatically the steps of the first exemplary method. [Figure 4] FIG. 4 is a flow chart illustrating steps of a second exemplary method for estimating error values ​​for a subset of primary images. [Figure 5A] FIG. 5A illustrates diagrammatically the steps of a second exemplary method. [Figure 5B] FIG. 5B illustrates diagrammatically the steps of a second exemplary method. [Figure 5C] FIG. 5C illustrates diagrammatically the steps of a second exemplary method. [Figure 6] FIG. 6 is a flow chart illustrating steps of a third exemplary method for minimizing error in an ambient light corrected image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] 1 shows an image capture apparatus 10 comprising an image capture device 12, a light 14, a processor 16, and a memory 18 containing computer readable instructions. The image capture device 12 is connected to the processor 16 and configured to capture images of an object 20 (in this example, a human being) that can be read and processed by the processor 16.

[0020] The light 14 is configured to illuminate the object 20 and is also connected to the processor 16. The light 14 is configured to emit illumination of variable intensity controlled by the processor 16. In other examples, the light 14 can be controlled by a separate processor 16.

[0021] The memory 18 includes computer readable instructions, and the processor 16 is configured to read the computer readable instructions to execute a method for minimizing error in an ambient light corrected (ALC) image.

[0022] Figure 2 is a flow chart illustrating the steps of a first exemplary method 100 for minimizing errors in an ALC image, and Figure 3 illustrates the method steps with respect to the output of captured and processed images.

[0023] At block 102, the first exemplary method 100 comprises capturing multiple primary images 30 of an object 20 over time, where the object 20 is illuminated by a light 14 and the illumination is controlled to vary over time. In Figure 3, the progression of time is represented by arrows 34. Each primary image 30 comprises a number of pixels 50, each of which has a pixel index corresponding to the position of the pixel 50 in the image, and a pixel value defining the intensity of light captured at the pixel 50.

[0024] Following block 102, method 100 includes identifying a plurality of subsets of primary images 30. Each subset of primary images 30 includes a predetermined number of temporally adjacent primary images 30 for which ambient light correction is performed to generate an ambient light corrected (ALC) image 42. In this example, an ALC image is generated from two primary images 30, and thus each subset of primary images 30 includes two primary images 30. In other examples, there may be any number of primary images from which a single ALC image is generated. The process of ambient light correction is described in more detail below.

[0025] Each subset of primary images 30 is offset by one temporally adjacent primary image 30 from the preceding subset of primary images 30. In other examples, there may be an offset of more than one primary image between the subsets of primary images.

[0026] Block 104 is followed by block 106, in which the method 100 includes estimating an error value due to motion for each of a plurality of subsets of the primary images 30. Block 106 of the method 100 includes blocks 108, 110, 112, 114, and 116. Because the steps in block 106 are applied to each subset of the primary images 30, these steps are described below with respect to a single subset of the primary images 30.

[0027] At block 108, the method 100 includes identifying pairs of primary images 30 within the subset of primary images 30. In this example, the subset of primary images 30 includes two primary images 30, so there are only pairs of primary images 30. Each pair of primary images 30 is defined as a first primary image 30a and a second primary image 30b, where the first primary image 30a is captured earlier in time than the second primary image 30b.

[0028] It will be appreciated that where a primary image subset includes more than two primary images, there will be more than one pair of primary images within that primary image subset.

[0029] Following block 108, the method 100 includes dividing the first primary image 30a into a first plurality of grid elements and dividing the second primary image 30b into a second plurality of grid elements, each of which may include one pixel 50 or multiple pixels 50.

[0030] At block 112, the method 100 includes matching a first grid element, each of which may represent a location on the object 20, with a second grid element corresponding to the same location on the object 20. The matched first and second grid elements may include pixels 50 with different pixel indices due to motion of the object 20 between the capture of the first primary image 30a and the capture of the second primary image 30b.

[0031] At block 114, the method 100 comprises determining a displacement vector between each matched first and second grid element, describing the movement of the object 20 in terms of movement from the first primary image 30a to the second primary image 30b, and determining a displacement vector matrix 36 describing the displacement vector of each matched grid element. The displacement vector matrix 36 may be a list of vectors that may be stored in an array corresponding to the grid elements, or may simply be a list of vectors that do not bear a correspondence with the grid elements.

[0032] In some examples, determining the displacement vectors of the matched first and second grid elements may comprise performing a displacement estimation based on phase correlation. Using phase information of the light received at each pixel or grid element to determine the displacement vector and ignoring the light intensity (i.e. light amplitude) means that the displacement estimation will be more accurate, especially when there are variations in the light.

[0033] At block 116, the method 100 includes estimating an error value 38 for the subset of primary images 30 based on the displacement vector matrix 36 between the pair of primary images 30. If there is more than one pair of primary images in the subset of primary images, the error value for the subset of primary images will be based on the displacement vector matrix for each pair of primary images in the subset.

[0034] In this example, estimating the error value 38 for the subset of primary images 30 includes selecting a maximum displacement vector in the displacement vector matrix 36 of the pair of primary images 30 and estimating the error value based on the maximum displacement vector. If there is more than one pair of primary images 30 in the subset of primary images 30, estimating the error value 38 may include selecting a maximum displacement vector in the displacement vector matrix 36 of all pairs of primary images 30 in the subset of primary images 30 and estimating the error value 38 based on the maximum displacement vector. In other examples, estimating the error value may include, for example, aggregating by summing the displacement vectors or determining an average absolute displacement vector.

[0035] Blocks 108 - 116 are repeated for each subset of primary images 30 , so that the output from block 106 is a plurality of error values ​​38 , each corresponding to the estimated error due to motion for a corresponding subset of primary images 30 .

[0036] At block 118, the method 100 includes selecting the error-minimized subset 40 of primary images having the smallest error value Ex and performing ambient light correction on the error-minimized subset 40 of primary images to generate an error-minimized ALC image 42. In a simplified example of ambient light correction, a first primary image may be captured with the lights on to illuminate the object, while a second primary image may be captured with the lights off such that only ambient light illuminates the object. For each pixel of the first primary image, the intensity of the second primary image at that pixel may be subtracted to generate an ALC pixel, all of which are combined to generate the ALC image. A more complex example of ambient light correction can be found in Kolaman, Amir & Hagege, Rami & Guterman, Hugo. (2014). Light source separation from image sequences of oscillating lights. 2014 IEEE 28th Convention of Electrical and Electronics Engineers in Israel, IEEEI 2014. 1-5. 10.1109 / EEEI.2014.7005874.

[0037] Figure 4 is a flow chart showing steps of a second exemplary method 200 for estimating error values ​​38 for a subset of primary images 30 having three or more primary images 30. The second exemplary method 200 can replace block 106 of the first exemplary method. Figures 5A, 5B and 5C show the steps of the second exemplary method 200 in a schematic manner.

[0038] Returning to FIG. 3, the second exemplary method 200 produces the same basic output as the first exemplary method 100 at various steps, and so the following description will refer to the numerals of FIG.

[0039] At block 202, the second exemplary method 200 includes identifying a plurality of pairs of primary images 30 in the subset. For example, if the subset includes three primary images, the subset would have three independent pairs of primary images including: (i) a first primary image and a second primary image, (ii) a second primary image and a third primary image, and (iii) a first primary image and a third primary image. In the example shown in Figures 5A-5C, the subset has six primary images 30, which is fifteen pairs of primary images 30.

[0040] In block 204, the method 200 includes determining a displacement vector matrix 36 for each pair of temporally adjacent primary images 30 in a manner similar to blocks 110-114 of the first exemplary method 100. FIG. 5A shows a first primary image 30a divided into a number of grid elements 46, with a displacement vector 48 indicated by an arrow for each grid element 46, where the start of the arrow is the position of the grid element on the first primary image 30a of the pair and the end of the arrow is the position of a matching grid element on the second primary image 30b of the pair. The displacement vectors 48 for the grid elements 46 form a displacement vector matrix 36 between the first and second primary images 30a, 30b of the temporally adjacent pair.

[0041] In block 206, the method 200 includes determining a displacement vector matrix 36 for a non-temporally adjacent pair of primary images 30 by adding the displacement vectors 48 for each grid element 46 of a temporally adjacent pair of primary images 30 to map a motion trajectory 52 between the pair of primary images 30 that are not temporally adjacent.

[0042] 5B illustrates a motion trajectory 52 for grid elements 46 between the first and sixth primary images by aggregating displacement vectors 48 between temporally adjacent pairs of primary images 30. In another example, the displacement vector matrix for non-temporally adjacent primary images can be calculated in a similar manner as the displacement vector matrix for temporally adjacent primary images. Determining the displacement vector matrix for non-temporally adjacent image pairs by mapping the motion trajectories requires less processing power than matching grid elements and calculating displacement vectors for the matched grid elements between non-temporally adjacent pairs of primary images.

[0043] FIG. 5C shows displacement vectors 48 between temporally non-adjacent pairs of primary images superimposed on the motion trajectories 52.

[0044] At block 208, the method 200 includes estimating an error value 38 for a subset of primary images 30 based on a displacement vector matrix 36 between each pair of primary images 30 in the subset. In this example, estimating an error value for a subset of a plurality of primary images 30 includes determining a maximum displacement vector for each pair of primary images 30 in the subset of primary images 30 and selecting the largest absolute displacement vector from these maximum displacement vectors. The error value 38 may be based on the largest absolute displacement vector.

[0045] The method 200 may be repeated for each subset of the primary images 30 to estimate the error value, and the method may proceed to block 118 of the first exemplary method 100, select the error-minimized subset 40 of the primary images having the smallest error value Ex, and perform ambient light correction on the error-minimized subset 40 of the primary images to generate an error-minimized ALC image 42.

[0046] FIG. 6 illustrates a third exemplary method 300 for minimizing errors in an ALC image due to motion.

[0047] In block 302 , the method 300 includes capturing a plurality of primary images 30 in a manner similar to block 102 of the first exemplary method 100 .

[0048] In block 304, the method includes identifying a subset of primary images 30 including a predetermined number of temporally adjacent primary images 30 for which ambient light correction is to be performed to generate an ambient light corrected (ALC) image 42 in a manner similar to block 104 of the first exemplary method 100.

[0049] Block 304 is followed by block 306, in which the method 300 includes estimating motion error values ​​38 for the subset of primary images 30. This may be in a manner similar to blocks 202-208 of the second exemplary method 200, or block 106 of the first exemplary method 100.

[0050] At block 308, it is determined whether the error values ​​38 of the subset of primary images 30 are the smallest error values ​​found so far. If it is determined that the error values ​​38 of the subset of primary images 30 are not the smallest so far, the method 300 returns to block 302 to capture at least one additional primary image 30 and identify another subsequent subset of primary images 30. The subsequent subset of primary images 30 may be offset from the previous subset of primary images 30 by one temporally adjacent primary image 30.

[0051] Each subset of primary images 30 is offset by one temporally adjacent primary image 30 from the previous subset of primary images 30. In other examples, there may be an offset of more than one primary image between the subsets of primary images.

[0052] If, in block 308, it is determined that the error values ​​of the subset of primary images 30 are the smallest so far (i.e., the subset of primary images 30 is an error-minimized subset 40 of primary images), the method 300 proceeds to block 310, where ambient light correction is performed on the error-minimized subset 40 of primary images to form an error-minimized ALC image 42.

[0053] The method 300 then returns to block 302 to continue capturing additional primary images 30. Thus, the third exemplary method 300 is very similar to the first exemplary method 00, except that rather than determining which of the subsets of primary images 30 has the smallest estimated error after multiple primary images 30 have been captured and performing ambient light correction on the subset with the minimized error, the third exemplary method 300 estimates error values ​​for the subset of primary images 30 in real time for temporally successive subsets of primary images and performs ambient light correction on the subset of primary images 30 each time each subset has an estimated error value 38 that is less than the previously estimated error value.

[0054] Variations of the disclosed embodiments can be understood and effected by those skilled in the art from a study of the drawings, the disclosure and the appended claims, in implementing the principles and techniques described herein. In the claims, the word "comprising" does not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program can be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided together with or as part of other hardware, but also in other forms, such as via the Internet or other wired or wireless communication systems. Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. 1. A method for minimizing motion-induced ambient light corrected image error, comprising: capturing a plurality of primary images of controlled illumination intensity that varies over time; estimating a motion error value for each of a plurality of subsets of the plurality of primary images; performing ambient light correction on the error-minimized subset of the primary images having the lowest estimated error values ​​to generate an error-minimized ambient light corrected image; having estimating an error value for each subset of the plurality of primary images identifying a pair of primary images in the subset including a first primary image and a second primary image, the first primary image being captured temporally before the second primary image; determining a displacement vector matrix between said pair of primary images; estimating error values ​​for a subset of the primary images based on the displacement vector matrix between the pairs of primary images; having determining a displacement vector matrix between the pair of primary images, Dividing the first primary image into a plurality of first grid elements and dividing the second primary image into a plurality of second grid elements; matching a first grid element with a second grid element corresponding to the same location on the object; determining a displacement vector between each matched first grid element and second grid element to determine the displacement vector matrix describing the displacement vectors of each grid element; The method comprising:

2. 2. The method of claim 1, wherein estimating an error value for a subset of the primary images comprises selecting a maximum displacement vector in a displacement vector matrix for the pair of primary images, and estimating the error value based on the maximum displacement vector.

3. estimating an error value for each of the subsets, identifying a plurality of pairs of primary images in the subset; determining said displacement vector matrix for each pair of primary images; estimating an error value for a subset of the plurality of primary images based on a displacement vector matrix for each pair of primary images in the subset; The method according to claim 1 or 2, comprising:

4. 4. The method of claim 2 or 3, wherein the step of estimating an error value for a subset of the plurality of primary images comprises the steps of selecting a largest absolute displacement vector from the maximum displacement vectors for each pair of primary images, and estimating the error value based on the largest absolute displacement vector in the subset of primary images.

5. 5. The method of claim 3 or 4, wherein the step of determining displacement vectors between a temporally adjacent pair of the primary images comprises matching the first grid element and the second grid element, and the step of determining displacement vectors between non-temporally adjacent pairs of the primary images comprises adding up the displacement vectors of the temporally adjacent pairs of the primary images to map motion trajectories between the non-temporally adjacent pairs of the primary images.

6. 6. The method of claim 1 , wherein determining a displacement vector between each matched first grid element and second grid element comprises performing a displacement estimation based on phase correlation.

7. 7. The method of claim 1, wherein each subset of primary images comprises a plurality of temporally adjacent primary images, and each subset of primary images is offset from other subsets of primary images by one temporally adjacent primary image.

8. 8. The method of claim 7, wherein the step of estimating motion error values ​​for each of the plurality of subsets is performed in real time for temporally successive subsets of the primary images, and each time an estimated error value for a subset of the primary images becomes smaller than any previously estimated error value, the method comprises the step of performing ambient light correction on a corresponding subset of the primary images to generate a minimized error ALC image.

9. A non-transitory computer readable storage medium having computer readable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

10. A computer program which, when read by a computer, causes the computer to carry out the method according to any one of claims 1 to 8.

11. 1. An image capture device, comprising: At least one processor; at least one memory containing computer readable instructions; having The at least one processor reads the computer readable instructions to perform the method of any one of claims 1 to 8. Image capture device.

Images