Method and image capturing apparatus for minimizing errors in an ambient light corrected image

Abstract

According to one aspect, a method for minimizing errors due to motion in an ambient light corrected image is provided, the method comprising capturing a plurality of primary images having controlled illumination intensity that varies over time, estimating an error value due to motion for each subset of a plurality of subsets of the plurality of primary images, and performing ambient light correction on a motion minimized subset of primary images having a lowest estimated error value to generate an ambient light corrected (ALC) image having minimized errors. Estimating the error value for each subset of the plurality of primary images comprises identifying a pair of primary images in the subset comprising a first primary image and a second primary image, wherein the first primary image is captured in time before the second primary image, determining a displacement vector matrix between the pair of primary images, and estimating the error value for the subset of primary images based on the displacement vector matrix between the pair of primary images. Determining the displacement vector matrix comprises dividing the first primary image into a plurality of first grid elements and the second primary image into a plurality of second grid elements, matching the first grid elements with the second grid elements corresponding to the same location on an object, and determining a displacement vector between each matched first grid element and second grid element to determine a displacement vector matrix describing the displacement vector for each grid element.

Description

Technical Field

[0001] The present invention relates to a method for minimizing errors in ambient light corrected (ALC) images and an image capture apparatus for the method. Background Technology

[0002] Images of objects captured by cameras or image capture devices are typically illuminated by ambient light as well as individual light sources. To accurately compare images of the same object or determine its color, ambient light must be corrected to produce an ambient light-corrected image. However, such images are usually generated by comparing pixels of the captured image, which introduces errors if there is any movement of the object between frames.

[0003] US2004 / 080623 discloses a system and method for minimizing motion clutter in an image generation apparatus. Temporally interleaved image subtraction reduces the magnitude of motion clutter without adversely affecting the desired ambient light cancellation of static images. Summary of the Invention

[0004] According to a first aspect, a method for minimizing motion-induced errors in an ambient light corrected image is provided, the method comprising: capturing a plurality of master images having controlled illumination intensities varying over time; estimating motion-induced error values ​​for each subset of the plurality of master images; and performing ambient light correction on the error-minimizing subset of master images having the lowest estimated error values ​​to generate an ambient light corrected (ALC) image with minimized error; wherein estimating the error values ​​for each subset of the plurality of master images comprises: identifying a pair of master images in the subset, the pair of master images including a first master image and a second master image, wherein the first master image was captured temporally prior to the second master image; determining a displacement vector matrix between the pair of master images, comprising: dividing the first master image into a plurality of first grid elements and dividing the second master image into a plurality of second grid elements, matching the first grid elements with second grid elements corresponding to the same position on an object, and determining a displacement vector between each matched first grid element and second grid element to determine a displacement vector matrix describing the displacement vector of each grid element; and estimating error values ​​for subsets of master images based on the displacement vector matrix between the pair of master images.

[0005] Estimating the error value for a subset of the main images may include: selecting the maximum displacement vector in the displacement vector matrix for the pair of main images, and estimating the error value based on the maximum displacement vector.

[0006] Estimating the error value for each subset of the main images may include: identifying multiple pairs of main images in the subset; determining the displacement vector matrix for each pair of main images; and estimating the error value for subsets of multiple main images based on the displacement vector matrix for each pair of main images in the subset.

[0007] Estimating the error value for a subset of multiple master images may include: selecting the highest absolute displacement vector from the maximum displacement vectors for each pair of master images, and estimating the error value based on the highest absolute displacement vector in the subset of master images.

[0008] Determining the displacement vector between temporally adjacent pairs of master images may include matching a first grid element and a second grid element. Determining the displacement vector between temporally non-adjacent pairs of master images may include adding the displacement vectors of temporally adjacent pairs of master images to map the motion trajectories between the temporally non-adjacent pairs of master images.

[0009] Determining the displacement vector between each matched first grid element and second grid element may include performing displacement estimation based on phase correlation.

[0010] Each subset of the main image can include multiple temporally adjacent main images. Each subset of the main image can be offset from another subset of the main image by a temporally adjacent main image.

[0011] The method can perform real-time estimation of motion-induced error values ​​for each of multiple subsets of a temporally continuous main image. Each time the estimated error value for a subset of the main image is lower than any previously estimated error value, the method may include performing ambient light correction on the corresponding subset of the main image to generate an ALC image with minimized error.

[0012] According to a second aspect, a non-transitory computer-readable storage medium is provided, including computer-readable instructions that, when executed by a processor, cause execution of the method according to the first aspect.

[0013] According to a third aspect, a computer program is provided that, when read by a computer, causes execution according to the method of the first aspect.

[0014] According to a fourth aspect, an image capture apparatus is provided, the image capture apparatus comprising: at least one processor; at least one memory including computer-readable instructions; the at least one processor being configured to read the computer-readable instructions and cause execution of the method according to the first aspect.

[0015] These and other aspects will be clear from the embodiments described below and will be explained with reference to the embodiments described below. Attached Figure Description

[0016] Exemplary embodiments will now be described by way of example only with reference to the following figures, in which:

[0017] Figure 1 An image capture device for minimizing errors in ambient light corrected images is schematically illustrated.

[0018] Figure 2 This is a flowchart illustrating the steps of a first example method for minimizing errors in ambient light corrected images;

[0019] Figure 3 The steps of the first example method are illustrated schematically;

[0020] Figure 4 This is a flowchart illustrating the steps of a second example method for estimating error values ​​for a subset of the main image;

[0021] Figures 5A-5C The steps of the second example method are illustrated schematically; and

[0022] Figure 6 This is a flowchart illustrating the steps of a third example method for minimizing errors in ambient light corrected images. Detailed Implementation

[0023] Figure 1 An image capture device 10 is shown, which includes an image capture device 12, a lamp 14, a processor 16, and a memory 18 including computer-readable instructions. The image capture device 12 is connected to the processor 16 and is configured to capture an image of an object 20, which in this example is a person, and the image can be read and processed by the processor 16.

[0024] Lamp 14 is configured to illuminate object 20 and is also connected to processor 16. Lamp 14 is configured to emit variable intensity illumination controlled by processor 16. In other examples, lamp 14 may be controlled by a separate controller.

[0025] The memory 18 includes computer-readable instructions, and the processor 16 is configured to read the computer-readable instructions to perform a method for minimizing errors in an ambient light correction (ALC) image.

[0026] Figure 2 This is a flowchart illustrating the steps of a first example method 100 for minimizing errors in an ALC image. Figure 3 The steps of the method in capturing and processing the output of the image are shown.

[0027] In box 102, the first example method 100 includes capturing multiple master images 30 of an object 20 over time, wherein the object 20 is illuminated by light 14, and wherein the illumination is controlled to vary over time. Figure 3 In the image, the passage of time is represented by arrow 34. Each main image 30 includes multiple pixels 50, each pixel having a pixel index corresponding to the position of pixel 50 in the image, and a pixel value defining the intensity of light captured at that pixel 50.

[0028] In box 104, following box 102, method 100 includes identifying multiple subsets of the master images 30. Each subset of the master images 30 includes a predetermined number of temporally adjacent master images 30 from which ambient light correction is performed to generate an ambient light corrected (ALC) image 42. In this example, the ALC image is generated from two master images 30, and therefore each subset of the master images 30 includes two master images 30. In other examples, any number of master images may exist from which a single ALC image can be generated. The ambient light correction process will be explained in more detail below.

[0029] Each subset of the main image 30 is offset from the previous subset of the main image 30 by a time-adjacent main image 30. In other examples, the offsets between subsets of the main images may be more than one main image.

[0030] In box 106, following box 104, method 100 includes estimating motion-induced error values ​​for each of a plurality of subsets of the main image 30. Box 106 of method 100 includes boxes 108, 110, 112, 114, and 116. Since the steps across 106 are applied to each subset of the main image 30, these steps will be described below for a single subset of the main image 30.

[0031] In box 108, method 100 includes identifying pairs of main images 30 within a subset of the main images 30. In this example, the subset of main images 30 includes two main images 30, and therefore there is only one pair of main images 30. Each pair of main images 30 is defined as a first main image 30a and a second main image 30b, wherein the first main image 30a is captured temporally before the second main image 30b.

[0032] It should be understood that if a subset of the main images includes more than two main images, then within the subset of the main images, there will be more than one pair of main images.

[0033] In box 110, following box 108, method 100 includes dividing a first main image 30a into a plurality of first grid elements and dividing a second main image 30b into a plurality of second grid elements. Each grid element may include one pixel 50 or more pixels 50.

[0034] In box 112, method 100 includes matching a first grid element, which may each represent a position on object 20, with a second grid element corresponding to the same position on object 20. The matched first and second grid elements may include pixels 50, which have different pixel indices due to the movement of object 20 between capturing a first main image 30a and a second main image 30b.

[0035] In box 114, method 100 includes determining a displacement vector between each matched first grid element and a second grid element, describing the movement of object 20 based on movement from the first main image 30a to the second main image 30b, and determining a displacement vector matrix 36 describing the displacement vector of each matched grid element. The displacement vector matrix 36 can be a list of vectors recorded as an array corresponding to grid elements, or it can simply be a list of vectors that does not maintain a correspondence with grid elements.

[0036] In some examples, determining the displacement vector for a matching first and second grid element may include performing displacement estimation based on phase correlation. Using the phase information of the light received at each pixel or grid element to determine the displacement vector, and ignoring the light intensity (i.e., the amplitude of the light), means the displacement estimation is more accurate, especially when there are variations in the light.

[0037] In block 116, method 100 includes estimating an error value 38 for a subset of the main images 30 based on the displacement vector matrix 36 between the pair of main images 30. If there are more than one pair of main images in the subset of main images, the error value for the subset of main images is based on the displacement vector matrix of each pair of main images in the paired main images within the subset.

[0038] In this example, estimating the error value 38 for a subset of the main images 30 includes selecting the largest displacement vector in the displacement vector matrix 36 for the pair of main images 30 and estimating the error value based on the largest displacement vector. If there are more than one pair of main images 30 in the subset of the main images 30, estimating the error value 38 may include selecting the largest displacement vector in the displacement vector matrix 36 of all pairs of main images 30 within the subset of the main images 30 and estimating the error value based on the largest displacement vector. In other examples, estimating the error value may include, for example, aggregating the displacement vectors by summing the displacement vectors, or determining the average absolute displacement vector.

[0039] Since boxes 108-116 are repeated for each subset of the main image 30, the output from box 106 is a plurality of error values ​​38, each error value corresponding to the motion-induced estimation error for the corresponding subset of the main image 30.

[0040] In box 118, method 100 includes selecting the value with the lowest error E. x The error of the primary image 40 is minimized, and ambient light correction is performed on the error-minimized subset of the primary image 40 to generate an ALC image 42 with minimized error. In a simplified example of ambient light correction, a first primary image can be captured with the lights on to illuminate the object, and a second primary image can be captured with the lights off so that only ambient light illuminates the object. For each pixel of the first primary image, the intensity of the second primary image at that pixel can be subtracted to generate an ALC pixel, and all ALC pixels are combined to generate an ALC image. A more complex example of ambient light correction can be found in "Kolaman, Amir & Hagege, Rami & Guterman, Hugo. (2014). Light source separation from image sequences of oscillating lights. 2014 IEEE 28th Convention of Electrical and Electronics Engineers in Israel, IEEE I 2014.1-5.10.1109 / EEEI.2014.7005874".

[0041] Figure 4 This is a flowchart illustrating the steps of a second example method 200 for estimating an error value 38 for a subset of main images 30 having more than two main images 30. The second example method 200 may replace block 106 of the first example method 100. Figure 5A , Figure 5B and Figure 5C The steps of the second example method 200 are illustrated schematically.

[0042] Refer again Figure 3 The second example method 200 produces the same basic output as the first example method 100 at each step, and therefore the following description will refer to... Figure 3 The accompanying figure labels.

[0043] In box 202, the second example method 200 includes identifying multiple pairs of master images 30 within a subset. For example, if the subset contains three master images, then the subset will include three independent pairs of master images, including (i) a first master image and a second master image, (ii) a second master image and a third master image, and (iii) a first master image and a third master image. Figures 5A-5C In the example shown, the subset includes six main images 30, which comprises 15 pairs of main images 30.

[0044] In box 204, method 200 includes determining a displacement vector matrix 36 for each pair of temporally adjacent master images 30 in a manner similar to boxes 110-114 of the first example method 100. Figure 5A A first master image 30a is shown, divided into multiple grid elements 46. For each grid element 46, a displacement vector 48 is represented by an arrow, where the beginning of the arrow is the position of the grid element in the first master image 30a of the pair, and the end of the arrow is the position of the matching grid element in the second master image 30b of the pair. The multiple displacement vectors 48 of the multiple grid elements 46 form a displacement vector matrix 36 between the first master image 30a and the second master image 30b in a pair of temporally adjacent master images 30.

[0045] In box 206, method 200 includes: determining a displacement vector matrix 36 for a pair of non-adjacent main images 30 by adding displacement vectors 48 for each grid element 46 of a pair of temporally adjacent main images 30, to map motion trajectories 52 between the pair of non-adjacent main images 30.

[0046] Figure 5B The motion trajectory 52 of the grid element 46 between the first and sixth main images is shown by aggregating the displacement vector 48 between temporally adjacent pairs of main images 30. In other examples, the displacement vector matrix for temporally non-adjacent main images can be calculated in a similar manner to that for temporally adjacent main images. Determining the displacement vector matrix for temporally non-adjacent pairs of main images by mapping motion trajectories requires less processing power compared to matching grid elements and calculating the displacement vectors for matching grid elements between temporally non-adjacent pairs of main images.

[0047] Figure 5C The displacement vector 48 between pairs of non-adjacent master images superimposed on motion trajectory 52 is shown.

[0048] In box 208, method 200 includes estimating an error value 38 for a subset of master images 30 based on a displacement vector matrix 36 between each pair of master images 30 within the subset. In this example, estimating the error value for a subset of multiple master images 30 includes determining the maximum displacement vector in each pair of master images 30 within the subset of master images 30, and selecting the highest absolute displacement vector from the maximum displacement vector. The error value 38 may be based on the highest absolute displacement vector.

[0049] Method 200 can be repeated for each subset of the main image 30 to estimate the error value, and the method can continue to box 118 of the first example method 100 to select the subset with the lowest error value E. xThe error-minimized subset of the main image 40 is obtained, and ambient light correction is performed on the error-minimized subset of the main image 40 to generate an ALC image 42 with minimized error.

[0050] Figure 6 A third example method 300 for minimizing motion-induced errors in ALC images is shown.

[0051] In box 302, method 300 includes capturing a plurality of master images 30 in a manner similar to box 102 of the first example method 100.

[0052] In block 304, the method includes identifying a subset of main images 30 comprising a predetermined number of temporally adjacent main images 30 in a manner similar to block 104 of the first example method 100, and performing ambient light correction on the subset of main images 30 to generate an ambient light corrected (ALC) image 42.

[0053] In box 306, following box 304, method 300 includes estimating motion-induced error values ​​38 for a subset of the main image 30. This can be done in a manner similar to boxes 202-208 of the second example method 200 or box 106 of the first example method 100.

[0054] In box 308, it is determined whether the error value 38 of a subset of the main image 30 is the lowest error value found to date. If it is determined that the error value 38 of a subset of the main image 30 is not the lowest to date, then method 300 returns to box 302 to capture at least one additional main image 30 and identify another later subset of the main image 30. This other later subset of the main image 30 can be offset from an earlier subset of the main image 30 by a time-adjacent main image 30.

[0055] Each subset of the main image 30 is offset from the previous subset of the main image 30 by a time-adjacent main image 30. In other examples, the offsets between subsets of the main images may be more than one main image.

[0056] If it is determined in box 308 that the error value of the subset 30 of the main image is the lowest so far (i.e., the subset of the main image 30 is the error-minimizing subset of the main image 40), then method 300 continues to box 310, in which ambient light correction is performed on the error-minimizing subset of the main image 40 to form an ALC image 42 with minimized error.

[0057] Method 300 then returns to box 302 to continue capturing another main image 30. Therefore, the third exemplary method 300 is very similar to the first exemplary method 100, except that, for a subset of main images 30 that are temporally consecutive, the error value of the subset is estimated in real time, and ambient light correction is performed on that subset of main images 30 whenever the estimated error value 38 of the corresponding subset is lower than the previously estimated error value, instead of determining which of the multiple subsets of main images 30 has the lowest estimated error and performing ambient light correction on the subset with the minimum error after multiple main images 30 have been captured.

[0058] By studying the accompanying drawings, this disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practice with respect to the principles and techniques described herein. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plural. A single processor or other unit can perform the functions of several items described in the claims. The fact that certain measures are listed only in mutually different dependent claims does not imply that combinations of these measures cannot be used for advantageous purposes. Computer programs can be stored or distributed on suitable media, such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A method for minimizing motion-induced errors in an ambient light (14) corrected image, the method comprising: Capture multiple master images with controlled illumination intensity that varies over time (30); Estimate the motion-induced error value (38) for each subset of the plurality of main images (30); and Ambient light (14) correction is performed on the error-minimizing subset of the main image with the lowest estimated error value (38) to generate an ambient light (14) corrected (ALC) image with minimized error; The estimation of the error value (38) for each subset of the plurality of main images includes: Identify a pair of main images (30) in the subset, the pair of main images (30) including a first main image (30a) and a second main image (30b), wherein the first main image (30a) was captured in time before the second main image (30b); Determining the displacement vector matrix (36) between the pair of main images (30) includes: The first main image (30a) is divided into a plurality of first grid elements (46), and the second main image (30b) is divided into a plurality of second grid elements (46). Match the first grid element (46) with the second grid element (46) at the same position on the object (20), and Determine the displacement vector (48) between each matched first mesh element (46) and second mesh element (46) to determine the displacement vector matrix (36) describing the displacement vector (48) of each mesh element (46); and Based on the displacement vector matrix (36) between the pair of main images (30), the error value (38) for a subset of the main images (30) is estimated.

2. The method of claim 1, wherein estimating the error value (38) for the subset of the main image (30) comprises: The maximum displacement vector (48) is selected in the displacement vector matrix (36) for the pair of main images (30), and the error value (38) is estimated based on the maximum displacement vector (48).

3. The method according to any one of claims 1 to 2, wherein estimating the error value (38) for each subset of the main image (30) comprises: Identify multiple pairs of main images (30) in the subset; Determine the displacement vector matrix (36) for each pair of main images (30); as well as Based on the displacement vector matrix (36) for each pair of main images (30) in the subset, the error value (38) for the subset of the plurality of main images (30) is estimated.

4. The method of claim 2, wherein estimating the error value (38) for the subset of the plurality of main images (30) comprises: The highest absolute displacement vector (48) is selected from the maximum displacement vectors (48) for each pair of main images (30), and the error value (38) is estimated based on the highest absolute displacement vectors (48) in a subset of the main images (30).

5. The method of claim 3, wherein determining the displacement vector (48) between temporally adjacent pairs of master images (30) comprises: Matching the first grid element (46) and the second grid element (46), wherein determining the displacement vector (48) between temporally non-adjacent pairs of main images (30) comprises: adding the displacement vectors (48) of temporally adjacent pairs of main images (30) to map the motion trajectory (52) between the temporally non-adjacent pairs of main images (30).

6. The method according to any one of claims 1 to 2, wherein determining the displacement vector (48) between each matched first mesh element (46) and second mesh element (46) comprises: Displacement estimation is performed based on phase correlation.

7. The method according to any one of claims 1 to 2, wherein each subset of the main images (30) comprises a plurality of temporally adjacent main images (30), and wherein each subset of the main images (30) is offset from another subset of the main images (30) by a temporally adjacent main image (30).

8. The method of claim 7, wherein estimation of the error value (38) due to motion is performed in real time for each subset of a plurality of subsets of a temporally continuous main image (30), and each time the estimated error value (38) for a subset of the main image (30) is lower than any previously estimated error value (38), the method comprising: Ambient light (14) correction is performed on a subset of the corresponding main image (30) to generate an ALC image (42) with minimized error.

9. A non-transitory computer-readable storage medium comprising computer-readable instructions that, when executed by a processor (16), cause execution of the method according to any one of claims 1 to 8.

10. A computer program product comprising a computer program that, when read by a computer, causes execution of the method according to any one of claims 1 to 8.

11. An image capture device (10), comprising: At least one processor (16); At least one memory (18) includes computer-readable instructions; The at least one processor (16) is configured to read the computer-readable instructions and cause execution of the method according to any one of claims 1 to 8.

Images