Image processing device, image processing method, imaging device, program, and recording medium

Abstract

To solve the problem in which brightness of a synthesized area is not necessarily appropriate for a subject when depth composition is performed.SOLUTION: To solve the above mentioned problem, image processing according to the present application is performed by an image processing apparatus that comprises: composing means that performs composition for a plurality of images having different focal positions; calculating means that calculates a value associated with contrast from the plurality of images; generating means that generates a composition map from the value associated with the contrast; dividing means that divides an area on the basis of a subject, for the plurality of images; and correcting means that corrects gradation for each area divided by the dividing means.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging device that captures a plurality of images with different focus positions and performs depth stacking. [Background technology]

[0002] When capturing images of multiple subjects at significantly different distances from a digital camera or a subject that is long in the depth direction, the depth of field may be insufficient to focus on only a portion of the subject. To solve this problem, Patent Document 1 discloses a so-called depth stacking technique that captures multiple images with different focus positions, extracts only the in-focus areas from each image, and combines them into a single image to generate a composite image in which the entire captured area is in focus. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-216532 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the depth stacking technology described above, multiple images with different focus positions are combined to generate an image that is focused on the entire imaging area, but it does not determine whether the brightness of the combined area is appropriate for the subject, and it is not necessarily possible to make the brightness of the subject appropriate.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to control gain so as to make the brightness of a subject appropriate in focus stacking. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention includes a synthesis means for synthesizing a plurality of images having different focus positions, a calculation means for calculating a value related to contrast from the plurality of images, a generation means for generating a synthesis map from the value related to contrast, a division means for dividing the plurality of images into regions based on a subject, and a correction means for correcting gradation for each of the regions divided by the division means. The dividing means divides the image into a region of a main subject and a region other than the main subject. An image processing device comprising: of provide. Effect of the Invention

[0007] According to the configuration of the present invention, in an image synthesized using a plurality of images with different focus positions, it is possible to control the brightness of the synthesized area so as to be appropriate. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a hardware configuration of a digital camera as an image processing device according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart for explaining generation of a composite image in the first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining an area subjected to photometry in step S201 in the first embodiment. [Figure 4] 4A to 4C are diagrams for explaining a synthesis map corresponding to an in-focus area generated in the first embodiment. [Diagram 5] FIG. 2 is a diagram illustrating subject region division in the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining correction by gain in the first embodiment. [Figure 7] 13A to 13C are diagrams for explaining an imaging scene and correction in the second embodiment. [Figure 8] 10 is a flowchart illustrating image synthesis in the second embodiment. [Figure 9]13A to 13C are diagrams for explaining an imaging scene and correction in the third embodiment. [Figure 10] 13 is a flowchart illustrating image synthesis in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0010] (First embodiment) 1 is an example of a block diagram showing the structure of a digital camera as an image processing device according to this embodiment. Digital camera 100 can capture still images, record information on the focus position, calculate contrast values, and synthesize images. Furthermore, digital camera 100 can perform enlargement or reduction processing on captured and saved images or images input from the outside.

[0011] The control unit 101 is, for example, a signal processor such as a CPU or MPU, and controls each part of the digital camera 100 while reading a program previously stored in a ROM 105 described later. For example, as described later, the control unit 101 issues commands to an imaging unit 104 described later to start and end imaging. Alternatively, the control unit 101 issues commands for image processing to an image processing unit 107 described later based on a program stored in the ROM 105. Commands from a user are input to the digital camera 100 by an operation unit 110 described later, and reach each part of the digital camera 100 through the control unit 101.

[0012] The driving unit 102 is constituted by a motor and the like, and mechanically operates the optical system 103 described below under the command of the control unit 101. For example, based on the command of the control unit 101, the driving unit 102 moves the position of a focus lens included in the optical system 103 to adjust the focal length of the optical system 103.

[0013] The optical system 103 is composed of a zoom lens, a focus lens, an aperture, etc. The aperture is a mechanism for adjusting the amount of light that passes through. The focal position can be changed by changing the position of the lens.

[0014] The imaging unit 104 is a photoelectric conversion element that performs photoelectric conversion to convert an incident optical signal into an electrical signal. For example, a CCD sensor or a CMOS sensor can be applied to the imaging unit 104. The imaging unit 104 is provided with a video imaging mode, and can capture a plurality of images that are continuous in time as individual frames of a video. The imaging unit 104 can measure the luminance of a subject through the optical system 103. Instead of photometry in the imaging unit 104, an AE sensor or the like can be used.

[0015] The ROM 105 is a read-only non-volatile memory serving as a recording medium, and stores operation programs for each block of the digital camera 100 as well as parameters and the like required for the operation of each block.

[0016] The RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operation of each block included in the digital camera 100 .

[0017] The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, filtering, and synthesis processing on the image output from the imaging unit 104 or on image signal data recorded in the internal memory 109 described later. In addition, the image processing unit 107 performs compression processing on the image signal data captured by the imaging unit 104 in accordance with a standard such as JPEG.

[0018] Image processing unit 107 is configured as an application specific integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, control unit 101 may perform part or all of the functions of image processing unit 107 by processing according to a program read from ROM 105 by control unit 101. When control unit 101 performs all of the functions of image processing unit 107, there is no need to have image processing unit 107 as hardware.

[0019] The display unit 108 is a liquid crystal display or an organic EL display for displaying images temporarily stored in the RAM 106, images stored in the internal memory 109 described below, or a setting screen of the digital camera 100.

[0020] The built-in memory 109 is a place for recording images captured by the imaging unit 104, images processed by the image processing unit 107, and information on the focal position when capturing an image, etc. A memory card or the like may be used instead of the built-in memory.

[0021] The operation unit 110 is, for example, a button, switch, key, or mode dial attached to the digital camera 100, or a touch panel that also serves as the display unit 108. Commands from a user reach the control unit 101 via the operation unit 110.

[0022] Fig. 2 is a flowchart for explaining generation of a composite image in this embodiment. Fig. 2(a) shows the overall processing in this embodiment. In step S201, the image capturing unit 104 measures the photometry of a subject that has passed through the optical system 103. The photometry processing will be described later.

[0023] In step S202, the image capturing unit 104 captures and develops a plurality of images with different focus positions.

[0024] In step S203, the control unit 101 performs position alignment on the image captured in step S202 by the imaging unit 104. Note that the position alignment process will be described later with reference to FIG.

[0025] In step S204, the image processing unit 107 synthesizes the images after the position adjustment to generate a synthesized image. Note that the image synthesis process will be described later with reference to FIG.

[0026] The photometry process will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining the area to be photometrized in step S201 in this embodiment. FIG. 3(a) shows the photometry area when a main subject is present, where area 301 shows the photometry area for the main subject, and area 302 shows the photometry area excluding the photometry area for the main subject. FIG. 3(b) shows the photometry area when a main subject is not present, where area 311 shows the photometry area for the entire surface not including the main subject. In step S205 in the flowchart shown in FIG. 2(b), the control unit 101 judges whether or not a main subject is present. If a main subject is present as shown in FIG. 3(a), the process proceeds to step S206, and if a main subject is not present as shown in FIG. 3(b), the process proceeds to step S208. In step S206, the area of ​​the main subject shown in area 301 is photometrized. In step S207, the area excluding the main subject shown in 302 is photometrized. In this case, the imaging unit 104 captures an image based on the photometry result of the main subject. In step S208, the entire area shown in area 311 is metered. The main subject photometry area 301 is used to determine the main subject area in the image synthesis process described below. The luminance values ​​obtained from the photometry results of the main subject and the area excluding the main subject are used to calculate a gain for correcting the synthesis area in the image synthesis process described below.

[0027] The alignment process will be described with reference to Fig. 2(c). In step S208, the control unit 101 acquires a reference image for alignment from among the images captured by the imaging unit 104 in step S202. The reference image for alignment may be, for example, the image captured earliest. Alternatively, since capturing images while changing the focus position causes a slight change in the angle of view between captured images, the image with the narrowest angle of view may be used.

[0028] In step S209, the control unit 101 acquires a target image for alignment processing. The target image is an image other than the reference image acquired in step S208, and is not yet subjected to alignment processing. If the image captured earliest in the image sequence is used as the reference image, the control unit 101 may acquire the target images in the order in which they were captured.

[0029] In step S210, the control unit 101 calculates the amount of positional deviation between the reference image and the target image. An example of the calculation method is described below. First, the control unit 101 sets a plurality of blocks in the reference image. It is preferable that the control unit 101 sets each block to have the same size. Next, the control unit 101 sets a search range in the target image at the same position as each block in the reference image, which is wider than the block in the reference image. Finally, the control unit 101 calculates a corresponding point in each search range of the target image that minimizes the sum of absolute difference (Sum of Absolute Difference, hereinafter referred to as SAD) of brightness with the block in the reference image. The control unit 101 calculates the positional deviation referred to in step S210 as a vector from the center of the block in the reference image and the corresponding point described above. In calculating the corresponding point described above, the control unit 101 may use the sum of squared difference (SSD, hereinafter referred to as SSD) or normalized cross correlation (NCC, hereinafter referred to as NCC) in addition to SAD.

[0030] In step S211, the control unit 101 calculates a conversion coefficient from the amount of positional deviation between the reference image and the target image. The control unit 101 uses, for example, a projection transformation coefficient as the conversion coefficient. However, the conversion coefficient is not limited to the projection transformation coefficient, and an affine transformation coefficient or a simplified conversion coefficient including only a horizontal and vertical shift may be used.

[0031] In step S212, the image processing unit 107 performs conversion on the target image using the conversion coefficients calculated in step S211.

[0032] For example, the control unit 101 can perform the transformation using the formula shown in formula (1).

number

[0033] In formula (1), (x', y') indicates the coordinates after transformation, and (x, y) indicates the coordinates before transformation. 1 The control unit 101 calculates the change exchange The coefficients are shown.

[0034] The image synthesis process will be described with reference to Fig. 2(d). In step S213, the image processing unit 107 calculates a contrast value for each image (including the reference image) after alignment. As an example of a method for calculating the contrast value, the image processing unit 107 first calculates a luminance Y from the color signals Sr, Sg, and Sb of each pixel using the following formula (2). Y=0.299Sr+0.587Sg+0.114Sb...Formula (2)

[0035] Next, a contrast value I is calculated for a matrix L of luminance Y of 3×3 pixels using a Sobel filter as shown in the following equations (3) to (5).

number

number

number

[0036] Furthermore, the above-mentioned method of calculating the contrast value is merely an example, and for example, it is also possible to use an edge detection filter such as a Laplacian filter or a bandpass filter that passes through a predetermined band.

[0037] In step S214, the image processing unit 107 generates a composite map by comparing the contrast values ​​of pixels at the same position in each image and calculating a composite ratio according to the magnitude of the contrast value.

[0038] In step S215, image processing unit 107 determines whether each region of the image is a main subject region and sets a target brightness value. In step S215, region division into the main subject and background subjects based on the synthesis map is also performed, which will be described below with reference to the drawings.

[0039] Fig. 4 is a diagram for explaining a composite map corresponding to an in-focus area generated in step S214 in this embodiment. Fig. 4(a), Fig. 4(b), and Fig. 4(c) show three images captured with the focus shifted and composite maps generated from each of the three images. Fig. 5 is a diagram for explaining the subject area division in step S215 in this embodiment.

[0040] In step S215, the control unit 101 judges whether the composite map 402, 412, 422 generated from the focused area 401, 411, 421 is the main subject area depending on whether it corresponds to the photometry area 301 of the main subject. Then, the image processing unit 107 sets the luminance value obtained by photometry of the main subject area and the target luminance value of the main subject area depending on whether it corresponds to the main subject area. If the composite map corresponds to the photometry area of ​​the main subject as shown in FIG. 5(a), the image processing unit 107 sets the luminance value of the main subject area obtained by photometry of the main subject area and the target luminance value of the main subject area in step S206. If the composite map does not correspond to the photometry area of ​​the main subject as shown in FIG. 5(b) and FIG. 5(c), the image processing unit 107 sets the luminance value obtained by photometry of the area excluding the main subject area and the target luminance value for the area excluding the photometry area of ​​the main subject in step S207. The target luminance value is a luminance value that makes the gradation of the photometry area of ​​the main subject and the area excluding the photometry area of ​​the main subject appropriate brightness. In step S216, the image processing unit 107 calculates a gain for adjusting the luminance value set in step S215 to the target luminance value.

[0041] In step S217, the image processing unit 107 corrects the gradation of the area to be synthesized based on the synthesis map generated in step S214, using the gain calculated in step S216.

[0042] FIG. 6 is a diagram for explaining the correction by gain in this embodiment.

[0043] In the case of Fig. 6(a) where the composite map corresponds to the photometric region of the main subject, the region 601 composited based on the composite map 602 can be corrected by a gain that brings the main subject to an appropriate brightness. In the case of Fig. 6(b) where the composite map does not correspond to the photometric region of the main subject, the regions 611 and 621 composited based on the composite maps 612 and 622 can be corrected by a gain that brings the regions excluding the main subject region to an appropriate brightness.

[0044] In step S218, the image processing unit 107 combines the regions 603, 613, and 623 corrected in step S217 in accordance with the combination maps 602, 612, and 622.

[0045] In the above explanation, it has been explained that the control unit 101 divides the areas of the main subject and the background subject based on the composite map, but this is not limited thereto, and the control unit 101 may divide the areas of the main subject and the background subject based on a contrast value or the like.

[0046] According to the first embodiment, when generating a depth stacking image, it is possible to improve the gradation so that each synthesized region has appropriate brightness.

[0047] Second embodiment The second embodiment will be described below with reference to the drawings. The second embodiment differs from the first embodiment in that correction may be performed with different gains even within the same subject region. The second embodiment will be described below with a focus on the differences from the first embodiment.

[0048] FIG. 7 is a diagram for explaining an imaging scene and correction in this embodiment.

[0049] Photometry, imaging, development, and alignment in this embodiment are the same as those in the first embodiment. Image synthesis in this embodiment is different from that in the first embodiment. FIG. 8 is a flowchart for explaining image synthesis in this embodiment. Step S816 in the flowchart shown in FIG. 8(a) in this embodiment is different from step S216 in the first embodiment. This will be described in detail below.

[0050] As shown in Fig. 7(a), when the distance between a main subject 701 and a background subject 702 is short, in step S214, in addition to the main subject, a part of the background subject may be generated as a composite map 711 as shown in Fig. 7(b). In this case, the gradation of the same subject 702 is corrected with a gain calculated from a different target luminance value, resulting in a luminance difference 721 between the captured images 701 and 702 as shown in Fig. 7(c). Therefore, in this embodiment, unlike the first embodiment, the gain calculation is switched depending on the conditions when the gain is calculated in step S816, thereby suppressing the luminance difference within the same subject.

[0051] FIG. 8B is a flow chart for explaining the gain calculation in this embodiment. In step S801, the control unit 101 judges whether or not there is an area used for synthesis in the image immediately before the image being processed. If there is an area used for synthesis in the image immediately before the image being processed, it is judged that there is a possibility that a brightness difference will occur within the same object if a gain in which the target area has a target brightness value is calculated as in the first embodiment. In this case, the process proceeds to step S802, and the image processing unit 107 calculates the gain by controlling the correction amount according to the gain at which the gradation was last corrected. This makes it possible to suppress a steep brightness difference within the same object, such as the area 731, in the image shown in FIG. 7D, which is a synthesis of the first image captured and the second image captured in the imaging scene shown in FIG. 7A. Conversely, if there is no area used for synthesis in the image immediately before the image being processed, the process proceeds to step S803, and the image processing unit 107 performs processing similar to the first embodiment. This eliminates the need to control the amount of correction according to the gain used to correct the gradation last for an image without a focused subject, such as the third captured image 741 in the imaging scene shown in Fig. 7(a). In addition, for the fourth captured image 742, the control unit 101 calculates a gain that corrects the luminance to the target luminance value, since the third image 741 was not used in the synthesis.

[0052] According to this embodiment, it is possible to switch between gain calculation means so as to prevent abrupt luminance differences caused by different gains within the same subject region.

[0053] (Third embodiment) The third embodiment will be described below with reference to the drawings. The third embodiment differs from the first and second embodiments in that it describes processing when there are multiple subjects focused on the same image. The third embodiment will be described below, focusing on the differences from the first embodiment.

[0054] FIG. 9 is a diagram for explaining an imaging scene and correction in this embodiment.

[0055] The photometry, image capture, development, and alignment in this embodiment are the same as those in the first embodiment. The image synthesis in this embodiment is different from that in the first embodiment. Fig. 10 is a flowchart for explaining the image synthesis in this embodiment.

[0056] In an image capture such as that shown in FIG. 9(a), a main subject 901 and a subject 902 other than the main subject are present at a close distance. In the case of FIG. 9(a), in the method described in the first embodiment, the main subject and the subject other than the main subject may be generated as in a composite map 911 in step S214 as shown in FIG. 9(b). In this case, in step S217, the area of ​​the subject other than the main subject is corrected by a gain for making the main subject area a target brightness. If a condition is added before step S215 and an area other than the main subject area exists in the composite map, a composite map in which the area other than the main subject is erased and a composite map in which the area corresponding to the main subject is erased are generated, and the respective areas are corrected by the gain.

[0057] In steps S1001 and S1002 in this embodiment, the same processes as those in steps S213 and S214 described in the first embodiment are performed.

[0058] In step S1003, the control unit 101 determines whether or not both the main subject and an area other than the main subject are present in the composite map generated in step S1002. For example, a method for determining whether both the main subject and an area other than the main subject are present may be object recognition using a convolutional neural network (CNN). FIG. 9(c) shows a case where both the main subject 922 and an area 923 other than the main subject are present. In a case like FIG. 9(c), the process proceeds to step S1004, and as shown in FIG. 9(d), a composite map 931 is generated from the composite map 911 generated in step S1002 by suppressing the area other than the composite area corresponding to the main subject area. Next, in step S1005, the image processing unit 107 corrects the gradation of the area corresponding to the composite map generated in step S1004.

[0059] In step S1006, a composite map 932 is generated by collapsing the composite area corresponding to the main subject area from the composite map generated in step S1002. In step S1007, the image processing unit 107 corrects the gradation of the area corresponding to the composite map generated in step S1006.

[0060] Also, in step S1003, if the control unit 101 determines that both the main subject and an area other than the main subject are not present, the process proceeds to step S1008, where the control unit 101 corrects the gradation of the area corresponding to the composite map generated in step S1002.

[0061] In step S1009, the image processing unit 107 synthesizes the regions with the gradation corrected in accordance with the synthesis map generated in step S1002, similarly to step S218 in the first embodiment.

[0062] The following describes the regional tone correction in steps S1005, S1007, and S1008. Fig. 10B is a flowchart for explaining the regional tone correction in this embodiment.

[0063] In step S1010, the control unit 101 determines whether the composite map to be processed corresponds to the main subject region. If it corresponds, the process proceeds to step S1011, and if it does not correspond, the process proceeds to step S1013.

[0064] In step S1011, image processing unit 107 sets a luminance value measured in the main subject region. Next, in step S1012, image processing unit 107 sets a target luminance value for the main subject region. Meanwhile, in step S1013, image processing unit 107 sets a luminance value measured in a region other than the main subject region. Next, in step 1014, image processing unit 107 sets a target luminance value for a region other than the main subject region. The target luminance value here is similar to the first embodiment, and is a luminance value that makes the gradation of the photometric region of the main subject or the photometric region other than the main subject region appropriate brightness.

[0065] Next, in step S1015, the image processing unit 107 calculates a gain for making the set luminance value equal to the set target luminance value, similarly to step S216 in the first embodiment.

[0066] In step S1016, the image processing unit 107 corrects the gradation of the area corresponding to the composite map to be processed, using the gain calculated in step S1015.

[0067] According to the third embodiment, even if different subject areas exist in a composite map, the gradation can be corrected so that each subject area has an appropriate brightness.

[0068] (Other embodiments) In the above embodiment, the description has been given based on a personal digital camera, but as long as the camera is equipped with a depth stacking function, the present invention can also be applied to a mobile device, a smartphone, or a network camera connected to a server. Alternatively, part of the above-described processing may be performed by a mobile device, a smartphone, or a network camera connected to a server.

[0069] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and run the program. Also, the present invention can be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0070] 100 Digital Camera 101 Control section 102 Drive unit 103 Optical system 104 Imaging unit 105 ROM 106 RAM 107 Image Processing Unit 108 Display section 109 Internal Memory 110 Operation section

Claims

1. A synthesis means for synthesizing a plurality of images having different focus positions; A calculation means for calculating a value related to contrast from the plurality of images; a generating means for generating a composite map from the contrast-related values; A dividing means for dividing the plurality of images into regions based on a subject; a correction means for correcting a gradation for each of the regions divided by the dividing means, The image processing device according to claim 1, wherein the dividing means divides the image into an area of ​​a main subject and an area other than the main subject.

2. 2. The image processing apparatus according to claim 1, wherein the correction means corrects the gradation by applying a gain.

3. 3. The image processing apparatus according to claim 2, wherein the correction means obtains a luminance for each of the regions, and obtains the gain based on the luminance.

4. 4. The image processing apparatus according to claim 3, wherein the correction means corrects the luminance for each of the regions to be appropriate.

5. 5. The image processing apparatus according to claim 1, wherein the dividing means divides the area based on the synthesis map.

6. The image processing device according to claim 5, characterized in that when a plurality of the subjects are present in a composite map corresponding to one of the plurality of images generated by the generating means, the dividing means divides each of the plurality of subjects into different regions.

7. 7. The image processing device according to claim 1, wherein, when correcting a first image of the plurality of images, the correction means changes the amount of correction used for the first image based on whether a second image adjacent to the first image has been used in the synthesis.

8. An imaging means for capturing a plurality of images with different focus positions; A synthesis means for synthesizing the plurality of images; A calculation means for calculating a value related to contrast from the plurality of images; a generating means for generating a composite map from the contrast-related values; A dividing means for dividing the plurality of images into regions based on a subject; a correction means for correcting a gradation for each of the regions divided by the dividing means, the dividing means divides the area into a main subject area and an area other than the main subject area,

9. A synthesis step of synthesizing a plurality of images having different focus positions; A calculation step of calculating a value related to contrast from the plurality of images; generating a composite map from the contrast related values; A division step of dividing the plurality of images into regions based on a subject; a correction step of correcting a gradation for each of the regions divided in the division step, 4. An image processing method comprising the steps of: dividing an area into a main subject area and an area other than the main subject area in said dividing step.

10. A computer program for causing a computer of an image processing device to operate, A synthesis step of synthesizing a plurality of images having different focus positions; A calculation step of calculating a value related to contrast from the plurality of images; generating a composite map from the contrast related values; A division step of dividing the plurality of images into regions based on a subject; a correction step of correcting a gradation for each of the regions divided in the division step; The program, wherein in the dividing step, the area is divided into an area of ​​a main subject and an area other than the main subject.

11. A computer-readable recording medium having the program according to claim 10 recorded thereon.

Images