Image processing device, method, program, and storage medium

Abstract

To provide an image processing device capable of suppressing an occurrence of halo near a boundary between high brightness and low brightness in the case of correcting image brightness.SOLUTION: An image processing device includes a generation unit for generating a first image and a second image being images having different spatial frequency bands from an input image, a calculation unit for calculating gains each corresponding to the first and second images on the basis of the first and second images, and a synthesis unit for synthesizing the gains each corresponding to the first and second images to generate a final gain to be applied to the input image. The synthesis unit synthesizes the gain corresponding to the first image and the gain corresponding to the second image on the basis of difference between the first gain and the second gain when a gain of a target pixel of the first image whose spatial frequency is a first band is defined as the first gain, and a gain having the minimum difference from the first gain among gains of pixels being within a prescribed range from a target pixel of the second image being a second band whose spatial frequency is lower than in the first band is defined as the second gain.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device that performs gradation processing. [Background technology]

[0002] Conventionally, gradation processing has been considered that controls the brightness of a dark subject, such as a backlit scene, to an appropriate level by performing a dodging process on the input image. In this process, gain processing is performed to brighten a dark subject to an appropriate level of brightness, but there is an issue that the contrast of the processed image is significantly impaired.

[0003] Therefore, a technology has been proposed to improve the reduction in contrast by performing gain processing using an image with a low spatial frequency of the input image (hereinafter referred to as a low-frequency image) to reduce sensitivity to local changes in brightness. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-154108 A [Patent Document 2] JP 2008-165312 A Summary of the Invention [Problem to be solved by the invention]

[0005] The dodging process using a low-frequency image has the problem that halos occur near the boundary between high and low brightness. This is because the use of a low-frequency image causes gain to be applied to areas that do not actually require gain, resulting in overshoot and undershoot.

[0006] As a measure to improve such a problem, Patent Document 1 proposes a technique for adding pixels in a predetermined range of the low-frequency image based on the result of adding pixels of interest when synthesizing an image with a high spatial frequency (hereinafter referred to as a high-frequency image) with a low-frequency image, and synthesizing the high-frequency image. However, when the difference between the pixels of interest is large, a pixel of the low-frequency image that is significantly different from the pixel of interest of the high-frequency image is synthesized, and it is not possible to suppress the occurrence of halos near the boundary.

[0007] Patent Document 2 proposes a technology for hierarchical synthesis, which calculates a synthesis coefficient based on the edge strength of the image. However, although only a high-frequency image can be output for a portion with high edge strength, synthesis is performed even if the difference between the pixel of interest of the low-frequency image and the high-frequency image is large near the boundary, so the generation of halos cannot be completely suppressed.

[0008] The present invention has been made in consideration of the above-mentioned problems, and its object is to provide an image processing device that can suppress the occurrence of halos near the boundary between high brightness and low brightness when correcting the brightness of an image. [Means for solving the problem]

[0009] The image processing device of the present invention comprises a generation means for generating a first image and a second image, the first image and the second image being images with different spatial frequency bands, from an input image; a calculation means for calculating gains corresponding to the first and second images based on the first and second images; and a synthesis means for synthesizing the gains corresponding to the first and second images to generate a final gain to be applied to the input image, wherein the synthesis means synthesizes the gain corresponding to the first image and the gain corresponding to the second image based on the difference between the first gain and the second gain, when a gain of a pixel of interest in the first image having a spatial frequency in a first band is defined as a first gain, and a gain of a pixel within a predetermined range of a pixel of interest in the second image having a spatial frequency in a second band lower than the first band is defined as a second gain, the gain having the smallest difference from the first gain. Effect of the Invention

[0010] According to the present invention, when correcting the brightness of an image, it is possible to suppress the occurrence of halos near the boundary between high brightness and low brightness. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an imaging apparatus which is an embodiment of an image processing apparatus of the present invention. [Diagram 2] FIG. 4 is a block diagram showing the configuration of an image processing unit. [Diagram 3] 4 is a flowchart showing the operation of the imaging apparatus. [Figure 4] FIG. [Diagram 5] 13A and 13B are diagrams for explaining a weight calculation method using a difference from a minimum differential gain. [Figure 6] FIG. 13 is a diagram for explaining a weight calculation method using distance. [Figure 7] 13A and 13B are diagrams for explaining a weight calculation method using the magnitude of the differential minimum gain. [Figure 8] FIG. 13 is a diagram for explaining a weight calculation method using additional information. [Figure 9] FIG. 13 is a block diagram showing a modified example of the image processing unit. [Figure 10] FIG. 13 is a block diagram showing a modified example of the image processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] FIG. 1 is a block diagram showing an example of the configuration of an image capturing apparatus 100 which is an embodiment of an image processing apparatus according to the present invention.

[0014] 1, a control unit 101 is, for example, a CPU, which reads out an operation program for each block included in the imaging device 100 from a ROM 102 described below, expands the program into a RAM 103 described below, and executes the program to control the operation of each block included in the imaging device 100. The ROM 102 is an electrically erasable and recordable non-volatile memory, and stores parameters and the like required for the operation of each block in addition to the operation program for each block included in the imaging device 100. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the imaging device 100.

[0015] The optical system 104 is composed of a group of lenses including a zoom lens and a focus lens, and forms an image of a subject on the imaging unit 105 described below. The imaging unit 105 includes an imaging element 105a, for example, a CCD or CMOS sensor, and photoelectrically converts the optical image formed by the optical system 104, and outputs the obtained analog image signal to the A / D conversion unit 106. In this embodiment, the imaging element 105a includes a Bayer array color filter of primary colors RGB. The A / D conversion unit 106 converts the input analog image signal into digital image data, and outputs the obtained digital image data to the RAM 103.

[0016] The image processing unit 107 applies various image processing such as noise reduction processing, gradation processing, and development processing to the image data stored in the RAM 103 .

[0017] The recording unit 108 is composed of a removable memory card or the like, and the image processed by the image processing unit 107 is recorded as a recorded image via the RAM 103 .

[0018] The display unit 109 is a display device such as an LCD, and displays images stored in the RAM 103 and the storage unit 108, and an operation user interface for receiving instructions from the user.

[0019] The configuration and basic operation of the imaging device 100 have been described above.

[0020] 2 is a block diagram showing the configuration of the image processing unit 107. The image processing unit 107 in this embodiment includes an input image acquisition unit 201, a hierarchical image generation unit 202, a gain calculation unit 203, a gain synthesis unit 204, a gain processing unit 205, and a signal processing unit 206.

[0021] In this embodiment, gain synthesis is performed taking into consideration the minimum value of the differences between hierarchical images with different spatial frequencies, and gradation processing is performed in which the synthesized gain is applied to the input image.

[0022] First, the input image acquisition unit 201 acquires an image captured by the imaging device 100 as an input image.

[0023] Next, the hierarchical image generating unit 202 performs band limiting processing on the input image to generate a plurality of images with different spatial frequencies.

[0024] The gain calculation unit 203 calculates, from the image output from the hierarchical image generation unit 202, a gain to be used in gradation processing of the input image.

[0025] The gain synthesis unit 204 synthesizes the gains generated from images with different spatial frequencies to generate a gain that is ultimately used in gradation processing.

[0026] The gain processing unit 205 applies the gain acquired from the gain synthesis unit 204 to the image acquired by the input image acquisition unit 201 to perform gradation processing.

[0027] The signal processing unit 206 performs signal processing such as development processing based on the signal that has been subjected to gain processing by the gain processing unit 205, and generates an output image.

[0028] The above operations are shown in the flowchart of Fig. 3 according to the process flow, and the details of the process will be described with reference to Fig. 4 to Fig. 7. The operations of the flowchart of Fig. 3 are realized by the control unit 101 expanding an operation program stored in ROM 102 into RAM 103 and executing it.

[0029] First, in step S301, the control unit 101 uses the input image acquisition unit 201 to acquire an image to be subjected to gradation processing.

[0030] In step S302, the control unit 101 generates multiple hierarchical images from the input image using the hierarchical image generation unit 202. Specifically, multiple images with different spatial frequencies are generated by performing band limitation on the input image. In generating the multiple images, a first image and a second image, band limitations of different spatial frequencies are performed, but the first image has a higher spatial frequency band than the second image.

[0031] The method of band limiting processing here is not particularly limited, but one possible example is a method of using a low-pass filter having desired frequency characteristics. In addition, in this embodiment, a process of generating two images with different spatial frequencies and synthesizing the gains will be described, but the number of images with different spatial frequencies is not limited to two, and may be three or more.

[0032] Next, in step S303, the control unit 101 uses the gain calculation unit 203 to calculate a gain for each pixel from the luminance signal of each layer image.

[0033] The gain calculation process will be described with reference to Fig. 4. Fig. 4 shows a first image 401, a second image 402, a graph 403 showing the tone characteristics used in the tone conversion process, and a calculated gain 404.

[0034] First, based on the pixel signals of the first image 401 and the second image 402, the input luminance Y of each pixel is calculated for each of the first image 401 and the second image 402 using the following equation (1).

[0035] Y=0.299×R+0.587×G+0.114×B…(1) Next, using the input luminance Y and gradation characteristics calculated for each pixel, the gain of each pixel is calculated for each of the first image 401 and the second image 402. The gradation characteristics used in this gain calculation process refer to a gain table, as shown in graph 403, with the input luminance signal on the horizontal axis and the gain signal on the vertical axis.

[0036] In the gain calculation process, a graph 403 is used to calculate a gain value for each pixel corresponding to the input luminance Y for each pixel, and as a result, a gain signal such as 404 corresponding to each pixel position is generated. Then, the gain signals calculated for each of the first image 401 and the second image 402 are output to the gain synthesis unit 204. Note that, in this embodiment, equation (1) is used to calculate the input luminance Y, but this is not limiting and other equations may be used.

[0037] In step S304, the control unit 101 calculates a gain signal of the second image that has a minimum difference from the gain signal of the first image, between the gain signal of the first image and the gain signal of the second image.

[0038] The process of calculating the minimum value of the gain signal difference will be described with reference to Fig. 5. Fig. 5(a) shows the images and pixel positions used in calculating the minimum value of the gain signal difference, and Fig. 5(b) shows a graph relating to the weighting using the gain signal difference.

[0039] To calculate the minimum difference between gain signals in two images, first calculate the difference between the gain at a position of interest (position of a pixel of interest) 501 in the first image and the gain (for example, pixel position 504 or pixel position 505) in a predetermined range 503 (within a predetermined range) centered on a position of interest (position of a pixel of interest) 502 in the second image. Then, the calculated differences are compared, and the gain signal at the position where the difference is smallest is recorded. Hereinafter, the gain signal at the position where the difference in gain values ​​is smallest is referred to as the minimum difference gain value.

[0040] More specifically, to calculate the smallest difference value, a difference value Min1 between the gain at the focus position 501 in the first image and the gain at the focus position 502 in the second image is compared with a difference value Min2 between the gain at the focus position 501 in the first image and the gain at the pixel position 504 in the second image, and a difference value Min3 between the gain at the focus position 501 in the first image and the gain at the pixel position 505 in the second image. From among these, the gain with the smallest gain difference value (the minimum difference gain value defined above) and its pixel position are determined for the second image. In this embodiment, the subsequent processing will be described for the case where the minimum value of the gain difference is Min2.

[0041] Note that, in this example, two gains at pixel positions 504 and 505 are used as gains in the predetermined range 503, but in reality, the calculation is performed using gains at other pixel positions in the range 503 as well.

[0042] In step S305, the control unit 101 determines the weight Wm of each gain in the predetermined range 503 of the second image based on the gain at the position where the gain difference determined in step S304 is minimum (the gain with the minimum difference).

[0043] The difference between the minimum differential gain calculated in step S304 and the gain of a predetermined range of the second image is calculated, and the weight is determined using graph 510 in Fig. 5(b). Graph 510 is a graph in which the horizontal axis indicates the difference value from the minimum differential gain, and the vertical axis indicates the weight.

[0044] Specifically, the difference between the gain at pixel position 504, which is the minimum differential gain, and the gain at focus position 502, and the difference between the gain at pixel position 504 and the gain at pixel position 505 are calculated. As a result, if the gain has a value close to the minimum differential gain, the weight of the weighted addition is increased, whereas if the gain has a value far from the minimum differential gain, the weight of the weighted addition is decreased. This makes it possible to increase the weight of the gain close to the gain of the first image for synthesis.

[0045] In step S306, the control unit 101 calculates a weight Wd using the distance of each gain for the gains in a predetermined range centered on the position of interest in the second image.

[0046] The process of calculating the weight using the distance will be described with reference to Fig. 6. Fig. 6(a) shows each image and pixel position used in calculating the weight using the distance, and Fig. 6(b) shows a graph related to the weight.

[0047] In calculating the weight using the distance, first, the distance between the gain at the position of interest 601 in the second image in Fig. 6(a) and the gain in a predetermined range 602 centered on the gain at the position of interest 601 (for example, pixel position 603 or pixel position 604) is calculated. Then, based on the calculated distance, the weight is determined using graph 610 in Fig. 6. Graph 610 is a graph in which the horizontal axis indicates the distance from the position of interest and the vertical axis indicates the weight.

[0048] Specifically, first, a distance D1 between the focus position 601 and pixel position 603, and a distance D2 between the focus position 601 and pixel position 604 are calculated using equation (2). Here, D1 and D2 are shown as DN (N=1 or 2). Here, N is 1 or 2, but in reality, calculations are performed for all pixel positions present in a predetermined range 602.

[0049] DN=√{(xc-xs) 2 +(yc-ys) 2} …(2) Where: xc, yc: coordinates of the point of interest xs, ys: Coordinates of pixel positions within a given range As a result of calculating the distances D1 and D2, the weight of the gain for the position close to the focus position is increased, while the weight of the gain for the position far from the focus position is decreased. This makes it possible to combine the gains by increasing the weight of the gain for the position close to the first image.

[0050] In step S307, the control unit 101 performs weighted addition of the gains in a predetermined range 602 of the second image including the focus position, using a weight Wm using the difference between the differential minimum gains calculated in steps S306 and S307, and a weight Wd using the distance. The weighted addition process is performed using the following formula (3).

[0051] GAINL(x, y)=Σgain(x, y)×Wm(x, y)×Wd(x, y) / ΣWm(x, y)×Wd(x, y) …(3) Where: GAINL(x,y): The weighted sum of the gains in the second image gain(x,y): Gain in a given range of the second image at coordinates (x,y). Wm(x,y): Weight using the difference between the minimum gain and the difference at coordinate (x,y) Wd(x,y): Weight using distance at coordinates (x,y) The GAINL obtained by weighting and adding the gain of the second image calculated in step S307 is used in step S309 as the gain to be combined with the gain of the first image.

[0052] In step S308, the control unit 101 calculates a weight Wv using the magnitude of the differential minimum gain. The process of calculating the weight using the magnitude of the differential minimum gain will be described with reference to Fig. 7. Fig. 7 shows a graph relating to the weight using the magnitude of the differential minimum gain.

[0053] The weight using the magnitude of the minimum differential gain is determined by calculating the difference between the minimum differential gain in the second image (the gain at the position where the gain difference with the gain of interest in the first image is minimum) and the gain of interest in the first image, and then determining it from graph 701 in Fig. 7. Graph 701 shows the difference value with respect to the gain of interest on the horizontal axis, and the weight on the vertical axis.

[0054] Specifically, the difference value Min2 calculated in step S304 is used as the difference value. As a result, if the difference is small, the weight of the weighted addition is increased and the weight of the gain of the second image is increased to improve contrast. On the other hand, if the difference is large, the weight of the weighted addition is decreased and the weight of the gain of the first image is increased to prevent the generation of halos.

[0055] In step S309, the control unit 101 combines the gains of the first image and the second image based on the weight Wv calculated in step S308. The combination based on the weight is performed using the following formula (4).

[0056] Gain(x, y)=Wv(x, y)×GAINL(x, y)+(1-Wv(x, y))×GAINU(x, y) …(4) Where: GAINL(x,y): Gain of the weighted sum of the second image GAINU(x,y): Gain of the focus position in the first image Wv(x,y): weight using the magnitude of the difference minimum gain Gain(x,y): Gain of the focus position to be applied to the input image Then, the gain calculated by equation (4) is sent to step S310.

[0057] In step S310, the control unit 101 performs gradation processing on the input image using the gain calculated in step S309, using the gain processing unit 205. The gradation processing is performed using the following formula (5).

[0058] out(x, y)=Gain(x, y)×in(x, y) …(5) Where: out(x,y): Output signal value after gain at coordinate (x,y) in(x,y): Input signal value at coordinates (x,y) Gain(x,y): Gain of the focus position to be applied to the input image Then, the pixel signal after gain processing calculated by the equation (5) is output to the signal processing unit 206.

[0059] Finally, in step S311, the control unit 101 inputs the gain-processed input image signal to the signal processing unit 206, which performs signal processing such as development processing, and outputs the processed and developed result.

[0060] In the above embodiment, the gains are synthesized using the weights calculated from the hierarchical images, but additional information may also be used. Specifically, the weights are calculated using subject information and distance information detected from the input image. The process of calculating the weights using the additional information will be described with reference to FIG. 8. FIG. 8 is a diagram for explaining the calculation of the weights using the additional information.

[0061] For example, when a person is photographed as a subject as in the input image 800 of FIG. 8, the area of ​​the identified person can be represented as 801. In 801, the area of ​​the person is represented in white, and the other areas are represented in black. In the synthesis of the gains of the first image and the second image, if the gains of different subjects are included in a predetermined range of the second image, the difference with the first image becomes large by synthesizing the gains, and halos are likely to occur. Therefore, by knowing the boundaries of the subjects as in 801, it is possible to distinguish between different subjects. Therefore, based on the information in 801, by increasing the weight when the same subject is included in a predetermined range of the second image and decreasing the weight when the subjects are different, it is possible to synthesize only the gains corresponding to the same subject, and to suppress the occurrence of halos.

[0062] As another example, when a landscape such as the input image 810 in FIG. 8 is captured, distance information can be obtained using a distance sensor or the like as shown in 811. In 811, close objects are represented in white, and far objects are represented in black. In the synthesis of the gains of the first image and the second image, it is desired to distinguish between different objects in a predetermined range of the second image. Therefore, if distance information of the object is known as shown in 811, objects with different distances can be separated. Therefore, based on the information in 811, if there are objects at the same distance within a predetermined range of the second image, the weight is increased, and if there are objects at different distances, the weight is decreased. This makes it possible to synthesize only gains corresponding to the same distance, thereby suppressing the occurrence of halos.

[0063] In the above embodiment, a plurality of images with different spatial frequencies are generated as hierarchical images by performing band limiting processing on the input image, but hierarchical images may be generated by shrinking. Specifically, the input image is shrinked using an arbitrary shrink ratio to generate an image in which only low-frequency components remain. At this time, the shrink ratio of the second image is set smaller than that of the first image to generate the second image as an image with a lower frequency. In this embodiment, the shrink ratio is not particularly limited, but it is desirable to estimate it from the frequency of the fine texture portion to which a uniform gain is to be applied.

[0064] When reducing an image, the process of calculating the gain from the luminance signal of each layer image in step S303 is the same as above, so a description thereof will be omitted. Also, since the first image and the second image need to be the same size before combining, the second image is enlarged before the gain combining process.

[0065] For example, if a reduction process is performed by 1 / N times, the size can be matched by performing an enlargement process by N times. Then, the gain of the enlarged second image is synthesized with the gain of the first image. This synthesis process is similar to the process after step S304 above, and therefore a description thereof will be omitted. In addition, the method of reduction process and the method of enlargement process are not particularly limited.

[0066] In the present embodiment, a process has been described in which hierarchical images are generated from an input image, gain calculation is performed on the hierarchical images, and then gain synthesis is performed. However, the hierarchical images may be synthesized first, and gain calculation may be performed on the synthesis result.

[0067] 9 is a block diagram showing the configuration of image processing unit 107 that synthesizes input image signals. Specifically, a hierarchical image generating unit 902 generates hierarchical images with different spatial frequencies from an input image, and a signal synthesizing unit 903 performs weighted addition based on the image signals. Specifically, weighted synthesis is performed using image signals of a first image and a second image.

[0068] As the image signal used here, it is assumed that the input luminance Y is calculated using equation (1), but is not limited to this. The concept of weighted addition used here is the same as the process from step S304 onwards in this embodiment, so a description thereof will be omitted. Then, the gain used in the gradation process is calculated for the combined signal by the gain calculation unit 904, and the gain used in the gradation process is finally generated.

[0069] In the present embodiment, a process has been described in which hierarchical images are generated from an input image, gains are calculated for the hierarchical images, and then the gains are synthesized. However, gains may be calculated before the hierarchical images are generated, and the hierarchical images may be generated based on the results of the gain calculation.

[0070] FIG. 10 is a block diagram showing the configuration of the image processing unit 107 in the process of generating a hierarchical image from a gain. Specifically, a gain calculation unit 1002 calculates a gain to be used in gradation processing of an input image. The concept of gain calculation is the same as that of step S303, and therefore a description thereof will be omitted. Next, a hierarchical image generation unit 1003 generates a hierarchical image. Here, a band limiting process is performed on the calculated gain image to generate a plurality of gain images with different spatial frequencies. Then, a gain synthesis unit 1004 performs weighting synthesis using the hierarchical images of the generated gains, and the concept of weighting addition used here is the same as that of the process after step S304 in the above embodiment, and therefore a description thereof will be omitted. Then, gradation processing is performed using the synthesized gain.

[0071] The disclosure of this specification includes the following image processing device, image processing method, program, and storage medium.

[0072] (Item 1) a generating means for generating a first image and a second image, which are images having different spatial frequency bands, from an input image; a calculation means for calculating gains corresponding to the first and second images based on the first and second images; a combining means for combining the gains corresponding to the first and second images to generate a final gain to be applied to the input image; Equipped with the combining means combines a gain corresponding to the first image and a gain corresponding to the second image based on a difference between the first gain and the second gain, when a gain of a pixel of interest in a first image having a first band of spatial frequency is defined as a first gain, and a gain of a pixel within a predetermined range from the pixel of interest in a second image having a second band of spatial frequency lower than the first band, the gain having the smallest difference from the first gain, among gains of pixels within a predetermined range from the pixel of interest in a second image having a second band of spatial frequency lower than the first band.

[0073] (Item 2) 2. The image processing device according to item 1, wherein the synthesis means weights and adds the gains of pixels within the predetermined range based on a difference between the second gain and a gain of a pixel within the predetermined range of the second image.

[0074] (Item 3) 2. The image processing device according to item 1, wherein the synthesis means weights and adds gains of pixels within a predetermined range based on a distance between a pixel of interest in the second image and pixels within the predetermined range from the pixel of interest.

[0075] (Item 4) 4. The image processing device according to item 2 or 3, wherein the combining means combines a gain corresponding to the first image with a gain corresponding to the second image obtained by performing the weighted addition.

[0076] (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the combining means combines a gain corresponding to the first image and a gain corresponding to the second image based on subject information about the input image.

[0077] (Item 6) 5. The image processing device according to any one of items 1 to 4, wherein the combining means combines a gain corresponding to the first image and a gain corresponding to the second image based on distance information of a subject in the input image.

[0078] (Item 7) 2. The image processing device according to item 1, wherein the synthesizing means synthesizes the first image and the second image, and then calculates a gain for the synthesized image.

[0079] (Item 8) 2. The image processing device according to claim 1, further comprising a second calculation means for calculating a gain image representing a gain to be applied to an image signal of the input image based on the input image, wherein the generation means generates a first image and a second image, which are gain images having different spatial frequency bands, based on the gain image.

[0080] (Item 9) The image processing device according to item 1, characterized in that the generation means generates a first image and a second image by reducing the input image at different reduction rates, and the reduction rates of the first image and the second image are set so that the spatial frequency band of the first image is higher than the spatial frequency band of the second image.

[0081] (Item 10) A generating step of generating a first image and a second image, which are images having different spatial frequency bands, from an input image; a calculation step of calculating gains corresponding to the first and second images based on the first and second images; combining the gains corresponding to the first and second images to generate a final gain to be applied to the input image; having an image processing method characterized in that, in the synthesis step, a gain of a pixel of interest in a first image having a first band of spatial frequency is defined as a first gain, and a gain that has the smallest difference from the first gain among gains of pixels within a predetermined range from the pixel of interest in a second image having a second band of spatial frequency lower than the first band is defined as a second gain, and the gain corresponding to the first image and the gain corresponding to the second image are synthesized based on a difference between the first gain and the second gain.

[0082] (Item 11) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 9.

[0083] (Item 12) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 9.

[0084] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0085] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0086] 100: imaging device, 101: control unit, 102: ROM, 103: RAM, 104: optical system, 105: imaging unit, 106: A / D conversion unit, 107: image processing unit, 108: recording unit, 109: display unit

Claims

1. a generating means for generating a first image and a second image, which are images having different spatial frequency bands, from an input image; a calculation means for calculating gains corresponding to the first and second images based on the first and second images; a combining means for combining the gains corresponding to the first and second images to generate a final gain to be applied to the input image; Equipped with The image processing device is characterized in that the combining means combines a gain corresponding to the first image and a gain corresponding to the second image based on a difference between the first gain and the second gain, when a gain of a pixel of interest in a first image having a first band of spatial frequency is set as a first gain, and a gain of a pixel within a predetermined range of the pixel of interest in a second image having a second band of spatial frequency lower than the first band, the gain having the smallest difference from the first gain, among gains of pixels.

2. 2. The image processing device according to claim 1, wherein the combining means weights and adds the gains of pixels within the predetermined range based on a difference between the second gain and a gain of a pixel within the predetermined range of the second image.

3. 2. The image processing device according to claim 1, wherein the synthesizing means weights and adds gains of pixels within a predetermined range based on a distance between a pixel of interest in the second image and pixels within the predetermined range from the pixel of interest.

4. 4. The image processing apparatus according to claim 2, wherein the combining means combines a gain corresponding to the first image with a gain corresponding to the second image obtained by performing the weighted addition.

5. 2 . The image processing apparatus according to claim 1 , wherein the synthesizing means synthesizes a gain corresponding to the first image and a gain corresponding to the second image based on subject information about the input image.

6. 2 . The image processing apparatus according to claim 1 , wherein the synthesizing means synthesizes the gain corresponding to the first image and the gain corresponding to the second image based on distance information of a subject in the input image.

7. 2 . The image processing apparatus according to claim 1 , wherein the synthesizing means synthesizes the first image and the second image, and then calculates a gain for the synthesized image.

8. 2. The image processing device according to claim 1, further comprising a second calculation means for calculating a gain image representing a gain to be applied to an image signal of the input image based on the input image, wherein the generation means generates a first image and a second image, which are gain images having different spatial frequency bands, based on the gain image.

9. 2. The image processing device according to claim 1, wherein the generating means generates a first image and a second image by reducing the input image at different reduction rates, and the reduction rates of the first image and the second image are set so that the spatial frequency band of the first image is higher than the spatial frequency band of the second image.

10. a generating step of generating a first image and a second image, which are images having different spatial frequency bands, from an input image; a calculation step of calculating gains corresponding to the first and second images based on the first and second images; combining the gains corresponding to the first and second images to generate a final gain to be applied to the input image; having an image processing method characterized in that, in the synthesis step, a gain of a pixel of interest in a first image having a first band of spatial frequency is defined as a first gain, and a gain that has the smallest difference from the first gain among gains of pixels within a predetermined range from the pixel of interest in a second image having a second band of spatial frequency lower than the first band is defined as a second gain, and the gain corresponding to the first image and the gain corresponding to the second image are synthesized based on a difference between the first gain and the second gain.

11. A program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

12. 2. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

Images