Imaging device, control method and program thereof

Abstract

To more properly perform exposure correction based on a region detection result.SOLUTION: The luminance of an image captured by an imaging element 103 is detected, a likelihood which is a specific region is calculated from the image, a first region and a second region are determined on the basis of the luminance and the likelihood, and a correction amount for correcting an exposure value with respect to the imaging element 103 is calculated on the basis of the first region and the second region. The determination of the first region and the second region is made by setting a region in which the likelihood is equal to or more than a first threshold value and the luminance is equal to or more than a second threshold value as the first region and a region in which the likelihood is less than the first threshold value and the luminance is less than the second threshold value as the second region.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging apparatus, a control method thereof, and a program. [Background technology]

[0002] In digital cameras and smartphones that capture images using an image sensor such as a CMOS sensor, automatic exposure correction is generally performed so that the image captured by the image sensor has appropriate overall brightness. Exposure correction is performed, for example, by detecting the brightness of the image captured by the image sensor for each pixel or for each predetermined section (an area consisting of multiple pixels divided into sections of a certain size on the imaging surface of the image sensor) and adjusting the average brightness of all pixels to a constant value. In this case, if there is a high-brightness area, such as the sky on a clear day, in addition to the main subject, the average brightness is significantly affected by the high-brightness area, and the exposure value determined by exposure correction is biased toward the high-brightness side, resulting in a problem of the main subject appearing dark.

[0003] To address this problem, a technology has been disclosed in which the brightness Yup of the upper region (sky region) of an image and the brightness Ydown of the lower region (region other than the sky region) are calculated, and the larger the value of Yup / Ydown, the more the exposure correction amount that suppresses the influence of the upper region is calculated (see Patent Document 1).

[0004] However, the technology disclosed in Patent Document 1 is based on the premise that the sky is at the top of the image and that a low-brightness subject is at the bottom of the image, and therefore has the problem that appropriate exposure correction cannot be performed in shooting scenes that do not meet this premise.

[0005] Therefore, in recent years, a technology has been used in which a neural network that has been trained on an area to be detected (hereinafter referred to as a "detection target area") is used to detect the area from an optical image of a subject, and the amount of exposure compensation is calculated based on the detected area (see Patent Document 2). By using a neural network that has been trained on images that include detection target areas in various states, it is believed that it will be possible to accurately detect sky areas in various shooting scenes and perform appropriate exposure compensation. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-177779 [Patent Document 2] Japanese Patent Application Publication No. 2019-029833 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the accuracy of area detection using a neural network depends on the neural network's training data, and insufficient training can lead to detection errors. Furthermore, small-scale neural networks tend to have poor detection accuracy in photographic scenes where it is difficult to determine whether an area is a target area. For example, because there are bright and dark skies, when the target area is a sky, it can be difficult to distinguish between the dark sky and the main subject. Given these circumstances, a method is needed that can obtain a more appropriate exposure value when detecting a target area and performing exposure correction.

[0008] An object of the present invention is to provide an imaging device that can more appropriately perform exposure correction based on the detection result of the detection target area. [Means for solving the problem]

[0009] The imaging device according to the present invention includes a detection means for detecting the luminance of an image captured by an imaging element, a calculation means for calculating the likelihood of a specific region from the image, a determination means for determining a first region and a second region based on the luminance and the likelihood, and an exposure determination means for determining an exposure value based on the first region and the second region. bid price and a correction means for calculating a correction amount for correcting the likelihood, wherein the first region is a region where the likelihood is equal to or greater than a first threshold and the brightness is equal to or greater than a second threshold, and the second region is a region where the likelihood is less than the first threshold and the brightness is less than the second threshold. [Effects of the Invention]

[0010] According to the present invention, it is possible to more appropriately perform exposure correction based on the detection result of the detection target area. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating the configuration of functional blocks according to the first embodiment that perform exposure correction. [Figure 3] 10 is a flowchart of a process for determining an exposure correction amount. [Figure 4] 10 is a flowchart of the process of S304. [Figure 5] 10A and 10B are diagrams illustrating an example of a brightness histogram and a brightness threshold detected in S302. [Figure 6] FIG. 10 is a diagram showing an example of an image acquired in S301, its sky region (correct answer), and the region detection result in S303. [Figure 7] FIG. 10 is a diagram illustrating the configuration of functional blocks according to a second embodiment that performs exposure correction. [Figure 8] 10 is a flowchart of a process for determining a corrected exposure compensation amount. [Figure 9] 10 is a flowchart of the process of S801. [Figure 10] 10A and 10B are diagrams illustrating an example of a frame image and its brightness histogram. [Figure 11] FIG. 10 is a diagram showing an example of an image used for training a neural network. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Fig. 1 is a block diagram showing a schematic configuration of an imaging system 10 according to the embodiment. The imaging system 10 is composed of an imaging device 100 and a lens barrel 150 that is detachable from the imaging device 100.

[0013] The imaging device 100 is a so-called digital camera, and includes a system control unit 101, a memory 102, an imaging element 103, a shutter 104, an A / D conversion unit 105, an image processing unit 106, a D / A conversion unit 108, a display unit 109, and a TG 110. The imaging device 100 also includes a release button 111, an operation unit 112, a storage medium 119, a detection unit 113, and a photometry unit 114. The lens barrel 150 is a so-called interchangeable lens, and includes a lens control unit 151, a lens group 152, and an aperture 153.

[0014] In the imaging device 100, a system control unit 101 is a microcomputer including, for example, a CPU and memories such as ROM and RAM, and performs overall control of the operation of the imaging system 10. A shutter 104 controls exposure of the imaging element 103 in accordance with a control signal from the system control unit 101. The imaging element 103 is a charge-accumulating photoelectric conversion device such as a CMOS, and photoelectrically converts an optical image incident from the lens barrel 150 and formed on the imaging surface to generate an analog image signal, which is output to an A / D conversion unit 105. The A / D conversion unit 105 converts the analog image signal sent from the imaging element 103 into a digital image signal, and sends the converted digital image signal to a memory control unit 107 and an image processing unit.

[0015] The image processing unit 106 generates image data by performing pixel interpolation, resizing, color conversion, and correction of saturated pixels and crushed black pixels on the digital image signal transmitted from the A / D conversion unit 105. The image processing unit 106 also performs similar processing on image data transmitted from the memory control unit 107. The memory 102 temporarily stores various data including the digital image signal output from the A / D conversion unit 105 and image data that has been subjected to predetermined processing by the image processing unit 106. The D / A conversion unit 108 converts the image data read from the memory 102 into an analog image signal for display and transmits the signal to the display unit 109.

[0016] The display unit 109 has a display device such as a liquid crystal panel, and displays a menu screen and an image based on an analog image signal for display transmitted from the D / A conversion unit 108. A live view display can be performed by D / A converting image data obtained by performing predetermined image processing on the image processing unit 106 after output from the A / D conversion unit 105 in the D / A conversion unit 108 and displaying the image data on the display unit 109. The TG 110 transmits timing related to operations within the camera, such as drive timing and frame rate changes for the image sensor 103, and timing for exposing and shielding the image sensor 103 by the shutter 104, to each unit of the imaging device 100.

[0017] The release button 111 is configured as a two-stage switch that generates an SW1 signal when pressed halfway (halfway down) and generates an SW2 signal when the operation is completed (fully down). When the system control unit 101 receives the SW1 signal, it executes shooting preparation operations such as distance measurement calculation processing and photometry calculation processing, and when it receives the SW2 signal, it performs shooting operations. The operation unit 112 is an operation member (excluding the release button 111) that allows the user to input various operation instructions to the system control unit 101, and is configured as switches, buttons, dials, etc., and specifically includes a power switch, menu button, direction instruction buttons, etc. The operation unit 112 includes a touch panel that is configured integrally with the display unit 109.

[0018] The detection unit 113 detects specific subjects such as the sky or people from the captured image (an image output from the A / D conversion unit 105 and subjected to a predetermined development process by the image processing unit 106). The photometry unit 114 sets a photometry frame (photometry area) within the captured image and performs photometry calculations using the captured image. The detection unit 113 and photometry unit 114 may be provided integrally with the system control unit 101 or the image processing unit 106. The storage medium 119 is, for example, a memory card that is insertable into or built into the imaging device 100, and stores image data of captured images (still images, videos).

[0019] In the lens barrel 150, the lens group 152 is composed of multiple lenses including an optical axis shift lens, a zoom lens, a focus lens, etc. The aperture 153 adjusts the amount of light transmitted through the lens group 152. When the lens barrel 150 is attached to the imaging device 100, the lens control unit 151 and the system control unit 101 are capable of bidirectional communication via an interface. The lens control unit 151 transmits information related to the configuration and functions of the lens barrel 150 to the system control unit 101. The lens control unit 151 also comprehensively controls the operation of the lens barrel 150 in accordance with commands from the system control unit 101 and notifies the system control unit 101 of the control results. The lens control unit 151 has actuators that drive the lens group 152 and the aperture 153, and controls the driving of the lens group 152 and the aperture 153 in accordance with commands from the system control unit 101.

[0020] First Embodiment Next, functional blocks for performing exposure correction during image capture according to a first embodiment of the image capture system 10 will be described. Fig. 2 is a diagram illustrating the configuration of functional blocks for performing exposure correction during image capture according to the first embodiment. The functional block for performing exposure correction during image capture includes a brightness detection unit 201, an area detection unit 202, a determination unit 203, and a correction amount calculation unit 204. The functions of the brightness detection unit 201, the determination unit 203, and the correction amount calculation unit 204 are executed by the photometry unit 114, and the function of the area detection unit 202 is executed by the detection unit 113.

[0021] Image data that has been output from the A / D conversion unit 105 and has undergone a predetermined development process in the image processing unit 106 is input to the brightness detection unit 201 and the area detection unit 202. Specifically, the image data input to the brightness detection unit 201 and the area detection unit 202 is image data of a frame image acquired when live view is performed (before actual shooting of a still image or video) or an image acquired by half-pressing the release button 111.

[0022] The brightness detection unit 201 detects brightness information (specifically, brightness values) of the input image data for each pixel or for each section consisting of a predetermined number of pixels divided into sections of a certain size on the imaging surface of the imaging element 103.

[0023] The area detection unit 202 detects a specific area (detection target area) from the input image data. In this embodiment, it is assumed that a sky area is detected as the detection target area. As a result of sky detection, the area detection unit 202 outputs the likelihood (probability) that each pixel or each section is sky as a sky determination score. The sky determination score can be, for example, an 8-bit number (0 to 255), and the larger the sky determination score, the higher the likelihood that the area is a sky area.

[0024] In this embodiment, the detection unit of the luminance information detected by the luminance detection unit 201 and the detection unit of the sky determination score determined by the area detection unit 202 are set to the same block unit, taking into consideration the processing in the subsequent determination unit 203. However, this is not limited to this, and the detection unit may be set to a pixel unit.

[0025] The region detection method may be a known method, such as a method using color information (e.g., JP 2016-151955 A) or a method using a neural network to identify regions (e.g., JP 2006-39666 A). In this embodiment, region detection is performed using a neural network. Image data obtained by executing the region detection process is stored in memory 102.

[0026] Here, an example of training a neural network for detecting an area estimated to be the sky will be described. FIGS. 11(a) and 11(b) are diagrams showing example images used for training the neural network. In the image of FIG. 11(a), there is a sky area between buildings on the left and right, and the sky is bright near the ground and dark high up, like a sunset sky. In the image of FIG. 11(b), the upper bodies of several people forming a circle are captured facing the sky from the ground, and there is a bright sky area between the people throughout the image. The sky areas in each of the images of FIGS. 11(a) and 11(b) are used as correct answer areas to train the neural network. In addition, the neural network is similarly trained using images with various sky areas (not shown).

[0027] There are various types of sky-containing photographic scenes, such as blue sky scenes, cloudy sky scenes, evening sky scenes, and night sky (starry sky) scenes. As the variety of photographic scenes that the neural network is trained on increases, the difficulty of training increases. However, considering usability, it is desirable to be able to perform exposure compensation for as many photographic scenes as possible. Therefore, in this embodiment, the neural network is trained on as many different sky photographic scenes as possible.

[0028] Returning to the explanation of Figure 2, determination unit 203 compares the luminance information obtained by luminance detection unit 201 with the sky determination score obtained by region detection unit 202 to determine a sky region and a non-sky region. A sky region refers to a region in an image that is determined to be sky based on both the luminance value and the sky determination score, while a non-sky region refers to a region in an image that is determined not to be sky based on both the luminance value and the sky determination score. The method for determining a sky region and a non-sky region will be described later.

[0029] The correction amount calculation unit 204 calculates a correction amount (hereinafter referred to as "exposure correction amount") for correcting the exposure value for the image sensor 103 based on the brightness and area of ​​each of the sky region and non-sky region determined by the determination unit 203. The method for calculating the exposure correction amount will be described later.

[0030] 3 is a flowchart showing the flow of processing for determining the exposure compensation amount in the imaging system 10. Each process (step) indicated by an S number in this flowchart is realized by the CPU of the system control unit 101 loading a predetermined program stored in its ROM into its RAM and comprehensively controlling the operation of each unit of the imaging system 10.

[0031] In S301, the system control unit 101 acquires image data to be input to the brightness detection unit 201 and the area detection unit 202 from the image processing unit 106, and transmits the image data to the brightness detection unit 201 and the area detection unit 202. The image data acquired from the image processing unit 106 is image data that is output from the A / D conversion unit 105 to the image processing unit 106 and generated by performing a predetermined development process in the image processing unit 106, that is, image data acquired before the actual shooting.

[0032] In S302, the brightness detection unit 201 detects brightness information (brightness value for each section in the image) from the image data acquired in S301.

[0033] In S303, the area detection unit 202 detects a sky area from the image data acquired in S301. The sky area is detected by determining a sky determination score for each section in the image.

[0034] In S304, the determination unit 203 determines a sky region and a non-sky region using the luminance information detected in S302 and the sky determination score determined in S303.

[0035] Here, we will explain how to determine sky regions and non-sky regions. Figure 4 is a flowchart of the processing in S304. This processing is performed for each partition of the image. In the following explanation, the partition being processed will be referred to as the "partition of interest."

[0036] In S401, the determination unit 203 determines whether the sky determination score of the section of interest is equal to or greater than the sky determination threshold (first threshold). As described above, the sky determination score ranges from 0 to 255, and the sky determination threshold is a fixed value set in advance between 0 and 255. For example, if the sky determination threshold is 128, a section of interest with a sky determination score of 128 or greater is determined to have a high likelihood of being a sky region, and as will be described later, brightness information is further taken into account to ultimately determine whether the section is a sky region. If the determination unit 203 determines that the sky determination score is equal to or greater than the sky determination threshold (YES in S401), it performs the process of S402. If the determination unit 203 determines that the sky determination score is less than the sky determination threshold (NO in S401), it performs the process of S405.

[0037] In S402, the determination unit 203 determines whether the luminance value of the section of interest is equal to or greater than the luminance threshold (second threshold). FIG. 5 is a diagram showing an example of the luminance histogram and luminance threshold detected in S302. The brightness of the sky varies depending on the scene, such as dawn, clear skies, cloudy skies, and dusk, and the luminance values ​​of bright and dark areas in the image change accordingly. For this reason, it is desirable to use the average luminance of the entire image as the luminance threshold. By determining whether the luminance value of the section of interest is equal to or greater than the luminance threshold, it is possible to determine whether it is a high-luminance area or a low-luminance area in the image.

[0038] If the determination unit 203 determines that the brightness value of the target section is greater than or equal to the brightness threshold (YES in S402), it executes the processing of S403, and if it determines that the brightness value of the target section is less than the brightness threshold (NO in S402), it executes the processing of S404.

[0039] In S403, the determination unit 203 determines that the section of interest is an empty area, and then ends this processing.

[0040] However, region detection using a neural network can result in detection errors. Detection errors include overdetection, in which a region that is not a detection target region is detected as a detection target region, and underdetection, in which a region that is a detection target region is not detected as a detection target region. A region of interest for which the determination in S402 is 'NO' constitutes an overdetected region. Therefore, in S404, the determination unit 203 excludes the region of interest from the sky region and terminates this process. In this way, through the determination processes in S401 and S402, only regions that are highly likely to be sky and have high brightness values ​​are determined to be sky regions.

[0041] In S405, the determination unit 203 determines whether the luminance value of the target section is less than the luminance threshold. If the determination unit 203 determines that the luminance value of the target section is less than the luminance threshold (YES in S405), it executes the process of S406, and if the determination unit 203 determines that the luminance value of the target section is equal to or greater than the luminance threshold (NO in S405), it executes the process of S407.

[0042] In S406, the determination unit 203 determines that the section of interest is a non-sky area, and then ends this processing.

[0043] As mentioned above, when detecting areas using a neural network, there is a possibility that an area may be detected as being sky but not sky, with a high likelihood of not being sky. A section of interest for which the determination in S405 is 'NO' constitutes an undetected area. Therefore, in S407, the determination unit 203 excludes the section of interest from non-sky areas and terminates this process. Thus, through the determination processes in S401 and S405, only areas with a low likelihood of being sky and low brightness values ​​are determined as non-sky areas. In this way, overdetection and underdetection of sky areas in S303 can be reduced, allowing sky areas and non-sky areas to be correctly determined.

[0044] In the flowchart of FIG. 4, the sky determination score is compared with the sky determination threshold, and then the brightness value is compared with the brightness threshold, but the order of these steps may be reversed.

[0045] The processing of the flowchart in Fig. 4 will be further explained using example images. Fig. 6(a) is a diagram showing an example image acquired in S301. Fig. 6(b) is a schematic diagram showing the results of area detection in S303 for the image in Fig. 6(a). As shown in Fig. 6(a), the ceiling of a building is captured along with an outdoor landscape including the sky. In Fig. 6(b), areas E1 and E2 are areas where the sky determination score is equal to or greater than the sky determination threshold, and area E3 is a schematic representation of an area where the sky determination score is less than the sky determination threshold.

[0046] Area E1 is correctly detected as a sky area, but area E2 is an overdetected area of ​​a part of the ceiling area. In this case, only area E1, which has a high sky determination score and is high in brightness, is determined to be a sky area in S403. Area E2 is excluded from the sky area because it is not high in brightness. Area E2 is also not treated as a non-sky area.

[0047] Areas other than areas E1 and E2 have sky determination scores below the sky determination threshold, and areas other than areas E1 to E3 have been correctly detected as non-sky areas, but area E3 has not been detected as a sky area. In this case, areas other than areas E1 to E3 with low brightness are determined to be non-sky areas in S406, and area E3 is excluded from the non-sky area because it is high brightness. Area E3 is not even treated as a sky area.

[0048] Returning to the explanation of Figure 3, in S305, the correction amount calculation unit 204 calculates the average value (hereinafter referred to as "average brightness value") of each brightness value of the sky region and non-sky region determined in S304. The average brightness of the sky region can be calculated by dividing the sum of the brightness values ​​of all sections in the sky region by the number of all sections, and the average brightness of the non-sky region can also be calculated using a similar calculation.

[0049] In S306, the correction amount calculation unit 204 calculates the exposure correction amount. In calculating the exposure correction amount, the difference between the average brightness value AveS of the sky region and the average brightness value AveNS of the non-sky region calculated in S305 and the current exposure control value EvC is calculated. The difference ΔEvS between the average brightness value AveS of the sky region and the current exposure control value EvC, and the difference ΔEvNS between the average brightness value AveNS of the non-sky region and the current exposure control value EvC are calculated using the following equations 1 and 2, respectively. Next, the exposure correction amount SC is calculated using the area nS of the sky region, the area nNS of the non-sky region, and the difference ΔEvAve between the average brightness of the entire image and the exposure control value, using the following equation 3. The difference ΔEvAve is calculated using the following equation 4.

[0050] In the following equation (3), 'α' and 'β' are preset coefficients that adjust the contribution of the sky area and non-sky area to the exposure compensation amount SC. For example, in a scene with a bright sky, where the main subject (non-sky area) is likely to be darkened due to backlighting, it is desirable to prevent the main subject from becoming too dark. In this case, the coefficients α and β are set so that the contribution of the non-sky area to the exposure compensation amount SC is greater than the contribution of the sky area to the exposure compensation amount SC. In other words, they are set to satisfy the relationship α<β. For example, by setting α=0.8 and β=1.2, the contribution of the non-sky area to the exposure compensation amount SC can be made 1.5 times that of the sky area.

[0051]

number

[0052] The coefficients α and β may be changed depending on the state of the non-sky region. For example, if the detection unit 113 detects a person (such as a person's face, head, upper body, or entire body) in a non-sky region, it is assumed that the person is likely to be the main subject. Therefore, if a person is present in the non-sky region, the coefficient β may be set to a larger value than the coefficient α. For example, if the default settings are [α=0.8, β=1.2], and a person is detected in the non-sky region, the settings are switched to [α=0.5, β=1.5]. This can strengthen the influence of the non-sky region on the exposure compensation amount SC.

[0053] As described above, according to this embodiment, by performing exposure correction while excluding overdetected and undetected areas in the sky area, which is an example of a detection target area, it is possible to determine more appropriate exposure conditions.

[0054] Second Embodiment In the second embodiment, a configuration for correcting the exposure compensation amount SC calculated in the first embodiment will be described. The imaging system 10 shown in Fig. 1 is applied to this embodiment as is. The functional block in this embodiment is configured by adding an exposure compensation amount correcting unit to the functional block shown in Fig. 2.

[0055] FIG. 7 is a diagram illustrating the configuration of functional blocks according to a second embodiment that perform exposure compensation during image capture. The functional block that performs exposure compensation during image capture includes a brightness detection unit 701, an area detection unit 702, a determination unit 703, a correction amount calculation unit 704, and a correction amount modification unit 705. The brightness detection unit 701, the area detection unit 702, the determination unit 703, and the correction amount calculation unit 704 are equivalent to the brightness detection unit 201, the area detection unit 202, the determination unit 203, and the correction amount calculation unit 204 in FIG. 2, respectively, and therefore description thereof will be omitted. The correction amount modification unit 705 modifies the exposure compensation amount SC calculated by the correction amount calculation unit 704 to determine the final correction amount (hereinafter referred to as the "modified exposure compensation amount"). The function of the correction amount modification unit 705, i.e., the calculation of the modified exposure compensation amount, is performed by the photometry unit 114.

[0056] 8 is a flowchart showing the flow of processing for determining a corrected exposure compensation amount in an imaging operation in the imaging system 10. Each process (step) indicated by an S number in this flowchart is realized by the CPU of the system control unit 101 loading a predetermined program stored in its ROM into its RAM and comprehensively controlling the operation of each unit of the imaging system 10.

[0057] The processing of S801 to S806 is the same as the processing of S301 to S306 in the flowchart of Fig. 3, and therefore description thereof will be omitted. In S807, the correction amount correction unit 705 performs calculation processing to correct the exposure correction amount SC calculated in S806, and calculates a corrected exposure correction amount.

[0058] Here, the calculation process of the modified correction amount in S807 will be described below. Fig. 9 is a flowchart of the process in S807.

[0059] In S901, the correction amount modification unit 705 acquires the variance of the luminance of the entire image. Here, as in the first embodiment, it is assumed that the luminance detection unit 701 detects luminance information for each section. At this time, the variance Ev var are the intensities Ev1, Ev2, . . . , Ev for each of n sections of the image. n and the average brightness value of the entire image, Ev ave Using the above, it can be calculated using the following formula 5. From the following formula 5, the variance Ev var It can be seen that σ becomes large for images with large variations in brightness (large contrast), and becomes small for images with small variations in brightness (small contrast).

[0060]

number

[0061] In S902, the correction amount modifying unit 705 obtains a coefficient (hereinafter referred to as "coefficient k") according to the variance calculated in S901. The coefficient k is, for example, k=Ev var ×γ. The coefficient k is the variance Ev varIt is defined such that the value of var becomes larger if it is large and smaller if it is small. 'γ' is a fixed value. For example, γ = 0.1 can be set, but it may also be defined to change according to the value of the variance Ev

[0062] In S903, the exposure correction amount SC calculated in S806 is corrected using the coefficient calculated in S902. Specifically, the coefficient k obtained in S902 is multiplied by the exposure correction amount SC obtained in S806 to obtain the corrected exposure correction amount SC R is obtained. That is, SC R = SC × k, and the corrected exposure correction amount SC R is calculated, and this process ends thereby.

[0063] Normally, the coefficient k is determined to take a value within the range represented by 0 < k ≤ 1. Also, as described above, the coefficient k is determined to be larger as the variance is larger and smaller as the variance is smaller. In other words, the correction of the exposure correction amount SC can be said to be a process of maintaining the exposure correction result for an image with a large brightness difference and suppressing the exposure correction based on the determination result of the empty region for an image with a small brightness difference.

[0064] Next, the effect of correcting the exposure correction amount SC obtained in S806 in S807 as described above will be explained. FIGS. 10(a) and (b) are diagrams showing an example of a frame image obtained in the live view.

[0065] FIG. 10(a) shows a frame image in a shooting scene looking from near the exit inside the tunnel in the direction of the exit. Region A1 indicates the dark region inside the tunnel, and region A2 indicates the empty region in the scenery visible from the tunnel exit. FIG. 10(c) shows the luminance histogram of the empty region (region where the empty determination score is greater than or equal to the empty determination threshold) detected in S803 for the frame image of FIG. 10(a).

[0066] The average luminance value of the entire frame image in Fig. 10(a) is used as the luminance threshold. The first luminance value group H1 represents the luminance values of regions (over-detected regions) detected as having a high likelihood of being empty within region A1 by the empty detection in S803. The second luminance value group H2 represents the luminance values of regions correctly detected as empty within region A2 by the empty detection in S803.

[0067] Since the over-detected regions detected as having a high likelihood of being empty within region A1 with respect to region A2 have low luminance values, the first luminance value group H1 and the second luminance value group H2 are divided into a high-luminance side and a low-luminance side. Also, because the brightness difference across the entire frame image is large, the luminance threshold, which is the average luminance of the entire image, is likely to be set in the middle of the first luminance value group H1 and the second luminance value group H2. Therefore, in an image where the bright region and the dark region are clearly distinguishable, even if an empty region is over-detected in S803, the over-detected regions can be excluded by the determination using the luminance threshold, and the correct empty regions can be determined.

[0068] In this case, basically, the need to correct the exposure correction amount SC obtained in S806 in S807 is reduced. Even when correction is performed in S807, var because the value of the variance Ev is large, the coefficient k is set to a value close to '1' within the range of 0 < k ≤ 1.

[0069] Fig. 10(b) shows a frame image in a shooting scene where most of the region is grassland and there is a small empty region at the upper part of the grassland. Region A3 represents the grassland region, and region A4 represents the empty region. Fig. 10(d) shows the luminance histogram of the empty regions (regions where the empty determination score is greater than or equal to the empty determination threshold) detected in S803 for the frame image in Fig. 10(b).

[0070] Here, the average brightness value of the entire frame image in Figure 10(b) is used as the brightness threshold. The third brightness value group H3 represents the brightness values ​​of areas within area A3 that are detected as areas with a high likelihood of being sky by sky detection in S803 (over-detected areas). The fourth brightness value group H4 represents the brightness values ​​of areas within area A4 that are correctly detected as sky by sky detection in S803.

[0071] As shown by the diagonal hatching, a portion of the third brightness value group H3 has brightness values ​​equal to or greater than the brightness threshold. Therefore, the overdetected area corresponding to the portion shown by the diagonal hatching is determined to be a sky area in S804. This is because the average brightness of the entire frame image is used as the brightness threshold. In other words, because the average brightness is heavily influenced by the brightness of the grassland (area A3) that occupies most of the frame image, it is not possible to set the brightness threshold to a value that can distinguish the sky area from other areas.

[0072] Therefore, the exposure compensation amount SC calculated in S806 is corrected by the process of S807. As described above, for an image with a small variance of brightness, the coefficient k is a small value. In other words, the corrected exposure compensation amount SC is calculated in S806 so as to reduce the exposure compensation amount SC. R As a result, when the contrast in the image is small, exposure compensation based on the detection result of the sky region is suppressed.

[0073] 10(b), even in an image in which the sky region occupies the majority or in which the difference in brightness between the sky region and other regions is small, the variance in brightness becomes small, so the coefficient k becomes small and exposure compensation based on the detection result of the sky region can be suppressed. In addition, in this embodiment, the exposure compensation amount SC is corrected using the coefficient k based on the variance of the brightness of the image, but an index (coefficient) expressing variation such as standard deviation or interquartile range may be used instead of the variance.

[0074] As described above, by modifying the exposure compensation amount SC in accordance with the variation in the brightness values ​​of the image, in shooting scenes where it is difficult to appropriately determine the brightness threshold, exposure compensation based on the detection results of the sky area can be suppressed, thereby obtaining a captured image with more appropriately exposure compensation.

[0075] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0076] For example, in the above embodiment, the present invention has been described with reference to an imaging system 10 including an imaging device 100 (digital camera) and a lens barrel 150 (interchangeable lens), but the present invention can also be applied to electronic devices capable of capturing images using an imaging element. Examples of such electronic devices include digital video cameras, smartphones, tablet PCs, and mobile phones with cameras.

[0077] In the above embodiment, the average brightness of the entire image is used as the brightness threshold, but a fixed value may be used as the brightness threshold, just as a fixed value is used as the sky determination threshold. Furthermore, in the above embodiment, the sky region is used as the detection target region, but the detection target region is not limited to the sky region. Regarding neural network training, if it is desired to lower the learning difficulty, it is possible to train only blue skies and cloudy skies, for example.

[0078] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A detecting means for detecting the brightness of an image captured by an imaging element, a calculating means for calculating the likelihood of a specific region from the image, a determining means for determining a first region and a second region based on the brightness and the likelihood, and an exposure determining means for determining an exposure value based on the first region and the second region. bid priceand a correction means for calculating a correction amount to correct the likelihood, wherein the first region is a region where the likelihood is equal to or greater than a first threshold and the brightness is equal to or greater than a second threshold, and the second region is a region where the likelihood is less than the first threshold and the brightness is less than the second threshold. (Configuration 2) The imaging device according to configuration 1, wherein the second threshold value is an average brightness of the entire image. (Configuration 3) The imaging device according to configuration 1 or 2, wherein the correction means modifies the amount of correction based on the respective contributions of the first area and the second area to the amount of correction. (Configuration 4) An imaging device according to Configuration 3, characterized in that the contribution of each of the first region and the second region to the correction amount is adjusted by a coefficient, and the value of the coefficient for the second region is greater than the value of the coefficient for the first region. (Configuration 5) The imaging device according to Configuration 3, further comprising a processing means for detecting a person from the image, wherein the correction means increases the contribution of the second region when the person is present in the second region. (Configuration 6) The imaging device according to any one of configurations 1 to 5, further comprising a correcting means for correcting the amount of correction based on variations in luminance of the image. (Configuration 7) The imaging device according to configuration 6, wherein the correction means reduces the amount of correction as the variation decreases. (Configuration 8) An imaging device described in any one of configurations 1 to 7, characterized in that the calculation means calculates the likelihood of the specific region being a sky region, and the determination means determines the first region to be a sky region and the second region to be a region that is not a sky region. (Configuration 9) The imaging device according to any one of configurations 1 to 8, wherein the calculation means calculates the likelihood for each pixel or for each partition. (Configuration 10) The imaging device according to any one of configurations 1 to 9, wherein the image is a live view frame image or an image acquired by half-pressing a release button provided on the imaging device. (Method 1) A step of detecting the brightness of an image captured by an image sensor, a step of calculating the likelihood of a specific region from the image, a step of determining a first region and a second region based on the brightness and the likelihood, and a step of determining exposure based on the first region and the second region. bid price and calculating a correction amount to correct the likelihood of the first region, wherein the first region is a region where the likelihood is equal to or greater than a first threshold and the brightness is equal to or greater than a second threshold, and the second region is a region where the likelihood is less than the first threshold and the brightness is less than the second threshold. (Program 1) A program that causes a computer to execute the control method described in Method 1.

[0079] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0080] 100 Imaging device 101 System control unit 103 Image sensor 106 Image processing section 113 Detection unit 114 Photometry section 201 Luminance detection unit 202 Area detection unit 203 Decision Section 204 Correction amount calculation section 705 Correction amount correction section

Claims

1. a detection means for detecting the luminance of an image captured by the imaging element; a calculation means for calculating a likelihood of a specific region from the image; a determining means for determining a first region and a second region based on the luminance and the likelihood; a correction unit that calculates a correction amount for correcting an exposure value based on the first area and the second area, the first region is a region in which the likelihood is equal to or greater than a first threshold and the luminance is equal to or greater than a second threshold, The imaging device, wherein the second region is a region where the likelihood is less than the first threshold and the luminance is less than the second threshold.

2. 2. The imaging device according to claim 1, wherein the second threshold is an average luminance of the entire image.

3. 3. The imaging apparatus according to claim 1, wherein the correction means modifies the amount of correction based on the degree of contribution of each of the first area and the second area to the amount of correction.

4. 4. The imaging device according to claim 3, wherein the contribution of each of the first and second regions to the correction amount is adjusted by a coefficient, and the value of the coefficient for the second region is greater than the value of the coefficient for the first region.

5. a processing means for detecting a person from the image; 4. The imaging device according to claim 3, wherein the correction means increases the contribution of the second region when the person is present in the second region.

6. 3. The imaging apparatus according to claim 1, further comprising a correction unit that corrects the amount of correction based on variations in luminance of the image.

7. 7. The imaging apparatus according to claim 6, wherein the correction means reduces the amount of correction as the variation decreases.

8. the calculation means calculates a likelihood of the specific region being an empty region, 3. The imaging device according to claim 1, wherein the determining unit determines that the first area is a sky area and the second area is a non-sky area.

9. 3. The imaging device according to claim 1, wherein the calculation means calculates the likelihood for each pixel or each partition.

10. 3. The imaging device according to claim 1, wherein the image is a frame image of a live view, or an image acquired by half-pressing a release button provided in the imaging device.

11. detecting the luminance of an image captured by the imaging element; calculating a likelihood of a specific region from the image; determining a first region and a second region based on the luminance and the likelihood; calculating a correction amount for correcting an exposure value based on the first area and the second area; the first region is a region in which the likelihood is equal to or greater than a first threshold and the luminance is equal to or greater than a second threshold, A method for controlling an imaging device, wherein the second region is a region where the likelihood is less than the first threshold and the luminance is less than the second threshold.

12. A program that causes a computer to execute the control method according to claim 11.

Images