Imaging device

Abstract

To apply good development parameters even in imaging scenes with different light sources.SOLUTION: An imaging apparatus includes an image sensor capable of making exposure conditions changeable for each exposure area, which is configured including a single pixel or a plurality of pixels and an image processing unit for digitally processing a captured video signal. The imaging apparatus generates a combined area that combines exposure areas from threshold values of exposure conditions for each exposure area, calculates an evaluation value of development parameters for each of the coupled areas, and applies the acquired development parameters for each of the exposure areas to the image processing unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] Conventionally, there is known a technique for performing white balance processing on a region-by-region basis by providing a configuration capable of changing exposure conditions on a region-by-region basis within an image sensor (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Application No. 2014-557907 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to apply favorable development parameters even in a shooting scene in which a different light source is present. [Means for solving the problem]

[0005] An imaging device according to one aspect of the present invention is configured with a single pixel or multiple pixels. Multiple Exposure conditions for each exposure area setting A possible image sensor; The image sensor The image was taken A white balance correction process is performed on the image for each image area corresponding to the plurality of exposure areas. Image Processing Unit and, having The image processing unit sets one or more image regions in which the brightness of a subject in the image regions is similar as one combined region based on the exposure condition for each of the exposure regions, and performs the white balance correction process individually for each of the combined regions. It is characterized by: Effect of the Invention

[0006] Good development parameters can be applied even in scenes where there are different light sources. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram showing the configuration of an image pickup apparatus according to a first embodiment; [Diagram 2] FIG. 2 is a diagram showing the relationship between an image sensor and an exposure area according to the present embodiment. [Diagram 3] Flowchart according to this embodiment [Figure 4] FIG. 2 is a diagram showing the relationship between the image sensor and the coupling region according to the embodiment; [Diagram 5] FIG. 1 is a configuration diagram of an imaging apparatus according to a second embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (First embodiment) Hereinafter, a method of acquiring and correcting a white balance evaluation value, which is one of the development parameters, according to the first embodiment will be described with reference to Figs. 1 to 4. Although all the explanations in the embodiments of this specification will be about white balance, the invention may also be about development parameters for correcting not only white balance but also color signal processing related to color, saturation, or hue. All the white balances in this embodiment are auto white balance (AWB), and the white balance evaluation value is acquired automatically.

[0009] In this embodiment, a method for calculating a white balance evaluation value and a method for image processing based on the calculated evaluation value will be described in a shooting environment where there are areas with different colors due to different light sources, and areas of different light sources are photographed simultaneously with a single imaging device.

[0010] The configuration of an image capturing apparatus 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of the image capturing apparatus 100 according to this embodiment. The image capturing apparatus 100 includes a lens 101, an image sensor 102, an image processing unit 103, an exposure region control unit 104, and an image output unit 105.

[0011] A lens 101 receives light within a photographing angle of view and collects the light on an image sensor 102 .

[0012] The image sensor 102 photoelectrically converts the collected light and transmits it as a video signal to the image processing unit 103. The image sensor 102 also has RGB color filters and generates a video signal having RGB color information. Furthermore, the image sensor 102 is an image sensor that can change exposure conditions for each single pixel or for each area (hereinafter referred to as exposure area) composed of multiple pixels, and details will be described later with reference to FIG. 2.

[0013] The image processing unit 103 performs image processing, including development processing, on the video signal, which is a digital signal. The video signal is converted into image data (JPEG format, etc.). At this time, the video signal having RGB color information transmitted from the image sensor 102 is converted into a color difference signal such as a YUV format and digitally processed. Finally, the video signal is converted (developed) into image data, and the image data is output via the image output unit 105. The development processing includes white balance correction, saturation correction, color tone correction, sharpness correction, gamma correction, gradation correction, etc. White balance is not a fixed development parameter, but a white balance evaluation value (development parameter) is calculated from the video signal. The calculated development parameter is applied to perform white balance correction. In addition, the image processing unit 103 can calculate luminance information and color difference information by YUV conversion. Based on this luminance information, exposure conditions that provide appropriate exposure are determined. The exposure conditions are determined by the exposure time and analog gain value of the image sensor 102. The calculated exposure conditions are transmitted to the exposure area control unit 104. In addition, a white balance evaluation value is calculated based on the luminance information and color difference information.

[0014] The exposure area control unit 104 assigns exposure conditions to each exposure area and controls the image sensor 102. In order to improve the dynamic range for the brightness of the subject, the exposure area control unit 104 sets the exposure conditions (exposure time and analog gain) so that, for example, the average luminance value in the exposure area becomes the median value of the data gradation that can be output.

[0015] The image output unit 105 is an interface related to the distribution of image data, and is capable of outputting image data from an imaging device via a network, for example.

[0016] With reference to FIG. 2, the exposure area of ​​the image sensor 102 in this embodiment will be described. FIG. 2 is a diagram showing the relationship between the image sensor 102 and the exposure area 201. In FIG. 2, the image sensor 102 is divided into nine exposure areas. For clarity, the exposure area 201 of the image sensor 102 will be described using symbols A, B, and C corresponding to the rows and symbols 1, 2, and 3 corresponding to the columns. For example, the exposure area A1 in FIG. 2 corresponds to the upper left exposure area of ​​the image sensor 102, and the exposure area B2 corresponds to the central exposure area of ​​the image sensor 102. The nine exposure areas can have their exposure conditions (exposure time and analog gain) set independently. In addition, position information of pixels corresponding to the exposure areas in the image sensor 102 is transmitted to the image processing unit 103 and the exposure area control unit 104, and is used for image processing and calculation of an evaluation value. The image processing unit 103 can perform image processing including calculation of an evaluation value and development processing from a video signal for each exposure area.

[0017] The development process in this embodiment will be described with reference to Fig. 3. Fig. 3 is a flow chart showing the procedure in this embodiment. Each step (hereinafter abbreviated as S) will be described.

[0018] In step S301, the exposure area control unit 104 determines the exposure conditions for each exposure area based on the brightness information for each exposure area. At this time, the exposure conditions are independently controlled to provide proper exposure within the exposure area. The brightness information is calculated from the video signal of the previous frame. The exposure conditions controlled by the exposure area control unit 104 are set for the image sensor 102.

[0019] In step S302, the exposure regions are combined to generate a combined region based on the threshold of the exposure conditions. The threshold of the exposure conditions makes it possible to distinguish between bright and dark regions of the subject being photographed. In other words, a combined region that divides regions with different light sources can be generated.

[0020] A detailed explanation of the exposure condition threshold and the method of calculating the brightness of the subject will be given later. For each exposure area, it is determined whether the brightness of the subject is brighter or darker than the threshold. Areas brighter than the threshold are collectively called the bright combined area, and areas darker than the threshold are collectively called the dark combined area. The bright combined area and dark combined area will be explained further using Figure 4.

[0021] In step S303, shooting is performed under the set exposure conditions.

[0022] In step S304, the image sensor 102 transmits the captured video signal to the image processing unit 103. The image processing unit 103 calculates a white balance evaluation value for each combined region from the captured video signal. At this time, the image processing unit 103 also calculates luminance information for each exposure region and transmits it to the exposure region control unit 104 so that the exposure conditions for the next frame can be set.

[0023] In step S305, white balance correction is performed for each combined region. At this time, development processing other than white balance correction is also performed for each exposed region.

[0024] In step S306, the developed image is output.

[0025] The combined area in this embodiment will be described with reference to FIG. 4. FIG. 4 is a diagram showing the relationship between the image sensor 102 and the combined area. FIG. 4 illustrates the exposure area of ​​the image sensor 102 in the same way as FIG. 2. The brightness of the subject for each exposure area is estimated from the average brightness value and the exposure condition for each exposure area. At this time, the image sensor 102 sets the exposure condition so that the brightness value is the same for each exposure area, so the brightness of the subject can be estimated by referring only to the exposure condition. That is, it can be said that an area where the exposure time is twice as long is an area with 1 / 2 the brightness, and an area where the analog gain is twice as large is an area with 1 / 2 the brightness. In this embodiment, the exposure condition is expressed by Equation 001. The exposure condition is X, the exposure time is T, and the analog gain is G. However, at this time, the exposure time T and the analog gain G are normalized with respect to the brightness of the subject, and the weighting of the exposure time T and the analog gain G is the same. That is, the value of X is the same whether T:G=2:1 or T:G=1:2. X=T×G Formula 001

[0026] At this time, a threshold value Xth for the exposure condition X is set to generate the combined region. The exposure condition X is held independently for each exposure region. Therefore, as shown in FIG. 4, the region where the exposure condition X is smaller than Xth becomes the combined region 401 of the bright part (white regions A2, A3, B2, B3, C3 in FIG. 4). On the other hand, when the exposure condition X is Xth The above The area becomes the dark combined area 402 (area with diagonal lines in FIG. 4, A1, B1, C1, C2). The exposure areas assigned to the same combined area are areas where the brightness of the photographed subject is similar and it is highly likely that they are illuminated by the same light source. For this reason, a white balance evaluation value is obtained for each combined area. Furthermore, the white balance evaluation value is applied to each corresponding combined area, and white balance correction is performed by digital gain.

[0027] As described above, it is possible to calculate a white balance evaluation value for each combined region, which is an area that is likely to be an abnormal light source. Furthermore, by applying the calculated white balance evaluation value to each exposure region (or combined region), it is possible to generate an image in which appropriate white balance correction for the abnormal light source has been applied to each region.

[0028] The following provides a supplementary explanation of the first embodiment.

[0029] White balance correction may be performed for each exposure area, not for each combined area. For example, since the exposure conditions are different for each exposure area, development processing such as data compression may be performed for each exposure area. In this case, signal processing can be made easier by performing development processing including white balance correction for each exposure area all at once.

[0030] Here is a supplementary explanation on how the white balance evaluation value is calculated. When calculating the white balance evaluation value, the R, G, and B ratios are calculated for each combined area. Therefore, the more pixels there are, the more samples there are, and the better the accuracy. In other words, if the combined area is too small, the number of samples will be reduced, and accuracy will decrease. For this reason, it is desirable to determine the minimum size of a block (minimum number of pixels).

[0031] In addition, if there are many blocked-up or blown-out pixels, it is difficult to calculate an appropriate white balance evaluation value. For this reason, it is desirable to calculate the white balance evaluation value from video signals of only pixels with intermediate luminance or exposed areas.

[0032] Furthermore, if the color of the subject is significantly biased, it is not possible to calculate an appropriate white balance evaluation value. Therefore, the area for calculating the evaluation value may be limited to an achromatic area within the combined area. In this case, it is possible that the color is significantly biased and the achromatic color of the subject is not detected. In this case, it is desirable to relax the exposure condition difference threshold for the combined area where the achromatic color cannot be detected. This makes it possible to detect the achromatic color and correct the white balance appropriately.

[0033] A method for generating the combined region will be explained in the supplementary section. When generating the imaging region, it may be limited to the case where the exposure regions are adjacent. When limiting the combined region to adjacent exposure regions, it is limited to regions that are close at least on the image, so there is a high possibility that they are the same light source. This improves the accuracy of the white balance evaluation value.

[0034] There is also an advantage to not limiting the area to adjacent areas. There are cases where the scene being photographed has depth, and the subject is in the background, causing the background area to be divided. In this case, if the area is not limited to adjacent areas, the divided areas can be determined to be the same combined area. This increases the number of samples in the combined area, improving accuracy.

[0035] A supplementary explanation will be given regarding the threshold value of the exposure condition. There may be multiple threshold values ​​Xth for the exposure condition X. By providing multiple threshold values, it becomes possible to appropriately correct the white balance even when there are three or more different light sources.

[0036] In addition, the threshold value of the exposure condition may be either an absolute value or a relative value. When using an absolute value, it is desirable to set the threshold value of the exposure condition in advance so that bright subjects and dark subjects can be separated. In this case, the calculation load is reduced because the threshold value is determined in advance.

[0037] When using relative values, for example, the maximum and minimum values ​​are calculated from the exposure conditions, and the median value is set as the threshold. This allows an appropriate combined area to be generated according to the shooting scene, improving the accuracy of white balance correction. Also, the histogram may be referenced, for example, by taking a histogram from brightness prediction for each pixel and setting the value where the histogram is a valley (where there is a brightness step) as the threshold.

[0038] A supplementary explanation will be given regarding exposure conditions. Since the imaging device is a single optical system, it is possible to predict the brightness of the subject for each exposure area from the brightness information for each exposure area and the exposure conditions for each exposure area. For example, under the same exposure conditions, the brightness information and the brightness of the subject are proportional. If the brightness information between exposure areas is doubled, it can be predicted that the brightness of the subject will also be doubled.

[0039] Furthermore, the analog gain may be converted into ISO sensitivity. The analog gain can be converted into ISO sensitivity based on the optical characteristics of the image sensor 102. Also, since the aperture value (F-number) is uniform for the image sensor 102, if the ISO sensitivity and aperture value are known in addition to the exposure condition X, it becomes possible to estimate the brightness of the subject.

[0040] Here is a supplementary explanation for Figure 3. In Figure 3, when continuous shooting is performed, the exposure conditions are set for each exposure area based on the brightness information acquired in S304. This makes it possible to shoot with exposure conditions that reflect the brightness of the subject in the previous frame.

[0041] The white balance evaluation value will be explained in more detail below.

[0042] It is desirable to reflect the white balance evaluation value over time in accordance with the rate of change in the exposure conditions. When the exposure conditions are changing abruptly, it is highly likely that the light source is changing, so white balance correction is also performed abruptly. On the other hand, when the exposure conditions are changing slowly, it is desirable to similarly perform white balance correction slowly. By performing white balance correction in this way in accordance with the rate of change in the exposure conditions, natural color changes are achieved when viewed as a video.

[0043] Furthermore, by reducing the frequency of white balance correction, the load on the image processing unit 103 can be reduced. By performing white balance correction only on combined areas where the exposure conditions have changed, the load on the image processing unit 103 can be reduced. At this time, a threshold value may be set for the amount of change in the exposure conditions of the exposure area, and it may be determined how much change is required before white balance correction (reacquisition and reapplication of development parameters) is performed again. Furthermore, when white balance correction is not performed, there is no need to calculate an evaluation value.

[0044] In this embodiment, a combined area is provided, and a different white balance evaluation value is applied to each combined area. However, if the white balance evaluation value differs significantly at the boundary of the combined area, the image will become unnatural. Therefore, it is desirable to apply a gradation to the white balance evaluation value at the boundary of the combined area to gradually change the color. For example, the white balance evaluation value at the center of the boundary may be set to the average value of the adjacent combined areas, and the evaluation value may be gradually changed up to 50 pixels inside each combined area. Also, it is desirable to widen the range of the gradation as the difference in the white balance evaluation value becomes larger. This results in a gradual change and a natural image.

[0045] 1, the exposure area control unit 104 has been described as an independent block, but it may be part of the function of the image sensor 102 or the image processing unit 103. In addition, in the description of the control of the exposure area control unit 104, an example of control in which the average luminance value is the median value of the data gradation has been given, but the invention is not limited to this. It is sufficient that the exposure conditions are determined so as to reduce crushed shadows and blown highlights.

[0046] 2, an example in which there are nine exposure regions 201 has been described, but this embodiment is applicable as long as there are two or more exposure regions. Furthermore, a single pixel may be one exposure region.

[0047] Second embodiment In the first embodiment, a case where all of the development processing is performed by the image processing unit 103 is described. In the second embodiment, a case where part of the development processing is performed by the image sensor 102 is described. Furthermore, in the second embodiment, the contents already described in the first embodiment will be omitted.

[0048] In the present embodiment, the image signal from the image sensor 102 having a plurality of exposure regions has different exposure conditions for each region, which complicates the processing by the image processing unit 103. Therefore, processing for each exposure region may not be possible with general-purpose image processing. In the second embodiment, the image sensor 102 having a plurality of exposure regions has some of the image processing functions shown in the first embodiment, so that even a general-purpose image processing unit 103 can perform development processing.

[0049] An imaging device 100 including an image sensor 102 having a development processing unit according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a configuration diagram of an imaging device 100 including an image sensor 102 having a development processing unit 501 according to this embodiment. The lens 101, image output unit 105, and exposure area control unit 104 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0050] The image sensor 102 in FIG. 5 includes a development processing unit 501. The image sensor 102 converts light collected through a color filter into an RGB video signal by photoelectric conversion. The image sensor 102 is a stacked sensor, and includes the development processing unit 501, which is a circuit that performs digital signal processing. The development processing unit 501 converts the RGB signal into a YUV signal (brightness information and color difference information). At this time, it is desirable to correct the brightness of the image using a digital gain based on the exposure conditions for each exposure area (hereinafter referred to as brightness correction). In brightness correction, a correction is made using a digital gain for each exposure area so that dark areas become darker and bright areas become brighter when viewed as a whole image.

[0051] It is desirable to carry out white balance correction before brightness correction. By carrying out the white balance correction by combined area described in the first embodiment before brightness correction, an appropriate white balance is applied to each exposure area. In this state, the digital gain of the brightness correction is applied, so that white balance correction is possible without shifting the color tone after development.

[0052] The image sensor 102 transmits the image signal that has been subjected to the brightness correction to the image processing unit 103. At that time, the YUV signal may be output as it is, or may be reconverted to an RGB signal. Since the image processing unit 103 handles the image signal that has been brightness corrected, it can process it in the same way as when the exposure conditions are uniformly set in the image sensor, so that even a general-purpose image processing unit can process it.

[0053] If white balance correction is performed after brightness correction, the digital gain of the brightness correction will cause the color to differ for each exposure area, making it impossible to perform appropriate white balance correction.

[0054] As described above, the image sensor 102 includes the development processing unit 501 and outputs a video signal on which white balance correction has been performed for each combined area, so that development processing can be performed even by a general-purpose image processing unit 103. In addition, even the general-purpose image processing unit 103 can generate an image to which appropriate white balance correction has been applied.

[0055] The output format of the image sensor 102 will be explained in more detail below. By outputting the signal as a YUV signal, it is possible to omit the process of converting the RGB signal to a YUV signal, which is performed by the image processing unit 103. Furthermore, the data conversion from the RGB signal is not limited to the YUV format. Other color difference signal formats, such as YCrCb, may also be used.

[0056] Furthermore, by reconverting the YUV signal into an RGB signal in the image sensor 102 before outputting it, the image sensor can output the signal in a general-purpose format, which makes it easier to develop software for the input of the video signal in the image processing unit 103.

[0057] Furthermore, the image sensor 102 may generate a combined region map indicating the size and position of the combined region separately from the video signal and transmit it to the image processing unit 103. This enables the image processing unit 103 to perform image processing and extract data of the video signal based on the combined region map.

[0058] In brightness correction, a compression process of the data amount (tone) may be performed separately from the development process. Also, a part or all of the processes performed by the image processing unit 103 may be performed by the development processing unit 501. Also, brightness correction may be performed by the image processing unit 103, and the roles of the development processing unit 501 and the image processing unit 103 may be freely changed.

[0059] A supplementary explanation of brightness correction is provided. The image sensor 102 is set to exposure conditions that provide the appropriate amount of light exposure for each exposure area. Therefore, if brightness correction is not performed using digital gain, the image as a whole will have no brightness gradations. By performing brightness correction, bright subjects can be rendered brighter and dark subjects can be rendered darker in the image.

[0060] Although the present invention has been described above with reference to the present embodiment, the present invention is not limited to the above embodiment, and various modifications and changes are possible within the scope of the present invention. [Explanation of symbols]

[0061] 100 Imaging device 101 Lens 102 Image Sensor 103 Image processing section 104 Exposure area control unit 105 Image output unit

Claims

1. an image sensor capable of setting exposure conditions for each of a plurality of exposure areas each composed of a single pixel or a plurality of pixels; an image processing unit that performs a white balance correction process for each image area corresponding to the plurality of exposure areas on the image captured by the image sensor, The image processing unit includes: setting one or more image regions having similar subject brightness in the image regions as one combined region based on the exposure conditions for each of the exposure regions; an imaging device that performs the white balance correction process for each of the combined regions individually;

2. The imaging device described in Claim 1, characterized in that the image processing unit sets adjacent image areas among multiple image areas that are determined to have similar subject brightness based on the exposure conditions for each exposure area as the single combined area.

3. The imaging device described in Claim 1 or 2, characterized in that the image processing unit calculates an evaluation value of the white balance for each combined area, and performs the correction processing for each combined area based on the evaluation value.

4. An imaging device as described in any one of claims 1 to 3, characterized in that the image processing unit sets one or more image areas having similar subject brightness as one combined area based on a comparison result with at least one threshold value for the exposure condition.

5. The imaging device as described in Claim 4, characterized in that at least one of the thresholds is adjusted so that an achromatic area is detected in the set combined area.

6. The image processing unit: Setting one or more image regions corresponding to an exposure region whose exposure condition is smaller than a predetermined threshold as one first combined region; setting one or more image regions corresponding to an exposure region whose exposure condition is equal to or greater than the predetermined threshold as one second combined region; 5. The imaging apparatus according to claim 1, wherein the white balance correction process is performed individually for each of the first combined area and the second combined area.

7. The imaging device according to claim 6, characterized in that the predetermined threshold is determined based on maximum and minimum values ​​of the exposure conditions of the exposure area.

8. An imaging device as described in claim 6 or 7, characterized in that the predetermined threshold is changed so that an achromatic area is detected in the set combined area.

Images