Imaging apparatus and white balance control method
Patent Information
- Application Number
- JP2022015303
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2022-02-03
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing imaging devices face challenges in accurately estimating the light source due to mismatched angles of view between infrared sensors and the captured image, and potential blockage of sensors by interchangeable lenses or user interference, leading to inaccurate white balance control.
An imaging apparatus with a detachable lens system that includes an infrared reflective filter and a separate infrared sensor outside the optical path, allowing for accurate detection of infrared rays from the shooting angle of view, and a control unit to calculate the light source index based on infrared and visible light signal intensities, with optional flicker detection and white balance adjustment.
Enables precise light source estimation and improved white balance control, avoiding flicker detection in natural environments and enhancing user experience by maintaining a steady through-the-lens image display.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to an imaging device and a white balance control method, and more particularly to a white balance control method that includes subject light source estimation using infrared detection performed in an imaging device. [Background technology]
[0002] 2. Description of the Related Art Conventionally, in a digital camera, which is a type of imaging device, a technology has been disclosed that effectively utilizes an infrared sensor to determine the type of light source in an imaging environment and appropriately control the white balance of an image to be captured.
[0003] Patent Document 1 discloses an imaging device, a white balance control method, and a white balance control program that enable white balance control by effectively utilizing an existing infrared sensor in the imaging device.
[0004] According to the invention disclosed in Patent Document 1, it is possible to control the white balance of an image to be captured by effectively utilizing an existing phase difference sensor required for autofocus control in an imaging device and an existing infrared sensor required for remote control. Furthermore, by effectively utilizing these existing infrared sensors, it is possible to determine the type of light source in the imaging environment and appropriately control the white balance of the image to be captured. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Patent No. 4033140 Summary of the Invention [Problem to be solved by the invention]
[0006] However, in the invention disclosed in Patent Document 1, the angle of view of infrared light acquired by the infrared sensor does not necessarily match the angle of view of the captured image, so when light source estimation is performed using infrared light acquired by the infrared sensor, there is a risk that the estimated light source will differ from the light source for the angle of view of the captured image in the actual capturing environment. Also, there is a risk that the light source for the angle of view of the captured image in the actual capturing environment cannot be estimated because the user may unintentionally block the remote control light receiving unit having the infrared sensor while operating the imaging device, or because the interchangeable lens of an interchangeable lens camera may block the remote control light receiving unit having the infrared sensor.
[0007] The present invention has been made in consideration of the above circumstances, and aims to provide an imaging device that can perform highly accurate light source estimation by detecting infrared rays incident from the shooting angle of view through a lens optical system. [Means for solving the problem]
[0008] In order to solve the above-mentioned problems, an imaging device according to a first aspect of the present invention is an imaging device equipped with a detachable photographing lens having a lens optical system, and is equipped with an imaging element that detects light incident through the lens optical system and performs photoelectric conversion on the detected light to output a signal, an infrared reflective filter arranged on the object side of the imaging element, an infrared sensor arranged between the lens optical system and the infrared reflective filter and outside the optical path of light passing through the lens optical system, a control unit, and an image processing unit that converts the signal output by the imaging element into image data and outputs it, and is characterized in that the infrared reflective filter reflects infrared light from the light incident through the lens optical system, the infrared sensor detects the infrared light reflected by the infrared reflective filter, the control unit obtains the intensity of the infrared signal detected by the infrared sensor, and the control unit obtains the intensity of the signal of visible light that has passed through the infrared reflective filter and been detected by the imaging element via the image processing unit.
[0009] An imaging device according to a second aspect of the present invention is characterized in that the control unit includes a light source estimation unit, which compares the intensity of the infrared signal acquired by the control unit with the intensity of the visible light signal, and calculates an index of the light source in the shooting environment based on the comparison result.
[0010] An imaging device according to a third aspect of the present invention is characterized in that the control unit includes a flicker detection unit, and the flicker detection unit changes the readout frame rate of the imaging element and performs flicker detection on image data output by the image processing unit when the light source index calculated by the light source estimation unit is greater than a threshold determined from the intensity of a visible light signal.
[0011] An imaging device according to a fourth aspect of the present invention is characterized in that the control unit includes a white point distribution setting unit and a white balance setting unit, the white point distribution setting unit calculates a white point distribution for image data output by the image processing unit based on a light source index calculated by the light source estimation unit, and the white balance setting unit calculates and changes a white balance setting value based on the white point distribution.
[0012] A fifth aspect of the present invention provides a white balance control method for an imaging device in which an infrared reflective filter reflects infrared light from light incident through a lens optical system, an infrared sensor disposed between the lens optical system and the infrared reflective filter and outside the optical path of light passing through the lens optical system detects the infrared light, and an imaging element detects visible light transmitted through the infrared reflective filter, the white balance control method comprising the steps of: The method includes the steps of: comparing the intensity of the infrared signal with the intensity of the visible light signal; calculating an index of the light source of the shooting environment based on the comparison result; performing flicker detection when the index of the light source is greater than a threshold determined from the intensity of the visible light; calculating a white spot distribution based on the index of the light source; and changing the white balance setting value based on the white spot distribution. [Effects of the Invention]
[0013] According to the present invention, since infrared rays incident through a lens optical system, i.e., infrared rays incident from the shooting angle of view, can be detected, it is possible to provide an imaging device capable of highly accurate light source estimation regardless of the lens attached, particularly in an interchangeable lens camera. Furthermore, it is possible to omit flicker detection in the imaging device in environments other than artificial light environments, thereby preventing a deterioration in the user's operational experience that accompanies control when flicker is detected in the imaging device. Furthermore, the highly accurate light source estimation makes it possible to improve the accuracy of white balance control. [Brief explanation of the drawings]
[0014] [Figure 1] 1 is a block diagram showing a main configuration of an imaging device according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating the arrangement of an infrared reflection filter 42 and an infrared sensor 43, and the behavior of the infrared reflection filter 42 in reflecting infrared light among light beams passing through the lens optical system 32. [Figure 3] 10 is a flowchart illustrating the flow of processing from light source estimation in the camera body 20 to white balance setting. [Figure 4] 1 is a diagram for explaining a white spot distribution under a natural light environment and a white spot distribution under an artificial light environment; DETAILED DESCRIPTION OF THE INVENTION
[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Also, the drawings are simplified to facilitate the explanation of the present invention.
[0016] 1 is a block diagram showing the main components of an imaging device 10 of this embodiment. The imaging device 10 is composed of a camera body 20 and a photographing lens 30, which is an interchangeable lens that can be attached to and detached from the camera body 20. The camera body 20 has a camera-side mount section 21, and the photographing lens 30 has a lens-side mount section 31, and the two mount sections are coupled together to be detachably fixed.
[0017] The camera body 20 includes an imaging element 41, an infrared reflection filter 42, an infrared sensor 43, a control unit 50, an image processing unit 60, and an image display unit 70.
[0018] The photographing lens 31 is provided with a lens optical system 32 having a plurality of lens groups (not shown) including a focus lens group. For simplicity, the lens optical system 32 is shown in Fig. 1 as a single lens.
[0019] The image sensor 41 receives light from the subject that is collected by the lens optical system 32, performs photoelectric conversion, and outputs an analog image signal of the subject image. In this embodiment, a CMOS image sensor is used for the image sensor 41. The light receiving surface of the image sensor 41 is made up of a large number of pixels.
[0020] The infrared reflecting filter 42 is disposed in front of the image sensor 41, and reflects the infrared component of the light guided to the image sensor 41 through the lens optical system 32, while transmitting the visible light component.
[0021] The infrared sensor 43 is located between the lens optical system 32 and the infrared reflection filter 42, outside the optical path of the light passing through the lens optical system 32, and detects infrared rays and outputs an infrared signal. The infrared sensor 43 is located, for example, on the outer edge of the back surface of the camera side mount unit 21 that contacts the lens side mount unit 31. Also, multiple infrared sensors may be located. By locating multiple infrared sensors, it is possible to improve the accuracy of infrared detection.
[0022] The image processing unit 60 performs various signal processing on the analog image signal output from the image sensor 41 and outputs it as digital image data. For example, it performs amplification processing to amplify the read color component signals, shading correction to ensure linearity between the amount of light received by the pixel and the output value, color reproduction processing, gamma correction processing, and development processing to convert image data into JPEG or TIFF format. It also performs white balance setting, which will be described later.
[0023] The control unit 50 has a CPU, a ROM storing programs and data, and a RAM used as a working area, and performs comprehensive control of the entire imaging device 10. For example, it controls the readout of the image sensor 41. The control unit 50 outputs a signal that determines the drive timing of the image sensor 41, thereby controlling the horizontal and vertical drive of each pixel and reading out each RGB color component signal from each pixel. In addition, in the process of detecting flicker, which will be described later, the control unit 50 changes the readout timing of the image sensor 41 and changes the exposure time.
[0024] The image display unit 70 displays, as through image data, the digital image data processed by the image processing unit 60. In addition, it displays images read from a recording medium (not shown), etc.
[0025] Next, we will explain how the infrared sensor 43 detects the infrared light reflected by the infrared reflection filter 42. Figure 2 is a diagram for explaining the arrangement of the infrared sensor 43 and the infrared reflection filter 42, and the behavior of the infrared reflection filter 42 when reflecting infrared light out of the light passing through the lens optical system 32.
[0026] Of the light Φ1 that passes through the lens optical system 32 and enters, the infrared reflective filter 42 reflects the infrared component Φ2. The infrared component Φ2 reflected by the infrared reflective filter 42 is detected by an infrared sensor 43 that is arranged between the lens optical system 32 and the infrared reflective filter 42. The visible light component Φ3 that passes through the infrared reflective filter 42 enters the image sensor 41. By arranging the infrared sensor 43 and the infrared reflective filter 42 in this manner, it is possible to detect infrared light that enters from the actual shooting angle of view regardless of the type of interchangeable lens.
[0027] 3 is a flowchart illustrating the flow of processing from light source estimation to white balance setting in camera body 20. Control unit 50 internally includes an infrared intensity calculation unit 51, a visible light intensity calculation unit 52, a light source estimation unit 53, a flicker detection unit 54, a white point distribution setting unit 55, and a white balance setting unit 56. The processing of each unit of control unit 50 will be described in detail below using the flowchart.
[0028] This flowchart starts when an operation equivalent to starting up the camera or switching from the camera setting screen to the shooting mode is performed in the camera body 20.
[0029] In S01, the infrared intensity calculation unit 51 acquires the infrared signal detected by the infrared sensor 43 and calculates the infrared intensity Bv_IR based on the strength of the infrared signal. If multiple infrared sensors are used, the strengths of the infrared signals from the multiple infrared sensors may be detected and the infrared intensity Bv_IR may be calculated from the maximum value.
[0030] In S02, the visible light intensity calculation unit 52 acquires the digital image data output by the image processing unit 60 and calculates the visible light intensity Bv_RGB. The visible light intensity Bv_RGB is calculated by accumulating and averaging the intensities Bv_RGB(x, y) of the R value (red), G value (green), and B value (blue) in the image signal of pixel blocks obtained by arbitrarily dividing the light receiving surface of the image sensor 41.
[0031] In S03, the light source estimation unit 53 calculates the artificial lighting reliability α from the infrared intensity Bv_IR calculated by the infrared intensity calculation unit 51 in S01 and the visible light intensity Bv_RGB calculated by the visible light intensity calculation unit 52 in S02.
[0032] The artificial lighting reliability α is an index obtained by comparing the values of the intensity Bv_IR of infrared rays and the intensity Bv_RGB of visible light. When the intensity Bv_IR of infrared rays is sufficiently small and the intensity Bv_RGB of visible light is sufficiently large, the artificial light source reliability α = 1. When the intensity Bv_IR of infrared rays is sufficiently large, the artificial lighting reliability α = 0. α takes a value between 0 and 1. When α is greater than or equal to a preset threshold T (0 < T ≤ 1), it is estimated that the environment is under artificial light. The threshold T takes a value preset corresponding to the intensity Bv_RGB of visible light.
[0033] The artificial lighting reliability α may be calculated for each intensity Bv_RGB(x,y) in pixel blocks obtained by arbitrarily dividing the light-receiving surface of the imaging element 41. By calculating the artificial lighting reliability α for each pixel block, it becomes possible to estimate the light source for the area corresponding to each pixel block on the imaging element 41 within the shooting angle of view.
[0034] In S04, the light source estimation unit 53 compares the artificial lighting reliability α calculated in S03 with a preset threshold T and performs light source estimation. When the artificial lighting reliability α is greater than or equal to the preset threshold T, it is estimated that the environment is under artificial light, and the process proceeds to S05. When the artificial lighting reliability α is smaller than the threshold T, it is estimated that the environment is under natural light, and the process proceeds to S06.
[0035] In S05, the flicker detection unit 54 changes the readout frame rate of the imaging element 41 and performs flicker detection. For the specific method of flicker detection in the present invention, a known technique can be adopted. For example, in a state where the frame rate is set to N + 0.5 times the flicker period to be detected (N is a non-negative integer), using the data of the difference in image data between two frames at an interval of N + 0.5 times the period of this flicker, it is determined whether the phases of the flicker stripes generated in each of the images between these two frames are in opposite phases. If they are in opposite phases, there is a method of determining that the light of the flicker period to be detected is the cause of the flicker generation.
[0036] When the flicker detection unit 54 detects flicker from the digital image data sent from the image processing unit 60, it changes the readout cycle and exposure time of the image sensor 41 in accordance with the flicker cycle, thereby avoiding the effects of flicker on the through image data and recorded images.
[0037] If it is determined in S04 that the image is taken under natural light, flicker detection in S05 is not performed. Because flicker detection involves the aforementioned timing control and readout speed change processes, it is not desirable for the user to check the live view image data on the image display unit 70 during flicker detection, as this would result in poor appearance due to the time lag caused by the process and a slower update rate for the live view image. Therefore, if it is determined that the shooting angle of view is taken under natural light, it is determined that flicker, which occurs under artificial light, is not occurring, and flicker detection is not performed, making it possible to provide good-looking live view image data.
[0038] In S06, the white spot distribution setting unit 55 sets the white spot distribution in the shooting environment using the artificial lighting reliability α calculated in S03.
[0039] The white spot distribution is a distribution chart obtained by calculating the color ratios R / G and B / G from the R value (red), G value (green), and B value (blue) of each pixel of the image sensor 41 output as digital image data by the image processing unit 60, and plotting the R / G value on the horizontal axis and the B / G value on the vertical axis, and refers to the distribution range of color component information (R / G, B / G) estimated to be calculated based on a white image signal. In this embodiment, the white spot distribution in the shooting environment is denoted as W(R / G, B / G).
[0040] FIG. 4 is a diagram for explaining the white spot distribution under a natural light environment and the white spot distribution under an artificial light environment. N is a white spot distribution map under natural light conditions, and W Aare white spot distribution maps under artificial light environments. In each distribution map, the horizontal axis represents the color ratio R / G value, and the vertical axis represents the color ratio B / G value. Here, the blackbody radiation locus is indicated by BR, and the white point regions of each light source under natural light environments and artificial light environments are determined in advance according to the blackbody radiation locus BR. These are data pre-stored in the ROM in the control unit 50, and differ depending on the sensitivity characteristics of the image sensor 41, the processing contents of the image processing unit 60, etc., and are set individually for each imaging device.
[0041] In Figure 4, W N The white point regions of each light source are N1, N2, N3, and N4. A A1 and A2 are shown as the white spot areas for each light source. N (R / G, B / G) and white spot distribution W under artificial light environment A In (R / G, B / G), it is assumed that each white spot area for each light source is calculated based on the white image signal. For example, in the white spot distribution map under natural light environment, the white spot distribution W under N1 light source is N1 (R / G, B / G) is an area that can be determined to be white digital image data from the color ratio R / G and color ratio B / G of the R value (red), G value (green), and B value (blue) from each pixel of the image sensor 41 output as digital image data by the image processing unit 60 under the N1 light source in a natural light environment.
[0042] The white spot distribution setting unit 55 calculates the white spot distribution W under the natural light environment based on the artificial light source reliability α calculated in S03. N (R / G, B / G) and white spot distribution W under artificial light environment A Using (R / G, B / G), calculate the white spot distribution W(R / G, B / G) in the shooting environment using the following formula:
number
[0043] As shown in the above formula, the white spot distribution W(R / G, B / G) is the white spot distribution W under natural light environment. N(R / G, B / G) and white spot distribution W under artificial light environment A The reliability of the artificial light source is calculated by weighting the reliability of the artificial light source α calculated in step S03 using (R / G, B / G). In other words, the light source is estimated by comparing infrared light and visible light in advance, so it is easy to distinguish, for example, between the white area N1 under natural light and the white area A1 under artificial light.
[0044] In S07, the white balance setting unit 56 sets the white balance gain based on the white point distribution W(R / G, B / G) calculated by the white point distribution setting unit 55 in S06.
[0045] The white balance gains (RGain, GGain, BGain) are values that correct the color of the R value (red), G value (green), and B value (blue) (R, G, B) from each pixel of the image sensor 41, converted as digital image data by the image processing unit 60, when reflected by a white object to achromatic colors (i.e., R=G=B).
[0046] Regarding the value for setting the white balance gain, the white balance setting unit 56 sets the horizontal direction of the R value (red), G value (green), and B value (blue) of each pixel of the image sensor 41 converted as digital image data by the image processing unit 60 as the x-th order and the vertical direction as the y-th order, and calculates color component information (R x,y / G x,y ,B x,y / G x,y ) is calculated using the following formula:
number
[0047] Furthermore, based on Equation 2, the white balance gain is set by the following equation.
number
[0048] The image processing unit 60 adjusts the white balance of the digital image signal by multiplying the digital image data obtained by converting the R value (red), G value (green), and B value (blue) (R, G, B) of each pixel of the image sensor 41 by the number 3 set by the white balance setting unit 56.
[0049] As described above, the imaging device and light source estimation method described in the present invention detect infrared light that has passed through a lens optical system and accurately estimates the light source at the shooting angle of view, thereby making it possible to improve the accuracy of white balance setting. [Explanation of symbols]
[0050] 10. Imaging device 20 Camera body 21 Camera side mount 30 Photographic Lens 31 Lens mount 32 Lens optical system 41 Image sensor 42 Infrared Reflection Filter 43 Infrared Sensor 50 control section 60 Image processing section 70 Image display unit
Claims
1. In an image pickup device equipped with a detachable photographing lens having a lens optical system, an image sensor that detects light incident through the lens optical system and outputs a signal by photoelectrically converting the detected light; an infrared reflective filter that is disposed on the object side of the image sensor; an infrared sensor that is disposed between the lens optical system and the infrared reflective filter and outside the optical path of light passing through the lens optical system; a control unit; and an image processing unit that converts the signal output by the image sensor into image data and outputs the image data, an infrared reflective filter reflects infrared light from the light incident through the lens optical system, the infrared sensor detects the infrared light reflected by the infrared reflective filter, the control unit acquires the intensity of the infrared signal detected by the infrared sensor, and the control unit acquires, via the image processing unit, the intensity of the visible light signal that passes through the infrared reflective filter and is detected by the imaging element.
2. The control unit includes a light source estimation unit, The imaging device according to claim 1 , wherein the light source estimation unit compares the intensity of the infrared signal and the intensity of the visible light signal acquired by the control unit, and calculates an index of a light source in the shooting environment based on a result of the comparison.
3. The control unit includes a flicker detection unit, 3. The imaging device according to claim 2, wherein the flicker detection unit changes a readout frame rate of the imaging element and performs flicker detection on the image data output by the image processing unit when the light source index calculated by the light source estimation unit is greater than a threshold determined from the intensity of a visible light signal.
4. The control unit includes a white point distribution setting unit and a white balance setting unit, 4. The imaging device according to claim 2, wherein the white point distribution setting unit calculates a white point distribution for the image data output by the image processing unit based on the light source index calculated by the light source estimation unit, and the white balance setting unit calculates and changes a white balance setting value based on the white point distribution.
5. 1. A white balance control method for an imaging device, comprising: an infrared reflective filter reflecting infrared light from among light incident through a lens optical system; an infrared sensor disposed between the lens optical system and the infrared reflective filter and outside an optical path of light passing through the lens optical system detecting infrared light; and an imaging element detecting visible light transmitted through the infrared reflective filter, A white balance control method comprising the steps of: detecting infrared rays with an infrared sensor; detecting visible light with an imaging element; comparing an intensity of the infrared signal with an intensity of the visible light signal; calculating a light source index of the shooting environment based on a result of the comparison; performing flicker detection when the light source index is greater than a threshold determined from the intensity of the visible light; calculating a white point distribution based on the light source index; and changing a white balance setting value based on the white point distribution. End
