Vehicle imaging device, vehicle imaging method, vehicle imaging program, and moving body

Abstract

To suppress the image quality deterioration.SOLUTION: An imaging apparatus comprises: a calculation unit; a white balance correction unit; and a reverse gain correction unit. The calculation unit calculates a white balance gain for correcting a white balance and a reverse gain of the white balance gain on the basis of a first image. The white balance correction unit corrects the white balance of a photographed image on the basis of the white balance gain. The reverse gain correction unit outputs the first image subjected to reverse gain correction for multiplying the photographed image whose white balance is corrected by the reverse gain to the calculation unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to For vehicle an imaging device, For vehicle an imaging method, For vehicle an imaging program, and a moving body.

Background Art

[0002] As a method for correcting the white balance of an image, a method of correcting the white balance by feedback control is known. In feedback control, a white balance gain is calculated using an image frame whose white balance has been corrected, and the white balance of the next image frame is corrected using the calculated white balance gain (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, image quality degradation may occur due to a shift in white balance caused by a change in the color temperature of the subject.

[0005] The problem to be solved by the present disclosure is to provide an imaging device, For vehicle an imaging method, For vehicle an imaging program, and a moving body that can suppress image quality degradation. For vehicle

Means for Solving the Problems

[0006] The imaging device according to the present disclosure includes an arithmetic unit, a white balance correction unit, and an inverse gain correction unit, For vehicle and a delay processing unit, ​It includes. The arithmetic unit calculates a white balance gain for correcting the white balance and an inverse gain of the white balance gain based on the first image. The white balance correction unit corrects the white balance of the captured image based on the white balance gain. The inverse gain correction unit outputs the first image that has undergone inverse gain correction, which is multiplying the inverse gain by the captured image whose white balance has been corrected, to the arithmetic unit. The delay processing unit controls the second timing so that a first timing at which the captured image whose white balance is corrected by the white balance gain is input to the inverse gain correction unit coincides with a second timing at which the inverse gain, which is the reciprocal of the white balance gain, is input to the inverse gain correction unit. 。

Advantages of the Invention

[0007] According to the present disclosure For vehicle imaging device, For vehicle imaging method, For vehicle imaging program, and a moving body, image quality degradation can be suppressed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5A

Figure 5B

Figure 6

Embodiments for Carrying Out the Invention

[0009] With reference to the accompanying drawings, embodiments of an imaging device, an imaging method, an imaging program, and a moving body according to the present disclosure will be described below.

[0010] FIG. 1 is a diagram showing an example of the imaging device 10 of the present embodiment.

[0011] In the present embodiment, a form in which the imaging device 10 is mounted on the moving body 1 will be described as an example.

[0012] The moving body 1 is an object that can move. The moving body 1 is, for example, a vehicle, a flyable object, a robot, or the like. The flyable object is, for example, a manned aircraft or an unmanned aircraft. The unmanned aircraft is, for example, a UAV (Unmanned Aerial Vehicle), a drone, or the like. Further, the moving body 1 is, for example, a moving body that travels through a driving operation by a person or a moving body that can autonomously travel without a driving operation by a person. In the present embodiment, a case where the moving body 1 is a vehicle will be described as an example. The vehicle is, for example, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, or the like.

[0013] Note that the imaging device 10 is not limited to the form mounted on the moving body 1. The imaging device 10 may be mounted on a stationary object. The stationary object is an object fixed to the ground. The stationary object is an object that cannot move or an object in a stationary state with respect to the ground. The stationary object is, for example, a building such as a structure.

[0014] The imaging device 10 images a subject and obtains imaging image data. The imaging image data is video data or moving image data composed of a plurality of consecutive frames or still images. In the present embodiment, a case where the imaging image data is video data composed of a plurality of frames will be assumed and described. Also, each of the plurality of frames will be simply referred to as an imaging image.

[0015] FIG. 2 is an example of a hardware configuration diagram of the imaging device 10.

[0016] The imaging device 10 has a hardware configuration using a normal computer, in which a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, an I / F 11D, etc. are interconnected by a bus 11E.

[0017] The CPU 11A is an arithmetic unit that controls the imaging device 10 of the present embodiment. The ROM 11B stores programs and the like for realizing various processes by the CPU 11A. The RAM 11C stores data necessary for various processes by the CPU 11A. The I / F 11D is an interface for transmitting and receiving data.

[0018] A program for executing the information processing executed by the imaging device 10 of the present embodiment is provided by being incorporated in advance in the ROM 11B or the like. Note that the program executed by the imaging device 10 of the present embodiment may be configured to be recorded and provided on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk) in a form installable or executable on the imaging device 10.

[0019] Returning to FIG. 1, the description will be continued. The imaging device 10 includes a lens 12, an image sensor 14, and a signal processing unit 16. The image sensor 14 and the signal processing unit 16 are connected so as to be able to exchange signals and data.

[0020] The image sensor 14 obtains a captured image of a subject by imaging. In the present embodiment, the image sensor 14 includes an imaging element 18, an A / D (Analog-to-Digital) conversion unit 20, a white balance correction unit 22, an image synthesis unit 24, and an I / F (InterFace) 26. These units are communicably connected. Note that the image sensor 14 may be configured not to include the image synthesis unit 24. In the present embodiment, a case where the image sensor 14 includes the image synthesis unit 24 will be described as an example.

[0021] The imaging device 18 outputs a signal corresponding to the intensity of the light received via the lens 12. The imaging device 18 is a known photoelectric conversion device. The light receiving device is, for example, a CCD (Charge Coupled Device), or a CMOS (Complementary Metal-Oxide-Semiconductor), etc. For example, when a CMOS sensor is used as the imaging device 18, an imaging image can be obtained by accumulating the charges generated by the incident light incident through the lens on the photodiode. The brightness of the imaging image is adjusted by appropriately controlling the charge accumulation time.

[0022] The A / D conversion unit 20 converts the analog signal output from the imaging device 18 into an imaging image which is digital data.

[0023] The white balance correction unit 22 corrects the white balance of the imaging image based on the white balance gain calculated by the signal processing unit 16, which will be described in detail later. The white balance correction unit 22 multiplies the signal of each color component of the pixels included in the imaging image received from the A / D conversion unit 20 by the white balance gain received from the signal processing unit 16. The signal of the color component of the pixel is represented by, for example, the signals of the respective color components of R (red), G (green), and B, which are primary color signals. Note that the signal of the color component of the pixel is not limited to the primary color signals of RGB. In the present embodiment, the case where the signal of the color component of the pixel is represented by the signals of the respective color components of the primary color signals of RGB will be described as an example. Hereinafter, the signal of R, which is one of the color components, may be simply referred to as R, the signal of G, which is one of the color components, may be simply referred to as G, and the signal of B, which is one of the color components, may be simply referred to as B for description. By multiplying the white balance gain to each of RGB of each pixel, the white balance of the imaging image is corrected. Note that hereinafter, the correction of the white balance may be simply referred to as white balance correction for description.

[0024] The image synthesizing unit 24 synthesizes a plurality of captured images with different exposures and corrected white balance. The synthesizing process by the image synthesizing unit 24 may be referred to as high-dynamic range (HDR) synthesizing process. That is, in the present embodiment, the imaging device 18 captures a plurality of captured images with different exposures. The image synthesizing unit 24 synthesizes these captured images with different exposures and corrected white balance as a captured image for one frame. The captured image synthesized by the image synthesizing unit 24 is output to the signal processing unit 16 via the I / F 26.

[0025] The signal processing unit 16 executes various signal processes on the captured image received from the image sensor 14. The signal processing unit 16 includes an I / F 28, an AGC 30, an image processing unit 32, an I / F 34, a storage unit 36, and a white balance processing unit 38. These units are communicably connected.

[0026] The AGC (Auto Gain Control) 30 adjusts the luminance or brightness of the captured image received from the image sensor 14 via the I / F 28 and outputs it to the image processing unit 32 and the white balance processing unit 38. The image processing unit 32 executes various known image processes such as gamma correction and contrast correction on the captured image received from the AGC 30. The image processing unit 32 outputs the image-processed captured image to the outside of the imaging device 10 via the I / F 34.

[0027] For example, the captured image output from the imaging device 10 is output to the power control unit of the moving body 1 or the like. The power control unit of the moving body 1 performs vehicle control, for example, using the captured image output from the imaging device 10, to avoid obstacles, maintain the current driving lane, and keep the inter-vehicle distance from the vehicle ahead at a predetermined distance or more.

[0028] The white balance processing unit 38 executes processes related to the calculation of the white balance gain.

[0029] The white balance processing unit 38 includes an inverse gain correction unit 40, an arithmetic unit 42, and a delay processing unit 48. The arithmetic unit 42 includes a detection unit 44 and a white balance calculation unit 46. These units are communicably connected to each other.

[0030] The inverse gain correction unit 40 receives the imaging image with white balance corrected from the white balance correction unit 22 via the AGC 30, the I / F 28, the I / F 26, and the image composition unit 24. The inverse gain correction unit 40 performs inverse gain correction on the imaging image with white balance corrected to generate a first image.

[0031] The inverse gain is the reciprocal of the white balance gain for correcting the white balance. The inverse gain correction means multiplying the imaging image with white balance corrected by the inverse gain. Specifically, the inverse gain correction unit 40 multiplies each of the RGB, which is the signal of each color component of the pixel included in the imaging image with white balance corrected, by the inverse gain calculated by the arithmetic unit 42 described later. By multiplying the inverse gain to each of the RGB of each pixel, the imaging image before white balance correction is generated. That is, the first image is the imaging image that has been returned to the white balance before white balance correction by the inverse gain correction unit 40 after the white balance has been corrected.

[0032] Based on the first image received from the inverse gain correction unit 40, the arithmetic unit 42 calculates the white balance gain for correcting the white balance and the inverse gain of the white balance gain. That is, the arithmetic unit 42 calculates the white balance gain and the inverse gain using the first image, which is the imaging image before white balance correction.

[0033] The arithmetic unit 42 includes a detection unit 44 and a white balance calculation unit 46.

[0034] The detection unit 44 detects the in-frame area in the first image that has been subjected to white balance correction by the inverse gain correction unit 40 before white balance correction. The in-frame area is the area of the color temperature within the white detection frame arranged in the color temperature space. The color temperature is a scale for quantitatively expressing the color of light emitted by various light sources as a numerical value, and takes a value correlated with the temperature of a black body. The unit of color temperature is K (Kelvin). The color temperature is represented by, for example, color difference information. The white detection frame will be described later.

[0035] FIG. 3A is an explanatory diagram of an example of white balance correction in the present embodiment. The color temperature space S is a multi-dimensional space representing the color temperature. The color temperature space S is represented by, for example, a multi-dimensional space with color difference information as the axis. In FIG. 3A, the color temperature space S with R / G and B / G, which are color difference information, as the axes is shown as an example. R / G indicates the ratio of R (red) to G (green). B / G indicates the ratio of B (blue) to G. Note that the color difference information of the coordinate axes representing the color temperature space S is not limited to the RGB ratio. For example, the color difference information may be the RGB signal difference such as the R - G signal and the B - G signal, or the color difference signal represented by R - Y and B - Y using the luminance signal Y obtained by adding RGB at a specific ratio.

[0036] The detection unit 44 sets the white detection frame 50 in the color temperature space S. The white detection frame 50 is a frame representing the correction target area of the white balance in the color temperature space S. In other words, the white detection frame 50 is a frame for restricting the range of the color temperature for which the white balance gain is calculated. The set value of the white detection frame 50 may be stored in advance in the storage unit 36, for example. The detection unit 44 may read the set value of the white detection frame 50 from the storage unit 36 and set it in the color temperature space S. Note that the set value of the white detection frame 50 may be made changeable according to an operation instruction by the user or the like.

[0037] The white detection frame 50 is set in a region along the blackbody radiation curve L in the color temperature space S. This is because the white detection frame 50 is a frame used to determine a white component that should bring a color temperature close to the blackbody radiation curve L closer to achromatic white. In the case of the color temperature space S with R / G and B / G as coordinate axes, the blackbody radiation curve L is plotted in a quadratic curve shape from the low color temperature side to the high color temperature side as shown in FIG. 3A.

[0038] In FIG. 3A, a rectangular white detection frame 50 is shown as an example. However, the white detection frame 50 is not limited to a rectangular shape such as a square.

[0039] The detection unit 44 detects a within-frame region in the first image where the color temperature is within the white detection frame 50. In this embodiment, the case where the detection unit 44 detects pixels in the first image where the color temperature is within the white detection frame 50 as the within-frame region will be described as an example.

[0040] Here, the color temperature of white in the captured image obtained by the imaging device 18 moves along the blackbody radiation curve L in the color temperature space S according to the change in the color temperature of the light source. Due to the color temperature moving along the blackbody radiation curve L, even an object that is actually white may be captured as a colored white in the captured image. For example, when the color temperature of the light source is low, a region that is originally white is captured as a reddish color. Also, when the color temperature of the light source is high, a region that is originally white is captured as a bluish color.

[0041] Therefore, the arithmetic unit 42 calculates a white balance gain for correcting the white balance so that the color temperature of the captured image captured with a reddish or bluish tint becomes achromatic white. In FIG. 3A, the target point P, which is the origin, becomes achromatic white. In other words, the target point P is the coordinate value obtained by converting the RGB luminance ratio targeted for calculating the white balance gain into color difference information.

[0042] First, the detection unit 44 of the arithmetic unit 42 divides the first image, which has been subjected to white balance correction by the inverse gain correction unit 40, into a plurality of areas. Then, the detection unit 44 calculates the integrated value of each of RGB for each area. The area may be, for example, an area of one pixel, or may be an area composed of a plurality of pixels. In the present embodiment, the case where each area corresponds to one pixel will be described as an example. The detection unit 44 converts the signal of each RGB color component of the pixel into color difference information using the calculated integrated value for each pixel. As described above, the color difference information is represented by, for example, R / G and B / G.

[0043] Next, the detection unit 44 plots the color difference information obtained for each pixel on the color temperature space S. In FIG. 3A, plots 601, 602, 603, 604, and 605 are shown as an example. These plots 601 to 605 are an example of plots of pixels with different color difference information in the first image.

[0044] Then, the detection unit 44 detects the pixels whose color difference information is within the white detection frame 50, thereby detecting the pixels in the first image whose color temperature is within the frame area of the white detection frame 50. In the example shown in FIG. 3A, the detection unit 44 detects plots 601 to 603, which are within the white detection frame 50 among plots 601 to 605, as the pixels of the color difference information to be used for calculating the white balance gain. On the other hand, for plots 604 and 605, since they are located outside the white detection frame 50, they are not subject to the calculation of the white balance gain.

[0045] The detection unit 44 integrates the detected pixels, that is, the luminance values of RGB for each pixel whose color difference information is within the white detection frame 50, for each of RGB, and outputs them to the white balance calculation unit 46. That is, the detection unit 44 calculates the integrated value of the luminance values for each of RGB for each pixel whose color difference information is within the white detection frame 50 in the first image, which is the captured image before white balance correction. Then, the detection unit 44 outputs the calculated integrated value of the luminance values for each of RGB to the white balance calculation unit 46. The integrated value of the luminance values for each of RGB corresponds to, for example, the current point C in FIG. 3A. The current point C is obtained by converting the integrated value obtained by integrating the luminance values of RGB within the white detection frame 50 in the first image for each of RGB into color difference information and plotting it on the color temperature space S. Note that the current point C' described later is obtained by converting the integrated value obtained by integrating the luminance values of RGB within the white detection frame 50 in the captured image after white balance correction for each of RGB into color difference information and plotting it on the color temperature space S.

[0046] Returning to FIG. 1, the description will be continued. The white balance calculation unit 46 calculates a white balance gain and an inverse gain of the white balance gain based on the color temperature of the in-frame region, which is the region within the white detection frame 50 in the first image. Specifically, the white balance calculation unit 46 calculates the white balance gain using the integrated value of the luminance values for each of RGB for each pixel whose color difference information is within the white detection frame 50 received from the detection unit 44.

[0047] For example, as shown in FIG. 3A, assume that the current point C of the color temperature within the white detection frame 50 of the first image, which is the captured image before white balance correction, is at the position of the low color temperature plot 60A. As described above, the current point C is obtained by converting the integrated value of the luminance values for each of RGB within the white detection frame 50 of the first image output from the detection unit 44 to the white balance calculation unit 46 into color difference information and plotting it on the color temperature space S. The white balance calculation unit 46 calculates a white balance gain for making the luminance ratio of the current point C approach the luminance ratio of the target point P.

[0048] For example, assume that the current point C is on the lower color temperature side compared to the target point P. Specifically, assume that the current point C is at the position of the plot 60A with a low color temperature. In this case, for the R gain and B gain to make the luminance ratio of the current point C approach that of the target point P, B gain ≥ 1 and R gain ≤ 1 respectively. Here, the R gain means the white balance gain of R, and the B gain means the white balance gain of B. In this embodiment, as an example, the case where the unit of these white balance gains is expressed as a magnification will be described. Note that the unit of the white balance gain is not limited to magnification.

[0049] On the other hand, assume that the current point C is on the higher color temperature side compared to the target point P. Specifically, assume that the current point C is at the position of the plot 62A with a high color temperature. In this case, for the R gain and B gain to make the luminance ratio of the current point C approach that of the target point P, B gain ≤ 1 and R gain ≥ 1 respectively.

[0050] Also, the white balance calculation unit 46 calculates the inverse gain by calculating the reciprocal of the calculated white balance gain. For example, assume that the white balance gain is a value represented by the following equations (1) and (2).

[0051] R_gain(t - 1)=1 / 1.2 ·· Equation (1) B_gain(t - 1)=1 / 0.8 ·· Equation (2)

[0052] In this case, the inverse gains of these white balance gains are values represented by the following equations (3) and (4).

[0053] R_gain_inverse(t)=1 / (1 / 1.2)=1.2 ·· Equation (3) B_gain_inverse(t)=1 / (1 / 0.8)=0.8 ·· Equation (4)

[0054] In the above formula (1), R_gain(t - 1) means the white balance gain of R calculated from the first image. In the above formula (2), B_gain(t - 1) means the white balance gain of B calculated from the first image. R_gain_inverse(t) in formula (3) means the inverse gain of R. B_gain_inverse(t) in formula (4) means the inverse gain of B.

[0055] Returning to FIG. 1 to continue the description. The white balance calculation unit 46 outputs the calculated white balance gain to the white balance correction unit 22 and outputs the calculated inverse gain to the delay processing unit 48.

[0056] The white balance correction unit 22 receives the white balance gain from the white balance calculation unit 46 via the I / F26 and I / F28. The white balance correction unit 22 corrects the white balance of the captured image of the next newly captured frame using the received white balance gain and outputs it to the image synthesis unit 24.

[0057] Therefore, in the imaging device 10 of the present embodiment, white balance correction by feedback control is performed. That is, the white balance correction unit 22 is arranged in front of the white balance processing unit 38 including the white balance calculation unit 46. For this reason, the imaging device 10 is configured to have a white balance correction function by a feedback control method.

[0058] On the other hand, the delay processing unit 48 that has received the inverse gain from the white balance calculation unit 46 controls the second timing so that the first timing and the second timing match. The first timing is the timing at which the captured image of the frame whose white balance has been corrected is input to the inverse gain correction unit 40. The captured image of this frame is the captured image that has been white balance corrected by the white balance correction unit 22 using the white balance gain calculated by the white balance calculation unit 46. The second timing is the timing at which the inverse gain, which is the reciprocal of the white balance gain, is input to the inverse gain correction unit 40.

[0059] Specifically, for example, the delay processing unit 48 holds the inverse gain received from the white balance calculation unit 46 for the exposure time of the image sensor 14, that is, for the time of one frame, and then outputs it to the inverse gain correction unit 40. Note that the delay processing unit 48 only needs to control the second timing so that the first timing and the second timing match, and the time for which the delay processing unit 48 holds the inverse gain varies depending on the configuration of the image sensor 14 and the like. Therefore, it is not limited to the form of holding for the time of one frame.

[0060] That is, the inverse gain correction unit 40 receives the captured image of the frame whose white balance has been corrected by the white balance correction unit 22 and the inverse gain of the white balance gain used for the white balance correction of the captured image of the frame at the same timing. Therefore, the inverse gain correction unit 40 can correct the captured image whose white balance has been corrected to the first image that is the captured image before white balance correction and output it to the arithmetic unit 42. As described above, the arithmetic unit 42 calculates the white balance gain and the inverse gain using the first image.

[0061] In this way, in the imaging device 10 of the present embodiment, the white balance gain is calculated using the first image that is the captured image before white balance correction by the inverse gain correction unit 40, rather than the captured image whose white balance has been corrected.

[0062] Therefore, in the imaging device 10 of the present embodiment, in an apparatus that performs white balance correction by feedback control, it is possible to suppress image quality degradation caused by, for example, a shift in white balance due to a change in the color temperature of a subject.

[0063] The above effect will be specifically described with reference to FIG. 3A.

[0064] For example, as shown in FIG. 3A, assume that the current point C of the captured image before white balance correction is at the position of the plot 60A with a low color temperature. In this case, by multiplying the white balance gain by the white balance correction unit 22, the current point C is corrected to the current point C' at the position of the plot 60B which is the target point P.

[0065] Here, the white balance gain used for this white balance correction is calculated using the first image of the captured image of the previous frame by the feedback control of the white balance processing unit 38 and input to the white balance correction unit 22. In FIG. 3A, Bgain(t - 1) represents the white balance gain of the color component B calculated from the first image of the captured image of the previous frame. Also, Rgain(t - 1) represents the white balance gain of the color component R calculated from the first image of the captured image of the previous frame.

[0066] In the imaging device 10 of the present embodiment, white balance correction by feedback control is performed. Also, in the imaging device 10 of the present embodiment, the white balance gain is calculated using the first image which is the captured image before white balance correction by the inverse gain correction unit 40 instead of the captured image with white balance corrected. That is, on the color temperature space S, it acts so that the current point C' is re - corrected from the target point P to the current point C before white balance correction. And in the present embodiment 10, the white balance gain is calculated using the integral value corresponding to the color difference information of the current point C before white balance correction.

[0067] Therefore, in the imaging device 10 of the present embodiment, when calculating the white balance gain to be used in the captured image of the next frame, the color difference information at the position of plot 60A that has been returned to before correction by inverse gain correction is used instead of plot 60B which is the plot after white balance correction, and the white balance gain is calculated. In other words, the white balance calculation unit 46 calculates the white balance gain using plot 60A which is the current point C of the first image that is the captured image before white balance correction, instead of plot 60B which is the current point C' of the captured image after white balance correction.

[0068] Therefore, in the imaging device 10 of the present embodiment, even when the color temperature between frames changes greatly on the color temperature space S, it is possible to suppress the current point C that should originally enter the white detection frame 50 from deviating from the white detection frame 50.

[0069] Thus, in the imaging device 10 of the present embodiment, it is possible to suppress image quality degradation due to white balance deviation.

[0070] Here, assume a case where the color temperature of the subject changes greatly between frames. Scenes where the color temperature of the subject changes greatly between frames are, for example, a scene where the moving body 1 equipped with the imaging device 10 is traveling near the entrance and exit of a tunnel, a scene where strong sunlight such as morning sun or evening sun suddenly enters the lens 12, a scene where rapid weather changes occur, and the like.

[0071] For example, assume a scenario where in the captured image of the previous frame, the current point C was at the position of the low color temperature plot 60A, but in the captured image of this frame, the current point C is at the position of the high color temperature plot 62A. In this case, the current point C' of the captured image of this frame after white balance correction, which is multiplied by the white balance gain calculated from the captured image of the previous frame, will shift to the position of plot 62B. Therefore, in the captured image before white balance correction, plot 62A, which is the current point C within the white detection frame 50, will be located at plot 62B outside the white detection frame 50 in the captured image after white balance correction. For this reason, when calculating the white balance gain using plot 62B, which is the current point C' of the captured image after this white balance correction, the image quality may deteriorate.

[0072] On the other hand, in the imaging device 10 of the present embodiment, instead of using the captured image with white balance corrected, the white balance gain is calculated using the first image that has been made into the captured image before white balance correction by the inverse gain correction unit 40. For this reason, in the imaging device 10 of the present embodiment, the white balance gain can be calculated using the color difference information of plot 62A, which is the current point C before white balance correction, instead of the color difference information of plot 62B, which is the current point C' after white balance correction. Therefore, even when the color temperature of the subject changes significantly between frames, the imaging device 10 of the present embodiment can calculate the white balance gain with high accuracy.

[0073] Also, similarly, assume a scenario where in the captured image of the previous frame, the current point C was at the position of plot 62A, but in the captured image of this frame, the current point C has shifted to the position of plot 64A with the color temperature of the blue sky. And, for example, assume that the white balance gain calculated from the captured image of the previous frame was the white balance gain for correcting the color temperature at the position of plot 62A to the position of the target point P. In this case, in the captured image before white balance correction, plot 64A, which is the current point C of this frame and was outside the white detection frame 50, will be positioned at plot 64B within the white detection frame 50 by being white balance corrected by this white balance gain. Therefore, when calculating the white balance gain using plot 64B, which is the current point C' of the captured image after this white balance correction, the image quality may deteriorate. This is because in the captured image before white balance correction, a plot that was outside the frame of the white detection frame 50 will be treated as an object for calculating the white balance gain.

[0074] On the other hand, in the imaging device 10 of the present embodiment, instead of using the captured image with white balance corrected, the white balance gain is calculated using the first image that has been made into the captured image before white balance correction by the inverse gain correction unit 40. Therefore, in the imaging device 10 of the present embodiment, a plot that was located outside the frame of the white detection frame 50, such as plot 64A, before white balance correction can be made into a plot that is not an object for calculating the white balance gain. For this reason, even when the color temperature of the subject changes significantly between frames, the imaging device 10 of the present embodiment can calculate the white balance gain with high accuracy.

[0075] As described above, in the imaging device 10 of the present embodiment, even when the color temperature of the subject changes significantly, it is possible to suppress the current point C, which is the integrated value of the RGB luminance values in the region to be white balance calculated, from shifting from the inside to the outside or from the outside to the inside of the preset white detection frame 50. Therefore, in the imaging device 10 of the present embodiment, even when the color temperature of the subject changes significantly, an accurate white balance gain can be calculated.

[0076] FIG. 3B is an explanatory diagram showing an example of changes in the RGB luminance values due to white balance correction. As shown in FIG. 3B, for example, it is assumed that the average value of the RGB luminance values at the current point C of the captured image is corrected to approximately the same value by white balance correction. In the present embodiment, the white balance gain is calculated after returning the average luminance value for each RGB to the state before white balance correction by inverse gain correction.

[0077] As described above, the calculation unit 42 of the present embodiment calculates the white balance gain based on the color difference information within the white detection frame 50 in the first image, which is the captured image before white balance correction rather than the captured image after white balance correction.

[0078] Therefore, in the imaging device 10 of the present embodiment, it is possible to suppress image quality degradation caused by, for example, a shift in white balance due to a change in the color temperature of the subject.

[0079] On the other hand, in the prior art, the white balance gain was calculated using the color difference information of the captured image after white balance correction. Therefore, in the prior art, image quality degradation may have occurred.

[0080] FIG. 4 is an explanatory diagram showing an example of conventional white balance correction. In the conventional white balance correction by feedback control, the white balance gain was calculated using the captured image that had been white balance corrected.

[0081] For example, assume that the color temperature of the subject changes significantly between frames. For example, assume a scenario where in the captured image of the previous frame, the current point C was at the position of the low color temperature plot 60A, but in the captured image of the current frame, the current point C is at the position of the high color temperature plot 62A. Specifically, assume a case where the color temperature of the subject at a certain time changes significantly from a low color temperature such as a road surface at dusk to a high color temperature such as an indoor white light source or a shaded road surface.

[0082] In this case, the current point C' of the captured image of the current frame after white balance correction, which is multiplied by the white balance gain calculated from the captured image of the previous frame, will shift from the position of plot 62A to the position of plot 62B. For this reason, in the captured image before white balance correction, the current point C that was within the white detection frame 50 will be outside the white detection frame 50 in the current point C' of the captured image after white balance correction. That is, the current point C' obtained from the color difference information of the frame immediately after the color temperature change will be outside the white detection frame 50. In this case, therefore, in the prior art, it was difficult to correctly calculate the white balance gain, and image quality degradation might occur.

[0083] Also, in the prior art, such image quality degradation occurred particularly significantly when the color temperature of the subject changed greatly between frames.

[0084] That is, in white balance correction by feedback control, the white balance gain is calculated from the captured image of the previous frame. In the prior art, since the white balance gain to be used in the captured image of the next frame is calculated using the captured image after white balance correction, significant image quality degradation might occur particularly in an environment where the color temperature of the light source of the subject changes rapidly.

[0085] Also, as a conventional technique for performing white balance correction by feedback control, a technique of expanding the white detection frame 50 in response to a change in the color temperature of a light source is disclosed. However, even when the white detection frame 50 is expanded in response to a change in the color temperature of the light source, it has been difficult to suppress image quality degradation.

[0086] FIGS. 5A and 5B are explanatory diagrams of a conventional technique in which the white detection frame 50 is expanded and used.

[0087] Also in this conventional technique, similar to the above-described conventional technique, the imaging image of the next frame was corrected using the white balance gain calculated from the imaging image corrected for white balance. Here, due to white balance correction, the plot of the color difference information after white balance correction may deviate from the reference white detection frame 50. In this case, in the conventional technique, the white detection frame 50 was expanded so as to include the plot of the color difference information after correction that was within the white detection frame 50 before white balance correction.

[0088] Specifically, this will be described with reference to FIG. 5A. For example, assume that in the imaging image of the previous frame, the current point C' was at the position of the low color temperature plot 60A. And assume that a white balance gain for correcting the current point C' of the low color temperature plot 60A to the target point P was calculated. Then, assume a scene where the current point C of the imaging image of the next frame is at the position of the high color temperature plot 62A. In this case, the current point C' of the imaging image of this frame after white balance correction, which is multiplied by the white balance gain calculated from the imaging image of the previous frame, will shift from the position of the plot 62A to the position of the plot 62B.

[0089] In this case, in the prior art, the white detection frame 50 was enlarged to the white detection frame 52 so as to include the plot 62B of the corrected current point C' corresponding to the plot 62A of the current point C that was within the white detection frame 50 before correction. Then, in the prior art, using the captured image after white balance correction and the enlarged white detection frame 52, the white balance gain for use in the captured image of the next frame was further calculated. For this reason, the white balance gain calculated using the plots within the enlarged white detection frame 52 is multiplied to the captured image of the next frame. For this reason, in the prior art, when the white balance correction by feedback control is repeated, there may be a mismatch between the range of the preset white detection frame 50 and the color temperature of the captured image used as the calculation target of the white balance gain.

[0090] Specifically, as shown in FIG. 5B, assume a case where the current point C of the captured image before white balance correction is at the position of the plot 64B outside the enlarged white detection frame 52. Also assume a case where the current point C' of the captured image after white balance correction of the previous frame is at the position of the plot 62B. In this case, the white balance gain is a value for bringing the color difference information at the position of the plot 62B closer to the target point P. For this reason, the captured image where the current point C is at the position of the plot 64B is corrected by white balance correction so as to become the current point C' at the position of the plot 64C within the expanded white detection frame 52.

[0091] In this case, further, when calculating the white balance gain for use in the captured image of the next frame using this current point C' of the captured image after white balance correction, the color difference information of the plot that should not originally be included in the calculation target of the white balance gain may enter within the white detection frame 52 and be used as the calculation target of the white balance gain. Similarly, in this prior art, the color difference information of the plot that should originally be included in the calculation target of the white balance may not be included within the white detection frame 52.

[0092] That is, in this prior art, for the captured image of the frame after white balance correction using the color temperature after white balance correction of the captured image captured in an environment of a light source that does not change over time, it is considered that the white balance gain converges. For example, it is considered that the white balance gain converges within several to several tens of frames. However, when the color temperature of the light source changes significantly, there may be a case where the white area that should originally be within the preset white detection frame 50 is not included within the white detection frame 52. For this reason, in the prior art, it may be difficult to calculate an appropriate white balance gain.

[0093] For this reason, also in the prior art of expanding and correcting the white detection frame 50 in white balance correction by feedback control, it may be difficult to perform an appropriate white balance calculation, and image quality degradation may occur.

[0094] On the other hand, in the imaging device 10 of the present embodiment, the inverse gain correction unit 40 calculates the white balance gain using the first image that is the captured image before white balance correction. For this reason, in the imaging device 10 of the present embodiment, even when the color temperature between frames changes significantly on the color temperature space S, it is possible to suppress the current point C that should not originally be within the white detection frame 50 from entering the white detection frame 50. Also, in the imaging device 10 of the present embodiment, it is possible to suppress the current point C that should originally be within the white detection frame 50 from deviating from the white detection frame 50. For this reason, in the imaging device 10 of the present embodiment, the white balance gain can be calculated accurately, and image quality degradation can be suppressed.

[0095] In addition, the color temperature of the subject varies significantly depending on the imaging environment and brightness. For this reason, conventionally, the setting of the white detection frame 50 has been changed according to the imaging environment. For example, the range of the white detection frame 50 is made variable in conjunction with AE (AUTO EXPOSURE). Specifically, when the illuminance is high during the day, sunlight becomes the main light source, so the white detection frame 50 is set to be narrow. On the other hand, at night, since there are various light sources including fluorescent lights, the white detection frame 50 is set to be wide. Setting the white detection frame 50 to be narrow means setting the white detection frame 50 in a region closer to the blackbody radiation curve L on the color temperature space S. Setting the white detection frame 50 to be wide means expanding the white detection frame 50 to a region farther from the blackbody radiation curve L in the color temperature space S.

[0096] Even in the case where the range of the white detection frame 50 is made variable in conjunction with AE in this way, by adopting a configuration including the white balance processing unit 38 including the inverse gain correction unit 40 of the present embodiment, white balance correction following the change in the color temperature of the subject becomes possible.

[0097] Next, the flow of information processing executed by the imaging device 10 of the present embodiment will be described.

[0098] FIG. 6 is a flowchart showing an example of information processing executed by the imaging device 10 of the present embodiment.

[0099] The imaging element 18 acquires an analog signal of the captured image (step S100). The A / D conversion unit 20 converts the analog signal output from the imaging element 18 into a captured image which is digital data (step S102).

[0100] The white balance correction unit 22 acquires the white balance gain calculated from the previous frame in step S124 described later from the white balance calculation unit 46 (step S104). The white balance correction unit 22 corrects the white balance of the captured image acquired in step S100 and A / D converted in step S102 using the white balance gain acquired in step S104 (step S106).

[0101] The image composition unit 24 composes a plurality of captured images with different exposures that have been white balance corrected in step S106 (step S108). The AGC 30 adjusts the brightness or luminance of the captured image composed in step S108 and outputs it to the image processing unit 32 and the white balance processing unit 38 (step S110). The image processing unit 32 performs various known image processes such as γ correction and contrast correction on the captured image received from the AGC 30 (step S112) and outputs it to the outside of the imaging device 10 (step S114). Note that the composition process of the captured image in step S108 is only necessary when multiple captures are involved, such as with an HDR-compatible sensor. In the case of a configuration where multiple captures are not performed, the process of step S108 is unnecessary.

[0102] On the other hand, the inverse gain correction unit 40 acquires the inverse gain calculated in step S124 described later from the delay processing unit 48 (step S116). The inverse gain correction unit 40 performs inverse gain correction by multiplying the inverse gain acquired in step S116 by the captured image after white balance correction that has been auto gain controlled in step S110 (step S118). Through the process of step S118, a first image is generated that returns the captured image white balance corrected in step S106 to the captured image before white balance correction.

[0103] The detection unit 44 detects the pixels in the in-frame area within the white detection frame 50 in the first image that has been subjected to inverse gain correction by the inverse gain correction unit 40 and is the captured image before white balance correction (step S120). That is, the detection unit 44 detects the pixels in the first image whose color difference information is within the white detection frame 50. Then, the detection unit 44 calculates the integrated value of the luminance values for each of the RGB values of each of the detected pixels (step S122).

[0104] The white balance calculation unit 46 calculates the white balance gain and the inverse gain of the white balance gain based on the color temperature of the in-frame area, which is the area within the white detection frame 50 in the first image generated in step S118 (step S124). That is, the white balance calculation unit 46 calculates the white balance gain and the inverse gain using the integrated values of the luminance values for each of the RGB values calculated in step S122.

[0105] Then, the white balance calculation unit 46 outputs the white balance gain calculated in step S124 to the white balance correction unit 22. Also, the white balance calculation unit 46 outputs the inverse gain calculated in step S124 to the inverse gain correction unit 40 via the delay processing unit 48 (step S126).

[0106] Next, the imaging device 10 determines whether imaging has ended (step S128). For example, the imaging device 10 executes the determination in step S128 by determining whether a signal indicating the end of imaging that has been predetermined has been received. If the determination in step S128 is negative (step S128: No), the process returns to step S100. If the determination in step S128 is positive (step S128: Yes), this routine ends.

[0107] As described above, the imaging device 10 of the present embodiment includes an arithmetic unit 42, a white balance correction unit 22, and an inverse gain correction unit 40. The arithmetic unit 42 calculates a white balance gain for correcting the white balance and an inverse gain of the white balance gain based on the first image. The white balance correction unit 22 corrects the white balance of the captured image based on the white balance gain. The inverse gain correction unit 40 outputs to the arithmetic unit 42 a first image obtained by performing inverse gain correction in which an inverse gain is multiplied by the captured image whose white balance has been corrected.

[0108] As described above, in the imaging device 10 of the present embodiment, in the white balance correction by feedback control, the white balance gain is calculated based on the first image that has been made into the captured image before white balance correction by inverse gain correction, rather than the captured image after white balance correction.

[0109] Therefore, in the imaging device 10 of the present embodiment, image quality degradation can be suppressed.

[0110] As described with reference to FIG. 1, in the present embodiment, an example of the configuration in which the image synthesis unit 24 is disposed between the white balance correction unit 22 and the inverse gain correction unit 40 in the data output direction is shown. The data output direction is the direction of arrow X in FIG. 1.

[0111] However, the image synthesis unit 24 may be disposed on the downstream side of the white balance correction unit 22 in the data output direction. For example, the image synthesis unit 24 may be disposed at a position A between the AGC 30 and the inverse gain correction unit 40, or at a position B between the inverse gain correction unit 40 and the image processing unit 32, etc. in the data output direction. Note that the image synthesis process is required only when multiple captures are involved, such as in an HDR-compatible sensor. Therefore, in the case of a configuration that does not perform multiple captures, the image synthesis unit 24 is unnecessary.

[0112] Here, an HDR image sensor, which is an example of the image composition unit 24, can capture high-quality video compared to a conventional single-exposure image sensor. However, since the HDR image sensor performs unique image processing for multiple exposures, the linearity of brightness and chroma may not be guaranteed. For example, when processing such as a brightness offset is performed, the linearity may be disrupted. For example, assume that the brightness levels of RGB are R:G:B = 20:30:10. If +20 is added to each of RGB as a brightness offset to this brightness level, the brightness levels of RGB will be R:G:B = 40:60:30. Therefore, the ratio of RGB may change from the original state, and it may become impossible to correctly calculate the white balance gain.

[0113] Therefore, when adopting a configuration provided with the image composition unit 24, it is preferable that the image composition unit 24 is arranged on the downstream side in the data output direction from the white balance correction unit 22. That is, it is preferable that the white balance correction unit 22 is arranged in the previous stage from the image composition unit 24. By arranging the white balance correction unit 22 in the previous stage from the image composition unit 24, the white balance correction unit 22 can perform white balance correction on the captured image in which the linearity of the brightness values of RGB is maintained.

[0114] Also, in the imaging device 10 having a configuration in which the white balance correction unit 22 is arranged in the previous stage from the white balance processing unit 38, assume a configuration in which the position of the image composition unit 24 in the image sensor 14 is in the subsequent stage from the white balance correction unit, or the position of the image composition unit 24 in the signal processing unit 16 is arranged at position A (see FIG. 1). In the imaging device 10 configured in this way, it is necessary to perform white balance correction by feedback control instead of feedforward control. In the imaging device 10 of the present embodiment, in the white balance correction by feedback control, the white balance gain is calculated using the first image that is the captured image before white balance correction.

[0115] Therefore, even when the imaging device 10 of the present embodiment has a configuration including the image synthesis unit 24, image quality degradation can be suppressed.

[0116] Note that the above describes the embodiments, but the above embodiments are presented as examples and are not intended to limit the scope of the invention. For example, the imaging device 10 of the present embodiment may have a configuration that does not include the image synthesis unit 24. Specifically, when using an image sensor 14 that does not support HDR, the imaging device 10 may have a configuration that does not include the image synthesis unit 24. Further, the above novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The above embodiments are included in the scope or gist of the invention and are included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0117] 1 Moving body 10 Imaging device 22 White balance correction unit 24 Image synthesis unit 40 Inverse gain correction unit 42 Arithmetic unit 44 Detection unit 46 White balance calculation unit 48 Delay processing unit

Claims

1. An arithmetic unit that calculates a white balance gain for correcting the white balance based on the first image and an inverse gain of the white balance gain; A white balance correction unit that corrects the white balance of the captured image based on the white balance gain; An inverse gain correction unit that outputs the first image that has undergone inverse gain correction in which the inverse gain is multiplied by the captured image whose white balance has been corrected to the arithmetic unit; A delay processing unit that controls the second timing so that the first timing at which the captured image whose white balance has been corrected by the white balance gain is input to the inverse gain correction unit and the second timing at which the inverse gain that is the reciprocal of the white balance gain is input to the inverse gain correction unit match; A vehicle imaging device comprising the above.

2. The delay processing unit outputs the inverse gain to the inverse gain correction unit after holding the inverse gain for a time corresponding to one frame. The vehicle imaging device according to Claim 1.

3. The arithmetic unit includes a detection unit that detects a region within a frame, which is a region of color temperature within a white detection frame representing a white balance correction target region in the color temperature space in the first image; a white balance calculation unit that calculates the white balance gain and the inverse gain of the white balance gain based on the color temperature of the region within the frame in the first image; The vehicle imaging device according to Claim 1.

4. The detection unit calculates an integrated value of luminance values for each of a plurality of color components of each pixel in the region within the frame in the first image; The white balance calculation unit calculates the white balance gain and the inverse gain using the integrated value. The vehicle imaging device according to Claim 3.

5. A composition unit that composes a plurality of the captured images whose white balances have been corrected and that have different exposures; The vehicle imaging device according to any one of Claims 1 to 3.

6. An arithmetic step of calculating a white balance gain for correcting the white balance based on the first image and an inverse gain of the white balance gain; A white balance correction step of correcting the white balance of the captured image based on the white balance gain; ​ An inverse gain correction step of outputting the first image that has undergone inverse gain correction by multiplying the captured image with the white balance corrected by the inverse gain; A delay processing step of controlling the second timing such that the first timing at which the captured image whose white balance is corrected by the white balance gain is input to the inverse gain correction step matches the second timing at which the inverse gain, which is the reciprocal of the white balance gain, is input to the inverse gain correction step; A vehicle imaging method including the above.

7. In the delay processing step, the inverse gain is output to the inverse gain correction step after being held for a time corresponding to one frame. The vehicle imaging method according to claim 6.

8. An arithmetic step of calculating a white balance gain for correcting the white balance based on the first image and an inverse gain of the white balance gain; A white balance correction step of correcting the white balance of the captured image based on the white balance gain; An inverse gain correction step of outputting the first image that has undergone inverse gain correction by multiplying the captured image with the white balance corrected by the inverse gain; A delay processing step of controlling the second timing such that the first timing at which the captured image whose white balance is corrected by the white balance gain is input to the inverse gain correction step matches the second timing at which the inverse gain, which is the reciprocal of the white balance gain, is input to the inverse gain correction step; A vehicle imaging program for causing a computer to execute the above.

9. In the delay processing step, the inverse gain is output to the inverse gain correction step after being held for a time corresponding to one frame. The vehicle imaging program according to claim 8.

10. An arithmetic unit that calculates a white balance gain for correcting the white balance based on the first image and an inverse gain of the white balance gain; A white balance correction unit that corrects the white balance of the captured image based on the white balance gain; An inverse gain correction unit that outputs the first image that has undergone inverse gain correction by multiplying the captured image with the white balance corrected by the inverse gain to the arithmetic unit; A delay processing unit that controls the second timing so that a first timing at which the captured image whose white balance is corrected by the white balance gain is input to the inverse gain correction unit and a second timing at which the inverse gain, which is the reciprocal of the white balance gain, is input to the inverse gain correction unit match. A mobile body comprising the same. According to claim 11, the delay processing unit outputs the inverse gain to the inverse gain correction unit after holding the inverse gain for a time corresponding to one frame. The mobile body according to claim 10.

Images