Image processing apparatus, image processing method, and computer program

Abstract

To provide an image processing apparatus capable of correcting black balance suitable for white balance and the like.SOLUTION: An image processing apparatus is provided, including detection means that detects a black balance deviation in an input image, acquisition means that acquires at least one of information on white balance or information on static black balance to be applied to an input image, calculation means that calculates a correction value for correcting the black balance deviation on the basis of at least one of the information on white balance or information on static black balance when the detection means detects a black balance deviation, and adjustment means that adjusts the black balance deviation in accordance with the correction value calculated by the calculation means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a computer program, and the like. [Background technology]

[0002] Conventionally, imaging devices are known that perform white balance (WB) control according to the light source of the shooting environment and output a color image, and imaging devices are also known that correct black balance (BB) according to the characteristics of the imaging sensor.

[0003] For example, Patent Document 1 discloses a technique for storing table data that associates offset values ​​of a luminance signal with offset values ​​of a color difference signal, and determining the offset value of the color difference signal according to the luminance signal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-208884 A Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, the input / output characteristics of an image sensor can change due to individual variations, sensitivity (sensor gain) characteristics, and other factors, as well as the influence of temperature and aging, and changes in the input / output characteristics of an image sensor can cause the black balance to shift. When the black balance shifts, colors that do not actually exist in the captured image appear, resulting in a problem of reduced color reproducibility of the subject.

[0006] In view of the above problem, an object of the present invention is to provide an imaging apparatus capable of performing black balance correction suited to white balance and the like. [Means for solving the problem]

[0007] In order to solve the above problem, an image processing device according to one aspect of the present invention has a detection means for detecting a black balance shift in an input image, an acquisition means for acquiring at least one of information regarding white balance to be applied to the input image or information regarding static black balance, a calculation means for calculating a correction value for correcting the black balance shift based on at least one of the information regarding the white balance or the information regarding the static black balance when the detection means detects the black balance shift, and an adjustment means for adjusting the black balance shift in accordance with the correction value calculated by the calculation means. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device capable of performing black balance correction suited to white balance and the like. [Brief description of the drawings]

[0009] [Figure 1] 1 is a functional block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 2] 4 is a diagram showing an example of an LUT of offset amounts for each of R, G, and B according to the first embodiment; FIG. [Diagram 3] FIG. 4 is a diagram showing an example of an effective range of white balance gain according to the first embodiment. [Figure 4] 5 is a flowchart showing an example of an image processing method for calculating a BB offset in the image processing device according to the first embodiment. [Diagram 5] 6A to 6C are diagrams showing an example of control of the black balance correction value (offset) when WB is shifted to the G side in the first embodiment. [Figure 6] 6A to 6C are diagrams showing an example of control of the black balance correction value (offset) when WB is shifted to the R and B sides in the first embodiment. [Figure 7] FIG. 11 is a functional block diagram showing an example of a functional configuration of an image processing device according to a second embodiment. [Figure 8] 6(A) to 6(C) are diagrams showing an example of control of the black balance correction value (offset) when infrared light is not captured in the second embodiment. [Figure 9] 13(A) to 13(C) are diagrams showing an example of control of the black balance correction value (offset) when infrared light is taken in the second embodiment. [Figure 10] FIG. 13 is a functional block diagram showing an example of a functional configuration of an image processing device according to a third embodiment. [Figure 11] 13(A) to 13(C) are diagrams showing an example of black balance control when WB is shifted to the G side in the third embodiment. [Figure 12] 13(A) to 13(C) are diagrams showing an example of black balance control when WB is shifted to the R and B sides in the third embodiment. [Figure 13] FIG. 13 is a functional block diagram showing an example of a functional configuration of an image processing device according to a fourth embodiment. [Figure 14] 13(A) to 13(C) are diagrams showing an example of black balance control according to a fourth embodiment. [Figure 15] FIG. 13 is a functional block diagram showing an example of a functional configuration of an image processing device according to a fifth embodiment. [Figure 16] 13 is a diagram showing an example of an LUT of offset amounts for each of R, G, and B according to the fifth embodiment. FIG. [Figure 17] 13(A) to 13(C) are diagrams showing an example of black balance control according to a fifth embodiment. [Figure 18] FIG. 13 is a functional block diagram showing an example of a functional configuration of an image processing device according to a sixth embodiment. [Figure 19] FIG. 23 is a diagram showing an example of an LUT according to the sixth embodiment. [Figure 20] 1 is a diagram illustrating an example of a hardware configuration of an image processing device according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. The embodiment described below is one example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. In addition, the present invention may be configured by appropriately combining parts of each embodiment described later.

[0011] <First embodiment> An image processing device according to a first embodiment of the present invention will be described below with reference to Fig. 1. Fig. 1 is a functional block diagram showing an example of a functional configuration of the image processing device according to the first embodiment. Note that some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like as a computer included in the image processing device to execute a computer program stored in a memory as a storage medium.

[0012] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0013] In addition, the functional blocks shown in Fig. 1 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 1 also applies to Figs. 7, 10, 13, 15, and 18.

[0014] An input image is captured by an imaging unit consisting of a lens and an imaging sensor (imaging element) (not shown). The input image is image data or an image signal consisting of a plurality of pixels, and includes a plurality of color information. The plurality of colors are, for example, red (Red:R), green (Green:G), and blue (Blue:B), and the image data (signal) corresponds to the amount of light that passes through color filters corresponding to the respective colors provided on the imaging sensor (not shown) and is converted into an electrical signal by the imaging sensor.

[0015] Color filters transmit not only visible light corresponding to red, green, and blue, but also some infrared light (invisible light). For this reason, in general imaging devices, an infrared cut filter (IRCF) is provided to remove infrared light components, so that images close to those seen by humans can be obtained.

[0016] The output image is an image in which the black balance (BB) is appropriately corrected by adding an offset value to the pixel value for each color of the input image, and the white balance is appropriately corrected by multiplying it by the white balance gain (WB gain).

[0017] The white balance gain includes, for example, a Red gain for adjusting the redness of an output image and a Blue gain for adjusting the blueness of an output image. There is an offset value for each of the R, G, and B colors.

[0018] In this embodiment, an offset amount for correcting black balance is determined based on the amount of deviation in black balance and the amount of deviation in white balance in a configuration in which an input image is multiplied by a white balance gain after an offset is added to the input image. Also, in this embodiment, when capturing an image by taking in infrared light, the white balance can be intentionally shifted according to a setting value.

[0019] A BB detection unit (detection means) 101 detects a deviation amount of black balance (BB) and outputs the deviation amount of black balance to a BB offset calculation unit 103.

[0020] Since the larger the gain (sensor gain) multiplied by the output signal of the sensor is, the larger the deviation in black balance is, in this embodiment, the amount of deviation in black balance is associated with the sensor gain. That is, the BB detection unit 101 outputs, for example, the value of the sensor gain as the amount of deviation in black balance to the BB offset calculation unit 103. The gain value is expressed, for example, in decibels (dB).

[0021] The WB correction setting unit 102 sets the amount of deviation of the white balance (WB) in accordance with the detection result of the infrared light detection unit 104 .

[0022] The amount of white balance deviation is expressed, for example, by a value between 0 and 10 (WB correction setting) that can be set by the user, where the smaller the value, the greater the deviation toward the magenta (Mg) side, and the larger the value, the greater the deviation toward the G side.

[0023] In addition, when the color of the input image is affected by infrared light based on the detection result obtained from the infrared light detection unit 104, the WB correction setting unit 102 outputs the set white balance deviation amount to the BB offset calculation unit 103 and the WB gain control unit 105.

[0024] If the color of the input image is not affected by infrared light, the set white balance shift amount is not output to the BB offset calculation unit 103 and the WB gain control unit 105, and information indicating that there is no white balance shift is output.

[0025] The BB offset calculation unit 103 acquires the amount of black balance deviation from the BB detection unit 101. Furthermore, the BB offset calculation unit 103 acquires the amount of white balance deviation from the WB correction setting unit 102, calculates offset amounts of R, G, and B, and outputs them to the BB offset addition unit 106. Here, the BB offset calculation unit 103 functions as a calculation unit that calculates a correction value for correcting the black balance deviation.

[0026] Fig. 2 is a diagram showing an example of an offset amount LUT for each of R, G, and B according to the first embodiment. The offset amount for each of R, G, and B may be determined by referring to a LUT (Look-Up Table) linked to the sensor gain (black balance deviation amount) and WB correction setting (white balance deviation amount) as shown in Fig. 2. In Fig. 2, M represents the maximum value of the WB correction setting value, and N represents the maximum value of the sensor gain.

[0027] The infrared light detection unit (determination means) 104 determines whether or not the color of the input image captured by the imaging sensor is affected by infrared light, and outputs the detection result to the WB correction setting unit 102 and the WB gain control unit 105 .

[0028] For example, when an IRCF (not shown) is inserted on the optical axis of the lens of the imaging unit, the infrared light detection unit 104 detects that the color of the input image is not affected by infrared light. On the other hand, when the IRCF is not inserted on the optical axis of the lens of the imaging unit (removed from the optical axis), the infrared light detection unit 104 detects that the color of the input image is affected by infrared light.

[0029] A WB gain control unit (control means) 105 acquires the amount of white balance deviation from the WB correction setting unit 102 and the detection result from the infrared light detection unit 104, determines parameters for calculating the white balance gain, and outputs the parameters to a WB gain calculation unit 108. The parameters for calculating the white balance gain include, for example, parameters for determining the effective range of the white balance gain.

[0030] 3 is a diagram showing an example of an effective range of white balance gain according to the first embodiment. The effective range of white balance gain is determined such that, for example, when an input image is affected by infrared light, the red gain and blue gain can have larger values ​​as the amount of deviation of the white balance toward the magnesium side increases (for example, A1 in FIG. 3).

[0031] On the other hand, the effective range of the white balance gain is determined so that when the input image is affected by infrared light, the greater the shift of the white balance toward the G side, the smaller the values ​​of the red gain and blue gain can be (for example, A2 in Figure 3).

[0032] The BB offset addition unit 106 acquires the offset amounts of R, G, and B from the BB offset calculation unit 103 , adds them to the input image, and outputs the image after the offset addition to the feature acquisition unit 107 and the WB gain multiplication unit 109 .

[0033] The feature amount acquiring unit 107 acquires the image after the offset addition from the BB offset adding unit 106, calculates feature amounts related to color, and outputs the calculated feature amounts to the WB gain calculating unit 108. More specifically, when the image is divided into a plurality of rectangular areas, the unit 107 calculates color information for each rectangular area determined by image data contained in each rectangular area. The color information is, for example, a representative value such as the average value or the most frequent value of the color difference signal for each rectangular area.

[0034] The WB gain calculation unit 108 acquires parameters that determine the effective range of the white balance gain from the WB gain control unit 105 and color information for each region from the feature acquisition unit 107, calculates the white balance gain (WB gain), and outputs it to the WB gain multiplication unit 109.

[0035] More specifically, the WB gain calculation unit 108 calculates a first white balance gain (for example, W0 in FIG. 3) such that the representative value of the color information for each area acquired by the feature acquisition unit 107 becomes a predetermined target value.

[0036] Furthermore, the WB gain calculation unit 108 determines the effective range of the white balance gain based on a parameter that determines the effective range of the white balance gain. Then, when the first white balance gain is included in the effective range of the white balance gain, the WB gain calculation unit 108 outputs the first white balance gain to the WB gain multiplication unit 109.

[0037] On the other hand, when the first white balance gain is not included in the effective range of the white balance gain, the WB gain calculation unit 108 outputs a second white balance gain (for example, W1 or W2 in FIG. 3) obtained by correcting the first white balance gain to the WB gain multiplication unit 109. The second white balance gain is, for example, a white balance gain that is included in the effective range of the white balance gain and is closest to the first white balance gain.

[0038] The WB gain multiplication unit 109 multiplies the image after the offset addition supplied from the BB offset addition unit 106 by the WB gain from the WB gain calculation unit 108 to generate an output image, and outputs the output image.

[0039] An example of a method for calculating a BB offset in the image processing device according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of an image processing method for calculating a BB offset in the image processing device according to the first embodiment.

[0040] In step S01, the infrared light detection unit 104 acquires (detects) whether or not the color of an input image captured by an image sensor is affected by infrared light, as information related to white balance.

[0041] Step S01 functions as an acquisition step (acquisition means) for acquiring information about the white balance to be applied to the input image, and also functions as a determination step (determination means) for determining whether or not the input image is affected by infrared light. If it is detected that the color of the input image is affected by infrared light, the process proceeds to step S02, and if it is detected that the color of the input image is not affected by infrared light, the process of FIG. 4 is terminated.

[0042] In step S02 (detection step), the BB detection unit 101 detects whether or not the black balance is shifted. If it is detected that the black balance is shifted, the process proceeds to step S03, and if it is detected that the black balance is not shifted, the process of FIG. 4 is terminated.

[0043] In step S03, the WB correction setting unit 102 detects whether the white balance is shifted toward green (G) or not. If it is detected that the white balance is shifted toward green, the process proceeds to step S04, and if it is detected that the white balance is not shifted toward green, the process proceeds to step S05.

[0044] In step S04, the BB offset calculation unit 103 outputs an offset value P that is suitable for the case where the white balance is shifted toward the green side, and the process in FIG. 4 ends.

[0045] In step S05, the WB correction setting unit 102 detects whether the white balance is shifted toward the purple (Mg) side. If it is detected that the white balance is shifted toward the purple side, the process proceeds to step S06, and if it is detected that the white balance is not shifted toward the purple side, the process proceeds to step S07.

[0046] In step S06, the BB offset calculation unit 103 outputs an offset value Q that is suitable for the case where the white balance is shifted toward the purple side, and the process in FIG. 4 ends.

[0047] In step S07, the BB offset calculation unit 103 outputs an offset value R suitable for the case where there is no deviation in the white balance, and ends the processing in Fig. 4. Here, steps S04, S06, and S07 function as calculation steps for calculating a correction value for correcting the deviation in the black balance.

[0048] According to the offset value (correction value) calculated by the BB offset calculation unit 103, an adjustment step is executed in which a BB offset addition unit (adjustment means) 106 adjusts the deviation of the black balance.

[0049] The effects of this embodiment will be described below. Figures 5(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) when WB is shifted to the G side in the first embodiment. Figures 6(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) when WB is shifted to the R and B sides in the first embodiment.

[0050] Figures 5(A)-(C) and 6(A)-(C) show examples of the characteristics of pixel values ​​of an input image and pixel values ​​of an output image when shooting a subject of achromatic colors ranging from black to white. Figures 5(A) and 6(A) show examples when black balance correction is not performed, while Figures 5(B), 6(B) and 5(C), 6(C) show examples when black balance correction is performed.

[0051] The offset amounts in Fig. 5(B), Fig. 6(B) and Fig. 5(C) and Fig. 6(C) are different, but the offset amounts in Fig. 5(B) and Fig. 6(B) and Fig. 5(C) and Fig. 6(C) are equal. The offset of B in Fig. 5(B) and Fig. 6(B) is represented by Bo1, the offset of R by Ro1, and the offset of G by Go1.

[0052] On the other hand, in Figure 5(C) and Figure 6(C), the offset of B is represented by Bo2, the offset of R by Ro2, and the offset of G by Go2. The magnitude of each offset is represented by the length of the arrow, with an upward arrow representing a positive offset (addition) and a downward arrow representing a negative offset (subtraction). Also, when there is no arrow, it represents no offset (0).

[0053] The offset amount in Figures 5(B) and 6(B) is smaller than that in Figures 5(C) and 6(C), and the black balance correction is weaker. On the other hand, the offset amount in Figures 5(C) and 6(C) is larger than that in Figures 5(B) and 6(B), and the black balance correction is stronger. Also, for simplicity, the characteristics of R and B are the same, and only G is different.

[0054] In the case of no correction in Fig. 5(A), the output pixel values ​​for Black are larger for Blue and Red than for Green, indicating that the black balance is shifted to Mg. This phenomenon is more likely to occur as the sensor gain increases, and the greater the sensor gain, the greater the shift in black balance.

[0055] On the other hand, the output pixel value for White is greater for Green than for Blue and Red, which indicates that the white balance is shifted to G based on the setting value of the WB correction setting unit 102. In other words, this is a case where the white balance and black balance are shifted in opposite directions.

[0056] In the case of no correction in FIG. 6(A), the output pixel values ​​for Black are greater for Blue and Red than for Green, indicating that the black balance is shifted towards Mg.

[0057] On the other hand, the output pixel values ​​for White are larger for Blue and Red than for Green, which indicates that the white balance is shifted to Mg based on the setting value of the WB correction setting unit 102. In other words, the white balance and black balance are shifted in the same direction.

[0058] In the case of Fig. 5(B), even after black balance correction, the output pixel values ​​for black are slightly larger for blue and red than for green, indicating that the black balance is shifted toward magnesium. In this case, since white and black are shifted toward different color sides, the shift in black balance is easily noticeable.

[0059] On the other hand, in the case of Fig. 6(B), the offset amount is the same as in Fig. 5(B), but since both White and Black are shifted to the Mg side, the shift in black balance is less noticeable.Similarly, in the case of Fig. 5(C), after black balance correction, both White and Black are shifted to the G side, so the shift in black balance is less noticeable.

[0060] On the other hand, in the case of FIG. 6C, the white balance is shifted to Mg and the black balance is shifted to G, and since white and black are shifted to different color sides, the shift in black balance is easily noticeable.

[0061] For these reasons, if white and black are off to different colors before black balance correction, the black balance shift will be more noticeable if the correction is insufficient.On the other hand, if white and black are off to the same color before black balance correction, the black balance shift will be more noticeable if the correction is overdone.

[0062] Therefore, in this embodiment, when white and black are shifted to different color sides, the offset amount is made larger, whereas when white and black are shifted to the same color side, the offset amount is made smaller. In other words, in the case of Fig. 5(A), the black balance is corrected with an offset amount as shown in Fig. 5(C), and in the case of Fig. 6(A), the black balance is corrected with an offset amount as shown in Fig. 6(B).

[0063] That is, in the black balance correction of this embodiment, the BB offset calculation unit 103 determines the offset amount by referring to the amount of deviation in black balance and the amount of deviation in white balance. Therefore, it is possible to apply an appropriate offset amount according to the degree of deviation in white balance.

[0064] Then, when white and black are shifted towards different colours, the offset amount is made larger, and when white and black are shifted towards the same colour, the offset amount is made smaller, thereby achieving black balance correction that is suitable for white balance.

[0065] In this embodiment, the amount of deviation in black balance is associated with the magnitude of the sensor gain, but this is not limiting. For example, the amount of deviation in black balance may be associated with, for example, the shutter speed, the sensor temperature, the aperture value of the iris, the environmental temperature, or the operation time of the imaging device. In other words, the amount of deviation in black balance may be associated with at least one of the sensor gain, the shutter speed (charge accumulation time), the sensor temperature, the aperture value, the environmental temperature, or the operation time of the imaging device.

[0066] The lower the illuminance of the shooting environment, the more likely the black balance is to be misaligned. When shooting in a low-illuminance environment, generally, the shutter speed (or charge accumulation time) tends to be slower (longer) and the aperture tends to be wider in order to ensure the brightness of the image. Therefore, it can be assumed that the slower (longer) the shutter speed (charge accumulation time) is, or the closer the iris aperture value is to the maximum aperture (the smaller the aperture value), the lower the illuminance of the shooting environment will be, and the greater the misalignment of the black balance will be.

[0067] Also, the higher the temperature around the sensor, the greater the deviation in black balance due to an increase in dark current. Or, the longer the operation time (power-on time) of the imaging device and the imaging sensor, the greater the deviation in black balance due to the rise in temperature around the sensor. Furthermore, the longer the operation time of the imaging device and the imaging sensor, the greater the deterioration of the imaging sensor, which increases the dark current and causes a greater deviation in black balance.

[0068] In addition, for example, in general, the higher the temperature of a semiconductor device, the more accelerated its deterioration, and in this case, the amount of deviation in black balance increases depending on the combination of temperature and operating time. Therefore, it is desirable to use a combination of multiple parameters for estimating the deviation in black balance as described above.

[0069] In this embodiment, the amount of white balance shift is the WB correction setting value set by the WB correction setting unit 102, but is not limited to this. The amount of white balance shift may be, for example, the difference between the first white balance gain and the second white balance gain calculated by the WB gain calculation unit 108 in the past (for example, one frame before).

[0070] Alternatively, the amount of white balance shift may be the white balance gain itself. The larger the white balance gain (red gain and blue gain), the more the white balance shifts toward the magenta side, and the smaller the white balance gain, the more the white balance shifts toward the green side.

[0071] The amount of white balance shift may also be the difference between shooting with the IRCF inserted and shooting with the IRCF removed. For example, the amount of white balance shift is the difference in color of the captured image when a white subject is shot with the IRCF inserted and when the IRCF is removed, when the white subject is shot with the same white balance gain.

[0072] Generally, when the IRCF is removed, the white balance is shifted to be reddish compared to when the IRCF is inserted. Alternatively, the amount of color correction for suppressing the color difference between when the IRCF is inserted and when the IRCF is removed may be used as the amount of white balance shift.

[0073] For example, when the amount of color correction is small, the redness of the image when the IRCF is removed remains, resulting in a large shift in the white balance.As the amount of color correction increases, the redness of the image when the IRCF is removed is reduced, resulting in a smaller shift in the white balance.

[0074] In this embodiment, the offset amount of the black balance correction is changed according to the degree of deviation of the white balance, but this is not limiting. The offset amount of the black balance correction may be changed according to the magnitude of the correction amount of the white balance.

[0075] For example, when an image is captured with the IRCF removed, the color of the captured image will shift toward magenta compared to when the IRCF is inserted; however, this color shift in the captured image can be reduced by correcting the white balance.

[0076] At this time, the offset amount of the black balance correction is changed according to the amount of white balance correction. More specifically, if the amount of white balance correction is small, the image captured when the IRCF is removed is shifted toward magenta, and even if the black balance is shifted toward the magenta side, there is little sense of incongruity, so the absolute values ​​of the subtraction amounts of the blue and red offsets are reduced, or the absolute value of the addition amount of the green offset is reduced.

[0077] As the white balance correction amount increases, the color shift in the image captured when the IRCF is removed decreases and the black balance shift becomes more noticeable, so the absolute values ​​of the blue and red offset subtraction amounts are increased, or the absolute value of the green offset addition amount is increased.

[0078] <Second embodiment> An image processing device according to a second embodiment of the present invention will be described below with reference to Fig. 7. In this embodiment, an offset amount for correcting black balance is determined based on the amount of black balance deviation and the presence or absence of the influence of infrared light in a configuration in which an input image is multiplied by a white balance gain after an offset is added to the input image. This embodiment also makes it possible to maintain an appropriate white balance even when capturing infrared light for imaging.

[0079] 7 is a functional block diagram showing an example of the functional configuration of an image processing apparatus according to the second embodiment. Note that the same functional blocks as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0080] The BB offset calculation unit 203 acquires the amount of black balance deviation from the BB detection unit 101 and the detection result of the influence of infrared light from the infrared light detection unit 204, calculates the offset amounts of R, G, and B, and outputs them to the BB offset addition unit 106.

[0081] The infrared light detection unit 204 detects whether or not the color of the input image captured by the image sensor is affected by infrared light, and outputs the detection result to the BB offset calculation unit 203 and the WB gain control unit 205 .

[0082] The WB gain control unit 205 acquires a detection result from the infrared light detection unit 204, determines parameters for calculating a white balance gain, and outputs the parameters to the WB gain calculation unit 108. The parameters for calculating the white balance gain include, for example, a parameter for determining an effective range of the white balance gain.

[0083] The effective range of the white balance gain is determined, for example, so that when an input image is affected by infrared light, the red gain and blue gain can be smaller than when there is no influence of infrared light. When an input image is affected by infrared light, the white balance shifts toward magenta, so by setting the red gain and blue gain to smaller values ​​than when there is no influence of infrared light, it is possible to maintain an appropriate white balance even when there is the influence of infrared light.

[0084] The effects of this embodiment will be described below. Fig. 8(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) when infrared light is not taken in the second embodiment. Fig. 9(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) when infrared light is taken in the second embodiment. That is, Fig. 8(A) to (C) show an example of a case where there is no effect of infrared light on an input image, and Fig. 9(A) to (C) show an example of a case where there is an effect of infrared light on an input image.

[0085] In the case of FIG. 8A where there is no black balance correction, the output pixel values ​​for Black are greater for Blue and Red than for Green, indicating that the black balance is shifted to Mg.

[0086] The greater the sensor gain, the greater the amount of black balance deviation. On the other hand, the output pixel value for White has an appropriate white balance, and the pixel values ​​for Red, Green, and Blue are almost the same, with no coloring.

[0087] Even in the case of FIG. 9A where there is no black balance correction, the output pixel values ​​for Black are greater for Blue and Red than for Green, indicating that the black balance is shifted to Mg.

[0088] However, due to the influence of infrared light, the black balance is shifted more than when there is no influence of infrared light. On the other hand, the output pixel value for White has an appropriate white balance, and the pixel values ​​for Red, Green, and Blue are almost the same, with no coloring.

[0089] In the case of Fig. 8(B), after black balance correction, the output pixel value for black is almost the same as the pixel values ​​for red, green, and blue, and no coloring occurs. On the other hand, in the case of Fig. 9(B), although the offset amount is the same as in Fig. 8(B), the black balance deviation has increased due to the influence of infrared light, resulting in insufficient correction and the black balance deviating to the Mg side.

[0090] Fig. 8C shows an example in which the amount of black balance correction is greater than in Fig. 8B. In Fig. 8C, the black balance is overcorrected, and the black balance is shifted toward the G side.

[0091] On the other hand, in the case of FIG. 9C, although the offset amount is the same as in FIG. 8C, the output pixel value for black is such that the pixel values ​​for red, green, and blue are almost the same, and the black balance is not shifted.

[0092] For the above reasons, if the offset amount is optimal when there is no influence of infrared light, the black balance will be under-corrected when there is influence of infrared light.On the other hand, if the offset amount is optimal when there is influence of infrared light, the black balance will be over-corrected when there is no influence of infrared light.

[0093] Therefore, in the second embodiment, when there is no influence of infrared light, an offset amount as shown in FIG. 8B is applied, and when there is an influence of infrared light, an offset amount as shown in FIG. 9C is applied.

[0094] That is, in the black balance correction of this embodiment, the BB offset calculation unit 203 calculates the offset amount of the black balance correction by referring to the amount of deviation of the black balance and the presence or absence of the influence of infrared light. Therefore, it is possible to apply an appropriate offset amount depending on the presence or absence of the influence of infrared light.

[0095] Specifically, as described above, by setting the offset amount to a larger value when there is an influence of infrared light, and setting the offset amount to a smaller value when there is no influence of infrared light, it is possible to perform suitable black balance correction according to whether or not there is an influence of infrared light.

[0096] <Third embodiment> Next, an image processing device according to a third embodiment of the present invention will be described with reference to Fig. 10. In this embodiment, an offset amount for correcting black balance is determined based on the amount of deviation in black balance and the amount of deviation in white balance in a configuration in which an input image is multiplied by a white balance gain and then an offset is added. In addition, this embodiment can intentionally shift the white balance according to a setting value when capturing infrared light for imaging.

[0097] Ideally, it would be desirable to adjust the black balance by adding an offset and then adjusting the white balance by multiplying the white balance gain. However, due to circumstances such as the system configuration, there are cases where offset addition for adjusting the black balance is performed after multiplying the white balance gain, and this embodiment is suitable for such cases.

[0098] 10 is a functional block diagram showing an example of the functional configuration of an image processing apparatus according to the third embodiment. Note that the same functional blocks as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0099] The BB offset addition unit 306 acquires an image multiplied by the white balance gain from the WB gain multiplication unit 309, and offset amounts of R, G, and B from the BB offset calculation unit 103. Then, an output image is generated by adding the offset amount to the image obtained by multiplying the input image by the white balance gain, and is output.

[0100] The WB gain multiplication unit 309 obtains the white balance gain from the WB gain calculation unit 108 , multiplies the input image by the white balance gain, and outputs the image multiplied with the white balance gain to the BB offset addition unit 306 .

[0101] The effects of this embodiment will be described below. Figures 11(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) when WB is shifted to the G side in the third embodiment. Figures 12(A) to (C) are diagrams showing an example of black balance control when WB is shifted to the R and B sides in the third embodiment.

[0102] In the case of FIG. 11A where there is no black balance correction, the output pixel values ​​for Black are greater for Blue and Red than for Green, indicating that the black balance is shifted to Mg.

[0103] This phenomenon is more likely to occur as the sensor gain increases, and the greater the sensor gain, the greater the black balance deviation. On the other hand, the output pixel value for White is greater for Green than for Blue and Red, which indicates that the white balance is shifted toward the G side based on the setting value of the WB correction setting unit 102. In other words, the white balance and black balance are shifted in opposite directions.

[0104] In the case of FIG. 12A where there is no black balance correction, the output pixel values ​​for Black are greater for Blue and Red than for Green, indicating that the black balance is shifted to Mg.

[0105] On the other hand, the output pixel values ​​for White are also larger for Blue and Red than for Green, which indicates that the white balance is shifted toward the Mg side based on the setting value of the WB correction setting unit 102. In other words, the white balance and black balance are shifted in the same direction.

[0106] In the case of Fig. 11(B), after black balance correction, the output pixel value in Black is almost the same as the pixel values ​​in Red, Green, and Blue, and no coloring occurs. On the other hand, in this embodiment, since an offset is added after multiplication by the white balance gain, the amount of white balance deviation is larger than in Fig. 11(A). As a result, in the case of Fig. 11(B), green coloring becomes noticeable in the output image.

[0107] On the other hand, in the case of Fig. 12(B), after black balance correction, the output pixel value for black is almost the same as the pixel values ​​for red, green, and blue, and no coloring occurs. Although the white balance cannot be maintained as in Fig. 12(A), the amount of deviation in the white balance is reduced, that is, the change is in the direction of reducing coloring, so there is little discomfort in the image quality.

[0108] Figure 11(C) shows an example in which the offset amount is smaller than that in Figure 11(B). In Figure 11(C), the deviation of the black balance and the white balance is intermediate between that in Figure 11(A) and that in Figure 11(B). In other words, while the black balance is prevented from deviating toward magenta compared to the case of Figure 11(A), the white balance can be prevented from deviating excessively toward green compared to the case of Figure 11(B).

[0109] Figure 12(C) also shows an example in which the offset amount is smaller than in Figure 12(B). In Figure 12(C), the deviation in black balance and white balance is intermediate between those in Figure 12(A) and Figure 12(B). That is, it is easier to maintain the white balance in Figure 12(A) compared to Figure 12(B) while preventing the black balance from deviating toward magenta compared to Figure 12(A).

[0110] As described above, when white and black are shifted to different color sides before black balance correction, correcting the black balance will increase the shift in white balance, and if the shift increases significantly, the sense of incongruity in the image quality will increase.

[0111] Therefore, in the third embodiment, when white and black are shifted to different color sides before black balance correction, the offset amount is made smaller. In other words, when white and black are shifted to different color sides before black balance correction as in Fig. 11(A), the offset amount is made smaller as in Fig. 11(C).

[0112] On the other hand, in the case where white and black are shifted to the same color side before black balance correction as in Fig. 12(A), correcting the black balance does not increase the white balance shift, so even if the offset amount is increased, there is little discomfort in the image quality. In other words, in the third embodiment, in the case where white and black are shifted to the same color side before black balance correction as in Fig. 12(A), the offset amount is set to a large amount as in Fig. 12(B).

[0113] In this manner, in this embodiment, when an offset is added after multiplying an input image by a white balance gain, the BB offset calculation unit 103 determines the offset amount by referring to the black balance shift amount and the white balance shift amount. Therefore, it is possible to apply an appropriate offset amount according to the degree of white balance shift.

[0114] Specifically, as described above, when white and black are shifted to different color sides, the offset amount is made smaller, and when white and black are shifted to the same color side, the offset amount is made larger. This makes it possible to perform black balance correction suited to the white balance.

[0115] In this embodiment, when the white balance is shifted to Green before black balance correction as shown in FIG. 11(A), the offset addition amount for Green is set to Go6 as shown in FIG. 11(C), which has an absolute value smaller than Go5 in FIG. 11(B).

[0116] On the other hand, when the white balance is shifted between blue and red before black balance correction as in Fig. 12(A), the offset addition amount for green is set to Go5 as in Fig. 12(B), which has a larger absolute value than Go6 in Fig. 12(C). However, this is not limited to this.

[0117] That is, when the white balance is shifted to green before black balance correction, the absolute values ​​of the blue and red offset subtraction amounts should be smaller than those when the white balance is shifted to blue and red before black balance correction.

[0118] On the other hand, when the white balance is shifted to blue and red before black balance correction, the absolute values ​​of the offset subtraction amounts for blue and red should be made larger than when the white balance is shifted to green before black balance correction.

[0119] <Fourth embodiment> Next, an image processing device according to a fourth embodiment of the present invention will be described with reference to Fig. 13. In this embodiment, one of intermediate gradations between black and white is set as a priority gradation, and black balance correction is performed by prioritizing the priority gradation.

[0120] In this embodiment, as in the third embodiment, an offset is added after the input image is multiplied by the white balance gain. Also, an offset amount for correcting the black balance is determined based on the amount of deviation in the black balance and the amount of deviation in the white balance. Also, in this embodiment, when capturing an image by taking in infrared light, the white balance can be intentionally shifted according to a setting value.

[0121] 13 is a functional block diagram showing an example of the functional configuration of an image processing device according to the fourth embodiment. Note that the same functional blocks as those in the first and third embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

[0122] The BB offset calculation unit 403 acquires the amount of deviation of black balance from the BB detection unit 101, the amount of deviation of white balance from the WB correction setting unit 102, and the priority gradation value from the priority setting unit 410. Then, the BB offset calculation unit 403 calculates offset amounts for black balance correction of R, G, and B, and outputs the offset amounts to the BB offset addition unit 106.

[0123] The priority setting unit 410 sets one of the intermediate gradations between black and white as a priority gradation, and outputs the priority gradation value to the BB offset calculation unit 403. The priority setting unit 410 functions as a setting unit for setting the priority of gradation values.

[0124] The operation of the BB offset calculation unit 403 according to this embodiment and the effects of this embodiment will be described with reference to Fig. 14. Figs. 14(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) according to the fourth embodiment.

[0125] 14A, in the case of no black balance correction, the output pixel values ​​for black are larger for blue and red than for green, indicating that the black balance is shifted toward magnesium. This phenomenon is more likely to occur the larger the sensor gain is, and the greater the sensor gain is, the greater the amount of black balance shift will be.

[0126] On the other hand, the output pixel value for White is larger for Green than for Blue and Red, which indicates that the white balance is shifted toward the G side based on the setting value of the WB correction setting unit 102. In other words, the colors are shifted in opposite directions in the white balance and black balance.

[0127] In such a case, if an offset is added after the input image is multiplied by a white balance gain, increasing the amount of offset reduces the deviation in black balance, but increases the deviation in white balance.

[0128] Conversely, if the offset amount is reduced, the black balance shift cannot be reduced, but the white balance shift can be prevented from becoming large. In other words, there is a trade-off between the black balance shift amount and the white balance shift amount.

[0129] Therefore, in this embodiment, the BB offset calculation unit 403 determines the offset amount by prioritizing the priority gradation acquired from the priority setting unit 410. Specifically, for example, the offset amount is determined so that there is no deviation in the white balance of the priority gradation.

[0130] Fig. 14B shows a case where a grayscale that is exactly halfway between black and white is set as the priority grayscale. In the case of Fig. 14B, the deviation in black balance is smaller than in Fig. 14A, and there is no deviation in white balance in the priority grayscale.

[0131] On the other hand, Fig. 14C shows a case where a lower gradation value is set as the priority gradation than in Fig. 14B. The offset amount is larger than in Fig. 14B. Therefore, although the white balance deviation in White is large, the black balance deviation is smaller than in Fig. 14B, and there is no white balance deviation in the priority gradation.

[0132] In this embodiment, the priority setting unit 410 directly sets the gradation value when setting the priority gradation, but is not limited to this. For example, the priority gradation may be set based on the brightness setting value of an image capture device (not shown), a setting value for shifting the exposure of the image capture device to the underexposure side or overexposure side, the most frequent value of the luminance histogram of the input image, etc.

[0133] <Fifth embodiment> An image processing device according to a fifth embodiment of the present invention will be described below with reference to Fig. 15. In this embodiment, in a configuration in which an offset is added to an input image and then a white balance gain is multiplied, an offset amount for correcting the black balance is determined based on the amount of black balance deviation and the color temperature of the light source in the shooting environment.

[0134] 15 is a functional block diagram showing an example of the functional configuration of an image processing apparatus according to the fifth embodiment. Note that the same functional blocks as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0135] The BB offset calculation unit 503 acquires the amount of black balance deviation from the BB detection unit 101 and the color temperature of the shooting environment from the WB gain calculation unit 508 , calculates offset amounts of R, G, and B, and outputs them to the BB offset addition unit 106 .

[0136] The offset amount for each color is determined by, for example, referring to an LUT as shown in Fig. 16. Fig. 16 is a diagram showing an example of an LUT for offset amounts for R, G, and B according to the fifth embodiment. As shown in Fig. 16, the offset amount for each color can be determined by referring to an LUT linked to the sensor gain (deviation amount of black balance) and the color temperature of the shooting environment.

[0137] The WB gain calculation unit 508 acquires color information for each region of the image from the feature amount acquisition unit 107, calculates a white balance gain (WB gain), and outputs it to the WB gain multiplication unit 109. In addition, the WB gain calculation unit 508 calculates the color temperature of the shooting environment from the color information acquired from the feature amount acquisition unit 107 or the calculated WB gain, and outputs it to the BB offset calculation unit 503.

[0138] The color temperature may be calculated by referring to a previously created LUT (not shown) that indicates the correspondence between white balance gain and color temperature, or a LUT (not shown) that indicates the correspondence between image color information and color temperature.

[0139] The effects of this embodiment will be described below. Fig. 17(A) to (C) are diagrams showing an example of control of the black balance correction value (offset) according to the fifth embodiment. Fig. 17(A) to (C) show the characteristics of the pixel values ​​of an input image and the pixel values ​​of each of the R, G, and B colors of an output image when an object of achromatic colors ranging from black to white is photographed.

[0140] 17A, in the case of no black balance correction, the output pixel values ​​for black are larger for blue and red than for green, and red is larger than blue, indicating that the black balance is shifted toward the red to purple side. This phenomenon is more likely to occur as the sensor gain increases, and the greater the sensor gain, the greater the amount of black balance shift.

[0141] 17B, ​​the output pixel value for black is greater for red than for green, and smaller for blue than for green, so the black balance is shifted toward the amber side.

[0142] That is, the black balance is shifted to a color close to that of a low color temperature illumination, and when the color temperature of the illumination in the shooting environment is low, there is little discomfort even if the black balance is shifted to the low color temperature side.On the other hand, when the color temperature of the illumination in the shooting environment is high, there is a large discomfort if the black balance is shifted to the low color temperature side.

[0143] In the case of Fig. 17(C), the offset amount is larger than in the case of Fig. 17(B). Also, in the case of Fig. 17(C), the output pixel value in Black is larger for Blue than for Green, and smaller for Red than for Green.

[0144] Therefore, the black balance is shifted to blue to cyan. That is, the black balance is shifted to a color close to the color of the lighting with a high color temperature, and when the color temperature of the lighting in the shooting environment is high, there is little discomfort even if the black balance is shifted to the high color temperature side. On the other hand, when the color temperature of the lighting in the shooting environment is low, there is a large discomfort if the black balance is shifted to the high color temperature side.

[0145] As described above, when the black balance is deviated from red to purple, if the color temperature of the lighting in the shooting environment is low, it is preferable that the offset amount be relatively small as in Fig. 17(B).On the other hand, when the black balance is deviated from red to purple, if the color temperature of the lighting in the shooting environment is high, it is preferable that the offset amount be relatively large as in Fig. 17(C).

[0146] However, when the black balance is shifted from cyan to blue, the situation is reversed. That is, when the black balance is shifted from cyan to blue, if the color temperature of the lighting in the shooting environment is low, it is preferable that the offset amount be relatively large as shown in Fig. 17(C). On the other hand, when the black balance is shifted from cyan to blue, if the color temperature of the lighting in the shooting environment is high, it is preferable that the offset amount be relatively small as shown in Fig. 17(B).

[0147] In this embodiment, the WB gain control unit 105 may control the white balance based on the deviation in the black balance detected by the BB detection unit 101. For example, the white balance is controlled according to a black balance offset value. This makes it easier to maintain the color tone of the subject at intermediate luminance and above as intended by the user by using the black balance offset.

[0148] Sixth embodiment An image processing device according to a sixth embodiment of the present invention will be described below with reference to Fig. 18. In this embodiment, an offset amount for correcting black balance is determined based on the amount of black balance deviation and the amount of white balance deviation in a configuration in which an input image is multiplied by a white balance gain after an offset is added to the input image. In addition, this embodiment determines the amount of black balance deviation or the amount of black balance correction based on temperature information, or temperature information and sensor gain.

[0149] 18 is a functional block diagram showing an example of the functional configuration of an image processing device according to the sixth embodiment. Note that the same functional blocks as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0150] The static BB correction unit 711 calculates the black levels of R, G, and B based on the acquired input image at a predetermined timing. Then, in order to correct the deviation of the black levels for each of R, G, and B, that is, the black balance, the static BB correction unit 711 calculates an offset amount of static black balance, which is an offset amount for correcting the black levels for each of R, G, and B, and outputs the calculated amount to the BB offset addition unit 706.

[0151] Moreover, the static BB correction unit 711 outputs temperature information when the static offset amount is calculated to the BB offset calculation unit 703. Here, the static offset amount is an offset amount that is calculated only at a predetermined timing and does not change sequentially. Note that, hereinafter, the black balance deviation that is measured only at a predetermined timing and corrected by the static offset amount is referred to as a "static black balance deviation."

[0152] The static black balance deviation refers to a black balance deviation that does not change gradually over a relatively long period of time, such as a black balance deviation caused by deterioration of a sensor over time. The predetermined timing includes, for example, the time of black balance adjustment performed at a factory before shipping during manufacturing, the time of starting up an imaging device that captures an input image, the time when a user intentionally performs black balance calibration, a predetermined time, etc.

[0153] The calibration of the static offset amount is performed, for example, with the sensor gain and the like set to predetermined values ​​and with the sensor shielded from light. The static offset amount measured in the calibration may be stored in a non-volatile storage device or a volatile storage device.

[0154] The BB detection unit (detection means) 701 detects (detects) the amount of black balance deviation, and outputs the amount of black balance deviation to the BB offset calculation unit 703. The higher the sensor gain, sensor temperature, and shooting environment temperature when the input image is captured, the greater the black balance deviation. Therefore, the BB detection unit 701 outputs, for example, the sensor gain and temperature values ​​when the input image is captured to the BB offset calculation unit 703 as the amount of black balance deviation.

[0155] The BB offset calculation unit 703 acquires a sensor gain and a temperature value as the amount of deviation of black balance from the BB detection unit 701. Furthermore, the BB offset calculation unit 703 acquires the amount of deviation of white balance from the WB correction setting unit 102, calculates offset amounts of R, G, and B, and outputs the calculated offset amounts to a BB offset addition unit 706.

[0156] The offset amount calculated by the BB offset calculation unit 703 is a dynamic offset amount that changes successively depending on the current gain and temperature. The dynamic offset amount is determined by referring to an LUT as shown in Fig. 19, for example. Fig. 19 is a diagram showing an example of an LUT according to the sixth embodiment. Fig. 19 shows an example of an LUT linked to the sensor gain and temperature that indicate the amount of deviation in black balance, and the WB correction setting (amount of deviation in white balance).

[0157] Such an LUT is provided for each of R, G, and B. In Fig. 19, M represents the maximum value of the WB correction setting value, and N represents the maximum value of the sensor gain. For temperature, the difference (Tc-Tr) between the temperature value (Tc) acquired from the BB detection unit 701 and the reference temperature value (Tr) is used.

[0158] The minimum value of (Tc-Tr) is -P, and the maximum value is Q. The reference temperature value (Tr) may be a predetermined value, may be the temperature at the time when the LUT data is created, or may be the temperature value at the timing when the static BB correction unit 711 calculates the static offset amount.

[0159] Alternatively, the reference temperature value (Tr) may be externally set. In that case, for example, a temperature value obtained by separately measuring the average temperature of the shooting environment may be set by the user through a setting operation.

[0160] In this embodiment, the temperature used when referring to the LUT is (Tc-Tr), but this is not limited to this. For example, the temperature value (Tc) itself obtained from the BB detection unit 701 may be used for reference.

[0161] The BB offset addition unit 706 acquires the dynamic offset amounts of R, G, and B from the BB offset calculation unit 703 , and acquires the static offset amounts of R, G, and B from the static BB correction unit 711 .

[0162] Then, the BB offset addition unit 706 adds the dynamic offset amount from the BB offset calculation unit 703 and the static offset amount from the static BB correction unit 711 to the input image, and outputs the image after the offset addition to the feature acquisition unit 107 and the WB gain multiplication unit 109.

[0163] The effects of this embodiment will be described below: In the black balance correction of this embodiment, the BB offset calculation unit 703 determines the dynamic offset amount by referring to the amount of deviation in black balance and the amount of deviation in white balance.

[0164] Therefore, it is possible to apply an appropriate offset amount according to the degree of white balance deviation, and to perform black balance correction suitable for the white balance. In addition, since the BB offset calculation unit 703 determines the dynamic offset amount based on the temperature, it is possible to appropriately correct the black balance deviation that occurs at high temperatures.

[0165] In addition, in the black balance correction of this embodiment, the BB offset calculation unit 703 determines the dynamic offset correction amount by referring to both the temperature and the sensor gain. Since the larger the sensor gain, the more likely it is that the black balance will be shifted at high temperatures, if the offset correction amount suitable for high gain and high temperature is applied to low gain and high temperature, there is a risk of overcorrection of the black balance.

[0166] Alternatively, if an offset correction amount suitable for low gain and high temperature is applied to high gain and high temperature, there is a risk of insufficient black balance correction. In other words, if the black balance correction amount is determined based only on temperature, there is a risk of overcorrection or undercorrection of black balance.

[0167] In the black balance correction of this embodiment, the offset correction amount is determined by referring to both the temperature and the sensor gain, so that different offset correction amounts can be applied to high gain and high temperature and low gain and high temperature. Therefore, it is possible to prevent overcorrection at low gain and high temperature or undercorrection at high gain and high temperature, and to appropriately correct the black balance deviation.

[0168] Furthermore, in the black balance correction of this embodiment, the BB offset calculation unit 703 calculates a dynamic offset amount for correcting the black balance based on (Tc-Tr), which is the difference between the temperature value (Tc) acquired from the BB detection unit 701 and a reference temperature value (Tr).

[0169] Generally, the higher the temperature, the greater the dark current of the image sensor, and the greater the deviation in black balance. Therefore, if (Tc) is a value that varies according to the current temperature and (Tr) is a fixed value, the larger (Tc-Tr) becomes, the greater the offset amount for correcting the black balance becomes.

[0170] Therefore, in this embodiment, for example, the LUT is designed so that when (Tc-Tr) is greater than 0, the offset amount is a value other than 0, and when (Tc-Tr) is equal to or less than 0, the offset amount is 0. In other words, the LUT is designed so that when (Tc-Tr) is greater than 0, black balance correction according to temperature is performed, and when (Tc-Tr) is equal to or less than 0, black balance correction according to temperature is not performed.

[0171] In other words, the LUT is designed to perform black balance correction only when the current temperature value (Tc) is greater than the reference temperature value (Tr). In this way, the black balance correction of this embodiment can change the temperature at which the temperature-dependent black balance correction is applied simply by changing (Tr). For example, by providing a separate means for the user to set (Tr), black balance correction can be applied when the temperature is equal to or greater than the temperature desired by the user.

[0172] In addition, in the black balance correction of this embodiment, the static BB correction unit 711 calculates a static offset amount when the user performs a black balance calibration operation, etc. This static offset amount is determined during the calibration operation, and maintains a constant value until the user performs another calibration operation.

[0173] That is, the static offset amount is an offset amount appropriate for the temperature at the time of the calibration operation. In this embodiment, the BB offset calculation unit 703 sets the temperature at the time of the calibration operation as a reference temperature value (Tr). Also, the BB offset calculation unit 703 calculates a dynamic offset amount for correcting the black balance based on the difference (Tc-Tr) between the current temperature value (Tc) and the reference temperature value (Tr).

[0174] That is, a dynamic offset amount for correcting the black balance is calculated according to the difference between the current temperature and the temperature at the time of calibration. Then, the BB offset adding unit 706 adds the dynamic offset amount and the static offset amount to the input image to correct the black balance.

[0175] As described above, black balance correction by user calibration corrects "static black balance deviation." "Static black balance deviation" is, for example, black balance deviation caused by sensor degradation over time. When measuring black balance deviation caused by sensor degradation over time, it is necessary to fix the settings of the imaging device at the time of measurement (for example, sensor gain, etc.) every time.

[0176] However, when performing user calibration, although it is possible to fix configurable measurement conditions such as the sensor gain, it is not possible to fix the temperature because it is dependent on the environment. In other words, the temperature when performing black balance correction by user calibration may be high or low.

[0177] When black balance correction by user calibration is performed at a high temperature, a static offset amount suitable for a high temperature is applied. In this case, if the current temperature is also high, the BB offset calculation unit 703 also applies a dynamic offset correction amount suitable for a high temperature.

[0178] That is, when black balance correction by user calibration is performed at a high temperature and the current temperature is also high, both the static offset amount and the dynamic offset amount become offset correction amounts suitable for high temperatures, resulting in overcorrection. In other words, when black balance correction by user calibration is performed at a high temperature, black balance correction suitable for high temperatures has already been applied, so dynamic offset correction at high temperatures is not necessary.

[0179] In contrast, in the black balance correction of this embodiment, the BB offset calculation unit 703 calculates a dynamic offset amount for correcting the black balance based on the difference (Tc-Tr) between the temperature at the time of the calibration operation (Tr) and the current temperature value (Tc).

[0180] And since the larger (Tc-Tr) is, the larger the dynamic offset amount for correcting the black balance is, the higher the current temperature value (Tc) is than the temperature (Tr) during the calibration operation, the larger the dynamic offset amount is. In other words, even if the static offset amount set at a low temperature is small, the dynamic offset amount set at a high temperature is correspondingly large, so that insufficient correction can be prevented.

[0181] On the other hand, the smaller (Tc-Tr) is, the smaller the dynamic offset amount for correcting the black balance is (closer to 0), so the lower the current temperature value (Tc) is than the temperature (Tr) during the calibration operation, the smaller the dynamic offset amount is (closer to 0). In other words, even if the static offset amount set at a high temperature is large, the dynamic offset amount set at a high temperature will be correspondingly smaller, preventing overcorrection.

[0182] Here, we will provide some additional information on static offset amounts and dynamic offset amounts. As mentioned above, static offset amounts are measured with the sensor shielded from light. In other words, it is an offset correction amount with higher correction accuracy that is obtained by accurately measuring and calculating the black levels for each of R, G, and B with the sensor's imaging area shielded from light, for example by attaching a lens cap or fully closing the aperture.

[0183] However, since it takes time to measure and shooting of the subject is interrupted by shading, it is not suitable for real-time offset correction. On the other hand, the dynamic offset amount estimates the amount of black balance deviation based on the sensor gain and temperature, and is an offset correction amount that can be calculated in real time based on information that can be obtained while shooting.

[0184] However, the dynamic offset amount is not suitable for estimating the amount of deviation in black balance that is determined by complex conditions such as the degree of sensor deterioration. For example, in a use case such as a surveillance camera, continuous shooting is required for surveillance, and the usage time is long, so correction for deterioration over time may be necessary.

[0185] In order to meet such demands, in this embodiment, for example, changes in black balance that occur over a relatively short period of time, such as changes in temperature or illuminance during the day, are corrected in real time based on a dynamic offset amount, while changes in black balance that occur over a relatively long period of time, such as deterioration over time, are accurately corrected based on a static offset amount.

[0186] Since the degree of deterioration over time varies depending on the installation environment of the imaging device, etc., correction based on the static offset amount measured by calibration, etc., can provide more accurate correction than simple estimation from the elapsed time, etc. In addition, by setting the timing for measuring the static offset amount to a timing specified by the user, it is possible to prevent the video from the surveillance camera from being interrupted at a timing unexpected by the user.

[0187] In this embodiment, the BB offset calculation unit 703 determines the dynamic offset correction amount by referring to both the temperature and the sensor gain, but is not limited to this. For example, the dynamic offset correction amount may be determined based on the temperature and the illuminance of the shooting environment.

[0188] One of the causes of black balance deviation is thought to be an increase in dark current generated in the sensor. In a low-illumination environment, the signal components derived from photoelectric conversion contained in the output signal of the sensor are few, and the signal components derived from dark current are relatively more numerous, so black balance deviation becomes more noticeable. In addition, the higher the temperature of the sensor, the greater the black balance deviation becomes because the dark current increases.

[0189] Therefore, the BB offset calculation unit 703 determines the dynamic offset correction amount based on both the temperature and the illuminance of the shooting environment, so that the black balance can be corrected more appropriately.

[0190] For example, when the gain, shutter speed (charge accumulation time), and aperture value (F-number) are automatically controlled according to the illuminance of the shooting environment, the illuminance of the shooting environment may be estimated from the gain, shutter speed, and iris aperture value.

[0191] That is, the illuminance of the shooting environment can be determined to be lower as the gain is larger, the shutter speed is slower, and the aperture value is smaller. In other words, the dynamic offset correction amount may be determined based on both the temperature and the illuminance of the shooting environment by referring to at least one of the gain, the shutter speed, and the aperture value, and the temperature.

[0192] The method of estimating the illuminance of the shooting environment is not limited to the method of estimating it from the gain, shutter speed, and aperture value. A separate illuminance sensor may be provided and its measured value may be used.

[0193] In this embodiment, the static BB correction unit 711 outputs the temperature information when the static offset amount is calculated to the BB offset calculation unit 703, but this is not limited to this. That is, the static BB correction unit 711 may output the average value of the static offset amount calculated to the BB offset calculation unit 703.

[0194] Alternatively, the average value of the black levels for each of R, G, and B of the input image used to calculate the static offset amount or the like may be output to the BB offset calculation unit 703. In this case, the BB offset calculation unit 703 may refer to an LUT that determines the dynamic offset amount based on the average value of the static offset amount or the average value of the black levels for each of RGB of the input image.

[0195] The method of calculating the offset amount for correcting the black balance may be changed depending on the type of sensor. The types of sensors include, for example, a sensor using an avalanche photodiode, a photon counting imaging sensor, and an SPD sensor. SPD is an abbreviation for Single Photon Avalanche Diode.

[0196] Other types of sensors include CMOS sensors and CCD sensors. The dark current characteristics of the sensor differ depending on the sensor type, so the degree of deviation of the black balance differs depending on the sensor type. The sensor type includes at least one of the above multiple sensors.

[0197] Therefore, for example, the LUT for calculating the offset amount for correcting the black balance may be switched depending on the type of sensor. In other words, by referring to an LUT having an offset correction amount suitable for the type of sensor, appropriate black balance correction can be performed depending on the type of sensor.

[0198] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the present invention, including the following combinations.

[0199] (Configuration 1) An image processing device comprising: a detection means for detecting a deviation in black balance in an input image; an acquisition means for acquiring at least one of information on white balance to be applied to the input image or information on static black balance; a calculation means for calculating a correction value for correcting the deviation in black balance based on at least one of the information on white balance or the information on static black balance when the deviation in black balance is detected by the detection means; and an adjustment means for adjusting the deviation in black balance in accordance with the correction value calculated by the calculation means.

[0200] (Configuration 2) The image processing device according to configuration 1, wherein the information relating to the white balance is information relating to at least one of a deviation of the white balance, a correction amount of the white balance, and a white balance gain.

[0201] (Configuration 3) The method further includes a determination unit for determining whether or not the input image is affected by infrared light, 3. The image processing device according to claim 1, wherein when the detection means detects a shift in the black balance, the calculation means calculates the correction value based on the presence or absence of an effect of infrared light on the input image.

[0202] (Configuration 4) The image processing device described in any one of configurations 1 to 3, characterized in that the acquisition means acquires information regarding a shooting environment of the input image, and the calculation means, when the detection means detects a shift in the black balance, calculates the correction value based on the information regarding the shooting environment.

[0203] (Configuration 5) The image processing device according to configuration 4, wherein the information regarding the shooting environment is a color temperature of a light source in the shooting environment.

[0204] (Configuration 6) The image processing device according to configuration 4 or 5, wherein the information about the shooting environment is an amount of infrared light in the shooting environment.

[0205] (Configuration 7) A control unit for controlling the white balance of an image is further provided, 7. The image processing device according to any one of configurations 1 to 6, wherein the control means performs control so as to shift the white balance of the image in accordance with information relating to the white balance.

[0206] (Configuration 8) The image processing device according to configuration 7, wherein the control means controls the white balance based on the deviation of the black balance detected by the detection means.

[0207] (Configuration 9) The image processing device described in any one of configurations 1 to 8, characterized in that the detection means detects the black balance deviation by detecting at least one of the following: a signal for each color output from the sensor that captures the input image when the sensor is shaded, or a sensor gain, shutter speed, iris aperture value when the input image is captured, the temperature of the imaging device that captures the input image, the operating time of the imaging device, the temperature of the shooting environment, the illuminance of the shooting environment, and the amount of infrared light in the shooting environment.

[0208] (Configuration 10) The image processing device described in Configuration 9, characterized in that the detection means detects a greater deviation in the black balance the greater the sensor gain when capturing the input image, the slower the shutter speed, the closer the iris aperture value is to the maximum aperture, the higher the temperature of the imaging device capturing the input image, the longer the operating time of the imaging device, the higher the temperature of the shooting environment, the lower the illuminance of the shooting environment, or the greater the amount of infrared light in the shooting environment.

[0209] (Configuration 11) The image processing device according to any one of configurations 1 to 10, further comprising a setting means for setting a priority of a gradation value, wherein the calculation means calculates the correction value according to the priority of the gradation value.

[0210] (Configuration 12) The image processing device according to any one of configurations 1 to 11, wherein the information regarding the static black balance is information detected at a timing different from the timing at which the detection means detects the deviation in the black balance.

[0211] (Configuration 13) The image processing device described in Configuration 12, characterized in that the information regarding the static black balance includes information regarding at least one of the deviations in the static black balance detected at the different times and temperature information at the different times.

[0212] (Configuration 14) The image processing device according to configuration 13, wherein the calculation means calculates the correction value in accordance with the temperature information detected at the different times and the deviation of the static black balance.

[0213] (Configuration 15) The image processing device described in any one of configurations 12 to 14, characterized in that the different timing is at least one of the timing when a user performs calibration, when an imaging device that captures the input image is started up, a predetermined time, or when the imaging device is adjusted during manufacture.

[0214] (Configuration 16) The image processing device according to any one of configurations 1 to 15, wherein the calculation means calculates the correction value for correcting the black balance deviation according to the type of sensor that captured the input image.

[0215] (Configuration 17) The image processing device according to configuration 16, characterized in that the type of the sensor includes at least one of a sensor using an avalanche photodiode as an image sensor, a photon counting sensor, a SPAD sensor, a CMOS sensor, and a CCD sensor.

[0216] (Method 1) An image processing method comprising: a detection step of detecting a black balance deviation in an input image; an acquisition step of acquiring at least one of information regarding white balance or information regarding static black balance to be applied to the input image; a calculation step of calculating a correction value for correcting the black balance deviation based on at least one of the information regarding the white balance or the information regarding the static black balance if the black balance deviation is detected in the detection step; and an adjustment step of adjusting the black balance deviation in accordance with the correction value calculated in the calculation step.

[0217] (Method 2) Further comprising a determination step of determining whether or not the input image is affected by infrared light, The image processing method described in method 1, characterized in that, in the calculation step, if a deviation in the black balance is detected in the detection step, the correction value is calculated based on the presence or absence of an effect of infrared light on the input image.

[0218] (Method 3) In the acquiring step, information regarding a shooting environment of the input image is acquired, and calculating the correction value based on information about the photographing environment when a deviation in the black balance is detected in the detecting step. 3. The image processing method according to method 1 or 2,

[0219] (Method 4) The image processing method according to any one of Methods 1 to 3, characterized in that in the calculation step, the correction value for correcting the black balance deviation is calculated depending on the type of sensor that captured the input image.

[0220] (Method 5) The image processing method described in Method 4, characterized in that the type of sensor includes at least one of a sensor that uses an avalanche photodiode as an image sensor, a photon counting sensor, a SPAD sensor, a CMOS sensor, and a CCD sensor.

[0221] (Program) A program for causing a computer to execute the image processing method according to any one of Methods 1 to 5.

[0222] 20 is a diagram showing an example of a hardware configuration of an image processing device according to the present invention. The image processing device has an input I / F 1, an output I / F 2, and a CPU (Central Processing Unit) 3 as a computer. It also has a RAM (Random Access Memory) 4 and a ROM (Read Only Memory) 5 as storage media.

[0223] The input I / F 1 is an interface for accepting an input image. The CPU 3 loads a computer program stored in the ROM 5 into the RAM 4 and executes it to realize one or more functions in the first to fifth embodiments described above. The output I / F 2 is an interface for outputting an image after the above-mentioned image processing has been performed on the input image.

[0224] <Other embodiments> In order to realize a part or all of the control in the above-described embodiment, a computer program for realizing the functions of the above-described embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0225] 101: BB detection unit 102: WB correction setting section 103: BB offset calculation unit 104: Infrared light detector 105: WB gain control section 106: BB offset addition unit 107: Feature acquisition unit 108: WB gain calculation section 109: WB gain multiplication section 410:Priority setting section 711: Static BB correction section

Claims

1. A generating means for generating an input image by multiplying an output signal output from an image sensor by a predetermined gain; an acquisition means for acquiring at least one of information on white balance to be applied to the input image and information on static black balance, and the predetermined gain value; a calculation unit that calculates a correction value for correcting the predetermined gain based on at least one of information related to the white balance and information related to the static black balance; an adjusting means for adjusting the value of the predetermined gain in accordance with the correction value calculated by the calculating means; a determination means for determining whether or not the input image is affected by infrared light, The calculation means calculates the correction value based on whether or not the input image is affected by infrared light.

1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the information relating to the white balance is information relating to at least one of the amount of correction of the white balance and the white balance gain.

3. the acquiring means acquires information about a shooting environment of the input image, 2. The image processing apparatus according to claim 1, wherein the calculation means calculates the correction value based on information about the shooting environment.

4. 4. The image processing device according to claim 3, wherein the information about the image capturing environment is a color temperature of a light source in the image capturing environment.

5. 4. The image processing apparatus according to claim 3, wherein the information about the photographing environment is the amount of infrared light in the photographing environment.

6. Further comprising a control means for controlling the white balance of the image, 2. The image processing apparatus according to claim 1, wherein the control means controls the white balance of the image to be shifted in accordance with information about the white balance.

7. 7. The image processing apparatus according to claim 6, wherein the control means controls the white balance based on the value of the predetermined gain.

8. 2. The image processing apparatus according to claim 1, further comprising a setting unit for setting a priority of a gradation value, wherein the calculation unit calculates the correction value in accordance with the priority of the gradation value.

9. 2. The image processing apparatus according to claim 1, wherein the calculation means calculates the correction value for correcting the predetermined gain value in accordance with the type of sensor that captured the input image.

10. 10. The image processing device according to claim 9, wherein the type of the sensor includes at least one of a sensor using an avalanche photodiode as an image pickup element, a photon counting sensor, a SPAD sensor, a CMOS sensor, and a CCD sensor.

11. An image processing method executed by one or more processors, comprising: a generating step of generating an input image by multiplying an output signal output from the image sensor by a predetermined gain; an acquisition step of acquiring at least one of information on white balance to be applied to the input image and information on static black balance and the predetermined gain value; a calculation step of calculating a correction value for correcting the value of the predetermined gain based on at least one of information related to the white balance and information related to the static black balance; an adjusting step of adjusting the value of the predetermined gain in accordance with the correction value calculated in the calculating step; a determination step of determining whether or not the input image is affected by infrared light, In the calculation step, the correction value is calculated based on whether or not the input image is affected by infrared light. An image processing method comprising:

12. A generation step of generating an input image by multiplying an output signal output from an image sensor by a predetermined gain; an acquisition step of acquiring at least one of information on white balance to be applied to the input image and information on static black balance and the predetermined gain value; a calculation step of calculating a correction value for correcting the value of the predetermined gain based on at least one of information related to the white balance and information related to the static black balance; an adjusting step of adjusting the value of the predetermined gain in accordance with the correction value calculated in the calculating step; a determination step of determining whether or not the input image is affected by infrared light, In the calculation step, the correction value is calculated based on whether or not the input image is affected by infrared light.

2. A program for causing a computer to execute an image processing method according to claim 1.

Images