Image processing device, image processing method, and program

Abstract

To allow appropriate white balance control even when a visible light component and an infrared light component are mixed.SOLUTION: An image processing apparatus has: acquisition means that acquires information related to a first control value based on an input image; determination means that determines if a predetermined first area can be changed where the first control value acquired by the acquisition means is present; control means that, based on a result from the determination means, changes the predetermined first area to a second area different from the first area; and gain calculation means that calculates a second control value to be applied to an output image in the second area resulting from the change made by the control means. The image processing apparatus determines a predetermined area for calculating an appropriate control value by gradually changing the predetermined first area based on the operation of the first control value.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] , Acquisition means for acquiring a first control value and a target value for the white balance gain applied to the input image based on the input image; estimation means for estimating the influence of the infrared light component in the input image; determination means for determining whether a first retraction region containing the first control value can be changed to a second retraction region different from the first retraction region, among a plurality of retraction regions pre-set in the white balance gain space; control means for changing the first retraction region to the second retraction region when the determination means determines that the first retraction region can be changed to the second retraction region; and when the control means changes the first retraction region to the second retraction region, the second control value for the white balance gain applied to the output image is set to the second The system includes a gain calculation means that calculates within the retraction region, and the determination means determines that the first retraction region can be changed to a second retraction region if the influence of the infrared light component estimated by the estimation means exceeds a predetermined threshold, the target value is outside the first retraction region, and the first control value is near the boundary line of the first retraction region. The control means changes the first retraction region in stages using a retraction region that overlaps with the first retraction region in at least part of the plurality of retraction regions so that the first control value is included in the second retraction region, and the control means controls the amount of change from the first retraction region to the second retraction region according to the difference between the first control value and the target value.

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Conventionally, a technique for switching the white balance control method according to the position of an infrared cut filter, which is a mechanism for preventing the capture of infrared light components, is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003] [[ID=2第三]]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to enable appropriate white balance control even when a visible light component and an infrared light component are mixed.

Means for Solving the Problems

[0005] The image processing apparatus is characterized in that... Acquisition means for acquiring a first control value and a target value for the white balance gain applied to the input image based on the input image; estimation means for estimating the influence of the infrared light component in the input image; determination means for determining whether a first retraction region containing the first control value can be changed to a second retraction region different from the first retraction region, among a plurality of retraction regions pre-set in the white balance gain space; control means for changing the first retraction region to the second retraction region when the determination means determines that the first retraction region can be changed to the second retraction region; and when the control means changes the first retraction region to the second retraction region, the second control value for the white balance gain applied to the output image is set to the second The system includes a gain calculation means that calculates within the retraction region, and the determination means determines that the first retraction region can be changed to a second retraction region if the influence of the infrared light component estimated by the estimation means exceeds a predetermined threshold, the target value is outside the first retraction region, and the first control value is near the boundary line of the first retraction region. The control means changes the first retraction region in stages using a retraction region that overlaps with the first retraction region in at least part of the plurality of retraction regions so that the first control value is included in the second retraction region, and the control means controls the amount of change from the first retraction region to the second retraction region according to the difference between the first control value and the target value. This is the feature...

Effects of the Invention

[0006] According to the present invention, it is possible to provide good white balance control considering the influence of infrared light components.

Brief Description of the Drawings

[0007] [Figure 1] It is a diagram showing a configuration example of an image processing apparatus according to the first embodiment. [Figure 2] It is a flowchart showing the flow of the main part of the image processing according to the present embodiment. [Figure 3] This figure shows a first example of WB gain control in the first embodiment. [Figure 4] This figure shows a second example of WB gain control in the first embodiment. [Figure 5] This figure shows a third example of WB gain control in the first embodiment. [Figure 6] This figure shows a fourth example of WB gain control in the first embodiment. [Figure 7] This figure shows an example configuration of an image processing apparatus according to the second embodiment. [Figure 8] This is a flowchart showing the main flow of the image processing in the second embodiment. [Figure 9] This figure illustrates an example of WB gain control in the second embodiment. [Figure 10] This figure shows an example configuration of the image processing apparatus according to the first and second embodiments. [Figure 11] This figure shows an example of the UI configuration in the third embodiment. [Figure 12] This figure shows an example of the UI in the third embodiment. [Figure 13] This figure shows the hardware configuration of the image processing apparatus according to the first to third embodiments. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings.

[0009] The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications and various conditions (usage conditions, usage environment, etc.) of the device to which the present invention is applied. Furthermore, parts of each embodiment described later may be combined as appropriate. In the following embodiments, the same components are denoted by the same reference numerals.

[0010] (First Embodiment) Figure 1 is a block diagram showing an example configuration of an image processing apparatus according to the first embodiment of the present invention. The image processing apparatus of the first embodiment will be described below with reference to Figure 1. The image processing apparatus of this embodiment is assumed to be a device built into or connected to an imaging device such as a digital camera or a surveillance camera.

[0011] The input image is an image captured by an imaging unit consisting of a lens and an image sensor built into or connected to this device. The input image is image data or an image signal consisting of multiple pixels and contains information on multiple colors. The multiple colors are, for example, red, green, and blue. The image data is data converted into an electrical signal by the image sensor after passing through color filters corresponding to each color, which are provided on the image sensor built into or connected to this device. The color filters transmit not only visible light corresponding to red, green, and blue, but also some infrared light (invisible light). In general imaging devices, to account for the effect of infrared light components, for example, an infrared light cut filter is provided to remove the infrared light component, thereby obtaining an image that is close to human vision. The image sensor consists of an image element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge-Coupled Device).

[0012] The output image shows an image with appropriate white balance correction by applying the white balance gain, which is the final white balance control value calculated as described later, to the pixel values ​​of the input image. In the following description, white balance will be referred to as WB as appropriate. The image processing device of this embodiment obtains an output image with appropriate white balance correction by controlling the WB depending on whether or not the input image is affected by the infrared light component captured by the built-in or connected image sensor.

[0013] The WB information acquisition unit 101 acquires the WB status and outputs it to the area shift determination unit 103. The WB status refers to the WB control method, WB gain control value, and WB gain target value of the connected digital camera or surveillance camera. The WB control method includes, for example, a manual method that sets a fixed WB gain control value, and an automatic method that automatically calculates the WB gain control value according to the shooting environment. The WB gain control value is a value obtained in accordance with the WB gain target value calculated within a predetermined area according to the color features of the input image. In this way, the WB information acquisition unit 101 acquires a first control value (WB gain control value) related to white balance based on the input image. The color features of the input image are the color information for each area determined by the image data contained within each area when the input image is equally divided into multiple areas. Specifically, the color features are representative values ​​of the color difference signals for each equally divided area, and the representative values ​​are, for example, the mean or the mode. The predetermined region is a region where it is predetermined how much color is drawn in and corrected to white, depending on the ambient light and the spectral characteristics of the sensor, in order to apply an appropriate WB gain control value. The region consists of three or more points that indicate the WB gain value, and in the following explanation, this predetermined region will be referred to as the draw-in region. The WB gain control value is calculated only within the draw-in region. By applying the WB gain control value to the input image, an output image with a more appropriately corrected WB than the input image can be obtained. The WB gain target value is the converged value of the WB gain control value obtained to appropriately control the WB of the output image. The WB gain target value may be determined from the color information of the input image, or a fixed value or limit value may be set. Specifically, a representative value of the input image's color information is calculated from the color information of each region of the input image, and the WB gain target value may be calculated so that the representative value of the output image's color information becomes a predetermined value. In addition, there are cases where a fixed value is set manually, and cases where a limit value is set within the draw-in region to prevent the WB gain control value from being calculated outside the draw-in region in environments unaffected by infrared light components.In this embodiment, the WB gain used for WB control is, for example, a Red (R) gain for adjusting the redness of an image (hereinafter referred to as R gain) and a Blue (B) gain for adjusting the blueness of an image (hereinafter referred to as B gain).

[0014] Based on the feature amount related to the color of the input image, the infrared light component estimation unit 102 estimates the degree to which the color of the input image is affected by the infrared light component captured by the imaging sensor, and outputs the estimation result to the region shift determination unit

[0015] . Specifically, the infrared light component estimation unit 102 calculates the average value of the color information of the input image, and estimates that the greater the red component of the calculated color average value, the greater the influence of the infrared light component. On the other hand, the smaller the red component of the calculated color average value, the smaller the influence of the infrared light component is estimated.

[0015] The region shift determination unit 103 determines whether the pull-in region for obtaining an appropriate WB gain control value can be shifted according to the inputs from the WB information acquisition unit 101 and the infrared light component estimation unit 102, and outputs the determination result to the region shift control unit 104. Specifically, based on the information on the WB control method obtained from the WB information acquisition unit 101, and the relationship between the WB gain control value, the WB gain target value, and the pull-in region, it is determined whether the current pull-in region can be shifted. More specifically, the relationship between the WB gain control value, the WB gain target value, and the pull-in region indicates at which point within the pull-in region the current WB gain control value is calculated, and at which point the WB gain target value is set with respect to the pull-in region. In other words, the relationship between the WB gain control value, the WB gain target value, and the pull-in region indicates at which points the calculated value of the WB gain control value and the set position of the WB gain target value are located with respect to the pull-in region. For example, as part of the conditions for being able to shift, there is a case where the control value of the WB gain is located at the edge of the pull-in region, that is, on the boundary line of the region. From the input of the infrared light component estimation unit 102, it is determined whether the pull-in region can be shifted based on the magnitude of the influence of the infrared light component received by the input image. Specifically, when the estimated value of the infrared light component received by the current input image exceeds a predetermined threshold, it is determined that the pull-in region can be shifted. The determination conditions for whether the pull-in region can be shifted are a combination of multiple conditions. Specifically, when the following conditions (a) to (d) are satisfied, it is determined that the current pull-in region can be shifted. (a) The control method of WB is the auto method. (b) The ratio of the infrared light component to the visible light component exceeds a predetermined threshold. (c) The WB gain target value is set outside the pull-in region. (d) The current WB gain control value is located at the edge of the pull-in region (on the boundary line between the inside and outside of the pull-in region).

[0016] The region shift control unit 104 controls the retraction region in response to inputs from the infrared light component estimation unit 102 and the region shift determination unit 103, and outputs information about the controlled retraction region to the gain calculation unit 105. Specifically, the region shift control unit 104 controls which of the adjacent regions to shift the current retraction region to, based on the ratio of the infrared light component and the determination result of whether the region can be shifted, in order to cancel out the influence of the infrared light component. For example, if the ratio of the infrared light component is large, the WB gain target value is set outside the current retraction region, and the WB gain control value is located on the boundary line of the retraction region, the retraction region will be shifted in a direction that cancels out the influence of the infrared light component.

[0017] The gain calculation unit 105 receives information from the region shift control unit 104 after the retraction region has been controlled, and calculates a WB gain control value so that the output image is properly WB controlled in the retraction region after the shift. Specifically, the WB gain control value is calculated in a manner that follows the WB gain target value as described above, and the WB gain control value is calculated in the retraction region after the shift.

[0018] The gain application unit 106 obtains an appropriate WB gain control value from the gain calculation unit 105, multiplies the input image by that WB gain control value to generate an output image to which WB control is applied, and outputs it.

[0019] Figure 2 is a flowchart showing an example of the main flow of image processing performed by the image processing apparatus of this embodiment. Hereinafter, an example of the image processing apparatus according to this embodiment will be described with reference to the flowchart in Figure 2. In the following description of the flowchart, processing steps are represented by the symbol "S".

[0020] First, in step S1, the WB information acquisition unit 101 acquires the WB status (WB control method, WB gain control value, and WB gain target value), i.e., WB gain information, from the settings of the connected device and information about the color of the input image.

[0021] Next, in step S2, the region shift determination unit 103 determines whether the retraction region A (hereinafter referred to as region A) can be shifted based on the WB control method, using the WB gain information acquired in step S1. For example, if the WB control method is set to allow the retraction region to be shifted, such as auto, the first condition for shifting region A is met, and the process proceeds to step S3. Alternatively, if the WB control method is set to manual or preset, and the retraction region is fixed, region A is not shifted, and the process proceeds to step S11.

[0022] In step S3, the infrared light component estimation unit 102 estimates the effect of the infrared light component and calculates the ratio R of the infrared light component to the visible light component.

[0023] In step S4, the shift determination unit 103 determines whether region A can be shifted based on the ratio R calculated in step S3. For example, if the ratio R exceeds a predetermined threshold, the second condition for shifting region A is met, and the process proceeds to step S5. Alternatively, if the ratio R does not exceed the predetermined threshold, region A is not shifted, and the process proceeds to step S11.

[0024] In step S5, the shift determination unit 103 determines whether region A can be shifted based on the WB gain target value from the WB gain information acquired in step S1. For example, if the WB gain target value is outside region A, the third condition for shifting region A is met, and the process proceeds to step S6. Alternatively, if the WB gain target value is set inside region A, region A is not shifted, and the process proceeds to step S11.

[0025] In step S6, the region shift determination unit 103 determines whether region A can be shifted based on the WB gain control value from the WB gain information acquired in step S1. For example, if the WB gain control value is located at the edge of region A, i.e., on the boundary line, the fourth condition for shifting region A is met, and the process proceeds to step S7. Alternatively, if the WB gain control value is not located on the boundary line of region A, region A is not shifted, and the process proceeds to step S11.

[0026] In step S7, the shift direction of region A is determined based on the ratio R of the infrared light component calculated by the infrared light component estimation unit 102. For example, if the ratio R is higher than a predetermined threshold for the direction that is more affected by the infrared light component, the direction in which region A is shifted is determined to cancel out the effect of the infrared light component. On the other hand, if the ratio R is lower than a predetermined threshold for the direction in which the effect of the infrared light component is reduced, the direction in which region A is shifted is determined to give more consideration to the effect of the visible light component than the effect of the infrared light component. Once the direction is determined, the process proceeds to step S8.

[0027] In step S8, the region shift determination unit 103 determines whether region A can be shifted based on the shift direction of region A determined in step S7. For example, if the next region is set on the side of region A in the shift direction, the fifth condition for shifting region A is met, and the process proceeds to S9. Alternatively, if the WB gain target value is set inside region A, region A is not shifted, and the process proceeds to step S11.

[0028] In step S9, the region shift control unit 104 shifts region A to a different region a based on the result of determining whether it is shiftable in step S8 and the shift direction of region A determined in step S7. Specifically, in a plurality of pre-set stepwise pull-in regions, there is a region a that can be shifted in the direction determined in step S7 based on the determination result in step S8 relative to the current region A. A detailed explanation will be given later with the help of a diagram, but region A is shifted to the shiftable region a, and the process proceeds to step S10.

[0029] In step S10, the gain calculation unit 105 calculates a new WB gain control value for the region a shifted in step S9 to perform appropriate WB control on the input image. Specifically, the WB gain control value is calculated within region a so as to follow the WB gain target value. At this time, the WB gain target value does not have to be set within the shifted region a. In that case, the WB gain control value is calculated on the boundary line of region a that is closest to the WB gain target value. After that, the process proceeds to step S12.

[0030] In step S11, if the region shift determination unit 103 determines that region A cannot be shifted, the gain calculation unit 105 calculates a WB gain control value for region A to perform appropriate WB control on the input image. Specifically, the WB gain control value is calculated within region A to follow the WB gain target value. At this time, the calculated WB gain control value may be the same as the WB gain control value obtained in step S1. For example, this may occur if the WB gain target value is set outside region A, the WB gain control value is located on the boundary line of region A, and a shift determination is not possible because there is one or more regions in the shift direction of region A. Once the WB gain control value is calculated, the process proceeds to step S12.

[0031] In step S12, the gain application unit 106 multiplies the input image by the WB gain control value calculated in step S10 or step S11 to output an output image with appropriately corrected white balance.

[0032] The following describes WB control and its effects in this implementation.

[0033] Figure 3 shows a first WB control example, which is an example of WB gain control in this embodiment.

[0034] In Figure 3, predetermined region A0 represents the region where the WB gain control value is calculated when the input image color is not affected by the infrared light component. Predetermined region A3 represents the region where the WB gain control value is calculated when the input image color is most affected by the infrared light component. That is, the more the input image color is affected by the infrared light component relative to the current retraction region, the more the retraction region shifts towards region A3 in the lower left of Figure 3. Also, the less the influence of the infrared light component, the more it shifts towards region A0. Regions A1 and A2 are pre-set regions for stepwise shifting when shifting between region A0 and region A3. In the example of this embodiment shown in Figure 3, regions A1, A2, and A3 are each set to the same size as region A0, and regions A1 and A2 are arranged so as to overlap at equal intervals along a straight line between region A0 and region A3.

[0035] The black dots in the figure represent the WB gain control value (Control Value), denoted as Cv. The black cross marks in the figure represent the WB gain target value (Target Value), denoted as Tv. In the following figures, the black dots Cv and black cross marks Tv represent the WB gain control value and the WB gain target value, respectively. In this embodiment, the WB gain is assumed to be the R gain, which adjusts the redness of the image, and the B gain, which adjusts the blueness of the image, as previously explained. The positions of Cv and Tv in the figure are determined by the respective R gain and B gain.

[0036] Under typical ambient light conditions, as illumination decreases, the infrared light component becomes more dominant than the visible light component, and the proportion of the infrared light component gradually increases. In this embodiment, as an example, we will describe the auto white balance control when infrared light is irradiated from a state with almost no infrared light component, resulting in a sudden increase in the influence of the infrared light component. The suction regions are pre-configured as regions A0, A1, A2, and A3 in Figure 3. Assume that before infrared light irradiation, Tv was set to the same position as Cv, and at this time the suction region is hardly affected by the infrared light component, so it becomes region A0, which is a normal suction region. When infrared light is irradiated here, the proportion of the infrared light component increases, and the image color becomes reddish, so the WB gain target value moves to the position of Tv in Figure 3 in order to correct the color. Since the WB gain control value is calculated within the suction region, if the value of Tv is at the position in Figure 3, then Cv is located on the boundary line of region A0, which is the current suction region, that is closest to Tv. At this point, the ratio of infrared light components exceeds a predetermined threshold for region A0. Furthermore, the WB gain target value is outside the current region A0, the WB gain control value is on the boundary line of the current region A0, and the shifted region A1 exists at the end of the region shift direction. Therefore, region A0 is determined to be shiftable. Since the region shift is performed in steps, region A0 is shifted to region A1. A new WB gain control value is calculated in the shifted region A1 to bring it closer to Tv. Cv gradually approaches Tv, and when it reaches the boundary line of region A1, the ratio of infrared light components exceeds a predetermined threshold for region A1. The WB gain target value is outside the current region A1, and the WB gain control value is on the boundary line of the current region A1. Furthermore, since the shifted region A2 exists at the end of the region shift direction, region A1 is determined to be shiftable. Since the region shift is performed in steps, region A1 is shifted to region A2. These processes are repeated until Cv can approach the limit of Tv. In the example in Figure 3, the data is shifted to region A3, and since Tv is included in region A3, Cv will have the same value as Tv.

[0037] When infrared light irradiation is stopped and the initial ambient light is returned, the proportion of infrared light decreases and the visible light component becomes dominant, so Tv returns to its initial position, i.e., the position of Cv in region A0 in Figure 3. At this time, Cv, which moved to the position of Tv shown in Figure 3 due to the influence of the infrared light component, begins to move towards Tv, which has returned to its initial position as the influence of the infrared light component disappears. Cv inside region A3 approaches Tv inside region A0, and when it reaches the boundary line of region A3 in the direction of region A0, the proportion of infrared light component exceeds the lower side of a predetermined threshold for region A3. The WB gain target value is located outside the current region A3, and the WB gain control value is located on the boundary line of the current region A3. Also, since the shifted region A2 exists at the end of the region shift direction, i.e., on the side with a smaller infrared light component, it is determined that region A3 can be shifted. Since the region shift is performed in steps, region A3 is shifted to region A2. These processes are repeated until Cv approaches Tv as closely as possible, and when it is shifted back to region A0 and returns to its initial position, Cv is calculated to have the same value as Tv.

[0038] In existing white balance control systems, the pull-in range is fixed to, for example, region A0 in Figure 3, regardless of whether the input image's color is affected by infrared light components. Therefore, when the input image's color is affected by infrared light components, the optimal white balance gain control value, which is the target white balance gain value in region A3 of Figure 3, cannot be applied. Instead, a white balance gain control value located within or on the boundary of region A0 is applied. If the difference between the actually applied white balance control value and the optimal white balance gain target value is large, the white balance may be disrupted, potentially degrading image quality.

[0039] As in this embodiment, by shifting the retraction region from the normal region A0 to retraction regions A1, A2, and A3 that take into account the influence of infrared light components, according to the strength of the infrared light component's influence, it is possible to perform appropriate WB control even for images affected by infrared light components. Furthermore, by shifting the retraction region in steps, the WB gain control value is always located within the retraction region, avoiding problems such as the WB gain control value being located outside the retraction region and therefore being unable to be calculated. In addition, when the image is suddenly affected by infrared light components, the WB gain control value is calculated gradually as the retraction region shifts in steps, resulting in a gradual change in the image's color. In auto white balance control, abrupt changes in perceived color are undesirable, so the WB control of this embodiment, which allows for gradual changes, is effective.

[0040] Furthermore, in the WB control of this embodiment, the retraction region A3 can be set according to the sensitivity of the imaging sensor, for when the input image color is most affected by the infrared light component. That is, in this embodiment, the higher the sensitivity of the imaging sensor, the further the position of region A3 can be set from the position of the normal retraction region A0 when the input image color is not affected by the infrared light component. Then, by setting one or more overlapping regions between region A0 and region A3, stepwise shifting becomes possible. The number of shift steps shown in this embodiment was three, including the shift limit, but the number of shift steps can be freely determined as long as the design ensures that the same WB gain control value is included in the regions before and after the shift when shifting the retraction region. Furthermore, in this embodiment, it was determined whether the current region could be shifted based on whether the WB gain control value was located on the boundary line of the region, but this is not limited to this. It is possible to determine that the region can be shifted if the WB gain control value is located near the boundary line of the region, and the user can freely decide how close it needs to be to the boundary line.

[0041] In the WB control of this embodiment, the infrared light component estimation unit 102 estimates the extent to which the color of the input image is affected by the infrared light component captured by the imaging sensor, based on the color feature quantity of the input image. However, it is not limited to this. For example, the extent to which the input image is affected by the infrared light component may be estimated according to the insertion / removal state of the IRCF built into or connected to the device of this embodiment, or, if an infrared illumination unit is connected, the ON / OFF state of the infrared illumination. That is, the ratio of the infrared light component may be estimated to be large when the IRCF is removed or when the infrared illumination unit is emitting infrared light. Conversely, the ratio of the infrared light component may be estimated to be small when the IRCF is inserted or when the infrared illumination unit is not emitting infrared light. Alternatively, the user may be able to set whether or not the color of the input image is affected by the infrared light component. That is, a WB control mode for when the color of the input image is affected by the infrared light component and a WB control mode for when the color of the input image is not affected by the infrared light component may be provided, and the user may be able to select either WB control mode. Furthermore, in the WB control mode when the input image's color is affected by infrared light components, the user may set the ratio of the infrared light components and control the retraction area according to the magnitude of the ratio. However, the method of shifting to the specified retraction area shall follow that of this embodiment, and the area shall be shifted in steps.

[0042] As mentioned above, the lower the illumination, the greater the ratio of infrared light to visible light in the ambient light, resulting in a stronger reddish tint in the captured image. To correct this strong redness in the captured image, it is necessary to apply a WB gain control value that is significantly different from the WB gain control value used when the input image's color is not affected by the infrared light component. Furthermore, the higher the sensitivity of the imaging sensor, the higher the signal-to-noise ratio (SNR) even in low-light environments, resulting in an image that retains the subject's original colors. Therefore, the higher the sensitivity of the imaging sensor, the more actively WB control can be applied in low-light environments, even when the input image's color is affected by the infrared light component, thereby improving color reproduction. On the other hand, if the imaging sensor's sensitivity is insufficient, actively controlling the WB may cause significant WB degradation due to noise. Therefore, even when the input image's color is affected by the infrared light component, applying a WB gain control value that is smaller in difference from the WB gain control value used when the infrared light component is not affected can prevent significant WB degradation.

[0043] Based on these considerations, the WB control of this embodiment actively controls the white balance (WB) even when the input image's color is affected by infrared light components, as the sensitivity of the imaging sensor increases. In other words, the WB control of this embodiment applies a WB gain control value that is significantly different from the WB gain control value applied when the input image's color is not affected by infrared light components, as the sensitivity of the imaging sensor increases. This prevents significant WB degradation while improving the color reproduction of captured images in low light conditions.

[0044] Figure 4 shows an example of a second WB control in this embodiment.

[0045] In the first WB control example, multiple regions were arranged to overlap linearly at equal intervals for the retraction regions that were pre-set in steps by the region shift control unit 104, but this is not the only option. In the second WB control example shown in Figure 4, regions A1 and A2 of the retraction regions that were pre-set in steps in the first WB control example are replaced with regions B1 and B2. The difference from Figure 3 is that the regions are not linearly shifted between the normal retraction region A0 and region A3, which is the limit of the region that can be shifted when the influence of the infrared light component is at its maximum. Since the way in which the image sensor is affected by the infrared light component differs depending on the sensitivity and spectral characteristics of the imaging sensor, it is possible to correct the WB in a way that takes the influence of the infrared light component into more consideration by setting the regions according to these characteristics. Specifically, for images that have a reddish tint due to the influence of the infrared light component, regions that reduce the R gain early are arranged, as shown in regions B1 and B2 in Figure 4, in order to more actively control the R gain, which adjusts the redness based on measured values ​​and the spectral characteristics of the imaging sensor. In addition to the steppedly arranged regions, the positions of the normal retraction region A0 and the shift limit region A3 may also be freely designed based on measured values ​​and the characteristics of the imaging sensor.

[0046] Figure 5 shows an example of a third WB control in this embodiment.

[0047] In the first WB control example, the retraction regions A1, A2, and A3 were set to the same size as region A0 in the region shift control unit 104, but this is not limited to this. In the third WB control example shown in Figure 5, regions A1 and A2 of the retraction region, which were set in stages in the first WB control example, are replaced with regions C1 and C2. The difference from Figure 3 is that the size of the shifted region is not constant between the normal retraction region A0 and region A3, which is the limit of the shift that can be achieved when the influence of the infrared light component is at its maximum. The color of an image affected by the infrared light component becomes reddish and does not represent the correct color. Therefore, from the time the influence of the infrared light component begins until the ratio of the infrared light component becomes constant, that is, between regions C1 and C2 where the retraction region is shifted, the size of the retraction region is expanded to actively draw in white, making it easier to get closer to the original color of the subject. The size of the retraction region can be freely designed from measured values ​​or the spectral characteristics of the image sensor. In addition to the stepped arrangement of regions, the sizes of the normal retraction region A0 and the shift limit region A3 may also be freely designed based on measured values ​​and the characteristics of the image sensor. Furthermore, there may be cases where only the size of the region expands compared to the normal retraction region A0.

[0048] Figure 6 shows an example of a fourth WB control in this embodiment.

[0049] In the first WB control example, the retraction regions A1, A2, and A3 were set to have the same shape as region A0 in the retraction regions that were pre-set in steps by the region shift control unit 104, but this is not limited to this. In the fourth WB control example shown in Figure 6, regions A1, A2, and A3 of the retraction region that were pre-set in steps in the first WB control example are replaced with regions D1, D2, and D3. The difference from Figure 3 is that the shape of the shift region is not constant when affected by the infrared light component. The retraction region for correcting the WB for an image affected by the infrared light component may be designed differently from the region for correcting the WB for an image affected by the visible light component. In other words, it is possible to design a retraction region that is specialized only for the effect of the infrared light component. Specifically, it is possible to design a retraction region according to the ratio of the infrared light component based on measured values ​​or the spectral characteristics of the imaging sensor. For example, if you want to allow for a wider range of adjustment with respect to the R gain that controls the redness, you can design wider regions along the R gain axis, as shown in regions D1, D2, and D3 in Figure 6.

[0050] As described above, the image processing apparatus of this embodiment can generate appropriate image data when the input image is affected by the infrared light component, according to the magnitude of the infrared light component ratio and the current WB gain information.

[0051] (Second embodiment) The following describes the image processing apparatus of the second embodiment. In the first embodiment, the retraction region, which is set in steps in advance, is controlled to shift to one of the adjacent regions. In contrast, in the WB control of the second embodiment, the method of shifting the retraction region is changed according to the convergence speed of the WB gain control value. The more steps in the retraction region that are set in advance, the more effective it is when the influence of the infrared light component changes gradually. However, when the influence of the infrared light component changes rapidly, the processing of shifting the retraction region increases, which may slow down the responsiveness of the WB control. In the second embodiment, appropriate WB control can be performed without impairing the responsiveness of the WB control even when the influence of the infrared light component changes rapidly.

[0052] Figure 7 is a configuration diagram showing an example of the functional configuration of the image processing apparatus according to the second embodiment. The same reference numerals are used for the same functional parts as in the first embodiment, and their descriptions are omitted. The following describes functional parts having different functions from those in the first embodiment.

[0053] The region shift control unit 701 controls the retraction region in response to inputs from the infrared light component estimation unit 102 and the region shift determination unit 103, and outputs information about the retraction region after control to the gain calculation unit 105. Specifically, the region shift control unit 701 controls which region to shift the current retraction region to, based on the ratio of the infrared light component and the determination result of whether the region can be shifted, in order to cancel out the influence of the infrared light component. As will be described in detail later, it is not necessary to shift to an adjacent region as long as the conditions for shifting the region in steps are met. For example, when the influence of the infrared light component changes rapidly and the current WB gain control value and the WB gain target value are more than a certain distance apart. When the conditions for shifting the retraction region in the first embodiment are met, the retraction region is shifted as much as possible in a direction that cancels out the influence of the infrared light component. As much as possible means that the retraction region is shifted so that the WB gain control value does not go outside the region. In other words, the influence of the infrared light component changes rapidly, and the larger the difference between the WB gain control value and the WB gain target value, the greater the shift in the retraction region.

[0054] Figure 8 is a flowchart illustrating an example of the main flow of image processing performed by the image processing apparatus of this embodiment. Hereinafter, an example of the image processing apparatus according to this embodiment will be described with reference to the flowchart in Figure 8. Note that the same processing steps as in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted. The following describes processing steps that differ from those in the first embodiment.

[0055] In step S13, the difference D between the WB gain control value and the WB gain target value acquired by the WB information acquisition unit 101 is calculated. Specifically, in the first embodiment, if the third and fourth conditions for shifting region A are met, a difference D will always occur between the WB gain control value and the WB gain target value, so a difference D that is not zero is calculated. Once the difference D is calculated, the process proceeds to step S14.

[0056] In step S14, the shift direction and amount of region A are determined based on the ratio R of the infrared light component calculated by the infrared light component estimation unit 102 and the difference D calculated in step S13. For example, if the ratio R is above a predetermined threshold on the side that is more affected by the infrared light component, the direction in which region A is shifted is determined to cancel out the effect of the infrared light component. On the other hand, if the ratio R is below a predetermined threshold on the side that is less affected by the infrared light component, the direction in which region A is shifted is determined to give more consideration to the effect of the visible light component than the effect of the infrared light component. In addition, the amount of shift in the direction determined by the ratio R is determined according to the size of the difference D. Specifically, the larger the difference D, the larger the shift amount, and the shifted region a is set at a position further away from the original region A. When the shifted region is set at the furthest position, the shifted region a is set in such a way that it satisfies the condition of shifting the region in steps, as will be described in detail later. Alternatively, the smaller the difference D, the smaller the shift amount, and the shifted region a is set at a position closer to the original region A. If the shifted region is set to the closest position, the shifted region a will be set to the adjacent region. Once the shift direction and shift amount are determined, the process proceeds to step S8.

[0057] In step S8, the same processing as in the first embodiment is performed, and if the fifth condition for shifting the region is met, the process proceeds to step S15. Alternatively, if the fifth condition for shifting the region is not met, the process proceeds to step S11.

[0058] In step S15, the region shift control unit 701 shifts region A to a different region a based on the result of determining in step S8 whether region A is shiftable, and the shift direction and shift amount of region A determined in step S14. Specifically, if there is one or more regions that can be shifted in the direction determined in step S14 based on the determination result in step S8 relative to the current region A, the shifted region a is determined from among the shiftable regions according to the shift amount determined in step S14. A detailed explanation will be given later with the help of a diagram, but region A is shifted to the shiftable region a, and the process proceeds to step S10.

[0059] The following describes the WB control and its effects in the second embodiment.

[0060] Figure 9 shows an example of WB gain control in this embodiment. In Figure 9, the black dot Cv and the black cross mark Tv are the same as those in Figure 3, so no explanation is given. The predetermined region A represents the region where the current WB gain control value is calculated. In Figure 9, Cv is calculated near the boundary of region A, and Tv is set to the lower left side of region A in the figure, influenced by the infrared light component. At this time, the shift direction of region A is determined to be in the direction of Tv, and it is shifted to one of the shiftable regions a1, a2, or a3. The reason why region a4 is not a candidate for the shifted region is as follows. If the retraction region is set in advance as in Figure 9, a stepwise region shift cannot be performed when shifting from region A to region a4. As a result, Cv may temporarily move outside the retraction region, and there is a risk that an appropriate WB gain control value will not be calculated. For example, when Cv is located above the position shown in Figure 9 along the boundary of region A, if the shifted region a3 does not contain Cv, it is excluded from the candidates for the shifted region. In other words, in this case, regions a1 and a2 are candidates for the shifted region. The method for determining the shifted region a, that is, the amount of shift of the current region A, is determined by the distance between Tv and Cv. Specifically, if the distance between Tv and Cv is large, the amount of shift of region A will also be large, and the region will shift to the region furthest from the current region among the candidate regions for the shifted region, satisfying the conditions for a stepwise shift. In the case of Figure 9, since the distance between Tv and Cv is large, the region will shift to region a3, which is the furthest from the current region A among the candidates. The relationship between the distance between Tv and Cv and the amount of shift is determined by the number of regions that are set in steps in advance. The fewer the number of regions that are set, the smaller the amount of shift when the distance between Tv and Cv increases. On the other hand, if the number of regions that are set is large, the larger the amount of shift when the distance between Tv and Cv increases. In this embodiment, the amount of shift does not represent an absolute distance, but rather the number of steps in which the region shifts from the current region to the next region. In the example of Figure 9, the amount of shift from region A to region a3 is 3. That is, it means that a shift of 3 steps has been performed.

[0061] If the speed at which the WB gain control value converges to the WB gain target value is constant regardless of the WB gain target value, then the shift to adjacent regions described in the first embodiment is possible. In general WB control, the speed at which the WB gain control value converges may be changed depending on the WB gain target value in order to adjust the WB faster. Specifically, when the WB gain target value is set to a value farther away than the current WB gain control value, the change in the WB gain control value per unit process is increased to make the WB gain control value converge to the WB gain target value faster. In this case, if the amount of change in the WB gain control value becomes larger than the interval between regions that have been set in stages in advance, the control method for shifting to adjacent regions may not be able to calculate an appropriate WB gain control value. To give an extreme example, in Figure 9, if the amount of change in Cv per unit process is larger than the interval between region A and region a1, then when region A shifts to region a1, Cv will exceed region a1 and go outside the pull-in region. Furthermore, if the magnitude of the change in the WB gain control value is limited and WB control is performed prioritizing the spacing of the pull-in region, the response will be slower than the WB control that is originally required.

[0062] Therefore, in this embodiment, when the convergence speed of the WB gain control value changes depending on the distance between Tv and Cv, the amount of shift of the retraction region is controlled according to the amount of change per unit processing of Cv, so that the WB gain control value is always located within the shifted region. In other words, even when the convergence speed of the WB gain control value is changed according to the WB gain target value, the WB gain control value is correctly calculated without falling outside the retraction region, and appropriate WB control can be performed without slowing down the response. In this embodiment, the shift format of the retraction region is not limited, and for example, WB control of this embodiment is possible even with region shift formats such as those shown in Figures 4, 5, and 6.

[0063] Figure 10 is a block diagram showing an example configuration for a WB control example that takes into account the influence of a specific light component. In the embodiments already described, WB control is performed only for the influence of the infrared light component. In contrast, in this WB control example, WB control is performed in a way that cancels out the influence of a specific light component. A specific light component is a specific color component that the user can freely decide. For example, this is effective when the user wants to control the retraction region for the blue component, assuming WB control is performed underwater. The same reference numerals are used for the same functional parts as in the first and second embodiments, and their descriptions are omitted. This WB control example shows that the user can arbitrarily design a shift limit region corresponding to region A3 in Figure 3 for the influence of any ambient light. Furthermore, the processing related to the control of gradually shifting the region for calculating the appropriate WG gain control value does not change as a result. In other words, WB control can be performed appropriately by gradually shifting the retraction region in the first and second embodiments, not only due to the influence of the infrared light component, and the user can freely design the shift limit region in that case.

[0064] (Third embodiment) Figure 11 shows an example of the GUI (Graphical User Interface) configuration in the third embodiment. The configuration of the image processing apparatus according to the third embodiment is the same as that of the first or second embodiment, so a description will be omitted. In this embodiment, the GUI is displayed by the CPU (display control means) of the hardware configuration of the image processing apparatus, which will be described later, executing display control to an external display means. In this embodiment, the method of displaying the GUI displayed by the display control means will be described.

[0065] GUI1101 outputs video of the shooting environment with the current camera settings. GUI1102 consists of items for setting camera functions such as white balance and exposure. For example, for the white balance function, it is possible to specify the white balance mode, the correction level to take into account the effects of infrared light components, and the R gain or B gain. GUI1103 shows the part of the white balance function that sets the correction level to take into account the effects of infrared light components. For example, in the first embodiment, when the WB gain control is shown in Figure 3, there are normally four stages in the retraction region from the retraction region A0 to the shift limit A3, so there are four stages in the correction level (WB gain control value and the second region including the said WB gain control value).

[0066] In Figure 3, the further the retraction region is shifted towards the direction with a larger infrared light component ratio (the intersection of the vertical and horizontal axes), the stronger the correction applied to cancel out the influence of the infrared light component. This may significantly disrupt the color balance depending on the shooting environment. Therefore, a UI is provided that allows the user to select the shift limit position in the actual shooting environment. In the example of WB control in Figure 3, when the normal retraction region A0 is set to correction level 0, the shift limit A3 becomes correction level 3, and the user can select from four levels from correction level 0 to 3. When selecting each correction level, an image of the correction result is generated and displayed as shown in GUI1104, allowing the user to visually judge the correction result in relation to the current shooting environment. Alternatively, as shown in Figure 12, a UI is provided that allows the user to determine the correction level by arranging the generated correction result image images in a separate window and selecting one of the arranged images. The selected image can be stored as an output image on a storage medium such as an SD card or auxiliary storage device, or output to an external information processing device.

[0067] The shift limit position is determined based on the correction level. Therefore, the user can freely determine the correction level to cancel out the influence of infrared light components in the actual shooting environment, and by providing a corresponding UI, it becomes possible to visually judge, making it easy for the user to select the desired correction level. In this example, based on the WB control example in Figure 3, the number of correction level steps, including no correction, was set to 4, but a minimum of 2 correction level steps is sufficient. For example, if no correction is included, a minimum of 2 steps, correction level 0 and correction level 1, is sufficient. If no correction is not included, a minimum of 2 steps, correction level 1 and correction level 2, is sufficient. Also, although the display example in Figure 12 is arranged in 2 rows and 2 columns, the arrangement pattern does not have to be fixed as long as the corrected images can be arranged. The same UI can be provided not only when considering the influence of infrared light components, but also when considering the influence of specific light components.

[0068] Although the image processing apparatus in the embodiment described above is an example applied to an imaging device, the image processing apparatus may also be implemented using an information processing device (computer) such as a PC (Personal Computer) or smartphone connected to the imaging device. In this case, the imaging device outputs to the computer the raw data captured by the imaging unit, along with shooting parameters such as exposure time, frame rate, and exposure setting value, as well as information indicating whether or not infrared light components have an effect on the color of the input image. The information indicating whether or not infrared light has an effect may be input by the user. Then, the same image processing as described in the embodiment described above is performed on the computer.

[0069] Figure 13 shows the hardware configuration of the image processing apparatus according to the first to third embodiments. The image processing apparatus according to the first to third embodiments includes an input interface 1301, an output interface 1302, a network interface 1303, a CPU 1304, a read-only memory 1305, a random access memory (RAM) 1306, and an auxiliary storage device 1307. Each part of the image processing apparatus transmits information via a bus.

[0070] The input interface 1301 is an interface for receiving user instructions to the image processing unit, and consists of a mouse, keyboard, joystick, etc. The input interface 1301 receives various instructions from the user and inputs them to the CPU 1304.

[0071] Output I / F1302 is an interface for outputting information from the image processing device (for example, processed images and control values ​​related to WB control), and is composed of, for example, an LCD display or an LED display. Output I / F1301 displays a GUI for the user to operate the image processing device.

[0072] The input interface 1301 and output interface 1302 may be configured as an integrated unit, such as a touch panel display. In each of the first to third embodiments, the input interface 1301 and output interface 1302 are not essential. For example, they may receive user operation instructions from the network interface 1303 or transmit output images from an image processing device. In this case, the input interface 1301 and output interface 1302 are not necessary.

[0073] The Network I / F1303 is an interface for communicating with information processing devices such as PCs and smartphones via a network such as a LAN (Local Area Network). Examples include terminals for plugging in LAN cables and antennas for wireless communication.

[0074] The CPU 1304 controls the entire image processing unit and performs the aforementioned WB control, etc., using computer programs and data stored in the ROM 1305 and RAM 1306. The image processing unit may also have one or more dedicated hardware components separate from the CPU 1304, and the dedicated hardware may perform at least a portion of the processing performed by the CPU. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 1305 stores programs that do not require modification. The RAM 1306 temporarily stores programs and data supplied from auxiliary storage devices, as well as data supplied from external sources via the network I / F 1303.

[0075] The auxiliary storage device 1307 consists of an HDD or the like and stores various data such as image data, shooting parameters, and information indicating whether or not infrared light is present.

[0076] Furthermore, the CPU 1304 also operates as an operation control means for controlling the input I / F 1301 and a display control means for controlling the output I / F 1302. Alternatively, it may be configured to send control information for controlling external operation units and display units via a network I / F. The CPU 1304 performs the control for displaying a GUI as shown in the third embodiment.

[0077] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0078] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0079] The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of Symbols]

[0080] 101 WB information acquisition department 102 Infrared light component estimation unit 103 Area Shift Determination Unit 104 Region Shift Control Unit 105 Gain Calculation Section 106 Gain application section

Claims

1. An acquisition means that acquires a first control value for the white balance gain applied to the input image and a target value for the white balance gain based on the input image. Estimation means for estimating the influence of infrared light components in the input image, A determination means for determining whether a first retraction region, which includes the first control value, can be changed to a second retraction region different from the first retraction region, among a plurality of retraction regions pre-set in the white balance gain space, If the determination means determines that the first retraction area can be changed to the second retraction area, a control means for changing the first retraction area to the second retraction area is provided. The system includes a gain calculation means that calculates a second control value for the white balance gain applied to the output image within the second retraction region when the first retraction region is changed to the second retraction region by the control means, The determination means determines that the first retraction region can be changed to a second retraction region if the influence of the infrared light component estimated by the estimation means exceeds a predetermined threshold, the target value is outside the first retraction region, and the first control value is near the boundary line of the first retraction region. When changing the first retraction region to the second retraction region, the control means changes the first retraction region in steps using a retraction region from among the plurality of retraction regions that overlaps with the first retraction region in at least a portion thereof, so that the first control value is included in the second retraction region. The control means controls the amount of change from the first retraction region to the second retraction region according to the difference between the first control value and the target value. An image processing apparatus characterized by the following:

2. The plurality of retraction regions are arranged in a stepwise manner in a direction that cancels out the influence of the infrared light component, as the influence of the infrared light component in the input image becomes greater. The image processing apparatus according to claim 1, characterized in that the control means sets a retraction region arranged in a direction that cancels out the influence of the infrared light component, based on the influence of the infrared light component estimated by the estimation means, as the second retraction region.

3. The image processing apparatus according to claim 1, characterized in that the control means increases the amount of change from the first retraction region to the second retraction region as the difference between the first control value and the target value increases, and determines the amount of change within the range in which the first control value is included in the second retraction region.

4. The acquisition means further acquires information regarding the method of controlling the white balance gain in the connected imaging device. The determination means determines, based on the fact that the control method is an automatic method that automatically calculates the control value for the white balance gain according to the shooting environment, that the first retraction area can be changed to the second retraction area. The image processing apparatus according to claim 2.

5. The image processing apparatus according to claim 4, characterized in that the acquisition means acquires the first control value calculated within the first pull-in region according to a representative value of color information based on image data contained in each region obtained by dividing the input image into a plurality of regions.

6. The target value is the convergence value of the first control value for controlling the white balance of the output image, The image processing apparatus according to claim 5, characterized in that the acquisition means acquires a value calculated based on the color information of the input image, a fixed value, or a limit value set within the first pull-in region as the target value.

7. The image processing apparatus according to claim 1, characterized in that the estimation means estimates the influence of the infrared light component in the input image based on the color feature quantity of the input image.

8. The image processing apparatus according to claim 7, characterized in that the estimation means estimates that the greater the red component in the representative value of the color information of the input image, the greater the influence of the infrared light component in the input image.

9. The image processing apparatus according to claim 1, wherein the estimation means estimates the influence of infrared light components in the input image based on at least one of the insertion / removal state of the infrared light cut filter, the irradiation state of the infrared illumination unit, or a user setting regarding whether or not the input image is affected by infrared light components.

10. The plurality of retraction regions are set according to the sensitivity of the imaging sensor that generates the input image, The image processing apparatus according to claim 1, characterized in that the higher the sensitivity of the imaging sensor, the more the retraction region used when it is most affected by the infrared light component is set to a position farther away from the retraction region used when it is not affected by the infrared light component.

11. The image processing apparatus according to claim 1, characterized in that at least one of the plurality of retraction regions is positioned, sized, or shaped in the white balance gain space based on measured values ​​for an image affected by the spectral characteristics of the imaging sensor that generates the input image or the infrared light component.

12. The image processing apparatus according to claim 1, characterized in that, among the plurality of retraction regions, the retraction region used when affected by the infrared light component is set such that the width in the R gain direction is greater than the width in the B gain direction in the white balance gain space.

13. The image processing apparatus according to claim 1, characterized in that the control means changes the first retraction region in steps in a direction that approaches the retraction region used when there is no influence from the infrared light component, when the influence of the infrared light component estimated by the estimation means becomes small.

14. An image processing method performed by an image processing device, An acquisition step that acquires a first control value for the white balance gain applied to the input image and a target value for the white balance gain based on the input image. An estimation step to estimate the influence of the infrared light component in the input image, A determination step of determining whether it is possible to change a first retraction region, which includes the first control value, from among a plurality of retraction regions pre-set in the white balance gain space, to a second retraction region different from the first retraction region, If the determination step determines that the first retraction area can be changed to the second retraction area, a control step of changing the first retraction area to the second retraction area is performed. The control step includes a gain calculation step of calculating a second control value for the white balance gain applied to the output image within the second retraction region when the first retraction region is changed to the second retraction region by the control step, In the determination step, if the influence of the infrared light component estimated in the estimation step exceeds a predetermined threshold, the target value is outside the first retraction region, and the first control value is near the boundary line of the first retraction region, it is determined that the first retraction region can be changed to the second retraction region. In the control step, when changing the first retraction area to the second retraction area, the first retraction area is changed stepwise using a retraction area from among the plurality of retraction areas that overlaps with the first retraction area in at least a portion, so that the first control value is included in the second retraction area. In the control step, the amount of change from the first retraction region to the second retraction region is controlled according to the difference between the first control value and the target value. An image processing method characterized by the following:

15. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 13.

Images