Image processing device, image processing method, and program

Abstract

To improve the accuracy of determining whether fog or haze is present in an image.SOLUTION: An image processing device estimates, from an image, an estimated value indicating the transmittance of light in the atmosphere at each position in the image, determines as to whether fog or haze is present in the image on the basis of the estimated value, and switches an image correction process from a first correction process to a second correction process when it is determined that fog or haze is present.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Surveillance cameras have been used as one of the crime prevention measures. Surveillance cameras are installed in various environments, and it is desirable for them to provide images with high visibility. However, when fog or haze occurs in the shooting environment of the surveillance camera, the contrast of the subject decreases, resulting in an image with poor visibility. In order to obtain an image with high visibility even in such a shooting environment where fog or haze occurs, there is a method of enhancing contrast when fog or haze occurs.

[0003] Fog or haze is not always constant, and the concentration and other conditions change over time. Therefore, the degree of contrast reduction due to fog or haze in the video is also constantly changing. For this reason, it is possible to make corrections according to the situation by determining whether fog or haze is occurring in the shooting environment where the surveillance camera is installed, and performing contrast enhancement only when fog or haze is occurring. The technology described in Patent Document 1 detects a luminance histogram, calculates the ratio of the luminance distribution within the range of a preset low-luminance side judgment point and a preset high-luminance side judgment point to the entire histogram, and determines that fog or haze is present when the ratio exceeds a preset threshold. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 700080 Summary of the Invention [Problem to be solved by the invention]

[0005] If the presence or absence of fog or haze is determined based solely on the luminance of an image, it may be erroneously determined that fog or haze exists in color regions that tend to have high luminance.

[0006] An object of the present invention is to improve the accuracy of determining whether or not there is fog or haze in an image. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, for example, an image processing device according to one embodiment includes the following configuration: an estimation means for estimating an estimated value indicating the atmospheric light transmittance for each position in an image from the image, a determination means for determining whether or not fog or haze is occurring in the image based on the estimated value, and a control means for switching a correction process for the image from a first correction process to a second correction process when it is determined that fog or haze is occurring. Effect of the Invention

[0008] To improve accuracy in determining the presence or absence of fog or haze in an image. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of an image processing apparatus according to a first embodiment. [Diagram 2] 5 is a flowchart showing an example of image processing by the image processing device according to the first embodiment. [Diagram 3] FIG. 13 is a diagram showing an example of a histogram of estimated values ​​of light transmittance through the atmosphere. [Figure 4] 10 is a flowchart showing an example of image processing by the image processing device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [Embodiment 1] The image processing device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the hardware configuration of an image processing device 100.

[0012] 1, the image processing device 100 includes an optical lens 101, an image sensor 102, an image processing unit 103, a transmittance generation unit 104, a CPU 105, a video output driving unit 106, a frame memory 108, an exposure control unit 109, a ROM 110, a RAM 111, an operation unit 112, and a display driving unit 113. The functional units included in the image processing device 100 are connected to each other via an internal bus 115 so as to be able to transmit and receive data. Moreover, the processing by the functional units including the image processing unit 103 and the transmittance generation unit 104 according to this embodiment is controlled by the CPU 105.

[0013] The optical lens 101 is an optical element having a lens and a motor for driving the lens. The optical lens 101 operates based on a control signal and can optically perform various processes such as enlarging or reducing an image or adjusting a focal length. The optical lens 101 can also adjust the amount of light (for example, to provide brightness according to the amount of operation by a user) by controlling the aperture area of ​​the iris. The light transmitted through the lens is imaged by the image sensor 102.

[0014] The image sensor 102 is a CCD sensor or a CMOS sensor, and converts an optical signal into an electrical signal. The image sensor 102 is driven based on a control signal to reset the charge in the pixel or control the timing of readout. The image sensor 102 can also perform gain processing on the pixel signal read out as an electrical analog signal (voltage value), or convert the analog signal into a digital signal.

[0015] The image processing unit 103 performs various image processing on the image output from the imaging element 102. For example, the image processing unit 103 can perform image processing such as correcting the amount of light in the peripheral part of the image generated due to the characteristics of the optical lens 101, correcting the sensitivity variation for each pixel of the imaging element 102, or performing color correction or flicker correction. The image processing unit 103 also performs sharpening processing using the estimated value estimated by the transmittance generation unit 104, the details of which will be described later. Note that, in this embodiment, the values ​​of the image to be processed are described as being expressed in the RGB color space, but this is not particularly limited as long as similar processing can be performed.

[0016] The image processing unit 103 according to this embodiment switches the correction process for the image from a first correction process to a second correction process when a transmittance generating unit 104 described later determines that fog or haze is occurring in the image. Here, when it is determined that fog or haze is occurring, the image processing unit 103 can switch from a state in which correction process based on a Dark Channel Prior (DCP) method described later is not performed to a state in which correction process based on the DCP method is performed.

[0017] In order to sharpen an image, the transmittance generating unit 104 generates an estimated value indicating the transmittance of atmospheric light (ambient light) using the DCP method. The DCP method is a technique for correcting contrast for each pixel of interest using an image (dark channel image) in which the minimum pixel value in all RGB channels in a predetermined range around the pixel of interest is extracted. The image processing unit 103 can perform correction to remove fog or haze from the image using the DCP method. Therefore, it is possible to correct an image in which visibility has been reduced due to scattering, and improve the contrast. Since the DCP method is a known technique, a detailed description of the image correction process using the DCP method will be omitted.

[0018] When a dark channel image is generated from an image (clear) in which no fog or haze occurs, the pixel values ​​of the dark channel image are close to 0 in most areas in many cases. On the other hand, a dark channel image generated from an image in which fog or haze occurs has a larger pixel value in accordance with the density of the fog or haze than an image generated from a clear image, and a brighter image is more likely to be obtained. In this way, by using a dark channel image, it is possible to evaluate the density of the fog or haze in the image (i.e., the distance to the subject). Therefore, the transmittance generation unit 104 can estimate a value that evaluates the density of the fog as an estimated value of the transmittance of light in the atmosphere for each position in the image using the pixel values ​​of the dark channel image. The transmittance generation unit 104 according to this embodiment generates a transmittance map based on the dark channel image and uses the value of the transmittance map as an estimated value of the transmittance, and the generation process of such an estimated value using the DCP method will be described later.

[0019] The exposure control unit 109 can control the exposure based on the image value. For example, the exposure control unit 109 can detect the brightness of the image based on a signal detected from the image, and control the exposure aperture, shutter speed, or gain to achieve a predetermined brightness. The exposure control unit 109 can also control the exposure based on a user's input. In this case, the exposure control unit 109 can detect the brightness of the image based on a signal detected from the image, calculate a value indicating how bright the current situation is relative to an appropriate brightness (which can be set as desired), and present the value to the user.

[0020] The frame memory 108 is a memory generally called RAM (Random Access Memory), which can temporarily store video signals and read them out when necessary. Video signals often have a huge amount of data, so the memory that stores them is required to be high-speed and large-capacity. For example, a DDR4-SDRAM (Dual Data Rate 4 - Synchronous Dynamic RAM) can be used as the frame memory 108. The frame memory 108 is used, for example, in various image processing such as synthesizing images that differ over time or cutting out only a necessary area from an image.

[0021] The CPU 105 is a central processing unit (CPU) for controlling each functional unit of the image processing device 100, and executes each process performed by the image processing device 100. In the image processing device 100, a read only memory (ROM) and a RAM are connected to drive the CPU 105. The ROM 110 is a non-volatile memory, and stores a program for operating the CPU 105 or various information such as various adjustment parameters. The program read from the ROM 110 is expanded in the volatile RAM 111 and executed. As the RAM 111, for example, the frame memory 108 or a low-speed, low-capacity memory is used.

[0022] The image generated by the image processing unit 103 is output to an external device of the image processing device 100 via a video output driving unit 106 and a video terminal 107. As an interface for inputting and outputting with the external device, it is possible to use SDI (Serial Digital Interface), HDMI (registered trademark) (High Definition Multimedia Interface), DisplayPort (registered trademark), or the like. Such an interface makes it possible to display real-time images on an external monitor or the like.

[0023] The image generated by the image processing unit 103 is displayed on a display device via a display drive unit 113 and a display unit 114. The display unit 114 is a display device that can be visually recognized by the user, and can display, for example, an image processed by the image processing unit 103 and a setting menu, and can confirm the operating status of the image processing device 100. The display unit 114 can use a small, low-power device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) as a display device. The display unit 114 may also include a resistive or capacitive thin-film element called a touch panel. The CPU 105 can generate display information such as a character string that notifies the user of the setting state of the image processing device 100 or a menu for setting the image processing device 100, and display the information on the display unit 114 by superimposing it on the image processed by the image processing unit 103. The information displayed on the display unit 114 may be a shooting assist display such as a histogram, a vector scope, a waveform monitor, a zebra, a peaking, or a false color.

[0024] The transmittance generating unit 104 calculates an estimated value (estimated transmittance) indicating the transmittance of the atmosphere. In the following, the transmittance generating unit 104 will be described as generating a transmittance map indicating the estimated transmittance at each position of the image. The transmittance generating unit 104 according to this embodiment can correct the image to be processed based on the estimated transmittance and generate an image from which fog or haze has been removed (post-removal image). Each process performed by the transmittance generating unit 104 will be described below.

[0025] The transmittance generating unit 104 can generate a post-removal image from the image to be processed, for example, by the process shown in the following formula (1). Formula (1) is an atmospheric model that shows the correspondence between an image I before processing (before removing fog or haze) and a post-removal image J. Note that it is assumed here that the pre-processing image and the post-removal image are the same size. I(x,y)=J(x,y)×t(x,y)+A(1-t(x,y)) Equation (1)

[0026] x and y represent the two-dimensional coordinate position in the horizontal and vertical directions in the image (for example, with the upper left corner as the origin), t(x, y) represents the value at (x, y) of the fog or haze transmittance map, and A represents the ambient light.

[0027] The transmittance map t(x,y) is a map that indicates the amount of light attenuation due to the atmosphere for each position. The transmittance map t is generated so that the greater the distance from the subject, the greater the value (pixel value) in the transmittance map, and the smaller the distance from the subject, the smaller the value (pixel value) in the transmittance map.

[0028] By estimating the ambient light A and the transmittance map t(x,y), it is possible to calculate the defrosted J(x,y) using equation (1). Here, we will first explain how to estimate the ambient light A using equations. Ambient light A is the average pixel value of the RGB components in the sky area, and is calculated, for example, using the following equation (2).

number

[0029] Here, the ave function is a function that calculates the average value in the arguments, d represents a color component, and Ωsky represents a local region in the sky region. The sky region according to this embodiment is a region that represents a space corresponding to the sky in the image. Here, the method of identifying the sky region in the image is not particularly limited, and it can be identified by any known method of estimating the sky region, such as a method of calculating based on the distribution of a histogram, a method of using a coordinate position specified in advance, or a method of using a position indicated by a user. Similarly, the method of identifying the local region in the sky region used here is not particularly limited. Here, the image I(x,y) in which fog or haze occurs is used to estimate the ambient light A, but the average value of the local region of Ωsky may be calculated using the dark channel value Idrk(x,y) described later. Alternatively, the average of a predetermined position of the dark channel value Idrk(x,y) described later (for example, a position where the dark channel value is in the top 10%) may be used as the ambient light A without identifying the sky region.

[0030] Next, a method for estimating the transmittance map t(x,y) will be described. The transmittance generation unit 104 according to this embodiment estimates the transmittance map t(x,y) based on the dark channel value. For example, when using an image in which the pixel value of at least one of the RGB components is locally very small, such as an outdoor image without fog or haze, the transmittance map can be efficiently estimated. The dark channel value Idrk(x,y) is calculated using Equation (3).

number

[0031] Here, d represents a color component, and Ω represents a local region including the target coordinates (x, y). The local region used here is assumed to be the same as that used in formula (2). As shown in formula (3), the dark channel value is the minimum value of the RGB components in the local region including the target pixel. By substituting the dark channel calculation formula in formula (3) into the atmospheric model in formula (1), formula (4) for calculating the dark channel value from the atmospheric model can be obtained. Idrk=Jdrk(x,y)×t(x,y)+A(1-t(x,y)) Equation (4)

[0032] In an image without fog or haze, it is considered that at least one of the RGB components is locally very small. For such an image without fog or haze, in which the pixel value of at least one of the RGB components is locally small, the dark channel value Jdrk(x,y) is a value that is very close to 0. Therefore, formula (4) can be approximated as formula (5). Idrk(x,y)≒A(1-t(x,y)) Equation (5)

[0033] The transmittance generating unit 104 can estimate the transmittance map t(x, y) by using, for example, equation (6) which is an approximation of equation (5).

number

[0034] Here, ω is a parameter that controls the degree of fog or haze correction by the transmittance generation unit 104, and the larger the value, the stronger the correction effect for removing fog or haze can be set. Here, ω is defined in the range from 0.0 to 1.0. The image processing unit 103 can obtain J(x, y) after fog and haze removal by substituting the ambient light A and the transmittance map t(x, y) calculated by equations (2) to (6) into the following equation (7).

number

[0035] Here, the estimated transmittance according to this embodiment refers to the transmittance map, and the description will be given assuming that it does not include the dark channel value.

[0036] Next, with reference to Figs. 2 and 3, a process performed by the image processing device 100 to estimate an estimated value indicating the transmittance of the atmosphere in an image and correct the image when it is determined that fog or haze is occurring based on the estimated value will be described. The process shown in Fig. 2 may be started automatically after the image processing device 100 is started, or may be started in response to a user operation. The process shown in Fig. 2 may also be started when settings for performing part or all of the process shown in Fig. 2 have been made in advance. In that case, a menu for making the settings may be displayed on the display unit 114, and a user input may be accepted. Furthermore, in this embodiment, when the process shown in Fig. 2 is set to be performed automatically, the process in Fig. 2 is repeatedly performed.

[0037] In S201, the CPU 105 acquires an image from the image sensor 102. Next, in S202, the CPU 105 determines whether the exposure is appropriate based on the brightness of the image calculated by the exposure control unit 109. If the exposure is determined to be appropriate, the process proceeds to S203, and if the exposure is determined to be inappropriate, the process ends. For example, an image in a state in which the charge in the pixels of the image sensor 102 is saturated has excessive exposure, and it is difficult to correctly estimate the ambient light using the image, so that the reliability of the transmittance map is low. When the exposure is manually controlled, it is considered that such exposure often becomes an inappropriate value. It is possible to determine whether the exposure is appropriate, and if it is not appropriate, to cut off and end the subsequent process, so that it is possible to determine whether fog or haze is occurring only in an appropriate state.

[0038] The CPU 105 may determine whether the exposure is appropriate based on, for example, the brightness of the image. For example, if the average brightness of a predetermined area in the image is equal to or greater than a predetermined range (which can be set arbitrarily), the exposure may be determined to be inappropriate. In this way, the determination of whether the exposure is appropriate can be made by arbitrarily using a known image processing technique. Also, the determination of whether the exposure is appropriate may be made based on the settings of the imaging device, depending on whether the exposure is within a predetermined setting range.

[0039] In S203, the CPU 105 generates a transmittance map using the DCP method from the image acquired by the transmittance generation unit 104 in S201. The transmittance generation unit 104 according to the present embodiment generates a transmittance map having, as elements, estimated values ​​indicating the transmittance of light in the atmosphere for each position in the image. Next, the transmittance generation unit 104 can determine that fog or haze is occurring in the image to be processed when the ratio of the estimated values ​​that fall within a predetermined range is equal to or greater than a threshold value. Hereinafter, the transmittance generation unit 104 illustrates each element of the transmittance map using a histogram as shown in FIG. 3, which will be described later, and determines whether the ratio of the elements between the lower limit judgment point 301 and the upper limit judgment point 302 to the whole is equal to or greater than a threshold value. Hereinafter, such a histogram illustrating the elements of the transmittance map will be simply referred to as a "histogram of the transmittance map."

[0040] FIG. 3 is a diagram showing an example of the range of the lower limit judgment point and the upper limit judgment point of the transmittance in the histogram according to the first embodiment. For example, when the lower limit judgment point 301 is set to a transmittance of 0.15 and the upper limit judgment point 302 is set to a transmittance of 0.3, a transmittance distribution within the transmittance range as shown in FIG. 3 is calculated. The lower limit judgment point 301 and the upper limit judgment point 302 are preset values, and can be set experimentally or empirically as design values ​​according to, for example, the use, performance, or shooting environment of the camera. In addition, the lower limit judgment point 301 and the upper limit judgment point 302 may be set or changed by a user operation via the operation unit 112. By changing the set values ​​of the lower limit judgment point 301 and the upper limit judgment point 302, it is possible to change the judgment criteria for what concentration of fog or haze has occurred.

[0041] In S204, the transmittance generation unit 104 generates a histogram of the transmittance map. In S205, the transmittance generation unit 104 calculates a transmittance distribution within a range between a lower limit judgment point and an upper limit judgment point in the histogram.

[0042] In S206, the transmittance generation unit 104 calculates the proportion of the transmittance distribution within the range between the lower limit judgment point 301 and the upper limit judgment point 302 calculated in S205 to the transmittance histogram of the entire image. In S207, the CPU 105 judges whether the proportion calculated in S206 is equal to or greater than a preset threshold. If it is equal to or greater than the threshold, the process proceeds to S208, and if it is less than the threshold, the process proceeds to S209. For example, if the threshold is 50%, the transmittance generation unit 104 judges that fog or mist is occurring when a transmittance distribution of 50% or more is included in the aforementioned range. This threshold may be set and changed by a user operation via the operation unit 112, for example.

[0043] When the range of the lower limit judgment point 301 and the upper limit judgment point 302 is widened or the threshold of the ratio of the transmittance distribution within the range to the entire transmittance histogram is lowered, the transmittance generation unit 104 is more likely to judge that fog or haze is occurring from the histogram of the image. When the range of the lower limit judgment point 301 and the upper limit judgment point 302 is narrowed or the threshold of the ratio of the transmittance distribution within the range to the entire transmittance histogram is raised, the transmittance generation unit 104 is less likely to judge that fog or haze is occurring from the histogram of the image. In this way, by changing the settings of the judgment points or thresholds (for example, by the user), it is possible to change the criteria for the above judgment according to desired conditions. Here, when the user sets the thresholds, the values ​​of the lower limit judgment point 301 and the upper limit judgment point 302 may be set via a menu screen by operating the operation unit 112, for example.

[0044] In the above description, the transmittance is evaluated based on the ratio of elements within a predetermined range using a histogram, but the method is not limited to this as long as the transmittance in the image can be evaluated. For example, the average value of the elements in the transmittance map may be calculated, and if the average value is equal to or greater than a predetermined threshold, it may be determined that fog or haze is occurring in the image.

[0045] In S208, the image processing unit 103 corrects the image acquired by the image sensor 102 using the ambient light estimate and the transmittance map generated by the transmittance generation unit 104, generates a post-mist or fog-removed image, and ends the process. In S209, the image processing unit 103 outputs the image acquired by the image sensor 102 without performing such image correction.

[0046] When the dark channel value Idrk(x,y) is used as it is to estimate the ambient light, the calculated value changes when the exposure changes (especially when the user changes it manually), which affects the result of the determination of whether fog or haze is occurring. On the other hand, the value t(x,y) of the transmittance map according to this embodiment is a value normalized using the ambient light A, as shown in formula (6). Therefore, by using such a transmittance map to determine whether fog or haze is occurring, it is possible to perform a determination that is less affected by changes in the brightness of the entire screen due to changes in exposure. Therefore, by using the transmittance map to perform the above determination, it is possible to appropriately determine whether fog or haze is occurring without changing the lower limit determination point 301 or the upper limit determination point 302, even when the exposure is changed manually.

[0047] According to such a process, an estimated value indicating the transmittance of light in the atmosphere in an image is estimated, and if it is determined that fog or haze is occurring in the image based on the estimated value, the image can be corrected. Therefore, regardless of the brightness and saturation values ​​(for example, even in the case of high brightness and high saturation), it is possible to more accurately determine whether or not fog or haze is occurring in the image. In particular, by making the above determination based on a transmittance map generated from the dark channel value, it becomes possible to make the above determination more accurately.

[0048] In this embodiment, the second correction process is performed as the image correction process by the DCP method, and the first correction process is controlled not to perform the correction process. However, as long as the correction process can be switched depending on the determination result of whether or not fog or haze is occurring, the present invention is not limited to such a switching process. For example, the first correction process may be performed as a level correction according to the brightness of the image, and when it is determined that fog or haze is occurring, the image correction process may be controlled to switch to the correction process by the DCP method.

[0049] [Embodiment 2] Hereinafter, a second embodiment of the present invention will be described. In the first embodiment, the average pixel value of RGB is used as the ambient light A, and the minimum value of the RGB components is used to calculate the dark channel value and the transmittance map, but in this embodiment, these calculations are performed for each RGB color component. That is, here, an R image, a G image, and a B image extracted for each component are prepared, and the same processing as in the first embodiment is performed on each of them. Hereinafter, each processing performed by the transmittance generation unit 104 and the image processing unit 103 according to this embodiment will be described.

[0050] In this embodiment, the ambient light A(c) for each color component is calculated using the following equation (8).

number

[0051] The ave function represents a function that calculates the average value in the argument. Furthermore, c represents a color component, and when c=1, it indicates that the processing is performed on the R image, when c=2, it indicates that the processing is performed on the G image, and when c=3, it indicates that the processing is performed on the B image. Ωsky represents a local area in the sky area. Here, as in the first embodiment, the method of identifying the sky area and the local area is not particularly limited. Here, the image I(x, y, c) in which fog or haze occurs is used to estimate the ambient light A(c), but the average value of the local area of ​​Ωsky may be calculated using the dark channel value Idrk(c) described later. Furthermore, without identifying the sky area, the average of a predetermined position (for example, a position where the dark channel value is in the top 10%) of the dark channel value Idrk(x, y, c) described later may be used as the ambient light A(c).

[0052] In addition, the dark channel Idrk(x, y, c) for each color component is calculated using the following equation (9).

number

[0053] c represents a color component, and when c=1, processing is performed on the R image, when c=2, the G image, and when c=3, the B image. Ω represents a local region including the target coordinates (x, y, c). The local region used here is assumed to be the same as that used in equation (8). As shown in equation (9), the dark channel value is the minimum value within the local region including the target pixel. By substituting the dark channel calculation formula in equation (9) into the atmospheric model in equation (1), equation (10) is obtained to calculate the dark channel value for each RGB color component from the atmospheric model. Idrk(c)=Jdrk(x,y,c)×t(x,y,c)+A(1-t(x,y,c)) Equation (10)

[0054] In an image without fog or haze, it is considered that at least one of the RGB components is locally very small. For such an image without fog or haze, in which the pixel value of at least one of the RGB components is locally small, the dark channel value Jdrk(x,y,c) is a value that is very close to 0. Therefore, formula (10) can be approximated as formula (11). Idrk(x,y,c)≒A(1-t(x,y,c)) Equation (11)

[0055] The transmittance generating unit 104 can estimate the transmittance map t(x, y, c) for each of the RGB color components by using, for example, equation (12) which is a modified version of the approximation equation of equation (11).

number

[0056] Here, ω is a parameter that controls the degree of fog or haze correction by the transmittance generation unit 104, and the larger the value, the stronger the fog / haze correction effect can be set. Here, ω is defined in the range from 0.0 to 1.0. The image processing unit 103 can obtain J(x, y, c) after fog / haze removal by substituting the ambient light A(c) and transmittance map t(x, y, c) calculated by equations (8) to (12) into the following equation (13).

number

[0057] By calculating J(x, y, c) for each color component using equation (13), it is possible to generate an image (hereinafter, scattered image) from which the components corresponding to Mie scattering (hereinafter, Mie scattering components) and the components corresponding to Rayleigh scattering (hereinafter, Rayleigh scattering components) have been removed.

[0058] Moreover, by using the minimum value between the ambient light estimation value A(c) calculated for each color component and the transmittance map t(x, y, c) and applying equation (13), an image in which the Mie scattering component has been removed (hereinafter, Mie scattering removed image) can be generated. Furthermore, the Mie scattering component can be extracted by subtracting the Mie scattering removed image from the original image acquired from the image sensor 102. Moreover, the Rayleigh scattering component can be extracted by subtracting the non-scattered image and the Mie scattering component from the original image acquired from the image sensor 102. The relative values ​​of the Mie scattering component and the Rayleigh scattering component can be calculated by dividing the extracted Mie scattering component and Rayleigh scattering component by the original image acquired from the image sensor 102.

[0059] Here, the estimated transmittance according to this embodiment refers to a relative Mie scattering component or Rayleigh scattering component, and is described as not including a dark channel value.

[0060] In this way, the transmittance generating unit 104 according to this embodiment estimates an evaluation value of the scattering components including the Mie scattering components or the Rayleigh scattering components based on the transmittance map, and uses the scattering components to determine whether or not fog or haze is occurring. Here, the transmittance generating unit 104 can perform this determination using the scattering components instead of the elements of the transmittance map in embodiment 1, and a duplicated description will be omitted.

[0061] Next, the process of correcting an image when it is determined that fog or haze is occurring, which is performed by the image processing device 100 according to this embodiment, will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the process performed by the image processing device 100 according to this embodiment. The process shown in Fig. 4 is performed in the same manner as that shown in Fig. 2, except that S401 to S409 are performed instead of S203 to S207, and therefore a duplicated description will be omitted.

[0062] In S401, the CPU 105 generates an ambient light estimate and a transmittance map for each color component using the DCP method from the image acquired in S201 by the transmittance generation unit 104. In S402, the CPU 105 generates a scattering-free image and a Mie-scattering removed image using the ambient light estimate and the transmittance map for each color component generated in S401 by the image processing unit 103 and the image acquired in S201.

[0063] In S403, the CPU 105 calculates relative values ​​of the Mie scattering component and the Rayleigh scattering component for the pixel values, using the non-scattering image and the Mie scattering image generated by the transmittance generating unit 104 in S402 and the image acquired in S201. Note that instead of the relative values ​​of the Mie scattering component and the Rayleigh scattering component for the pixel values, a relative value of the scattering component that is a combination of Mie scattering and Rayleigh scattering for the pixel value may be calculated.

[0064] In S404, the CPU 105 generates a histogram of the scattered components calculated in S403 by the transmittance generating unit 104. Note that, although the following description assumes that a histogram is generated for the scattered components in S404, a histogram of a Mie scattered component or a Rayleigh scattered component may also be used.

[0065] In S405, the transmittance generating unit 104 calculates the distribution of scattered components within the range between the lower limit judgment point and the upper limit judgment point in the histogram. The histogram is generated in the same manner as in FIG. 3 described in the first embodiment, so a duplicated description will be omitted.

[0066] In S406, the transmittance generation unit 104 calculates the proportion of the scattering component distribution within the range between the lower limit judgment point and the upper limit judgment point calculated in S405 to the scattering component histogram of the entire image. In S407, the CPU 105 determines whether the proportion calculated in S406 is equal to or greater than a preset threshold. If it is equal to or greater than the threshold, the process proceeds to S208, and if it is less than the threshold, the process proceeds to S209. For example, if the threshold is 50%, the transmittance generation unit 104 determines that fog or mist is occurring when 50% or more of the scattering component distribution is included within the aforementioned range. This threshold may be set and changed by a user operation via the operation unit 112, for example.

[0067] When the range of the lower limit judgment point 301 and the upper limit judgment point 302 is widened or the threshold of the ratio of the scattering component distribution within the range to the entire scattering component histogram is lowered, the transmittance generation unit 104 is more likely to judge that fog or haze is occurring from the histogram of the image. When the range of the lower limit judgment point and the upper limit judgment point is narrowed or the threshold of the ratio of the scattering component distribution within the range to the entire scattering component histogram is raised, the transmittance generation unit 104 is less likely to judge that fog or haze is occurring from the histogram of the image. In this way, by changing the settings of the judgment points and the thresholds (for example, by the user), it is possible to change the criteria for the above judgment according to desired conditions. Here, when the user sets the thresholds, the values ​​of the lower limit judgment point 301 and the upper limit judgment point 302 may be set via a menu screen by operating the operation unit 112, for example.

[0068] Although the histogram is used to evaluate the relative values ​​of the scattered components in the image, the method is not limited to this as long as it is possible to evaluate the scattered components in the image. For example, the average value of the scattered components may be calculated, and if the average value is equal to a predetermined threshold value, it may be determined that fog or haze is occurring in the image.

[0069] In S208, the image processing unit 103 corrects the image acquired by the image sensor 102 using the ambient light estimate and the transmittance map generated by the transmittance generation unit 104, generates a post-mist or fog-removed image, and ends the process. In S209, the image processing unit 103 outputs the image acquired by the image sensor 102 without performing such image correction.

[0070] According to such a process, an estimated value indicating the transmittance of light in the atmosphere in an image is estimated, and if it is determined that fog or haze is occurring in the image based on the estimated value, the image can be corrected. Therefore, regardless of the brightness and saturation values ​​(for example, even in the case of high brightness and high saturation), it is possible to more accurately determine whether or not fog or haze is occurring in the image. In particular, by making the above determination based on a transmittance map generated from the dark channel value, it becomes possible to make the above determination more accurately.

[0071] Since the dark channel value is an absolute value, the value also changes when the exposure changes. Therefore, when the dark channel value is used as it is to estimate the ambient light, the calculated dark channel value also changes when the exposure changes, which affects the result of the determination of whether fog or haze is occurring. On the other hand, by estimating the ambient light using the relative value of the scattered component, it is possible to perform a determination that is less affected by the change in brightness of the entire screen due to the change in exposure. Therefore, by performing the above determination using the relative value of the scattered component, it is possible to appropriately determine whether fog or haze is occurring without changing the lower limit determination point 301 or the upper limit determination point 302, even when the exposure is changed manually.

[0072] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) an estimation means for estimating from the image an estimate value indicative of the light transmittance through the atmosphere for each position in the image; a first determination means for determining whether or not fog or haze is occurring in the image based on the estimated value; a control means for switching a correction process of the image from a first correction process to a second correction process when it is determined that the fog or haze is occurring; An image processing device comprising: (Item 2) 2. The image processing device according to claim 1, wherein the first determination means determines that fog or haze is occurring in the image when a proportion of positions in the image where the estimated value is within a predetermined range is equal to or greater than a first threshold value. (Item 3) 2. The image processing device according to claim 1, wherein the first determination means determines that fog or haze is occurring in the image when the average value of the estimated values ​​in the image is equal to or greater than a second threshold value. (Item 4) 2. The image processing device according to item 1, wherein the estimation means estimates, as the estimated value, a value obtained by evaluating a concentration of fog in the atmosphere. (Item 5) 2. The image processing device according to item 1, wherein the estimation means estimates an evaluation value of a scattered component of light as the estimated value. (Item 6) 2. The image processing device according to item 1, wherein the scattered components of the light include a Mie scattered component or a Rayleigh scattered component. (Item 7) The method further includes a second determination unit that determines whether or not the exposure of the image is appropriate based on the image, 2. The image processing device according to claim 1, wherein the estimation means estimates the estimated value when the exposure is determined to be appropriate. (Item 8) estimating from the images an estimate indicative of the atmospheric light transmittance for each location in the images; determining whether fog or haze is present in the image based on the estimate; switching a correction process for the image from a first correction process to a second correction process when it is determined that the fog or haze is occurring; An image processing method comprising: (Item 9) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 7.

[0073] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0074] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0075] 101: optical lens, 102: imaging element, 103: image processing unit, 104: transmittance generating unit, 105: CPU, 106: video output driving unit, 107: video terminal, 108: frame memory, 109: exposure control unit, 110: ROM, 111: RAM, 112: operation unit, 113: display driving unit, 114: display unit

Claims

1. an estimation means for estimating from the image an estimate value indicative of the light transmittance through the atmosphere for each position in the image; a first determination means for determining whether or not fog or haze is occurring in the image based on the estimated value; a control means for switching a correction process of the image from a first correction process to a second correction process when it is determined that the fog or haze is occurring; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the first determination means determines that fog or haze is occurring in the image when a proportion of positions in the image where the estimated value is within a predetermined range is equal to or greater than a first threshold value.

3. 2 . The image processing device according to claim 1 , wherein the first determination means determines that fog or haze is occurring in the image when the average value of the estimated values ​​in the image is equal to or greater than a second threshold value.

4. 2. The image processing apparatus according to claim 1, wherein the estimation means estimates, as the estimated value, a value obtained by evaluating a concentration of fog in the atmosphere.

5. The image processing apparatus according to claim 1 , wherein the estimation means estimates an evaluation value of a scattered component of light as the estimated value.

6. The image processing device according to claim 1 , wherein the scattered components of the light include a Mie scattered component or a Rayleigh scattered component.

7. The image processing method further includes a second determination unit that determines whether or not the exposure of the image is appropriate based on the image, The image processing apparatus according to claim 1 , wherein the estimation means estimates the estimated value when the exposure is determined to be appropriate.

8. estimating from the images an estimate indicative of the atmospheric light transmittance for each location in the images; determining whether fog or haze is present in the image based on the estimate; switching a correction process for the image from a first correction process to a second correction process when it is determined that the fog or haze is occurring; An image processing method comprising:

9. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 7.

Images