Image processing apparatus, and image processing method

Abstract

To provide a technique allowing the execution of image sharpening processing at a smaller processing cost than before.SOLUTION: An image processing apparatus calculates an atmosphere transmittance distribution based on an input image, and sharpens the input image based on an illumination distribution calculated based on the transmittance distribution.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to techniques for sharpening an input image. [Background technology]

[0002] In the field of surveillance cameras and in-vehicle cameras, the decrease in visibility in captured images due to the influence of fog between the camera and the subject is a problem. This is because the contrast of the captured image is decreased due to light scattering by fine particle components when light passes through the atmosphere. Since the degree of scattering of this phenomenon changes depending on the distance to the subject, a captured image of a scene in which the distance to the subject is mixed will have a different degree of contrast decrease for each image region. As a method for correcting such a decrease in contrast, there is a method such as that described in Patent Document 1. In the method described in Patent Document 1, it is assumed that black floating occurs due to the influence of fog, etc., and the transmittance distribution of the atmosphere is calculated from the minimum value of the color channel in the region (hereinafter referred to as the dark channel), and the influence of fog is removed based on an atmospheric model (hereinafter referred to as fog removal). In addition, in the method described in Non-Patent Document 1, in addition to fog removal using the dark channel, dark area correction and bright area correction are performed based on the Retinex theory to improve visibility. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Pat. No. 8,340,461 [Non-patent literature]

[0004] [Non-Patent Document 1] Xinggang Liu, Sichuan University, "Dehaze Enhancement Algorithm Based on Retinex Theory for Aerial Images Combined with Dark Channel", Acess Library Journal,2020,Volume 7 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method of Patent Document 1 had problems such as crushed shadows and blown-out highlights after fog removal. On the other hand, the method of Non-Patent Document 1 had high processing costs because it had to calculate the illumination distribution used in the Retinex theory after calculating the transmittance distribution used for fog removal. The present invention provides a technology that can perform image sharpening processing at a lower processing cost than conventional techniques. [Means for solving the problem]

[0006] One aspect of the present invention is characterized by comprising a calculation means for calculating an atmospheric transmittance distribution based on an input image, and a processing means for sharpening the input image based on an illumination distribution calculated based on the transmittance distribution. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that is capable of executing image sharpening processing at lower processing costs than in the past. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration applicable to the camera 100. [Diagram 3] FIG. 2 is a block diagram showing an example of the functional arrangement of an image processing unit 208. [Figure 4]4 is a flowchart of a process performed by the camera 100. [Diagram 5] FIG. 13 is a diagram showing a process for generating a dark channel image from a developed image J′. [Figure 6] FIG. 13 is a diagram showing a shaping process of a transmittance distribution. [Figure 7] FIG. 4 is a block diagram showing an example of the functional configuration of a processing unit 306. [Figure 8] 10 is a flowchart showing details of the process in step S408. [Figure 9] 11A and 11B are diagrams for explaining the relationship between fog density and contrast. [Figure 10] FIG. 4 is a diagram for explaining calculation of fog density. [Figure 11] 5A and 5B are diagrams for explaining gain values ​​for dark area correction and light area correction. [Figure 12] FIG. 11 is a diagram for explaining a maximum value of an applied gain. [Figure 13] 1A and 1B are diagrams for explaining dark area correction and bright area correction based on the Retinex theory. [Figure 14] FIG. 2 is a block diagram showing an example of the functional arrangement of an image processing unit 208. [Figure 15] 4 is a flowchart of a process performed by the camera 100. [Figure 16] FIG. 16 shows an example of a GUI 1601 display. [Figure 17] FIG. 14 is a block diagram showing an example of the functional arrangement of a processing unit 1406. [Figure 18] FIG. 1 is a diagram showing an example of a system configuration. [Figure 19] FIG. 18 is a block diagram showing an example of the functional configuration of an image processing unit 1857. [Figure 20] 18 is a flowchart of a process performed by a computer device 1800. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] [First embodiment] In this embodiment, an example of an image processing device will be described that calculates an atmospheric transmittance distribution based on an input image, and sharpens the input image based on an illumination distribution calculated based on the transmittance distribution.

[0011] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a camera 100 as an image processing device, a computer device 102, and a display 103. The camera 100 and the computer device 102, and the computer device 102 and the display 103 are connected to each other so as to be able to communicate with each other via a video transmission cable 101 such as an HDMI or SDI (serial digital interface) cable. Note that the method of connection between the camera 100 and the computer device 102, and between the computer device 102 and the display 103 is not limited to a specific connection method, and for example, the devices may be connected to each other via a wired or wireless LAN.

[0012] First, the camera 100 will be described. The camera 100 is an example of an imaging device that can control its own pan, tilt, and zoom in response to an instruction from a computer device 102 or autonomously, and captures moving images or still images. When capturing a moving image, the camera 100 outputs an image obtained by performing a sharpening process described later on an image of each frame in the moving image as a captured image. When capturing a still image, the camera 100 outputs an image obtained by performing a sharpening process described later on a still image captured periodically or irregularly as a captured image. The captured image output from the camera 100 is input to the computer device 102 via a video transmission cable 101. The captured image output from the camera 100 may be output to a display 103 via a communication path (not shown) to display the captured image on the display 103.

[0013] Next, a description will be given of the display 103. The display 103 has a liquid crystal screen or a touch panel screen, and displays images and characters output from the computer device 102 and the camera 100. Furthermore, if the display 103 has a touch panel screen, operational input from the user onto the touch panel screen is notified to the computer device 102 via a communication path (not shown).

[0014] Next, the computer device 102 will be described. The computer device 102 is a computer device such as a PC (personal computer), a smartphone, a tablet terminal device, etc. The computer device 102 transmits an image capture instruction to the camera 100, sets parameters of the camera 100, and also holds captured images output from the camera 100 and transfers them to other devices.

[0015] Next, an example of a hardware configuration applicable to the above-mentioned camera 100 will be described with reference to the block diagram of Fig. 2. The CPU 201 executes various processes using computer programs and data stored in the RAM 202. In this way, the CPU 201 controls the overall operation of the camera 100, and also executes or controls various processes that will be described as processes performed by the camera 100.

[0016] The RAM 202 has an area for storing computer programs and data loaded from the ROM 203 or the recording medium 204, and an area for storing computer programs and data received from the computer device 102 via the communication I / F 205. The RAM 202 also has an area for storing RAW images acquired by the image input unit 206 from the imaging sensor 212. The RAM 202 also has a work area used when the CPU 201 and the image processing unit 208 execute various processes. In this way, the RAM 202 can provide various areas as appropriate.

[0017] The ROM 203 stores setting data for the camera 100, computer programs and data related to the startup of the camera 100, computer programs and data related to the basic operations of the camera 100, and the like.

[0018] The recording medium 204 is a memory device such as an SSD, an SD card, or a USB memory, and can store the results of processing by the camera 100, and computer programs and data received from the computer device 102 via the communication I / F 205. The computer programs and data stored in the recording medium 204 are loaded into the RAM 202 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201.

[0019] The communication I / F 205 is an interface such as HDMI or SDI, and is an interface for performing data communication with the computer device 102 and the display 103 via the video transmission cable 101 .

[0020] Light from the outside world enters the imaging sensor 212 via the lens 211. The imaging sensor 212 photoelectrically converts the incident light to generate a RAW image and outputs the generated RAW image. The image input unit 206 acquires the RAW image output from the imaging sensor 212. The imaging control unit 207 performs drive control of the lens 211 and operation control of the imaging sensor 212 under the control of the CPU 201.

[0021] The image processing unit 208 is a hardware circuit that performs various processes under the control of the CPU 201 to generate a sharpened image based on the RAW image acquired by the image input unit 206, which has been subjected to fog removal processing and correction processing of bright and dark areas.

[0022] The operation unit 210 is a user interface such as a button, a switch, a touch panel, etc., and a user can operate it to input various instructions to the CPU 201. The image output unit 213 outputs the sharpened image generated by the image processing unit 208 to the computer device 102 or the display 103 via the communication I / F 205.

[0023] The CPU 201 , RAM 202 , ROM 203 , recording medium 204 , communication I / F 205 , image input unit 206 , imaging control unit 207 , image processing unit 208 , operation unit 210 , and image output unit 213 are all connected to a system bus 214 .

[0024] Next, an example of the functional configuration of the image processing unit 208 is shown in the block diagram of Fig. 3. In the following, the functional units shown in Fig. 3 will be described as being all implemented in hardware. However, one or more of the functional units shown in Fig. 3 may be implemented in software (computer program), in which case the functions of the one or more functional units are realized by the CPU 201 executing the computer program corresponding to the one or more functional units.

[0025] The process performed by the camera 100 to generate a sharpened image based on the RAW image acquired by the image input unit 206 from the imaging sensor 212 will be described with reference to the flowchart of FIG.

[0026] In step S401, the CPU 201 loads into the RAM 202 "various parameters to be used in subsequent processes" stored in the ROM 203 or the recording medium 204 and sets them.

[0027] The imaging sensor 212 receives light incident through the lens 211, which is driven and controlled by the imaging control unit 207, and generates a RAW image based on the received light under the control of the imaging control unit 207, and outputs the generated RAW image. In step S402, the image input unit 206 acquires the RAW image J output from the imaging sensor 212, and inputs the acquired RAW image J to the image processing unit 208. Here, J(x, y) represents the pixel value of the pixel at the pixel position (x, y) in the RAW image J. In the RAW image J, any one of the colors R, G, and B is arranged in a Bayer arrangement at each pixel.

[0028] In step S403, the processing unit 301 performs development processing on the RAW image J to generate a developed image (input image) J'. Here, J'(x, y, c) represents the pixel value of the color channel c (c=R, G, B) of the pixel at the pixel position (x, y) in the developed image J'. The processing unit 301 performs image processing such as white balance, debayer, noise reduction, sharpness, and color conversion on the RAW image J.

[0029] In step S404, the calculation unit 302 generates a dark channel image from the developed image J', in which the pixel value is the minimum value of the color channel of the local area in the developed image J'. Here, the process for generating a dark channel image from the developed image J' will be described with reference to FIG. 5. In order to generate a dark channel image from the developed image J', the calculation unit 302 sets an image area of ​​3 pixels x 3 pixels centered on a pixel of interest in the developed image J' as a search range. FIG. 5(a) shows the R (red) pixel value of each pixel included in the search range, FIG. 5(b) shows the G (green) pixel value of each pixel included in the search range, and FIG. 5(c) shows the B (blue) pixel value of each pixel included in the search range. The calculation unit 302 specifies the minimum pixel value among the R pixel value, G pixel value, and B pixel value of each pixel included in the search range, and sets the specified pixel value as the pixel value of the pixel corresponding to the pixel of interest in the dark channel image. In the case of Fig. 5, the minimum pixel value among the R pixel value, the G pixel value, and the B pixel value of each pixel included in the search range is "115". Therefore, as shown in Fig. 5(d), the calculation unit 302 sets the pixel value "115" as the pixel value of the pixel corresponding to the pixel of interest in the dark channel image. By performing the above process with each pixel in the developed image J' as the pixel of interest, the pixel value of each pixel in the dark channel image can be determined.

[0030] In step S405, the calculation unit 303 calculates (estimates) the ambient light A(c) of the color channel c using the dark channel image generated in step S404. The ambient light is a component of light from the sun, the sky, etc., scattered by fog. Here, A(c) is a component of light corresponding to the color channel c. A method for calculating the ambient light from an image is well known, and it is possible to calculate the ambient light using, for example, the method described in Patent Document 1. Specifically, an area in which the pixel values ​​are in the top 0.1% of the dark channel image is extracted, and the ambient light A(c) is obtained for that area from the average RGB of the developed image J' developed in step S403.

[0031] In step S406, the calculation unit 304 calculates the atmospheric transmittance distribution t using the developed image J' and the ambient light A(c) according to the following (Equation 1), where t(x, y) represents the atmospheric transmittance corresponding to the pixel position (x, y) in the developed image J'.

[0032]

number

[0033] …(Formula 1) Here, D(·) is a function for calculating a dark channel image, and is a function for generating the dark channel image in step S404. Also, ω is a real parameter set within the range of 0 to 1 (a parameter for controlling the transmittance of distant objects from becoming too large), and is set in step S401. The lower the value of ω, the stronger the control for increasing the transmittance of distant objects, and the lower the amount of correction. In this embodiment, ω=0.95.

[0034] In step S407, the processing unit 305 performs a shaping process on the transmittance distribution t using the developed image J' to generate a shaped transmittance distribution t'. Here, t'(x, y) represents the transmittance of the atmosphere corresponding to the pixel position (x, y) in the developed image J'. Here, the shaping process of the transmittance distribution will be described with reference to FIG. 6. In the transmittance distribution t, the block shape of the region remains because it is calculated for each rectangular region. Therefore, in the shaping process, a shaped transmittance distribution t' in which the block shape of the transmittance distribution t is reduced is generated using the developed image J' as a guide image. In this shaping process, a guided filter is used to reduce the block shape, but a cross bilateral filter or the like may also be used, and the shaping process is not limited to a specific method.

[0035] In step S408, the processing unit 306 performs a sharpening process on the developed image J' using the ambient light A(c) and the shaped transmittance distribution t' to generate a sharpened image I. Here, I(x, y, c) represents the pixel value of the color channel c (c=R, G, B) of the pixel at the pixel position (x, y) in the sharpened image I. Details of the process in step S408 will be described later.

[0036] In step S409, the processing unit 307 performs gamma correction on the sharpened image I generated in step S408 to generate an output image (sharpened image subjected to gamma correction) I'. In step S410, the image output unit 213 outputs the output image I' generated in step S409 to the computer device 102 via the communication I / F 205.

[0037] In step S411, CPU 201 determines whether a processing end condition has been satisfied. Various conditions can be applied as the processing end condition. For example, there are "detection that camera 100 has been powered off," "the length of time during which image input unit 206 has not acquired a RAW image has exceeded a certain period of time," "detection that the user has operated operation unit 210 to input an instruction to end processing," and the like.

[0038] If the result of such determination is that the end condition of the process is satisfied, the process according to the flowchart in FIG. 4 ends, and if the end condition of the process is not satisfied, the process proceeds to step S401.

[0039] Next, details of the process in step S408 above will be described with reference to the flowchart in Fig. 8. An example of the functional configuration of the processing unit 306 is shown in the block diagram of Fig. 7. In this embodiment, in addition to fog removal processing, the processing unit 306 performs dark area correction and light area correction based on the Retinex theory in the sharpening process to generate a sharpened image with improved visibility of the developed image.

[0040] Here, we will explain the Retinex theory. The Retinex theory is a model of human vision, and is a method used to compress the dynamic range of captured images. In the Retinex theory, an image II is considered to be divided into a reflected light distribution RR and an illumination light distribution LL, as shown in the following (Equation 2).

[0041] II=LL·RR … (Formula 2) At this time, by utilizing the fact that human visual characteristics are insensitive to the illumination light distribution LL, only LL is compressed. Here, as shown in Fig. 13, the input LL is converted to LL' in which the dark areas are brightened (hereinafter, dark area correction) and the bright areas are darkened (hereinafter, bright area correction). At this time, the corrected image II' can be generated according to the following (Equation 3).

[0042] II'=LL'·RR … (Formula 3) (Equation 3) can be expressed as shown in (Equation 4) below using (Equation 2).

[0043] II'=II·LL' / LL … (Equation 4) The above is the correction method using the Retinex theory.

[0044] In step S801, the estimation unit 701 estimates the fog density (fog concentration) s based on the developed image J'. As shown in Fig. 9, the contrast of an image of a foggy scene generally decreases as the fog density increases, and increases as the fog density decreases. Therefore, the estimation unit 701 identifies the maximum brightness value Ymax and the minimum brightness value Ymin in the developed image J', and calculates the contrast C of the developed image J' using the identified maximum brightness value Ymax and minimum brightness value Ymin according to the following (Equation 5).

[0045] C=(Ymax-Ymin) / (Ymax+Ymin) … (Formula 5) Next, the estimation unit 701 identifies (estimates) the fog density s corresponding to the contrast C obtained according to (Equation 5) using the "correspondence relationship between contrast C and fog density s" shown in Fig. 10. The correspondence relationship shown in Fig. 10 is an example of the correspondence relationship between contrast C and fog density s, in which "the lower the contrast C, the higher the fog density s, and the higher the contrast C, the lower the fog density s." The correspondence relationship in Fig. 10 may be implemented as a lookup table or a function.

[0046] In step S802, the estimation unit 702 obtains the illumination distribution L using the shaped transmittance distribution t'. Here, L(x, y) represents the intensity of the illumination light corresponding to the pixel position (x, y) in the developed image J'. In general, the distribution value of the illumination illuminating the subject tends to be larger in the dark areas as the illumination becomes stronger, and tends to be smaller in the dark areas as the illumination becomes weaker. Therefore, the estimation unit 702 calculates the illumination distribution L by inverting the shaped transmittance distribution t' calculated from the amount of black floating caused by fog, as shown in the following (Equation 6).

[0047] L(x,y)=1.0-α·t'(x,y) … (Equation 6) Here, α is a parameter for adjusting the illumination distribution and takes a real value between 0 and 1.0. The closer α is to 0, the more uniform the illumination distribution L can be.

[0048] In step S803, the calculation unit 703 calculates the maximum gain gd to be applied in the fog removal process using the developed image J', the ambient light A(c), the fog density s, and the shaped transmittance distribution t', as shown in the following (Equation 7).

[0049]

number

[0050] …(Formula 7) Here, gd(x, y) represents the maximum gain corresponding to the pixel position (x, y) in the developed image J'. minis a coefficient to prevent division by zero, and in this embodiment, t min = 0.01. Here, the fog density s can be used to adjust the maximum gain applied to fog elimination.

[0051] In step S804, the calculation unit 704 specifies the dark correction / light correction gain gr corresponding to the illumination distribution L calculated in step S802, using the "correspondence relationship between illumination distribution L and dark correction / light correction gain gr" shown in FIG. 11. Here, gr(x, y) represents the dark correction / light correction gain corresponding to the pixel position (x, y) in the developed image J'. The correspondence relationship shown in FIG. 11 is only an example. The correspondence relationship in FIG. 11 may be implemented as a lookup table or a function.

[0052] In step S805, the adjustment unit 705 uses the illumination distribution L, the maximum gain gd, and the dark area correction / light area correction gain gr to determine an application gain gr' to be applied to the dark area correction and the light area correction.

[0053] The defogging and shadow / light correction are digital gains. Therefore, noise is amplified after application. In this process, the amount of amplified noise is controlled to reduce the decrease in visibility caused by noise amplification. The adjustment unit 705 adjusts the composite gain g applied by the defogging and shadow / light correction. total is calculated according to the following (Equation 8).

[0054] g total (x,y)=gd(x,y)*gr(x,y) … (Equation 8) In addition, the adjustment unit 705 adjusts the gain g max The lighting distribution L and gain g shown in Fig. 12 max The correspondence relationship shown in FIG. 12 is an example. The correspondence relationship in FIG. 12 may be implemented as a lookup table or a function. Then, the adjustment unit 705 determines the application gain gr′ to be applied in the dark area correction and the light area correction according to the following (Equation 9).

[0055]

number

[0056] …(Formula 9) Here, g max (L(x, y)) represents the gain gmax corresponding to L(x, y), and gr'(x, y) represents the applied gain corresponding to the pixel position (x, y) in the developed image J'.

[0057] In step S806, the correction application unit 706 performs sharpening processing on the developed image J' using the developed image J', the shaped transmittance distribution t', the fog density s, the applied gain gr', and the ambient light A(c) in accordance with the following (Equation 10) to generate a sharpened image I.

[0058]

number

[0059] …(Formula 10) In this way, in this embodiment, the illumination distribution for the dark correction and the light correction is calculated by a simple calculation method of inverting the shaped transmittance distribution t', and the obtained illumination distribution is used to reduce the blackout and whiteout caused by the fog removal process. In addition, noise amplification is suppressed by suppressing the gain to be applied. Also, the amount of fog removal process applied can be adjusted depending on the fog density.

[0060] [Second embodiment] In each of the following embodiments including this embodiment, the difference from the first embodiment will be described, and unless otherwise specified below, it is assumed that the embodiment is the same as the first embodiment. In the first embodiment, the estimated fog density is used as the amount of fog removal processing to be applied, but in this embodiment, a value set according to a user operation is used as the amount of fog removal processing to be applied.

[0061] An example of the functional configuration of the image processing unit 208 according to this embodiment is shown in the block diagram of Fig. 14. In Fig. 14, the same functional units as those shown in Fig. 3 are given the same reference numerals, and the description of these functional units will be omitted. In the following description, it is assumed that all of the functional units shown in Fig. 14 are implemented in hardware. However, one or more of the functional units shown in Fig. 14 may be implemented in software (computer program), and in that case, the functions of the one or more functional units are realized by the CPU 201 executing the computer program corresponding to the one or more functional units.

[0062] The process performed by the camera 100 to generate a sharpened image based on the RAW image acquired by the image input unit 206 from the imaging sensor 212 will be described with reference to the flowchart in Fig. 15. In Fig. 15, the same process steps as those shown in Fig. 4 are assigned the same step numbers, and descriptions of those process steps will be omitted.

[0063] In step S1501, the CPU 201 receives, via the communication I / F 205, the “amount of fog elimination processing to be applied (amount of fog elimination to be applied)” transmitted from the computer device 102 in addition to the process of step S401 described above.

[0064] 16 is displayed on the display screen of the computer device 102, the user can increase or decrease the amount of fog removal to be applied by operating a user interface such as a keyboard or mouse to move the indicator 1602 left or right. The user can specify an amount of fog removal to be applied that is closer to 0 by operating the user interface to move the indicator 1602 further to the left, and can specify an amount of fog removal to be applied that is closer to 1 by moving the indicator 1602 further to the right. When the user operates the user interface to input a confirmation instruction, the computer device 102 transmits the amount of fog removal to be applied that corresponds to the current position of the indicator 1602 to the camera 101.

[0065] In step S1508, the processing unit 1406 performs a sharpening process on the developed image J' using the ambient light A(c) and the shaped transmittance distribution t' to generate a sharpened image I. An example of the functional configuration of the processing unit 1406 is shown in the block diagram of Fig. 17. In Fig. 17, the same reference numerals are used for the same functional units as those shown in Fig. 7, and descriptions of these functional units will be omitted.

[0066] The estimation unit 702 obtains the illumination distribution L by using the shaped transmittance distribution t'. The calculation unit 1703 obtains the maximum gain gd by performing calculation processing according to (Equation 7) using the amount of fog removal processing applied received from the computer device 102 instead of the fog density s. The calculation unit 704 specifies the dark correction / light correction gain gr corresponding to the illumination distribution L. The adjustment unit 705 obtains the applied gain gr' by using the maximum gain gd, the dark correction / light correction gain gr, and the illumination distribution L. The correction application unit 1706 generates a sharpened image I by performing calculation processing according to (Equation 10) using the amount of fog removal processing applied received from the computer device 102 instead of the fog density s.

[0067] Thus, according to this embodiment, in addition to being able to achieve the effects of the first embodiment, the amount of fog removal processing applied can be adjusted in response to user operation, and a sharpened image can be generated in response to user operation.

[0068] [Third embodiment] An example of the configuration of a system according to this embodiment will be described with reference to Fig. 18. As shown in Fig. 18, the system according to this embodiment has a camera 1801 and a computer device 1800 as an image processing device. The camera 1801 and the computer device 1800 are configured to be able to communicate data with each other via a video transmission cable 101.

[0069] Camera 1801 is an example of an imaging device that captures moving images or still images. When capturing moving images, camera 1801 outputs images of each frame in the moving images as captured images. When capturing still images, camera 1801 outputs still images captured periodically or irregularly as captured images. The captured images output from camera 1801 are input to computer device 1800 via video transmission cable 101.

[0070] The computer device 1800 performs various image processing including the sharpening processing described above on the captured image output from the camera 1801 to generate an output image, and outputs the generated output image.

[0071] The CPU 1850 executes various processes using computer programs and data stored in the RAM 1851 and the ROM 1852. As a result, the CPU 1850 controls the operation of the entire computer device 1800, and executes or controls various processes that will be described as processes performed by the computer device 1800.

[0072] The RAM 1851 has an area for storing computer programs and data loaded from the ROM 1852 or the storage device 1855, and an area for storing various data including captured images received from the camera 1801 via the I / F 1856. The RAM 1851 also has a work area used when the CPU 1850 or the image processing unit 1857 executes various processes. In this way, the RAM 1851 can provide various areas as appropriate.

[0073] The ROM 1852 stores setting data for the computer device 1800, computer programs and data related to the startup of the computer device 1800, computer programs and data related to the basic operation of the computer device 1800, and the like.

[0074] The operation unit 1853 is a user interface such as a keyboard, a mouse, or a touch panel, and the user can input various instructions to the CPU 1850 by operating it.

[0075] The display unit 1854 has a liquid crystal screen or a touch panel screen, and can display, as images or characters, the results of processing by the CPU 1850 or the image processing unit 1857. The display unit 1854 may be a projection device such as a projector that projects images and characters.

[0076] The storage device 1855 is a large-capacity information storage device that is a non-volatile memory. The storage device 1855 stores an OS (operating system), computer programs and data for causing the CPU 1850 and the image processing unit 1857 to execute or control various processes described as processes performed by the computer device 1800, and the like. The computer programs and data stored in the storage device 1855 are loaded into the RAM 1851 as appropriate under the control of the CPU 1850, and become targets for processing by the CPU 1850 and the image processing unit 1857. The I / F 1856 is a communication interface for performing data communication with the camera 1801 via the above-mentioned video transmission cable 101.

[0077] The image processing unit 1857 performs various processes including the above-mentioned sharpening process on the captured image received from the camera 1801 to generate an output image. An example of the functional configuration of the image processing unit 1857 is shown in the block diagram of Fig. 19. In Fig. 19, the same reference numerals are used for the same functional units as those shown in Fig. 3, and the description of these functional units will be omitted.

[0078] In the following, the functional units shown in Fig. 19 will be described as being implemented in hardware. However, one or more of the functional units shown in Fig. 19 may be implemented in software (computer program), in which case the functions of the one or more functional units are realized by CPU 1850 executing the computer program corresponding to the one or more functional units.

[0079] The CPU 1850 , RAM 1851 , ROM 1852 , operation unit 1853 , display unit 1854 , storage device 1855 , I / F 1856 , and image processing unit 1857 are all connected to a system bus 1858 .

[0080] The processing performed by computer device 1800 to generate a sharpened image based on a captured image received from camera 1801 (a developed image (input image) generated by camera 1801 performing development processing on a RAW image) will be described with reference to the flowchart of FIG. 20.

[0081] In step S2001, the CPU 1850 loads into the RAM 1851 "various parameters to be used in the subsequent processes" stored in the ROM 1852 or storage device 1855 and sets them.

[0082] In step S 2002 , the CPU 1850 receives the developed image J′ transmitted from the camera 1801 via the I / F 1856 , and stores the received developed image J′ in the RAM 1851 or the storage device 1855 .

[0083] In step S2003, the processing unit 2001 performs degamma processing, which is processing for linearizing the developed image J' received in step S2002, on the developed image J' received in step S2002. The developed image J' thereafter is a degamma image obtained by performing degamma processing on the developed image J' received in step S2002.

[0084] In step S2004, the calculation unit 302 generates a dark channel image from the developed image J' in the same manner as in the first embodiment. In step S2005, the calculation unit 303 calculates (estimates) the ambient light A(c) of the color channel c using the dark channel image generated in step S2004 in the same manner as in the first embodiment.

[0085] In step S2006, the calculation unit 304 calculates the atmospheric transmittance distribution t using the developed image J' and the ambient light A(c) in the same manner as in the first embodiment. In step S2007, the processing unit 305 performs shaping processing on the transmittance distribution t using the developed image J' to generate a shaped transmittance distribution t' in the same manner as in the first embodiment.

[0086] In step S2008, the processing unit 306 performs sharpening processing on the developed image J' using the ambient light A(c) and the shaped transmittance distribution t' to generate a sharpened image I, in the same manner as in the first embodiment.

[0087] In step S2009, the processing unit 307 performs gamma correction on the sharpened image I generated in step S2008 in the same manner as in the first embodiment to generate an output image (sharpened image subjected to gamma correction) I'.

[0088] In step S2010, the processing unit 307 displays the output image I′ generated in step S2009 on the display unit 1854. Note that the output destination of the output image is not limited to the display unit 1854.

[0089] In step S2011, CPU 1850 determines whether or not the end condition of the process is satisfied. If the end condition of the process is satisfied as a result of such determination, the process according to the flowchart of Fig. 20 ends, and if the end condition of the process is not satisfied, the process proceeds to step S2001.

[0090] [Fourth embodiment] In the first and second embodiments, the camera 100 and the computer device 102 are separate devices. However, the camera 100 and the computer device 102 may be integrated to configure one image processing device having the functions of the camera 100 and the functions of the computer device 102. This is also true for the third embodiment, where the camera 1801 and the computer device 1800 may be integrated to configure one image processing device having the functions of the camera 1801 and the functions of the computer device 1800. Also, one device may be implemented by multiple devices, and the processing described as the processing performed by the single device may be executed by distributed processing by the multiple devices.

[0091] In the first and second embodiments, the output destination of the sharpened image I is the display 103, but the output destination of the sharpened image I is not limited to the display 103. For example, the sharpened image I may be transmitted to an external device via a network line such as a LAN or the Internet, or may be stored in the memory of the computer device 102 or another device.

[0092] In the first and second embodiments, the image processing unit 208 acquires a developed image obtained by performing development processing on a RAW image output from the imaging sensor 212 as a target (input image) for sharpening processing. However, the method of acquiring the input image is not limited to a specific acquisition method. For example, the image processing unit 208 may acquire an image stored in the recording medium 204 or an image received from an external device such as the computer device 102 via the communication I / F 205 as the input image. This is similar to the third embodiment, and the method of acquiring the input image is not limited to a specific acquisition method.

[0093] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0094] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0095] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) A calculation means for calculating an atmospheric transmittance distribution based on an input image; a processing means for sharpening the input image based on an illumination distribution calculated based on the transmittance distribution; An image processing device comprising: (Item 2) The calculation means is generating a dark channel image based on the input image, the dark channel image having a pixel value that is a minimum value of a color channel in a local region of the input image; Calculating ambient light based on the dark channel image; Calculating the transmittance distribution based on the input image and the ambient light 2. The image processing device according to item 1, (Item 3) The processing means includes: Calculating a first gain based on the input image, the ambient light, the amount of fog removal processing applied, and the transmittance distribution; determining a second gain based on the illumination distribution; 3. The image processing device according to item 2, characterized in that the input image is sharpened based on a third gain calculated based on the first gain, the second gain, and the illumination distribution, the input image, the transmittance distribution, the application amount, and the ambient light. (Item 4) 4. The image processing device according to item 3, wherein the processing means uses a fog density obtained based on a luminance value of the input image as the application amount. (Item 5) 4. The image processing device according to item 3, wherein the processing means uses a value set in response to a user operation as the amount of application. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the processing means performs gamma correction on the input image that has been sharpened. (Item 7) moreover, An imaging means; a developing means for performing a development process on a RAW image obtained by imaging using the imaging means to generate the input image; 7. The image processing device according to any one of items 1 to 6, comprising: (Item 8) moreover, 7. The image processing device according to any one of items 1 to 6, further comprising a unit for acquiring an image captured by an imaging device as the input image. (Item 9) An image processing method performed by an image processing device, comprising: a calculation step in which a calculation means of the image processing device calculates a transmittance distribution of the atmosphere based on an input image; a processing step in which a processing means of the image processing device sharpens the input image based on an illumination distribution calculated based on the transmittance distribution; An image processing method comprising: (Item 10) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 6 and 8.

[0097] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0098] 301: Processing unit 302: Calculation unit 303: Calculation unit 304: Calculation unit 305: Processing unit 306: Processing unit 307: Processing unit

Claims

1. A calculation means for calculating an atmospheric transmittance distribution based on an input image; a processing means for sharpening the input image based on an illumination distribution calculated based on the transmittance distribution; An image processing device comprising:

2. The calculation means is generating a dark channel image based on the input image, the dark channel image having a pixel value that is a minimum value of a color channel in a local region of the input image; Calculating ambient light based on the dark channel image; Calculating the transmittance distribution based on the input image and the ambient light 2. The image processing device according to claim 1,

3. The processing means includes: Calculating a first gain based on the input image, the ambient light, the amount of fog removal processing applied, and the transmittance distribution; determining a second gain based on the illumination distribution; 3. The image processing device according to claim 2, further comprising: a third gain calculated based on the first gain, the second gain, and the illumination distribution; the input image; the transmittance distribution; the application amount; and the ambient light, and the input image is sharpened based on the third gain calculated based on the first gain, the second gain, and the illumination distribution.

4. 4. The image processing apparatus according to claim 3, wherein the processing means uses a fog density obtained based on a luminance value of the input image as the application amount.

5. 4. The image processing apparatus according to claim 3, wherein the processing means uses a value set in response to a user operation as the amount of application.

6. 2. The image processing apparatus according to claim 1, wherein said processing means performs gamma correction on the input image that has been sharpened.

7. moreover, An imaging means; a developing means for performing a development process on a RAW image obtained by imaging using the imaging means to generate the input image; The image processing device according to claim 1 , further comprising:

8. moreover, 2. The image processing apparatus according to claim 1, further comprising: means for acquiring an image captured by an imaging device as the input image.

9. An image processing method performed by an image processing device, comprising: a calculation step in which a calculation means of the image processing device calculates a transmittance distribution of the atmosphere based on an input image; a processing step in which a processing means of the image processing device sharpens the input image based on an illumination distribution calculated based on the transmittance distribution; An image processing method comprising:

10. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 6 and 8.

Images