Image processing device, image processing method, and image processing program

Abstract

This image processing device includes a calculation unit, a black level correction unit, and a tone correction unit. The calculation unit uses image data acquired by an imaging device to calculate a reference black level and a reference illumination light component, on the basis of pixel values of surrounding pixels positioned around an image processing target pixel. The black level correction unit corrects the black level of the target pixel on the basis of the reference black level calculated by the calculation unit. The tone correction unit performs, on the basis of the reference illumination light component calculated by the calculation unit, tone correction of the target pixel for which the black level was corrected. Due to the foregoing, it is possible to generate a highly visible image with reduced whitening even if a large gain is applied to a dark portion of an image in local tone correction.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] In recent years, image sensors capable of capturing HDR (High Dynamic Range) images have been developed, enabling them to capture images without overexposure even in scenes with a wide dynamic range, such as backlit scenes. Because the dynamic range of HDR images captured by these image sensors is wider than that of SDR (Standard Dynamic Range), HDR images cannot be displayed on standard display devices. Therefore, local tone mapping (LTM) is performed to correct HDR images and generate SDR images that can be displayed on standard display devices. Black level correction is also known as a method for correcting image data captured by an image sensor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-76804 [Patent Document 2] U.S. Patent No. 10,803,565 [Patent Document 3] U.S. Patent No. 10,757,339 [Patent Document 4] Patent Publication No. 2021-16103 [Patent Document 5] International Publication No. 2019 / 193807 [Patent Document 6] Japanese Patent Application Publication No. 2020-3998 [Patent Document 7] Japanese Patent Application Laid-Open No. 2015-212978 [Non-patent literature]

[0004] [Non-Patent Document 1] Guided Image Filtering, IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume: 35, Issue: 6, June 2013) Summary of the Invention [Problem to be solved by the invention]

[0005] Local tone correction applies a large gain to dark areas of an image. Therefore, if the black level in dark areas is not properly corrected and pixel values ​​are shifted to the higher side, the local tone correction emphasizes the black level, resulting in a washed-out image. Furthermore, black level processing, which cancels out the dark current of the image sensor, is performed uniformly across the entire image. Therefore, it is difficult to correct local black level deviations around strong light sources, such as backlight, using black level processing across the entire image. If local black level deviations cause washed-out images, visibility may be reduced, and subsequent image processing may not be performed properly.

[0006] The present invention has been made in consideration of the above points, and aims to generate an image with high visibility in which the occurrence of whitening is suppressed even when a large gain is applied to dark areas of an image in local tone correction. [Means for solving the problem]

[0007] In one aspect of the present invention, an image processing device uses image data acquired by an imaging device to determine a reference black level and a reference illumination light component based on pixel values ​​of surrounding pixels located around a pixel to be processed. For each of the plurality of target pixels a black level correction unit that corrects the black level of the target pixel based on the reference black level calculated by the calculation unit; and a tone correction unit that performs tone correction on the target pixel whose black level has been corrected based on the reference illumination light component calculated by the calculation unit. [Effects of the Invention]

[0008] According to the disclosed technology, it is possible to generate an image with high visibility in which the occurrence of whitish color is suppressed even when a large gain is applied to dark areas of an image in local tone correction. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of an image processing system including an image processing device according to a first embodiment. [Figure 2] 2 is a block diagram showing an outline of the configuration of various devices mounted on the moving body of FIG. 1. FIG. [Figure 3] 3 is a block diagram showing an example of the configuration of the image processing device and the information processing device shown in FIG. 2. FIG. [Figure 4] FIG. 2 is a functional block diagram showing an example of the configuration of the image processing device in FIG. 1. [Figure 5] 5 is a functional block diagram illustrating an example of a black level estimation amount calculation unit in FIG. 4. FIG. [Figure 6] 6 is an explanatory diagram showing an example of a method for calculating the minimum pixel value of peripheral pixels for each target pixel using the minimum value filter of FIG. 5. FIG. [Figure 7] 5 is a functional block diagram illustrating an example of an illumination light component calculation unit in FIG. 4. FIG. [Figure 8] 5 is a functional block diagram illustrating an example of a local black level correction unit in FIG. 4. [Figure 9] 5 is a functional block diagram showing an example of the function of the local tone correction unit of FIG. 4. FIG. [Figure 10] 10 is an explanatory diagram showing an example of the compression processing of the illumination light component and the enhancement processing of the reflectance component in FIG. 9. FIG. [Figure 11] 3 is a flowchart showing an example of processing for reducing the dynamic range of image data, which is performed by the image processing device of FIG. 1. FIG. [Figure 12] 10A and 10B are explanatory diagrams showing an example of an image when local black level correction is performed before local tone correction, and an example of an image when local black level correction is not performed before local tone correction. [Figure 13]FIG. 10 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described with reference to the drawings. In the following description, image data may be simply referred to as an image.

[0011] (First embodiment) FIG. 1 illustrates an example of an image processing system including an image processing device according to a first embodiment. The image processing system 100 illustrated in FIG. 1 is mounted on a moving object 200, such as an automobile. Imaging devices 19A, 19B, 19C, 19D, and 19E, such as cameras, are installed at the front, rear, left, and right sides of the moving object 200 in a traveling direction D, and at the front of the interior of the moving object 200. Hereinafter, when the imaging devices 19A, 19B, 19C, 19D, and 19E are not to be distinguished from one another, they are also referred to as imaging devices 19. For example, an image sensor mounted on the imaging device 19 includes R pixels, G pixels, and B pixels that detect red light R, green light G, and blue light B, respectively. For example, the R pixels, G pixels, and B pixels are arranged in a so-called Bayer array.

[0012] The number and installation positions of the imaging devices 19 installed on the moving body 200 are not limited to those shown in Fig. 1. For example, one imaging device 19 may be installed only in front of the moving body 200, or two imaging devices 19 may be installed only in front and rear of the moving body 200. Alternatively, the imaging devices 19 may be installed on the ceiling of the moving body 200.

[0013] Furthermore, the mobile object 200 on which the image processing system 100 is mounted is not limited to an automobile, and may be, for example, a transport robot or a drone operating in a factory. The image processing system 100 may also be a system that processes images acquired from an imaging device 19 other than the imaging device 19 installed in the mobile object 200, such as a surveillance camera, a digital still camera, or a digital camcorder.

[0014] Each imaging device 19 is connected to the image processing device 10 by wire or wirelessly. Furthermore, the distance between each imaging device 19 and the image processing device 10 may be longer than the distance imagined in Fig. 1. For example, image data acquired by the imaging device 19 may be transmitted via a network to the image processing device 10 installed outside the mobile object 200. In this case, at least one of the image processing device 10 and the information processing device 11 may be realized by cloud computing.

[0015] The image processing system 100 has an image processing device 10, an information processing device 11, and a display device 12. In FIG. 1, for ease of understanding, the image processing system 100 is depicted superimposed on an image diagram of a moving object 200 viewed from above. However, in reality, the image processing device 10 and the information processing device 11 are mounted on a control board or the like mounted on the moving object 200, and the display device 12 is installed in a position within the moving object 200 that is visible to people such as the driver. The image processing device 10 may be mounted on the control board or the like as part of the information processing device 11.

[0016] Fig. 2 shows an overview of the configuration of various devices mounted on the mobile object 200 of Fig. 1. The mobile object 200 has an image processing device 10, an information processing device 11, a display device 12, at least one ECU (Electronic Control Unit) 13, and a wireless communication device 14, which are interconnected via an internal network. The mobile object 200 also has a sensor 15, a drive device 16, a lamp device 17, a navigation device 18, and an imaging device 19. For example, the internal network is an in-vehicle network such as a CAN (Controller Area Network) or Ethernet (registered trademark).

[0017] The image processing device 10 receives image data (frame data) acquired by the imaging device 19 and performs image processing using the received image data. The image processing device 10 may record the image data generated by the correction in an external or internal recording device.

[0018] The information processing device 11 performs processing such as image recognition using image data that has been image-processed by the image processing device 10. For example, the information processing device 11 may recognize objects such as people, signals, or signs outside the moving object 200 based on the image generated by the image processing device 10, and may track the recognized objects. The information processing device 11 may function as a computer that controls each part of the moving object 200. Furthermore, the information processing device 11 may control the entire moving object 200 by controlling the ECU 13.

[0019] The display device 12 displays an image, a corrected image, etc. using the image data generated by the image processing device 10. The display device 12 may display an image of the moving object 200 in the backward direction in real time when the moving object 200 moves backward (backing up). The display device 12 may also display an image output from the navigation device 18.

[0020] The ECUs 13 are provided corresponding to respective mechanical parts such as an engine or a transmission. Each ECU 13 controls the corresponding mechanical part based on instructions from the information processing device 11. The wireless communication device 14 communicates with devices external to the mobile object 200. The sensors 15 are sensors that detect various types of information. The sensors 15 may include, for example, a position sensor that acquires current position information of the mobile object 200. The sensors 15 may also include a speed sensor that detects the speed of the mobile object 200.

[0021] The driving device 16 is various devices for moving the moving body 200. The driving device 16 may include, for example, an engine, a steering device, and a braking device. The lamp device 17 is various lighting devices mounted on the moving body 200. The lamp device 17 may include, for example, a headlamp, a turn signal lamp, a backlight, and a brake lamp. The navigation device 18 is a device that provides audio and visual guidance on the route to a destination.

[0022] The imaging device 19 has, for example, an image sensor IMGS with a Bayer array, which is equipped with R pixels, G pixels, and B pixels, each of which includes a plurality of types of filters that transmit red light R, green light G, and blue light B, respectively. That is, the image sensor IMGS includes a plurality of types of pixels that detect light in different wavelength ranges. As described above, the image data acquired by the imaging device 19 is processed by the image processing device 10.

[0023] Fig. 3 shows an example of the configuration of the image processing device 10 and the information processing device 11 of Fig. 2. Since the image processing device 10 and the information processing device 11 have similar configurations, the configuration of the image processing device 10 will be described below. For example, the image processing device 10 has a CPU 20, an interface device 21, a drive device 22, an auxiliary storage device 23, and a memory device 24, which are interconnected by a bus BUS.

[0024] The CPU 20 performs various types of image processing, which will be described later, by executing an image processing program stored in the memory device 24. The interface device 21 is used to connect to a network (not shown).

[0025] The drive device 22 has an interface for connecting the recording medium 30, and transfers the image processing program stored in the recording medium 30 to the auxiliary storage device 23, for example, based on an instruction from the CPU 20. The drive device 22 may also transfer image data, etc. stored in the auxiliary storage device 23 to the recording medium 30.

[0026] The auxiliary storage device 23 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and stores image processing programs, image data, various parameters used in image processing, etc. The memory device 24 is, for example, a dynamic random access memory (DRAM), and stores image processing programs, etc., transferred from the auxiliary storage device 23.

[0027] Fig. 4 shows an example of the configuration of the image processing device 10 in Fig. 1. The image processing device 10 has an image data acquisition unit 102, a black level estimation amount calculation unit 104, an illumination light component calculation unit 106, a local black level correction unit 108, a local tone correction unit 110, and an image data output unit 112. The black level estimation amount calculation unit 104 and the illumination light component calculation unit 106 are examples of a calculation unit.

[0028] The image data acquisition unit 102 acquires image data showing an image of the surroundings of the moving object 200 captured by at least one of the imaging devices 19. For example, the image data acquired by the image data acquisition unit 102 is frame image data constituting a moving image captured by the imaging device 19. Note that the image data acquisition unit 102 may acquire image data from a storage device in which image data captured by the imaging device 19 or the like is stored.

[0029] The black level estimator calculation unit 104 receives as input image data the image data acquired by the image data acquisition unit 102. The black level estimator calculation unit 104 calculates the minimum pixel value of the pixels in a surrounding area located around a target pixel for image processing in the image represented by the input image data as the black level estimate of the target pixel.

[0030] By using the minimum pixel value of the surrounding pixels as the black level estimate, the black level estimate calculation unit 104 can calculate an appropriate black level estimate, even if the target pixel has a high black level due to the influence of a surrounding light source or the like. This allows the local black level correction unit 108, which will be described later, to appropriately correct the black level of the target pixel. The black level estimate of the target pixel calculated by the black level estimate calculation unit 104 is an example of a reference black level.

[0031] The illumination light component calculation unit 106 calculates the average value of pixel values ​​of surrounding pixels located in the surrounding area of ​​the target pixel as the illumination light component of the target pixel. By using the average value of pixel values ​​of surrounding pixels as the illumination light component, the illumination light component calculation unit 106 can calculate an appropriate illumination light component, for example, even when the pixel value of the target pixel changes due to the influence of a surrounding light source, etc. This allows the local tone correction unit 110, described later, to appropriately perform tone correction of the target pixel. The illumination light component of the target pixel calculated by the illumination light component calculation unit 106 is an example of a reference illumination light component.

[0032] The local black level correction unit 108 receives, for each target pixel, which is a pixel in the image represented by the input image data, a black level estimation amount from the black level estimation amount calculation unit 104. Then, the local black level correction unit 108 performs local black level correction (LBC) for each target pixel based on the black level estimation amount, and generates image data with corrected black levels.

[0033] Based on the illumination light components from the illumination light component calculation unit 106, the local tone correction unit 110 performs local tone correction (LTM: Local Tone Mapping) on ​​the image data whose black level has been corrected by the local black level correction unit 108, thereby generating corrected image data. The image data output unit 112 outputs the corrected image data output from the local tone correction unit 110 to, for example, the display device 12.

[0034] Typically, a correction unit that performs local tone correction reduces the dynamic range by changing the method of performing tone correction for each local region of the image, and generates image data that can be displayed on the display device 12, etc. For example, the correction unit applies a large gain to dark areas to brighten the image, and a small gain to bright areas to prevent the image from becoming too bright, thereby making the brightness of the entire image uniform. This prevents the loss of image information due to local tone correction that narrows the dynamic range of the image.

[0035] However, for example, if the black level in a dark area is shifted to the bright side, applying a large gain to the dark area through local tone correction may result in excessive pixel value increase. This may result in an image that appears washed out, reducing the visibility of the image. For example, a shift in the black level in a dark area may occur due to the effects of light scattering around a strong light source.

[0036] The local black level correction unit 108 of this embodiment performs local black level correction for partial whitening of pixel values ​​caused by optical effects such as halation due to the subject, rather than black level correction for the entire image to cancel dark current in the image sensor IMGS. Note that correction for black level whitening caused by dark current in the image sensor IMGS may be performed by a correction unit separate from the local black level correction unit 108. In this case, the correction performed by the separate correction unit is preferably performed before the black level correction by the local black level correction unit 108.

[0037] In the image processing device 10, the local black level correction unit 108 corrects the black level before the local tone correction unit 110 performs tone correction on the image. As a result, even when whiteout or the like occurs in an image in an area adjacent to a strong light source, the local black level correction unit 108 can generate image data in which whiteout or the like has been suppressed by the local black level correction, and supply the image data to the local tone correction unit 110. As a result, the image processing device 10 can prevent the local tone correction unit 110 from excessively increasing pixel values. In other words, it is possible to prevent the tone correction by the local tone correction unit 110 from deteriorating the visibility of the image.

[0038] Fig. 5 shows an example of the function of the black level estimator 104 of Fig. 4. The black level estimator 104 has a minimum value filter 1041 and a first edge-preserving smoothing processor 1042.

[0039] The minimum value filter 1041 sequentially sets each pixel in the image represented by the input image data as a target pixel, and calculates the minimum pixel value of the surrounding pixels in the surrounding area of ​​the target pixel for each target pixel.The minimum value filter 1041 then assigns each of the calculated minimum pixel values ​​to the position of the corresponding target pixel to generate input image data Pi (minimum value image data).

[0040] The first edge-preserving smoothing processor 1042 can use a guided filter or the like that performs edge-preserving smoothing based on the input image data Pi and the guide image data Ii. The first edge-preserving smoothing processor 1042 performs squaring, multiplication, calculation of an average value using multiple box filters, calculation of coefficients a k, calculation of coefficients b k, and calculation of an output pixel value to obtain a black level estimate. Hereinafter, the input image data Pi and the guide image data Ii will also be referred to as the input image Pi and the guide image Ii.

[0041] The box filter calculates the average value of pixel values ​​of surrounding pixels. Although not particularly limited, for example, the surrounding pixels are N pixels vertically and N pixels horizontally including the target pixel, where "N" is, for example, "32". The symbol μk indicates the average value of pixel values ​​of surrounding pixels located around the target pixel of the guide image Ii. The symbol σ2k indicates the average value of squared pixel values ​​of surrounding pixels of the guide image Ii. The symbol ΣIi*pi (the symbol * indicates multiplication) indicates the average value of the values ​​obtained by multiplying the pixel values ​​of surrounding pixels of the input image Pi by the pixel values ​​of surrounding pixels of the guide image Ii. The symbol pk indicates the average value of pixel values ​​of surrounding pixels of the input image Pi. The symbols ak and bk indicate coefficients to be applied to the guide image Ii.

[0042] The coefficient ak is expressed by equation (1), and the coefficient bk is expressed by equation (2). The symbol qi indicating the black level estimation amount is expressed by equation (3).

[0043]

number

[0044]

number

[0045]

number

[0046] The minimum pixel value obtained by processing by the minimum value filter 1041 does not take into account edges of the image. In this embodiment, the first edge-preserving smoothing processor 1042 smoothes the input image data Pi of the minimum pixel value output from the minimum value filter 1041 in accordance with the guide image, thereby improving the accuracy of the minimum value at the boundary between areas with different light sources. For example, the accuracy of the minimum value at the edge portion of the image, such as the boundary between the inside and outside of a tunnel, can be improved. This allows the black level estimator calculation unit 104 to calculate a black level estimate for each target pixel over the entire image to reduce the effects of flare, halation, and the like. Note that the edge-preserving smoothing process may be performed using a joint bilateral filter or the like.

[0047] Fig. 6 shows an example of a method for calculating the minimum pixel value of the surrounding pixels for each target pixel using the minimum value filter 1041 in Fig. 5. In Fig. 6, the target pixel is indicated by a white rectangle, and the minimum pixel value of the surrounding pixels is indicated by a black rectangle. For example, the minimum value filter 1041 shifts the target pixel and the surrounding area of ​​the target pixel by one pixel at a time using raster scanning or the like in the image IMG represented by the input image data, and finds the minimum pixel value in each surrounding area.

[0048] The surrounding area for finding the minimum pixel value includes a predetermined number of pixels surrounding the target pixel. For example, the number of pixels surrounding the target pixel is N 2 Here, N is an arbitrary natural number. However, in the areas close to the four corners and each side of the image IMG, the number of pixels for which the minimum pixel value is to be found is N 25, the target pixel and the surrounding area of ​​the target pixel are shifted by one pixel at a time, and the average pixel value in each surrounding area is found. As shown in FIG. 6, the image processing device 10 can find the minimum pixel value and the average pixel value for each surrounding pixel located around the target pixel by raster scanning the target pixel. The number of pixels in the surrounding area for which the minimum pixel value is found is N 2 The number is not limited to pieces, but may be A pieces vertically and B pieces horizontally (A and B are natural numbers with different values).

[0049] Fig. 7 shows an example of the function of the illumination light component calculation unit 106 in Fig. 4. The illumination light component calculation unit 106 has a second edge-preserving smoothing processing unit 1061. The second edge-preserving smoothing processing unit 1061 performs edge-preserving smoothing processing based on the input image Pi and the guide image Ii by using, for example, a guided filter or the like, and calculates the illumination light component.

[0050] When calculating the illumination light component, the input image Pi and the guide image Ii are the same. The second edge-preserving smoothing processor 1061 calculates the illumination light component by squaring, calculating the average value using multiple box filters, calculating the coefficients a k and b k , and calculating the output pixel value. The illumination light component can be estimated by averaging the pixel values ​​of the image, but as with the black level estimation amount, the accuracy of the average value may be low at the boundary between areas with different light sources. Therefore, by performing smoothing processing on the input image data using the second edge-preserving smoothing processor 1061, it is possible to suppress the decrease in accuracy of the average value at the boundary between the illumination light components.

[0051] The image processing device 10 can calculate an appropriate black level estimate and an appropriate illumination light component by selectively using the first edge-preserving smoothing processor 1042 and the second edge-preserving smoothing processor 1061.

[0052] Fig. 8 shows an example of the function of the local black level correction unit 108 in Fig. 4. The local black level correction unit 108 corrects the black level of the input image data using the black level estimate calculated by the black level estimate calculation unit 104, and outputs the corrected image data. For example, in the local black level correction process (LBC), the local black level correction unit 108 sets pixel values ​​equal to or less than the black level estimate to "0".

[0053] That is, the local black level correction unit 108 estimates that pixel values ​​equal to or less than the black level estimate are false pixel values ​​due to, for example, whitening caused by the influence of a high-luminance light source. The local black level correction unit 108 then corrects the false pixel values ​​to correct pixel values ​​that would occur if whitening or the like did not occur. This makes it possible to prevent pixel values ​​of subjects located near high-luminance light sources in the image from becoming high (i.e., whitening) through subsequent processing by the local tone correction unit 110.

[0054] For example, the local black level corrector 108 may perform black level correction using equation (4). Y = (XK) * Max / (Max-K) ... (4)

[0055] In equation (4), symbol X represents an input pixel value, symbol K represents a black level estimate, symbol Max represents the maximum value of the input pixel value and the maximum value of the output pixel value, and symbol Y represents the output pixel value. Note that the local black level correction unit 108 may correct the image data using a lookup table that holds various values ​​for each parameter in equation (4).

[0056] Fig. 9 shows an example of the function of the local tone corrector 110 in Fig. 4. The local tone corrector 110 performs division, emphasis processing, compression processing, and multiplication using a technique known as Retinex theory, and outputs locally tone-corrected output image data.

[0057] The local tone correction unit 110 calculates a reflectance component, which is a signal representing the texture component of an object, by dividing the input image data by the illumination light component. The local tone correction unit 110 performs an enhancement process on the reflectance component calculated by the division, making the texture component more visible.

[0058] The local tone correction unit 110 performs compression processing on the illumination light component to convert it into image data with minimal changes in illumination. The illumination light component to be compressed uses the data calculated by the illumination light component calculation unit 106 in Figure 4. The local tone correction unit 110 then multiplies the enhanced reflectance component by the compressed illumination light component to generate locally tone-corrected output image data.

[0059] This allows the local tone correction unit 110 to generate image data with little blocked-up shadows and blown-out highlights (a clearly visible image) even when the image data has a small number of bits and a narrow dynamic range. Note that the local tone correction unit 110 may also perform tone correction processing that changes the shape of the tone for each local region without using Retinex theory. In this case, the local tone correction unit 110 can also generate locally tone-corrected output image data in the same way as output image data generated using Retinex theory.

[0060] Furthermore, since the emphasis processing, compression processing, etc. performed in the local tone correction processing are performed using the illumination light component, it is possible to eliminate the need for additional calculation processing, thereby suppressing an increase in the processing load on the image processing device 10.

[0061] FIG. 10 shows an example of the compression process of the illumination light component and the enhancement process of the reflectance component in FIG.

[0062] For example, in the compression process of the illumination light component, the local tone correction unit 110 uses the tone curve shown in graph A to brighten the dark parts of the illumination light component and darken the bright parts of the illumination light component. This makes it possible to make the brightness of the entire image uniform in the local tone correction.

[0063] Furthermore, in the reflectance component enhancement process (1), the local tone correction unit 110 can perform enhancement processing such as multiplication of the logarithm of the reflectance component, as shown in graph B, to make the texture of the image more visible. For example, graph B is shown in equation (5). Enhanced reflectance component = exp(ln(reflectance component) * gain) ... (5)

[0064] Furthermore, the local tone correction unit 110 may perform a reflectance component enhancement process (2) instead of the reflectance component enhancement process (1). For example, in the reflectance component enhancement process (2), the enhancement process shown in graph C is performed instead of graph B.

[0065] Fig. 11 shows an example of a processing flow for reducing the dynamic range of image data, which is performed by the image processing device 10 of Fig. 1. That is, Fig. 11 shows an example of an image processing method by the image processing device 10. The flow shown in Fig. 11 may be realized, for example, by the CPU 20 (Fig. 3) of the image processing device 10 executing an image processing program.

[0066] 11 may be realized by hardware such as an FPGA or an ASIC mounted on the image processing device 10. Alternatively, the flow shown in FIG. 11 may be realized by cooperation between hardware and software.

[0067] First, in step S10, the image data acquisition unit 102 in Fig. 4 acquires image data, for example, from the imaging device 19, and outputs it as input image data to be used in image processing. Next, in step S20, the black level estimation amount calculation unit 104 in Fig. 4 calculates a black level estimation amount by finding the minimum pixel value of the surrounding pixels for each target pixel in the input image data.

[0068] Next, in step S30, the local black level correction unit 108 in Fig. 4 performs local black level correction (LBC) on the input image data based on the black level estimate, generating image data with corrected black levels. In step S40, the illumination light component calculation unit 106 in Fig. 4 calculates, for each target pixel in the input image data, the average value of pixel values ​​of surrounding pixels as the illumination light component. Note that step S40 may be performed anywhere between step S10 and step S50.

[0069] After steps S30 and S40 are performed, in step S50, local tone correction unit 110 in Fig. 4 performs local tone correction (LTM) on the image data whose black level has been corrected based on the illumination light component to generate corrected image data. Next, in step S60, image data output unit 112 in Fig. 4 outputs the corrected image data that has undergone local tone correction to, for example, display device 12, and the process shown in Fig. 11 ends.

[0070] FIG. 12 shows an example of an image where local black level correction is performed before local tone correction, and an example of an image where local black level correction is not performed before local tone correction.

[0071] As described above, when local black level correction is performed before local tone correction, an image with less whiteout, higher contrast, and better visibility can be obtained, which allows, for example, appropriate image processing using the corrected image data.

[0072] On the other hand, if local black level correction is not performed before local tone correction, the image will appear whitish in dark areas, resulting in a hazy overall image. Furthermore, the visibility of the image will be reduced due to factors such as light scattering around a strong light source. This can lead to issues with image processing using the corrected image data.

[0073] As described above, in this embodiment, the image processing device 10 performs local black level correction processing on image data acquired from the imaging device 19 or the like before performing local tone correction processing. This makes it possible to suppress blown-out appearance of the image due to local tone correction, for example, in cases where a strong light source is reflected in the image. In other words, even when a large gain is applied to dark areas of the image in local tone correction, it is possible to generate an image with high visibility in which blown-out appearance is suppressed.

[0074] The first edge-preserving smoothing processor 1042 smoothes the input image data Pi having the minimum pixel value output from the minimum value filter 1041 in accordance with the guide image, thereby suppressing a decrease in accuracy of the minimum value at boundaries. This allows the black level estimator calculation unit 104 to calculate a black level estimate for the entire image for each target pixel to reduce the effects of flare, halation, or the like.

[0075] By performing smoothing processing on the input image data using the second edge-preserving smoothing processor 1061, it is possible to suppress a decrease in accuracy of the average value at the boundary portion of the illumination light component.

[0076] Furthermore, it is possible to reduce flare and halation that occur due to the influence of the lens of the image pickup device 19. Also, by performing black level correction processing before local tone correction processing, it is possible to reproduce optically correct colors compared to when only local tone correction processing is performed.

[0077] (Second embodiment) Fig. 13 shows an example of the functional configuration of an image processing device in the second embodiment. Elements similar to those in the above-described embodiments are given the same reference numerals, and detailed description thereof will be omitted. The image processing device 10A shown in Fig. 13 is similar to the image processing device 10 shown in Figs. 1 to 3, and is installed in an image processing system 100 together with an information processing device 11 and a display device 12. The functions shown in Fig. 13 may be realized, for example, by the CPU 20 (Fig. 4) of the image processing device 10 in Fig. 3 executing an image processing program.

[0078] The image processing device 10A has an illumination light component calculation unit 106A instead of the illumination light component calculation unit 106 in Fig. 4. The illumination light component calculation unit 106A has a second edge-preserving smoothing processing unit 1061A. The other configuration of the image processing device 10A is the same as that of the image processing device 10 in Fig. 4, except that the black level estimation amount calculation unit 104 outputs a part of the calculation result to the illumination light component calculation unit 106A.

[0079] As shown in Fig. 7, part of the processing of the second edge-preserving smoothing processing unit 1061 of the illumination light component calculation unit 106 is common to the processing of the first edge-preserving smoothing processing unit 1042 of the black level estimation amount calculation unit 104 shown in Fig. 5. For example, the squaring processing, the box filter that receives input image data (guide image data), and the box filter that receives the output of the squaring processing are common to each other.

[0080] 5 outputs the output of a box filter that receives the guide image data Ii and the output of a box filter that receives the output of the squaring process to the second edge-preserving smoothing processor 1061A. The second edge-preserving smoothing processor 1061A has the squaring process and the two box process functions removed from the second edge-preserving smoothing processor 1061 in FIG.

[0081] The second edge-preserving smoothing processor 1061A then calculates the coefficients a k and b k using the outputs of the two box filters received from the first edge-preserving smoothing processor 1042. That is, the second edge-preserving smoothing processor 1061A uses a part of the processing result of the first edge-preserving smoothing process in the second edge-preserving smoothing process.

[0082] This allows the processing amount of the second edge-preserving smoothing processing unit 1061A to be reduced compared to the processing amount of the second edge-preserving smoothing processing unit 1061 in Fig. 7. As a result, for example, the processing time of the illumination light component calculation unit 106A can be shortened compared to the processing time of the illumination light component calculation unit 106 in Fig. 4, and the time required for image processing can be shortened. Furthermore, since the load on the image processing device 10A can be reduced compared to the image processing device 10, for example, the power consumption of the image processing device 10A can be reduced.

[0083] As described above, this embodiment can also achieve the same effects as the above-described embodiments. For example, by performing local black level correction processing before performing local tone correction processing on image data acquired from the image capture device 19, etc., it is possible to suppress whitening of the image caused by local tone correction. In other words, even when a large gain is applied to dark areas of the image in local tone correction, it is possible to generate an image with high visibility in which whitening is suppressed. can.

[0084] Furthermore, in this embodiment, the processing time of the illumination light component calculation unit 106A can be shortened, thereby shortening the time required for image processing. Also, the load on the image processing device 10A can be reduced, thereby reducing the power consumption of the image processing device 10A.

[0085] In the above-described embodiment, the image processing device 10, 10A performs local black level correction and local tone correction on the entire image captured by the imaging device 19. However, the local black level correction and local tone correction may be applied to a partial region of the image. For example, the image processing device 10 may perform local black level correction and local tone correction only on the region surrounding a light source that is likely to cause halation. In this case, the image processing device 10, 10A can determine the presence or absence of a light source that is likely to cause halation based on whether or not there is a surrounding region where pixel values ​​are greater than a predetermined threshold.

[0086] Furthermore, the above-described embodiments do not specify conditions for performing local black level correction and local tone correction. However, for example, the image processing device 10, 10A may perform local black level correction and local tone correction only in environments that are susceptible to the influence of light sources, such as at night or in strong backlight. In this case, the image processing device 10, 10A can determine that it is nighttime when the entire image is dark, and can determine that it is backlit when the entire image is bright or when the entire image is locally bright.

[0087] Furthermore, the image processing device 10, 10A may perform the estimation of the black level by the black level estimator calculation unit 104 based on luminance calculated from the pixel values ​​of R pixels, G pixels, and B pixels. Alternatively, the image processing device 10, 10A may perform the estimation of the black level for each pixel value of R pixels, G pixels, and B pixels. In this case, for example, the image processing device 10, 10A can reduce colored flares and the like that occur around light sources of each color in a traffic light in an image.

[0088] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These requirements can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0089] 10, 10A Image processing device 11 Information processing equipment 12 Display device 13 ECU 14 Wireless communication devices 15 sensors 16 Drive unit 17 Lamp unit 18 Navigation devices 19 (19A, 19B, 19C, 19D, 19E) Imaging device 20 CPU 21 Interface Device 22 Drive device 23 Auxiliary storage device 24 Memory Device 30 Recording media 100 Image Processing System 102 Image data acquisition unit 104 Black level estimation calculation unit 106, 106A Illumination light component calculation unit 108 Local black level correction section 110 Local Tone Correction Unit 112 Image data output unit 200 Mobile 1041 Minimum Filter 1042 First edge-preserving smoothing processing unit 1061, 1061A Second edge-preserving smoothing processing unit BUS Ii Guide Image Data Image IMG IMGS image sensor Pi Input image data PX pixels

Claims

1. a calculation unit that calculates a reference black level and a reference illumination light component for each of a plurality of target pixels for image processing based on pixel values ​​of surrounding pixels located around the target pixel, using image data acquired by the imaging device; a black level correction unit that corrects the black level of the target pixel based on the reference black level calculated by the calculation unit; a tone correction unit that performs tone correction on the target pixel whose black level has been corrected based on the reference illumination light component calculated by the calculation unit; An image processing device having:

2. The calculation unit calculates the minimum value of the pixel values ​​of the surrounding pixels as the reference black level. The image processing device according to claim 1 .

3. The calculation unit calculates an average value of pixel values ​​of the peripheral pixels as the reference illumination light component.

3. The image processing device according to claim 1.

4. The calculation unit performs a first edge-preserving smoothing process when calculating the reference black level, and performs a second edge-preserving smoothing process when calculating the reference illumination light component.

4. The image processing device according to claim 1.

5. The calculation unit uses a part of the processing result of the first edge-preserving smoothing processing in the second edge-preserving smoothing processing. The image processing device according to claim 4 .

6. a calculation process for calculating a reference black level and a reference illumination light component for each of a plurality of target pixels for image processing based on pixel values ​​of surrounding pixels located around the target pixel using image data acquired by the imaging device; a black level correction process for correcting the black level of the target pixel based on the reference black level calculated by the calculation process; a tone correction process for performing tone correction on the target pixel whose black level has been corrected based on the reference illumination light component calculated by the calculation process; An image processing method that performs the above.

7. a calculation process for calculating a reference black level and a reference illumination light component for each of a plurality of target pixels for image processing based on pixel values ​​of surrounding pixels located around the target pixel using image data acquired by the imaging device; a black level correction process for correcting the black level of the target pixel based on the reference black level calculated by the calculation process; a tone correction process for performing tone correction on the target pixel whose black level has been corrected based on the reference illumination light component calculated by the calculation process; An image processing program that causes a computer to perform the above.

Images