Image Processing Device

Abstract

To provide an image processing device capable of restraining occurrence of black clipping while restraining white clipping.SOLUTION: An image processing device includes a histogram creation unit for creating a histogram H1 representing the relationship between brightness and the number of pixels based on a captured image, and an image correction unit for estimating the brightness caused by reflected light from a hood included in the histogram H1 by multiplying correction brightness derived from the histogram H1 by a correction value, deriving estimated maximum brightness by subtracting brightness caused by the reflected light from the maximum brightness of the histogram H1, deriving estimated minimum brightness by subtracting the brightness caused by the reflected light from lowest brightness in the histogram H1, creating a correction histogram H2 in which the maximum brightness is equal to the estimated maximum brightness and the lowest brightness is equal to the estimated minimum brightness by translating the histogram H1, and forming a corrected image based on the corrected histogram H2.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing device that is mounted on a moving object such as a vehicle. [Background technology]

[0002] Conventionally, for example, Patent Document 1 proposes an image processing device mounted on a moving body such as a vehicle. Specifically, this image processing device includes an imaging unit, an image correction unit that corrects the captured image captured by the imaging unit, and an image display unit that displays the captured image. In addition, a hood is usually placed below the imaging unit to suppress the incidence of light from below the imaging unit (i.e., from inside the moving body). Note that the light from below the imaging unit refers to reflected light caused by the sun.

[0003] Here, although a hood is placed on the image captured by the imaging unit, it is difficult to make the reflectance of the hood zero, and there is a possibility that whitening occurs due to flare caused by reflected light from the hood. An image in which whitening occurs is likely to have low contrast and is difficult to accurately recognize. For this reason, a method has been proposed for the above-mentioned image processing device, in which a corrected image is created by adjusting the amplification value of the captured image in an image correction unit, and the corrected image is displayed on an image display device to suppress the inclusion of whitening. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-183907 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the imaging unit generally adjusts the exposure time (i.e., shutter time) for the next captured image based on the previous captured image, and adjusts the amount of light taken in. Therefore, if the above-mentioned whitening phenomenon is merely suppressed, when reflected light suddenly disappears or appears, there is a possibility that the occurrence of whitening phenomenon cannot be sufficiently suppressed in the next captured image (i.e., corrected image). Furthermore, in the current situation, there is also a demand to suppress the inclusion of black crush in the captured image.

[0006] In view of the above, an object of the present invention is to provide an image processing device capable of suppressing whitening while suppressing the occurrence of black crush. [Means for solving the problem]

[0007] Claim 1 for achieving the above object provides an image processing device, comprising: an image acquisition unit (101) mounted on a moving body, which is provided with a hood (60) for suppressing incidence of light from inside the moving body, and which acquires an image captured by an imaging unit (20) that captures an image with an exposure time based on a control signal; a histogram creation unit (102) that creates a histogram (H1) representing the relationship between luminance and the number of pixels based on the captured image; a correction unit (102) that multiplies a correction luminance derived from the histogram by a correction value to estimate a luminance (L2a) due to reflected light (L2) from the hood included in the histogram; and a correction unit (103) that calculates a luminance (L2b) due to reflected light from the hood based on the maximum luminance (MB) of the histogram. the histogram to create a corrected histogram (H2) in which the maximum luminance becomes the estimated maximum luminance and the minimum luminance becomes the estimated minimum luminance; and the histogram is translated to create a corrected image based on the corrected histogram. The image correction unit (103) is provided with an exposure time adjustment unit (104) that creates a control signal for adjusting the exposure time of the imaging unit based on the corrected histogram, and an output unit (104) that outputs the corrected image and the control signal.

[0008] According to this, the image processing device derives an estimated maximum luminance and an estimated minimum luminance by subtracting the luminance caused by reflected light from a histogram created for a captured image.The image processing device then creates a corrected histogram based on the estimated maximum luminance and the estimated minimum luminance, and creates a corrected image based on the corrected histogram.As a result, it is possible to suppress the influence of the luminance caused by reflected light, and to suppress the occurrence of whitening and crushed black.

[0009] The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and specific components described in the embodiments described below. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an image processing system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing the location of an in-vehicle camera. [Diagram 3] FIG. 1 is a block diagram showing a configuration of an image processing device. [Figure 4] 11A and 11B are diagrams for explaining a histogram, an estimated maximum luminance, and an estimated minimum luminance. [Diagram 5] 4 is a flowchart showing an operation performed by the image processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, parts that are the same or equivalent to each other will be denoted by the same reference numerals.

[0012] (First embodiment) The first embodiment will be described with reference to the drawings. The image processing device of this embodiment is preferably mounted on a moving body such as a vehicle and used to configure an image processing system together with an on-board camera, an image recording and display device, and a driving support device mounted on the moving body. In the following, an example in which the image processing device of this embodiment is mounted on a vehicle to configure an image processing system will be described.

[0013] The image processing system is a system mounted on a vehicle, and as shown in Fig. 1, includes an image processing device 10, an in-vehicle camera 20, an image recording and display device 30, and a driving support device 40. The in-vehicle camera 20, the image recording and display device 30, and the image recording and display device 30 are connected to the image processing device 10 via signal lines or the like.

[0014] The vehicle-mounted camera 20 is configured to have a CMOS (abbreviation of Complementary Metal Oxide Semiconductor) image sensor or the like as an imaging element. The vehicle-mounted camera 20 is disposed near a windshield 50 installed in front of the vehicle so that a predetermined range in front of the vehicle is the imaging range, as shown in FIG. 2, for example. The vehicle-mounted camera 20 of this embodiment repeatedly captures images at a preset time interval, for example, 60 times per second, and outputs the captured images to the image processing device 10. However, the vehicle-mounted camera 20 is configured to adjust the exposure time (i.e., shutter time) based on a control signal from the image processing device 10. The longer the exposure time, the more light is taken in, making the captured image brighter, and the shorter the exposure time, the less light is taken in, making the captured image darker.

[0015] The mounting position of the vehicle-mounted camera 20, the imaging range of the vehicle-mounted camera 20, and the imaging period of the vehicle-mounted camera 20 may be determined arbitrarily. The captured image is an image in which brightness is held in each pixel, and each pixel has a brightness in the range of 0 to 495, for example, captured in HDR (abbreviation of High Dynamic Range). The captured image may be a color image having red brightness, green brightness, and blue brightness as data for each pixel, or may be an image having only the brightness of one of the colors, or only the brightness that becomes a black and white image. In this embodiment, the vehicle-mounted camera 20 corresponds to an imaging unit.

[0016] A hood 60 is disposed below the vehicle-mounted camera 20 to prevent light from below the vehicle-mounted camera 20 (i.e., from the inside of the vehicle) from entering the vehicle-mounted camera 20. The hood 60 is made of, for example, a black anti-reflective material or plastic, and is generally plate-shaped. However, it is difficult to make the reflectance of the hood 60 zero, and as described below, light from the sun 70 may be reflected by the hood 60. For this reason, in addition to incident light L1 from the subject S, reflected light L2 reflected by the hood 60 also enters the vehicle-mounted camera 20.

[0017] The image recording and display device 30 is provided within the visual field of the driver, for example, located near a windshield 50 installed at the front of the vehicle, and displays the corrected image output from the image processing device 10. The image recording and display device 30 may be connected to a display device applied to a car navigation system or the like, so that the corrected image is displayed on the display device.

[0018] The driving support device 40 is an ECU, and is configured with a microcomputer or the like equipped with a memory unit configured with a CPU, a ROM, a RAM, a flash memory, etc. CPU is an abbreviation for Central Processing Unit, ROM is an abbreviation for Read Only Memory, and RAM is an abbreviation for Random Access Memory. Storage media such as ROM are non-transient substantial storage media.

[0019] The driving support device 40 performs driving support such as, for example, warning output control and driving control based on the corrected image output from the image processing device 10. The warning output control is a control that outputs a warning when the vehicle is likely to depart from the lane. The driving control is a control of steering and braking of the vehicle that is performed so as to drive within the lane. The driving support device 40 may be configured with one microcomputer or may be configured with multiple microcomputers.

[0020] The image processing device 10 is composed of a microcomputer having a storage unit 12 composed of a CPU 11, a ROM, a RAM, a flash memory, etc. The image processing device 10 has each unit as shown in FIG. 3 so that the CPU 11 executes a program stored in the storage unit. In this embodiment, the storage unit 12 corresponds to a non-transient tangible recording medium that stores a computer program. The execution of the computer program executes a method corresponding to the computer program. The image processing device 10 may be composed of one microcomputer or multiple microcomputers.

[0021] Specifically, the image processing device 10 of this embodiment includes an image acquisition unit 101, a histogram creation unit 102, an image correction unit 103, an exposure time adjustment unit 104, and an output unit 105, as shown in FIG.

[0022] The image acquisition unit 101 acquires a captured image captured by the vehicle-mounted camera 20. Note that the captured image may be a color image having red, green, and blue component values ​​as data for each pixel, as described above, or may be a black and white image having only any one of the color component values.

[0023] The histogram creation unit 102 creates a histogram H1 based on the captured image. The histogram H1 represents the number of pixels for each luminance, where the luminance is a pixel value. If the captured image is a color image, the histogram creation unit 102 creates a histogram H1 for each of the red, green, and blue components.

[0024] The image correction unit 103 performs correction processing based on the histogram H1 of the captured image to make it difficult for whitening due to flare and black crushing in dark areas to occur. For example, assume that a histogram H1 of a captured image as shown in FIG. 4 is created. In this case, the captured image contains incident light L1 from the subject S and reflected light L2 due to reflection by the hood 60. Therefore, the histogram H1 created based on the captured image is created to include the influence of the reflected light L2 due to reflection by the hood 60 as a whole. Furthermore, the created histogram H1 also includes the influence of flare due to the reflected light L2 of the hood (i.e., whitening). In other words, the created histogram H1 includes the luminance due to the reflected light L2 as a whole.

[0025] For this reason, the image correction unit 103 first estimates the luminance caused by the reflected light L2 from the created histogram H1. In this embodiment, as described above, it is assumed that the created histogram H1 includes the direct influence of the reflected light L2, and furthermore, whitish appearance occurs due to flare caused by the reflected light L2. For this reason, first, a value obtained by excluding the luminance caused by the influence of the reflected light L2 from the maximum luminance MB of the created histogram H1 is set as the estimated maximum luminance EMB. Specifically, if the luminance caused by the influence of the reflected light L2 is L2a, the image correction unit 103 calculates the following formula 1 to estimate the luminance L2a of the reflected light L2. Then, the image correction unit 103 calculates the following formula 2 to set the estimated maximum luminance EMB.

[0026] (Equation 1) L2a = MB × correction value

[0027] (Equation 2) EMB = MB - L2a The correction value is derived in advance by performing an experiment or the like taking into consideration the material and angle of the windshield 50, the angle and reflectance of the hood 60, and is stored in the storage unit 12. In this embodiment, the maximum luminance MB corresponds to the correction luminance.

[0028] After that, the image correction unit 103 sets the estimated minimum luminance ELB in the histogram H1 based on the luminance L2a caused by the reflected light L2. More specifically, the created histogram H1 includes the influence of the reflected light L2. For this reason, the image correction unit 103 sets the estimated minimum luminance ELB by subtracting the luminance L2a caused by the reflected light L2 from the minimum luminance LB in the created histogram H1, as shown in the following formula 3.

[0029] (Equation 3) ELB = LB - L2a Then, the image correction unit 103 creates a corrected histogram H2 by translating the created histogram H1 so that the maximum luminance becomes the estimated maximum luminance EMB and the minimum luminance becomes the estimated minimum luminance ELB. The corrected histogram H2 is a histogram obtained by subtracting the luminance L2a of the reflected light L2 from the initially created histogram H1.

[0030] After that, the image correction unit 103 creates a corrected image based on the correction histogram H2. In this embodiment, the image correction unit 103 creates a corrected image while performing tone curve adjustment using the correction histogram H2 as necessary. By creating a corrected image using the correction histogram H2 from which the luminance L2a of the reflected light L2 is subtracted in this way, it is possible to suppress whitening and to make it difficult for a black-crushed image to be created even if the reflected light L2 suddenly disappears. Note that an example of a case in which the reflected light L2 suddenly disappears is when the vehicle enters a tunnel while traveling on a road. In addition, the tone curve adjustment is performed by multiplying all pixels by a conversion function so that the output characteristics in the high luminance range are greater than those in the low luminance range.

[0031] The exposure time adjustment unit 104 creates a control signal that adjusts the next exposure time of the vehicle-mounted camera 20 based on the corrected histogram H2. The exposure time is adjusted so that the histogram created for the next captured image is broadened overall as shown in FIG.

[0032] The output unit 105 outputs the corrected image to the image recording and display device 30 and the driving support device 40. As a result, the corrected image is displayed on the image recording and display device 30, and the driving support device 40 performs driving support based on the corrected image. The output unit 105 also outputs a control signal with an adjusted exposure time to the vehicle-mounted camera 20. As a result, the vehicle-mounted camera 20 creates a captured image with an exposure time according to the luminance L2a of the reflected light L2.

[0033] Next, the operation performed by the image processing device 10 will be described with reference to the flowchart of Fig. 5. Note that the image processing device 10 of this embodiment performs the following operation every time a captured image is acquired by the vehicle-mounted camera 20.

[0034] First, in step S1, the image processing device 10 causes the image acquisition unit 101 to acquire a captured image from the vehicle-mounted camera 20.

[0035] Next, in step S2, the image processing device 10 creates a histogram H1 of the acquired captured image in the histogram creation unit 102. Note that, as described above, if the acquired captured image is a color image, a histogram H1 is created for each of the red component, the green component, and the blue component.

[0036] Next, in step S3, the image processing device 10 derives the luminance L2a of the reflected light L2 from the created histogram H1 in the image correction unit 103. In this embodiment, as described above, the luminance L2a of the reflected light L2 is determined as the value obtained by multiplying the maximum luminance MB of the histogram H1 by the correction value.

[0037] Next, in step S4, the image processing device 10 derives an estimated maximum luminance EMB from the created histogram H1. In this embodiment, as described above, the estimated maximum luminance EMB is a value obtained by subtracting the luminance L2a caused by the reflected light L2 from the maximum luminance MB in the created histogram H1.

[0038] Next, in step S5, the image processing device 10 derives the estimated minimum luminance ELB from the created histogram H1 in the image correction unit 103. In this embodiment, as described above, the estimated minimum luminance ELB is determined as the value obtained by subtracting the luminance L2a caused by the reflected light L2 from the minimum luminance LB.

[0039] Thereafter, in step S6, the image processing device 10 creates a corrected histogram H2 in the image correction unit 103. Specifically, the histogram H1 created in step S2 is translated to create a corrected histogram H2 so that the maximum luminance of the histogram H1 becomes the estimated maximum luminance EMB and the minimum luminance of the histogram H1 becomes the estimated minimum luminance ELB.

[0040] Next, in step S7, the image processing device 10 creates a corrected image using the correction histogram H2. In this embodiment, the image correction unit 103 creates a corrected image while performing tone curve adjustment using the correction histogram H2.

[0041] Furthermore, in step S8, the image processing device 10 generates a control signal in which the exposure time is adjusted based on the corrected histogram H2 in the exposure time adjustment unit 104.

[0042] Then, in step S9, the image processing device 10 outputs the corrected image from the output unit 105 to the image recording and display device 30 and the driving support device 40, and outputs a control signal with the adjusted exposure time to the vehicle-mounted camera 20, and then ends the processing.

[0043] According to the present embodiment described above, the image processing device 10 derives the estimated maximum luminance EMB and the estimated minimum luminance ELB by subtracting the luminance L2a caused by the reflected light L2 from the histogram H1 created for the captured image. Then, the image processing device 10 creates a corrected histogram H2 based on the estimated maximum luminance EMB and the estimated minimum luminance ELB, and creates a corrected image based on the corrected histogram H2. This makes it possible to suppress the influence of the luminance L2a caused by the reflected light L2, suppress the occurrence of whitening, and suppress the occurrence of black crush even if the reflected light L2 suddenly disappears or appears.

[0044] Furthermore, in this embodiment, since the corrected image is created taking into account the reflectance of the hood 60, the material of the hood 60 is not particularly limited, and the selectivity of the hood 60 can be improved.

[0045] (1) In this embodiment, when deriving the estimated maximum brightness EMB, the maximum brightness MB in the histogram H1 is multiplied by a correction value to derive the brightness L2a of the reflected light L2. Therefore, the brightness L2a of the reflected light L2 can be easily derived by taking into account the influence of the actual reflected light L2.

[0046] (Other embodiments) Although the present disclosure has been described based on the embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also encompasses various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more than one element, or less than one element, are also within the scope and concept of the present disclosure.

[0047] For example, in the above-described first embodiment, a vehicle is taken as an example of a moving body, but the moving body may be a train or the like.

[0048] In addition, in the first embodiment, the method for deriving the luminance L2a of the reflected light L2 can be changed as appropriate. For example, in the first embodiment, when deriving the luminance L2a of the reflected light L2, the average luminance in the histogram H1 may be derived, and the luminance L2a of the reflected light L2 may be derived by multiplying the average luminance by a predetermined correction value. In this way, the luminance L2a of the reflected light L2 is derived from the average luminance of the captured histogram H1, so that the luminance L2a of the reflected light L2 can be derived with higher accuracy. Note that the correction value here is derived in advance by performing an experiment or the like. Also, here, the average luminance corresponds to the luminance for correction.

[0049] The control unit and the method described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described in the present disclosure may be implemented by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. The computer program may also be stored in a computer-readable non-transitory tangible recording medium as instructions executed by the computer. [Explanation of symbols]

[0050] 10 Image processing device 20 Imaging unit (vehicle camera) 50 Food 101 Image acquisition unit 102 Histogram Creation Section 103 Image correction section 104 Output section

Claims

1. An image processing device, an image acquisition unit (101) mounted on a moving body, in which a hood (60) suppresses incidence of light from inside the moving body, and which acquires an image captured by an imaging unit (20) that captures an image with an exposure time based on a control signal; a histogram creation unit (102) that creates a histogram (H1) representing a relationship between luminance and the number of pixels based on the captured image; an image correction unit (103) that performs the following operations: multiplying a correction luminance derived from the histogram by a correction value to estimate a luminance (L2a) due to reflected light (L2) from the hood included in the histogram; subtracting the luminance due to the reflected light from the maximum luminance (MB) of the histogram to derive an estimated maximum luminance (EMB); subtracting the luminance due to the reflected light from the minimum luminance (LB) in the histogram to derive an estimated minimum luminance (ELB); shifting the histogram in parallel to create a corrected histogram (H2) in which the maximum luminance becomes the estimated maximum luminance and the minimum luminance becomes the estimated minimum luminance; and creating a corrected image based on the corrected histogram; an exposure time adjustment unit (104) that creates the control signal for adjusting the exposure time of the imaging unit based on the corrected histogram; an output unit (104) that outputs the corrected image and the control signal.

2. The image processing device according to claim 1 , wherein the luminance due to the reflected light is estimated by multiplying a maximum luminance in the histogram as the luminance for correction by a predetermined correction value.

3. The image processing device according to claim 1 , wherein the luminance due to the reflected light is estimated by multiplying an average luminance of the histogram serving as the correction luminance by a predetermined correction value.

Images