Image processing device and image processing method

Abstract

This image processing device detects, in first corrected image data generated by carrying out auto white balance processing on image data, a first area including a pixel wherein a first luminance, which is a luminance obtained by adding a color-difference component quantity to the luminance, is greater than a first threshold value. The image processing device corrects color information of the pixel included in the first area in accordance with the degree of correction to the color information of the pixel included in the first area calculated according to the first luminance. Thus, even if a subject area irradiated with illumination by a main light source includes a self-luminous subject, the present invention can suppress lowering of color reproducibility of the self-luminous subject by the auto white balance processing.

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] In an image processing device, a method is known in which, when performing white balance processing on an image captured under a light source that has a strong color cast effect, the light source is estimated and the color of the image is corrected using a color profile created for each type of light source (see, for example, Patent Document 1).When performing white balance processing on an image captured under illumination, if the subject image contains a specific light-emitting object, a method is known in which the saturation of the image data is suppressed based on the color temperature identified from the subject image (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-254311 [Patent Document 2] International Publication No. 2019 / 111921 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, if a subject area including a self-luminous subject such as a headlight or traffic light is photographed under the illumination of a street light such as a sodium lamp, performing auto white balance processing on the photographed image may reduce the color reproducibility of the self-luminous subject.

[0005] The present invention has been made in consideration of the above points, and aims to suppress the deterioration of color reproducibility of a self-luminous subject due to auto white balance processing, even when the self-luminous subject is included in the subject area illuminated by the main light source. [Means for solving the problem]

[0006] In one aspect of the present invention, an image processing device includes an image data acquisition unit that acquires first image data, a white balance value calculation unit that calculates a white balance value to be used in auto white balance processing according to the first image data, a first white balance correction unit that performs auto white balance processing on the first image data according to the white balance value and generates first corrected image data, a region detection unit that calculates a first luminance, which is a luminance obtained by adding a color difference component amount to luminance, in the first corrected image data and detects a first region including pixels where the first luminance is greater than a first threshold, a correction degree calculation unit that calculates a correction degree of color information of pixels included at least in the first region in the first corrected image data according to the first luminance, and a color information correction unit that corrects color information of pixels included at least in the first region in the first corrected image data according to the correction degree. [Effects of the Invention]

[0007] According to the disclosed technology, even when a self-luminous subject is included in a subject area illuminated by the main light source, it is possible to suppress deterioration in color reproducibility of the self-luminous subject due to auto white balance processing. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of an image processing system including an image processing device according to a first embodiment. [Figure 2] 2 is a block diagram showing an outline of the configuration of various devices mounted on the moving body of FIG. 1. FIG. [Figure 3] FIG. 3 is a block diagram showing an example of the configuration of the image processing device in FIG. 2. [Figure 4] 3 is a functional block diagram showing an example of a functional configuration of the image processing device of FIG. 2. FIG. [Figure 5] 3 is a flowchart showing an example of the operation of the image processing device in FIG. 2. [Figure 6] 5 is an explanatory diagram showing an example of processing by the point light source region detection unit in FIG. 4. FIG. [Figure 7]5 is an explanatory diagram showing an example of processing by a correction degree calculation unit and processing by a point light source area correction unit in FIG. 4. FIG. [Figure 8] FIG. 10 is a functional block diagram illustrating an example of a functional configuration of an image processing apparatus according to a second embodiment. [Figure 9] 9 is a flowchart showing an example of the operation of the image processing device in FIG. 8. FIG. [Figure 10] FIG. 10 is a functional block diagram illustrating an example of a functional configuration of an image processing apparatus according to a third embodiment. [Figure 11] FIG. 10 is a flowchart showing an example of the operation of the image processing device according to the fourth embodiment. [Figure 12] FIG. 12 is an explanatory diagram showing an example of the process of step S65 in FIG. [Figure 13] FIG. 12 is an explanatory diagram showing an example of image correction according to the processing flow of FIG. [Figure 14] FIG. 12 is an explanatory diagram showing another example of the process in step S65 of FIG. [Figure 15] FIG. 13 is a functional block diagram illustrating an example of a functional configuration of an image processing apparatus according to a fifth embodiment. [Figure 16] 16 is an explanatory diagram showing an example of a point light source detection process performed by the point light source region detection unit of FIG. 15. FIG. [Figure 17] 16 is a flowchart showing an example of the operation of the image processing device of FIG. 15. [Figure 18] FIG. 18 is a flow chart showing a continuation of the operation of FIG. 17. [Figure 19] FIG. 13 is an explanatory diagram illustrating an example of a correction degree calculation process performed by the image processing device according to the sixth embodiment. [Figure 20] FIG. 13 is an explanatory diagram illustrating an example of a correction degree calculation process performed by the image processing device according to the seventh embodiment. [Figure 21] FIG. 13 is a functional block diagram illustrating an example of a functional configuration of an image processing apparatus according to an eighth embodiment. [Figure 22] 22 is an explanatory diagram showing an example of processing by the point light source region detection unit in FIG. 21. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. In the following description, image data may be simply referred to as an image. Furthermore, pixel values ​​or pixel data may be simply referred to as pixels. Note that the embodiments described below are particularly effective in situations where a subject is illuminated by streetlights at night or inside a tunnel, but they can also be effective in dimly lit situations such as early morning or evening, or in bad weather such as fog.

[0010] (First embodiment) Fig. 1 shows an example of an image processing system including an image processing device according to the first embodiment. The image processing system 100 shown in Fig. 1 is mounted on a moving body 500 such as an automobile. Imaging devices 507A, 507B, 507C, 507D, and 507E such as cameras are installed at the front, rear, left, and right sides of the moving body 500 and at the front of the interior of the moving body 500. Hereinafter, when the imaging devices 507A, 507B, 507C, 507D, and 507E are not to be distinguished from one another, they are also referred to as imaging devices 507.

[0011] The number and installation positions of the imaging devices 507 installed in the mobile body 500 are not limited to those shown in FIG. 1 . For example, one imaging device 507 may be installed only at the front of the mobile body 500, or two imaging devices 507 may be installed at the front and rear. Alternatively, the imaging device 507 may be installed on the ceiling of the mobile body 500. Furthermore, the mobile body 500 on which the image processing system 100 is mounted is not limited to an automobile, and may be, for example, a transport robot or a drone operating in a factory. Furthermore, the image processing system 100 may be a system that processes images acquired from an imaging device other than the imaging device installed in the mobile body 500, such as a surveillance camera, a digital still camera, or a digital camcorder.

[0012] The image processing system 100 includes an image processing device 200, an information processing device 300, and a display device 400. For ease of understanding, FIG. 1 illustrates the image processing system 100 superimposed on an image diagram of a moving object 500 viewed from above. However, in reality, the image processing device 200 and the information processing device 300 are mounted on a control board or the like mounted on the moving object 500, and the display device 400 is installed in a position visible to people inside the moving object 500. The image processing device 200 may be mounted on the control board or the like as part of the information processing device 300. Each imaging device 507 is connected to the image processing device 200 via a signal line or wirelessly.

[0013] Fig. 2 shows an overview of the configuration of various devices mounted on the mobile object 500 of Fig. 1. The mobile object 500 has an image processing device 200, an information processing device 300, a display device 400, at least one ECU (Electronic Control Unit) 501, and a wireless communication device 502, which are interconnected via an internal network. The mobile object 500 also has a sensor 503, a drive device 504, a lamp device 505, a navigation device 506, and an imaging device 507. For example, the internal network is an in-vehicle network such as a CAN (Controller Area Network) or Ethernet (registered trademark).

[0014] The image processing device 200 corrects image data (frame data) acquired by the imaging device 507 and generates corrected image data. For example, the image processing device 200 displays the generated corrected image data on the display device 400. Note that the image processing device 200 may record the generated corrected image data in an external or internal recording device.

[0015] The information processing device 300 includes a computer such as a processor that performs recognition processing and the like based on image data received via the image processing device 200. For example, the information processing device 300 mounted on the mobile object 500 performs recognition processing on the image data to detect other mobile objects, traffic lights, signs, white lines on the road, people, and the like, and determines the situation around the mobile object 500 based on the detection results.

[0016] Moreover, the information processing device 300 controls the entire moving body 500 by controlling the ECU 501. Furthermore, the information processing device 300 may include an automatic driving control device that controls the movement, stopping, right turns, left turns, etc. of the moving body 500. In this case, the information processing device 300 may have a function of recognizing an object outside the moving body 500 based on the image generated by the image processing device 200, and may have a function of tracking the recognized object.

[0017] The display device 400 displays the image generated by the image processing device 200, the corrected image, and the like. The display device 400 is, for example, a side mirror monitor, a rearview mirror monitor, or a display of a navigation device installed in the mobile object 500. The display device 400 may be a display provided on a dashboard or the like, or a head-up display that projects an image onto a projection board, a windshield, or the like. The display device 400 may also display an image of the backward direction of the mobile object 500 in real time when the mobile object 500 is moving backward (backing up). Furthermore, the display device 400 may display an image output from a navigation device 506.

[0018] The ECUs 501 are provided corresponding to mechanical parts such as an engine or a transmission. Each ECU 501 controls the corresponding mechanical part based on instructions from the information processing device 300. The wireless communication device 502 communicates with devices external to the mobile object 500. The sensors 503 are sensors that detect various types of information. The sensors 503 may include, for example, a position sensor that acquires current position information of the mobile object 500. The sensors 503 may also include a speed sensor that detects the speed of the mobile object 500.

[0019] The driving device 504 is various devices for moving the mobile object 500. The driving device 504 may include, for example, an engine, a steering device, and a braking device. The lamp device 505 is various lighting devices mounted on the mobile object 500. The lamp device 505 may include, for example, a headlamp, a turn signal lamp, a backlight, and a brake lamp. The navigation device 506 is a device that provides audio and visual guidance on the route to a destination.

[0020] Fig. 3 shows an example of the configuration of the image processing device 200 in Fig. 2. Note that the information processing device 300 in Fig. 2 also has a configuration similar to that in Fig. 3. For example, the image processing device 200 has a CPU 20, an interface device 21, a drive device 22, an auxiliary storage device 23, and a memory device 24, which are interconnected by a bus BUS.

[0021] The CPU 20 executes various types of image processing, which will be described later, by executing an image processing program stored in the memory device 24. The interface device 21 is used to connect to a network (not shown). The auxiliary storage device 23 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores the image processing program, image data, various parameters used in image processing, and the like.

[0022] The memory device 24 is, for example, a dynamic random access memory (DRAM) or the like, and holds an image processing program and the like transferred from the auxiliary storage device 23. The drive device 22 has an interface for connecting the recording medium 30, and transfers the image processing program stored in the recording medium 30 to the auxiliary storage device 23 based on, for example, an instruction from the CPU 20. The drive device 22 may also transfer image data and the like stored in the auxiliary storage device 23 to the recording medium 30.

[0023] Fig. 4 shows an example of the functional configuration of the image processing device 200 in Fig. 2. Each functional unit of the image processing device 200 shown in Fig. 4 may be realized by an image processing program executed by the image processing device 200, or may be realized by hardware such as an FPGA (Field-Programmable Gate Array) mounted on the image processing device 200. Furthermore, each functional unit of the image processing device 200 shown in Fig. 4 may be realized by cooperation between software and hardware.

[0024] Image processing device 200 has an image data acquisition unit 210, a WB (white balance) value calculation unit 220, a WB (white balance) correction unit 230, and an image correction unit 240. Image correction unit 240 has a point light source area detection unit 250, a correction degree calculation unit 260, and a point light source area correction unit 270. Hereinafter, white balance will also be referred to as WB, and auto white balance will also be referred to as AWB.

[0025] The image data acquisition unit 210 performs image data acquisition processing to acquire image data (e.g., frame data) representing an image captured by, for example, the image capture device 507E or the image capture device 507A out of the multiple image capture devices 507 shown in Fig. 1. The image data acquired by the image data acquisition unit 210 is an example of first image data. Note that the image data acquisition unit 210 may acquire image data representing an image captured by an image capture device 507 other than the image capture devices 507E and 507A, or may acquire image data representing images captured by all of the image capture devices 507.

[0026] The WB value calculation unit 220 performs detection to identify the main light source illuminating the subject based on the image data acquired by the image data acquisition unit 210, and performs white balance value calculation processing to calculate a WB value corresponding to the identified light source. In other words, the WB value calculation unit 220 calculates the white balance value to be used in the auto white balance processing.

[0027] The WB correction unit 230 performs auto white balance processing on each pixel of the image data according to the WB value calculated by the WB value calculation unit 220, and performs first white balance correction processing to generate first-corrected image data. For example, each pixel on which auto white balance processing is performed includes one red pixel, one blue pixel, and two green pixels arranged in a Bayer array. The WB correction unit 230 is an example of a first white balance correction unit.

[0028] The point light source region detection unit 250 performs region detection processing to detect a point light source region (pixel region) included in the first corrected image data. The point light source region detection unit 250 is an example of a region detection unit, and the point light source region is an example of a first region.

[0029] Here, the point light source includes a self-luminous subject such as a vehicle headlight or a traffic light. The point light source area may include not only the area of ​​the self-luminous subject, but also an area where halation occurs due to the self-luminous subject and an area of ​​the subject that reflects light emitted from the self-luminous subject. The point light source area detection unit 250 outputs point light source area information indicating the detected point light source area to the correction degree calculation unit 260. An example of the point light source detection process by the point light source area detection unit 250 is described in FIG. 6.

[0030] Note that the point light source area detection unit 250 may output, as point light source area information, a luminance A (described later) that is a luminance including a color difference component amount and a threshold value of the luminance A used to detect the point light source area for each pixel of the first corrected image data to the correction degree calculation unit 260. The point light source area detection unit 250 may also output the luminance A for each pixel of the first corrected image data to the correction degree calculation unit 260 as point light source area information. In this case, the process of detecting an area of ​​pixels having a luminance A equal to or greater than a predetermined threshold as a point light source area is performed by the correction degree calculation unit 260, and therefore the point light source area detection function is included in the correction degree calculation unit 260. The point light source area detection unit 250 then functions as a luminance calculation unit that calculates the luminance A.

[0031] The correction degree calculation unit 260 performs a correction degree calculation process to calculate the correction degree α used to correct the color information of the pixels in the point light source area, according to the point light source area information. Note that the correction degree calculation unit 260 may calculate the correction degree α not only for the point light source area, but also for each pixel of the first corrected image data including the point light source area. An example of the calculation of the correction degree α by the correction degree calculation unit 260 will be described with reference to FIG. 7.

[0032] The point light source area correction unit 270 performs color information correction processing to correct color information of each pixel in the point light source area included in the first corrected image data using the correction degree α calculated by the correction degree calculation unit 260. The point light source area correction unit 270 is an example of a color information correction unit. A method for further correcting the first corrected image data that has undergone auto white balance correction using the correction degree α will be described with reference to FIG. 7. The point light source area correction unit 270 outputs corrected image data, in which color information of the pixels in the point light source area included in the first corrected image data, to at least one of the display device 400 and the information processing device 300.

[0033] Fig. 5 shows an example of the operation of the image processing device 200 in Fig. 2. That is, Fig. 5 shows an example of an image processing method by the image processing device 200, and an example of an image processing program executed by the image processing device 200.

[0034] First, in step S10, the image processing device 200 acquires image data captured by each imaging device 507, for example, using the image data acquisition unit 210. Next, in step S20, the image processing device 200 performs detection processing for performing auto white balance processing based on the acquired image data, and calculates a WB value, for example, using the WB value calculation unit 220. Next, in step S30, the image processing device 200 performs white balance processing on the image data in accordance with the WB value, for example, using the WB correction unit 230, and generates first corrected image data.

[0035] Next, in steps S40, S50, and S60, the image processing device 200 performs correction processing on the first corrected image data, for example, by the image correction unit 240. First, in step S40, the image processing device 200 detects a point light source region included in the first corrected image data, for example, by the point light source region detection unit 250. The point light source region may be represented by a luminance A and a threshold value.

[0036] Next, in step S50, the image processing device 200, for example, calculates a correction degree α used to correct color information of at least the pixels in the point light source region of the first corrected image data, according to the luminance A of the point light source region, using the correction degree calculation unit 260. Next, in step S60, the image processing device 200, for example, corrects color information of at least the pixels in the point light source region of the first corrected image data, using the correction degree α calculated by the correction degree calculation unit 260. Then, the point light source region correction unit 270 generates corrected image data, and the processing shown in FIG. 5 ends.

[0037] 6 shows an example of processing by the point light source area detection unit 250 of FIG. 4. First, the point light source area detection unit 250 calculates, in the color difference space CbCr, the distance ΔC from the origin of the color of each pixel of the first corrected image data that has been subjected to white balance processing. The distance ΔC indicates, for example, the color density of the pixel. Next, the point light source area detection unit 250 calculates the luminance A by adding the luminance Y of the pixel and the calculated distance ΔC using equation (1). Luminance A = Y + k ΔC (1)

[0038] The luminance A is an example of a first luminance, which is the sum of the luminance Y, which is the luminance information of a pixel expressed in the YCbCr color space, and the distance ΔC, which is an index of the color depth (vividness) of the pixel, multiplied by a coefficient k. The coefficient k is used to adjust the weighting of the distance ΔC and is a value greater than "0." The coefficient k is set in advance when the image processing device 200 is designed or when performance is evaluated.

[0039] For example, as shown in the graph of luminance Y versus distance ΔC in Fig. 6, the luminance A of a self-luminous subject with a high luminance Y and a relatively long distance ΔC is high. The luminance Y of a main light source such as a street lamp that irradiates the subject photographed by the image capture device 507 with illumination light is high. However, because the color of the main light source becomes achromatic due to auto white balance processing, the luminance A of the main light source is sufficiently smaller than the luminance A of the self-luminous subject.

[0040] Because black pixels have a small luminance Y and distance ΔC, the luminance A of black pixels is sufficiently smaller than the luminance A of a self-luminous subject. Also, the luminance A of a pixel with a large distance ΔC (dark color) but a small luminance Y is also sufficiently smaller than the luminance A of a self-luminous subject.

[0041] The point light source area detection unit 250 determines that a pixel whose luminance A is equal to or less than a predetermined threshold value VT0 is not a point light source area, and determines that a pixel whose luminance A is greater than the threshold value VT0 is a point light source area. For example, the threshold value VT0 is set to a value greater than the luminance A after auto white balance processing of a main light source such as a street lamp.

[0042] The graph in Fig. 7 shows an example of processing by correction degree calculation unit 260 and point light source area correction unit 270 in Fig. 4. Correction degree calculation unit 260 sets the correction degree α for pixels whose luminance A is less than threshold value VT0 to "0," and sets the correction degree α for pixels whose luminance A is greater than threshold value VT1 to "1." Furthermore, point light source area correction unit 270 sets the correction degree α for pixels whose luminance A is greater than or equal to threshold value VT0 and less than threshold value VT1 to be larger as the luminance A becomes higher, according to equation (2). Threshold value VT0 is an example of a first threshold value, and threshold value VT1 is an example of a second threshold value. Correction degree α = (luminance A-VT0) / (VT1-VT0) (2)

[0043] Here, the correction degree α indicates the application rate of correction of the first corrected image data in which color information is corrected by the point light source area correction unit 270. In this way, the correction degree calculation unit 260 calculates the application rate of correction of color information of the first corrected image data in accordance with the luminance A.

[0044] For pixels with a correction degree α of "0", no correction is performed on the first-corrected image data as a point light source region. For pixels with a correction degree α of "1", 100% correction is performed on the first-corrected image data as a point light source region. Similarly, for pixels with a correction degree α of "0.3", 30% correction is performed on the first-corrected image data as a point light source region, and for pixels with a correction degree α of "0.7", 70% correction is performed on the first-corrected image data as a point light source region.

[0045] Fig. 7(a) shows an example of an image captured by the imaging device 507E. Fig. 7(b) shows an example of an image based on first corrected image data that has been subjected to auto white balance processing by the WB correction unit 230. Fig. 7(c) shows an example of an image based on corrected image data that has been corrected by the point light source area correction unit 270.

[0046] In the captured image shown in Figure 7(a), illumination from a primary light source such as a sodium lamp is irradiated onto the subject area, and the image of the road and the subject on the road has the color of the illumination from the primary light source. Images of point light source areas, including self-luminous subjects such as vehicle headlights and traffic light signals and their surroundings, are not affected by illumination from the primary light source, and therefore have colors similar to those under daylight (have color reproducibility).

[0047] In the first-corrected image data on which auto white balance processing shown in Fig. 7(b) has been performed, the image in the area excluding the point light source area has colors similar to those under daylight (color reproducibility is good). On the other hand, the image in the point light source area has colors different from those under daylight due to auto white balance processing (reduction in color reproducibility). The point light source area detection unit 250 calculates the luminance A of each pixel from the first-corrected image data corresponding to the image shown in Fig. 7(b) and detects the point light source area.

[0048] In the corrected image data in which the color information of the point light source area in Figure 7(c) has been corrected, the color corrected by the auto white balance processing in the image of the point light source area (0≦α) is corrected according to the correction degree α, and the color is converted to the same color as under daylight. The image of the area other than the point light source area (α<0) reflects the results of the auto white balance processing as is, and has the same color as under daylight.

[0049] This allows the point light source area correction unit 270 to correct the color information of each pixel after auto white balance processing of the point light source area including the self-luminous subject excluding the main light source. As a result, it is possible to prevent the color reproducibility of the point light source area from being reduced by auto white balance processing. Furthermore, because the target of color information correction by the point light source area correction unit 270 does not include main light sources such as street lights, it is possible to prevent the color of the main light source corrected by auto white balance processing from becoming unnatural.

[0050] As described above, in this embodiment, the color of a self-luminous subject, whose color reproducibility is reduced by auto white balance processing, can be corrected to the color under daylight. Therefore, both an image of a road and an object on the road, which is colored by illumination from a main light source, and an image of a point light source area including the self-luminous subject and its surroundings, can be corrected to the same color under daylight. As a result, even if a self-luminous subject is included in an image that is auto white balance processed under illumination from a main light source, an image with correctly reproduced colors can be generated.

[0051] In this case, even in areas where halation occurs due to a self-luminous subject and areas of a subject that reflect light emitted from the self-luminous subject, the color of pixels whose color reproducibility has decreased due to auto white balance processing can be corrected to a color similar to that under daylight.

[0052] By calculating luminance A, which is luminance including the amount of color difference components, using point light source area detection unit 250, it is possible to make the luminance A of the main light source smaller than threshold value VT0. For example, point light source area detection unit 250 determines luminance A to be the sum of the luminance Y of a pixel expressed in the YCbCr color space and the distance ΔC (amount of color difference components) from the origin in the color difference space CbCr. As a result, it is possible to prevent the color of the pixel of the main light source, which has been corrected by white balance processing, from appearing unnatural when corrected by point light source area correction unit 270.

[0053] By calculating the correction degree α used to correct the color information of pixels in the point light source area using correction degree calculation unit 260, it is possible to prevent sudden changes in pixel color in the image at the boundary between the point light source area and the non-point light source area, thereby reducing the unnaturalness of the image at the boundary.

[0054] (Second embodiment) Fig. 8 shows an example of the functional configuration of an image processing device according to the second embodiment. Elements similar to those in Fig. 4 are given the same reference numerals, and detailed description thereof will be omitted. The image processing device 200A shown in Fig. 8 is similar to the image processing device 200 shown in Figs. 1 to 3, and is installed in the image processing system 100 together with the information processing device 300 and the display device 400.

[0055] Each functional unit of the image processing device 200A shown in FIG. 8 may be realized by software such as an image processing program executed by the image processing device 200A, may be realized by hardware, or may be realized by software and hardware working together.

[0056] Image processing device 200A has an image correction unit 240A instead of image correction unit 240 in Fig. 4. Image correction unit 240A has a point light source area correction unit 270A instead of point light source area correction unit 270 in Fig. 4. Other configurations of image processing device 200A are similar to those of image processing device 200 in Fig. 4.

[0057] Point light source area correction unit 270A has an inverse WB conversion unit 271, a WB correction unit 272, and a blend processing unit 273. Point light source area correction unit 270A is an example of a color information correction unit. Using the WB value calculated by WB value calculation unit 220, inverse WB conversion unit 271 converts the first corrected image data generated by WB correction unit 230 into the original image data before white balance processing.

[0058] The WB correction unit 272 performs white balance processing on each pixel of the original image data using, for example, the WB value of standard illuminant D65 (daylight illuminant) to generate second corrected image data. The WB correction unit 272 is an example of a second white balance correction unit. For example, the WB value of standard illuminant D65 is a fixed WB value that is set in advance. By performing white balance processing using standard illuminant D65, the colors of self-luminous subjects such as headlights and traffic light signals included in the original image data can be made to look natural in daylight.

[0059] The blending processing unit 273 uses equation (3) to blend the pixel values ​​of the first corrected image data (a) and the pixel values ​​of the second corrected image data (b) for each pixel according to the correction degree α calculated by the correction degree calculation unit 260 to generate corrected image data. a×(1-α)+b×α ‥(3)

[0060] As a result, pixels with a correction degree α of "0" (pixels with brightness A in FIG. 7 smaller than the threshold value VT0) become pixels of the first corrected image data (a), and pixels with a correction degree α of "1" (pixels with brightness A in FIG. 7 larger than the threshold value VT1) become pixels of the second corrected image data (b).

[0061] Furthermore, for pixels where the correction degree α is between "0" and "1" (pixels between thresholds VT0 and VT1), the pixel values ​​of the first corrected image data (a) and the second corrected image data (b) are blended according to the correction degree α. For example, for a pixel where the correction degree α is between "0.3", 70% of the pixel value of the first corrected image data (a) is blended with 30% of the pixel value of the second corrected image data (b). For a pixel where the correction degree α is between "0.7", 30% of the pixel value of the first corrected image data (a) is blended with 70% of the pixel value of the second corrected image data (b).

[0062] The blending unit 273 may output the first corrected image data as the corrected image data without performing blending on pixels where α=0 (brightness A<threshold value VT0). Similarly, the blending unit 273 may output the second corrected image data as the corrected image data without performing blending on pixels where α=1 (brightness A>threshold value VT1). In this case, the blending unit 273 only needs to perform blending on pixels where the correction degree α is greater than "0" and less than "1", thereby reducing the calculation load.

[0063] Fig. 9 is a flow diagram showing an example of the operation of the image processing device 200A of Fig. 8. That is, Fig. 9 shows an example of an image processing method by the image processing device 200A, and shows an example of an image processing program executed by the image processing device 200A. The same elements as in Fig. 5 are given the same step numbers, and detailed explanations will be omitted.

[0064] The processes of steps S10, S20, S30, and S50 are respectively the same as the processes of steps S10, S20, S30, and S50 in Fig. 5. In Fig. 9, steps S41 and S42 are performed instead of step S40 in Fig. 5, and steps S61, S62, and S63 are performed instead of step S60. Note that the processes of steps S61 and S62 may be performed in parallel with the processes of steps S41, S42, and S50.

[0065] After step S30, in step S41, the image processing device 200A calculates the luminance A for each pixel using, for example, the above-mentioned formula (1) by using the point light source area detection unit 250. Next, in step S42, the image processing device 200A determines whether the luminance A for each pixel is greater than a threshold value VT0. The image processing device 200A performs the process of step S50 for pixels whose luminance A is greater than the threshold value VT0, and ends the process shown in FIG. 9 for pixels whose luminance A is equal to or less than the threshold value VT0.

[0066] In step S50, the image processing device 200A calculates the correction degree α according to the luminance A of the pixel selected in step S42, for example, by the correction degree calculation unit 260. Next, in step S61, the image processing device 200A converts the first corrected image data into the original image data before white balance processing, for example, by the inverse WB conversion unit 271.

[0067] Next, in step S62, the image processing device 200A performs white balance processing on each pixel of the original image data using the WB value of standard light source D65, for example, using the WB correction unit 272, to generate second corrected image data. Next, in step S63, the image processing device 200A generates corrected image data by blending the pixel values ​​of the first corrected image data and the pixel values ​​of the second corrected image data for each pixel, for example, using the blending processing unit 273, according to the correction degree α. Then, the image processing device 200A ends the processing shown in FIG. 9.

[0068] As described above, this embodiment can also achieve the same effects as the above-described embodiments. For example, even when an image that is subjected to auto white balance processing under illumination by a main light source includes a self-luminous subject, an image that reproduces the correct colors can be generated.

[0069] Furthermore, in this embodiment, the image processing device 200A blends the first corrected image data that has been subjected to auto white balance processing and the second corrected image data that has been subjected to white balance processing using standard illuminant D65 according to the correction degree α. This makes it possible to make the colors of self-luminous subjects, such as headlights and traffic light signals, included in the original image data appear more natural. Furthermore, the blending processing unit 273 does not blend pixels where α=0 and α=1, thereby reducing the calculation load on the blending processing unit 273.

[0070] (Third embodiment) Fig. 10 shows an example of the functional configuration of an image processing device according to the third embodiment. Elements similar to those in Fig. 4 and Fig. 8 are denoted by the same reference numerals, and detailed description thereof will be omitted. The image processing device 200B shown in Fig. 10 is similar to the image processing device 200 shown in Figs. 1 to 3, and is installed in the image processing system 100 together with the information processing device 300 and the display device 400.

[0071] Each functional unit of the image processing device 200B shown in FIG. 10 may be realized by software such as an image processing program executed by the image processing device 200B, may be realized by hardware, or may be realized by software and hardware working together.

[0072] The image processing device 200B has a point light source area correction unit 270B instead of the point light source area correction unit 270A in Fig. 8. The point light source area correction unit 270B has the same configuration and function as the point light source area correction unit 270A in Fig. 8, except that the inverse WB conversion unit 271 is deleted from the point light source area correction unit 270A in Fig. 8. The point light source area correction unit 270B is an example of a color information correction unit. The other configurations of the image processing device 200B are the same as those of the image processing device 200 in Fig. 8.

[0073] In this embodiment, the WB correction unit 272 uses the WB value of the standard light source D65 to perform white balance processing on each pixel of the image data acquired by the image data acquisition unit 210 from the imaging device 507, and generates second corrected image data. Other functions and operations of the image processing device 200B are similar to those of the image processing device 200A in FIG. 8.

[0074] As described above, this embodiment can also achieve the same effects as the above-described embodiments. Furthermore, in this embodiment, by performing white balance processing using standard light source D65 using image data acquired by image data acquisition unit 210, it is possible to omit inverse WB conversion unit 271 in FIG. 8. As a result, the processing load of image processing device 200B can be reduced compared to the processing load of image processing device 200A, and other processing can be performed using the reduced processing load, thereby improving the processing performance of image processing device 200B. Furthermore, when point light source area correction unit 270B is configured as hardware, it is possible to reduce the circuit size by the circuit of inverse WB conversion unit 271.

[0075] In environments such as nighttime, the brightness of a self-luminous subject is higher than the brightness of other subjects, so white balance processing using the D65 WB value can make colors too dark, resulting in an unnatural image of the self-luminous subject after correction. In such cases, you can slightly shift the characteristics of the D65 WB value to reduce the colors.

[0076] (Fourth embodiment) FIG. 11 shows an example of the operation of the image processing device of the fourth embodiment. An image processing device 200C (not shown) that executes the flow of FIG. 11 has a configuration similar to that of the image processing device 200 of FIG. 4. The image processing device 200C is similar to the image processing device 200 shown in FIGS. 1 to 3, and is installed in the image processing system 100 together with the information processing device 300 and the display device 400. FIG. 11 shows an example of an image processing method by the image processing device 200C, and shows an example of an image processing program executed by the image processing device 200C. Elements similar to those in FIGS. 5 and 9 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0077] The processes of steps S10, S20, S30, and S50 are respectively the same as the processes of steps S10, S20, S30, and S50 in Fig. 5. The processes of steps S41 and S42 are respectively the same as the processes of steps S41 and S42 in Fig. 9. In Fig. 11, step S65 is performed instead of step S60 in Fig. 5.

[0078] After step S50, in step S65, the image processing device 200C calculates a color-corrected pixel value C' using equation (4) by using a point light source area correction unit similar to the point light source area correction unit 270 in Fig. 4. Hereinafter, the point light source area correction unit (not shown) included in the image processing device 200C will be referred to as point light source area correction unit 270C. The point light source area correction unit 270C is an example of a color information correction unit. C'=C×M(α) ‥(4)

[0079] In equation (4), symbol C indicates a pixel value of the first corrected image data, and symbol M(α) indicates a function that converts the color of a pixel using correction degree α as a variable. Correction degree α is the same as correction degree α described in FIG. 7. Function M(α) will be described in FIG. 12. Symbol C' indicates a pixel value of the corrected image data. Point light source area correction unit 270C then corrects color information of pixels in the point light source area included in the first corrected image data, and outputs the corrected image data to at least one of display device 400 and information processing device 300.

[0080] Fig. 12 shows an example of the process of step S65 in Fig. 11. The point light source area correction unit 270C of the image processing device 200C converts the first corrected image data expressed in the YCbCr color space into a conversion matrix M R to convert the image data into RGB color space.

[0081] Next, the point light source area correction unit 270C calculates the transformation matrix M R The image data in the RGB color space converted using the transformation matrix M C Convert the image data into RGB color space using the transformation matrix MC In the above formula, the variables Wr and Wb are calculated by detecting the range of pixels in the point light source area using auto white balance. For example, the variables Wr and Wb are calculated using formulas (5) and (6), respectively, based on the ratio of the R (red), G (green), and B (blue) components of the pixels adjacent to the point light source area.

[0082] Wr = (average of G (green) components of pixels in the vicinity of a pixel where α is "1") / (average of R (red) components of pixels in the vicinity of a pixel where α is "1") (5) Wb = (average of G (green) components of pixels in the vicinity of a pixel where α is "1") / (average of B (blue) components of pixels in the vicinity of a pixel where α is "1") (6)

[0083] Furthermore, the point light source area correction unit 270C uses the transformation matrix M C The image data in the RGB color space is converted using the transformation matrix M Y To summarize the above, the function M(α) that converts pixel values ​​of the first corrected image data into pixel values ​​of the corrected image data is expressed by equation (7). M(α)=M Y M C (α)M R ...(7)

[0084] For example, when the correction degree α is 1.0, in the color difference space CbCr of the first corrected image data, the color of each pixel of the first corrected image data (indicated by the distance ΔC) is converted to an achromatic color, which is the origin of the color difference space CbCr, by the function M(α).The closer the correction degree α of a pixel is to 1.0, the closer the color of the pixel is to an achromatic color.

[0085] Therefore, even when a subject area illuminated by street lights such as sodium lamps includes a self-luminous subject such as a headlight, and the self-luminous subject (point light source area) is subjected to auto white balance processing, the color of the self-luminous subject can be converted to an achromatic color. As a result, it is possible to prevent the color reproducibility of the point light source area from being reduced by the auto white balance processing.

[0086] 12, the point light source area correction unit 270C sets pixels with a correction degree α of 1.0 (i.e., 100%) to an achromatic color. However, the point light source area correction unit 270C may, for example, set pixels with a correction degree α of 1.0 to a color close to an achromatic color. Alternatively, the point light source area correction unit 270C may, for example, set pixels with a correction degree α of 0.9 or more to an achromatic color.

[0087] FIG. 13 shows an example of image correction using the processing flow of FIG. 11. Detailed descriptions of elements and processes similar to those in FIG. 7 will be omitted. FIG. 13(a) shows an example of an image captured by the imaging device 507E. FIG. 13(b) shows an example of an image based on first corrected image data on which auto white balance processing has been performed by a WB correction unit (not shown) included in the image processing device 200C. The WB correction unit of the image processing device 200C has the same function as the WB correction unit 230 of FIG. 4.

[0088] FIG. 13(c) shows an example of an image based on image data corrected by the point light source area correcting unit 270C of the image processing device 200C.

[0089] Fig. 14 shows another example of the process of step S65 in Fig. 11. For example, instead of performing the color correction conversion in the RBG color space, the color correction conversion may be performed in a color space in which colors are easier to correct than in the RGB color space, such as the HSV color space.

[0090] As described above, this embodiment can also achieve the same effects as the above-described embodiments. For example, as in Fig. 7, it is possible to correct the color information of each pixel in the point light source area including the self-luminous subject such as the headlight, excluding the main light source. As a result, it is possible to prevent the color reproducibility of the headlight and the point light source area around it from being reduced by the auto white balance processing.

[0091] (Fifth embodiment) Fig. 15 shows an example of the functional configuration of an image processing device according to the fifth embodiment. Elements similar to those in Fig. 8 are given the same reference numerals, and detailed description thereof will be omitted. The image processing device 200D shown in Fig. 15 is similar to the image processing device 200 shown in Figs. 1 to 3, and is installed in the image processing system 100 together with the information processing device 300 and the display device 400.

[0092] Each functional unit of the image processing device 200D shown in FIG. 15 may be realized by software such as an image processing program executed by the image processing device 200D, by hardware, or by a combination of software and hardware.

[0093] Image processing device 200D has an image correction unit 240D instead of image correction unit 240A in Fig. 8. Image correction unit 240D has a point light source area detection unit 250D and a correction degree calculation unit 260D instead of point light source area detection unit 250 and correction degree calculation unit 260 in Fig. 8, and also newly has a point light source area correction unit 280D and an image selection unit 290D. The other configuration of image processing device 200D is similar to that of image processing device 200A in Fig. 8. Note that image processing device 200D may have point light source area detection unit 250B in Fig. 10 instead of point light source area correction unit 270A.

[0094] The point light source area detection unit 250D has a function to detect, for example, high-brightness point light source areas such as preset traffic light lights, and a function to detect ultra-high-brightness point light source areas such as vehicle headlights, in the first corrected image data. For example, the point light source area detection unit 250D calculates a luminance A (first luminance) for each of a plurality of types of light sources, which is a luminance obtained by adding a color difference component amount to the luminance, and detects a first area including pixels where the luminance A (first luminance) is greater than a first threshold. The point light source area detection unit 250D is an example of an area detection unit. An example of the point light source area detection process by the point light source area detection unit 250D is described in FIG. 16.

[0095] The correction degree calculation unit 260D calculates a first correction degree α1 used to correct color information of pixels in a preset high-luminance point light source area (for example, traffic light, etc.). The correction degree calculation unit 260D also calculates a second correction degree α2 used to correct color information of pixels other than the preset high-luminance point light source area. The preset high-luminance pixels, such as traffic light, are an example of specific pixels containing specific color information. The pixels other than the specific pixels are an example of normal pixels.

[0096] Point light source area correction unit 270A uses a first correction degree α1 instead of the correction degree α in Fig. 8 to blend pixel values ​​of the first corrected image data (a) and pixel values ​​of the second corrected image data (b) for each pixel to generate provisional first corrected image data. Point light source area correction unit 280D corrects pixel values ​​of the first corrected image data according to the second correction degree α2 using the technique shown in Fig. 12 to generate provisional second corrected image data. Point light source area correction unit 270A is an example of a first color information correction unit, and point light source area correction unit 280D is an example of a second color information correction unit.

[0097] Image selection unit 290D selects, for each pixel, either the first provisional corrected image data generated by point light source area correction unit 270A or the second provisional corrected data generated by point light source area correction unit 280D, and outputs it as corrected image data. For example, image selection unit 290D selects the first provisional corrected image data for pixels to be corrected with the first correction degree α1, and selects the second provisional corrected image data for pixels to be corrected with the second correction degree α2.

[0098] Fig. 16 shows an example of the detection process of a point light source by the point light source area detection unit 250D of Fig. 15. For example, the image processing device 200D of Fig. 15 sets the light source (self-luminous subject) to be corrected, for further correcting the color information of pixels that have undergone auto white balance correction processing, to at least one of the green light, red light, and yellow light of a traffic light and the headlight of a vehicle. Then, the image processing device 200D changes the white balance processing for each of the light source that is assumed to be white and the light source that is assumed to be colored.

[0099] The point light source region detection unit 250D limits the range of the argument θ, the distance r from the center (color intensity), and the range of luminance in the polar coordinates (r, θ) of the color difference space CbCr, and regards the limited range as the light source, where Cb=rsinθ and Cb=rcosθ.

[0100] For example, the point light source area detection unit 250D determines that an area where the distance r is greater than a threshold th(Y) is a high saturation area, and determines that pixels included in the high saturation area are specific pixels suitable for white balance processing (D65WB) using the standard light source D65. On the other hand, the point light source area detection unit 250D determines that an area where the distance r is equal to or less than the threshold th(Y) is a low saturation area, and determines that pixels included in the low saturation area are normal pixels suitable for auto white balance (AWB) processing. The threshold th(Y) is a first threshold that changes depending on the luminance, and becomes smaller as the luminance increases. The threshold th(Y) is an example of a first threshold.

[0101] Furthermore, in the example of detecting a high-saturation light source area shown in FIG. 16, the point light source area detection unit 250D detects pixels that are within a range of a predetermined deflection angle θ and are included in an area of ​​a color whose distance r is equal to or greater than a predetermined value as, for example, a red light of a traffic light.

[0102] For example, the correction process for color information of pixels including red lights of a traffic light and pixels other than the red lights is as follows. "if(θ>θRedMin && θ<θRedMax){ if(r>c*th(Y)) {Apply D65WB or apply WB to a specific traffic light color} else{AWB applied} } else { if(r>th(Y)){D65WB applied} else{AWB applied} }"

[0103] Here, the range from θRedMin to θRedMax indicates the range of red light under red light. The variable c is a value smaller than 1.0. Note that by changing the range of the argument θ and the distance r, correction processing for blue light or yellow light can also be applied. Furthermore, correction processing for red light, blue light, and yellow light may be performed sequentially.

[0104] Figures 17 and 18 show an example of the operation of the image processing device 200D in Figure 15. That is, Figures 17 and 18 show an example of an image processing method by the image processing device 200D, and show an example of an image processing program executed by the image processing device 200. Elements similar to those in Figure 5 are given the same step numbers, and detailed descriptions thereof will be omitted.

[0105] The processes of steps S10, S20, S30, S41, S42, S62, and S63 are similar to the processes of steps S10, S20, S30, S41, S42, S62, and S63, respectively, in Fig. 9. The process of step S65 is similar to the process of step S65 in Fig. 11.

[0106] In step S42, if the brightness A is greater than the threshold value VT0, the image processing device 200D determines that the area is a point light source area, and performs step S43 in Fig. 18. If the brightness A is equal to or less than the threshold value VT0, the image processing device 200D determines that the area is a main light source area illuminated by street lamps, and performs step S70. In step S70, the image processing device 200D performs auto white balance processing on the main light source area, which is the entire image data acquired by the image data acquisition unit 210, and ends the processing shown in Figs. 17 and 18.

[0107] 18, the image processing device 200D, for example, uses the point light source area detection unit 250D to determine the polar coordinates (r, θ) in the color difference space CbCr of each pixel of the first corrected image data. Next, in step S44, the image processing device 200D, for example, uses the point light source area detection unit 250D to determine for each pixel whether the distance r is greater than a threshold th(Y). Pixels whose distance r is greater than the threshold th(Y) are treated as specific pixels and are therefore suitable for white balance processing using the standard light source D65 (D65WB), and therefore the process of step S51 is carried out (first condition). Pixels whose distance r is equal to or less than the threshold th(Y) are treated as normal pixels and are therefore suitable for auto white balance (AWB), and therefore the process of step S52 is carried out (second condition).

[0108] In step S51, the image processing device 200D, for example, by the correction degree calculation unit 260D, calculates a first correction degree α1 for each pixel applicable to the D65WB processing, and proceeds to step S62. In step S52, the image processing device 200D, for example, by the correction degree calculation unit 260D, calculates a second correction degree α2 for each pixel applicable to the AWB processing, and proceeds to step S64.

[0109] In steps S62 and S63, the image processing device 200D executes the same processes as steps S62 and S63 in FIG. 9 by, for example, the point light source area correction unit 270A, and ends the processes shown in FIGS.

[0110] In step S64, the image processing device 200D, for example, causes the point light source area correction unit 280D to perform detection processing for implementing auto white balance processing, and calculates the WB value (weighted with the second correction degree α2).

[0111] Next, in step S65, the image processing device 200D performs processing similar to step S65 in Figure 11, for example, by using the point light source area correction unit 280D, calculates the pixel value C' after color correction using the above-mentioned equation (4), and terminates the processing shown in Figures 17 and 18.

[0112] As described above, this embodiment also achieves the same effects as the above-described embodiments. Furthermore, in this embodiment, by setting the detection method for multiple types of point light sources according to the characteristics of the color information of each point light source, each of the multiple types of point light sources can be detected with high accuracy. This allows the colors of the pixels in the point light source area corresponding to each point light source to be reproduced on the image with high accuracy.

[0113] Note that the image processing device 200D can also correct color information of pixels in a detected point light source area for self-luminous subjects other than traffic light lamps and headlights by detecting them as point light source areas using the same method as in Fig. 16. This makes it possible to prevent the color reproducibility of point light source areas corresponding to each of a plurality of types of point light sources from being reduced by auto white balance processing.

[0114] (Sixth embodiment) Fig. 19 shows an example of correction degree calculation processing by the image processing device of the sixth embodiment. An image processing device 200E (not shown) that performs the processing of Fig. 19 has a configuration similar to that of the image processing device 200A of Fig. 8. For example, the image processing device 200E has the image data acquisition unit 210, WB value calculation unit 220, WB correction unit 230, point light source area detection unit 250, and point light source area correction unit 270A of Fig. 8. However, the function of the correction degree calculation unit (not shown) included in the image processing device 200E differs from the function of the correction degree calculation unit 260 of Fig. 8.

[0115] Hereinafter, the correction degree calculation unit of the image processing device 200E will be referred to as a correction degree calculation unit 260E. The image processing device 200E is similar to the image processing device 200 shown in Figures 1 to 3, and is installed in the image processing system 100 together with the information processing device 300 and the display device 400.

[0116] The correction degree calculation unit 260E in this embodiment sets the correction degree α of pixels that are within a distance D0 from the light source to "1." Furthermore, the correction degree calculation unit 260E sets the correction degree α of pixels that are at a distance from the light source that is greater than or equal to D0 and less than D1 so that the correction degree α approaches "0" as the distance increases. Furthermore, the correction degree calculation unit 260E sets the correction degree α of pixels that are farther away from the light source than D1 to "0."

[0117] The pixel region where the distance from the light source is equal to or greater than D0 and equal to or less than D1 is a blending region where D65WB processing and AWB processing are blended according to the correction degree α. Note that, for example, the distance from the light source may be the distance from the center of the light source if the light source is small on the image, or may be the distance from the outer periphery of the light source that is closest to the pixel whose distance is being measured if the light source is large on the image.

[0118] As shown in Fig. 19, the blending region may be rectangular. For example, the dashed-dotted line frame indicates that the distance from the light source is D0, and the dashed-line frame surrounding the dashed-dotted line frame indicates that the distance from the light source is D1. The region inside the dashed-dotted line frame is an example of a first region. The region between the dashed-dotted line frame and the dashed-line frame is an example of a second region within a predetermined range outside the first region.

[0119] When the blending region is rectangular, strictly speaking, the correction degree α is set not in accordance with distance but in accordance with pixels on a frame of a predetermined shape that gradually expands outward from the light source. The shape of the frame may be a circle, ellipse, polygon, or curve, or may be a shape similar to the outline of the light source on the image. In this case, too, the correction degree α is set to "1" near the light source and gradually decreases with increasing distance from the light source, finally being set to "0." Furthermore, as shown in FIG. 19, the correction degree α is set for each light source.

[0120] Then, the image processing device 200E generates corrected image data by blending the first corrected image data and the second corrected image data for each pixel according to the correction degree α set by the correction degree calculation unit 260E. For example, the correction degree α is a blending ratio of pixel values ​​of the first corrected image data and the second corrected image data.

[0121] Here, the first corrected image data is generated by AWB processing, and the second corrected image data is generated by D65WB processing. For example, the blending process for each pixel of the first corrected image data and the second corrected image data is performed by a point light source area detection unit (not shown) provided in the image processing device 200E and corresponding to the point light source area detection unit 250A in FIG. 8.

[0122] As described above, this embodiment also achieves the same effects as the above-described embodiments. Furthermore, in this embodiment, by setting the correction degree α according to the distance from the light source, it is possible to correct the color tone of a self-luminous subject, the color reproducibility of which would be reduced by auto white balance processing, to the color tone under daylight.

[0123] The correction degree calculation unit 260E of this embodiment may be used in place of the correction degree calculation unit 260 of Figures 8, 10, and Figure 21 described later, and may be used in the calculation method of the correction degree α of the correction degree calculation unit 260D of Figure 15.

[0124] (Seventh embodiment) FIG. 20 shows an example of correction degree calculation processing by the image processing device of the seventh embodiment. An image processing device 200F (not shown) that performs the processing of FIG. 20 has a configuration similar to that of the image processing device 200 of FIG. 4. However, the functions of a point light source area detection unit and a correction degree calculation unit (not shown) of the image processing device 200F differ from those of the point light source area detection unit 250 and the correction degree calculation unit 260 of FIG. 4. Hereinafter, the point light source area detection unit of the image processing device 200F will be referred to as the point light source area detection unit 250F, and the correction degree calculation unit of the image processing device 200F will be referred to as the correction degree calculation unit 260F. The image processing device 200F is similar to the image processing device 200 shown in FIGS. 1 to 3 and is installed in the image processing system 100 together with the information processing device 300 and the display device 400.

[0125] The point light source region detection unit 250F outputs information indicating pixels in the point light source region where the luminance A is greater than the threshold value VT0 to the correction degree calculation unit 260F as point light source region information.

[0126] In Method 1, when an image represented by the first corrected image data is divided into a plurality of blocks, the correction degree calculation unit 260F calculates the correction degree according to the number of pixels in the point light source area included in each block. For example, as shown in Method 1 in Fig. 20, the correction degree calculation unit 260F sets the correction degree α for each block from "0" (0%) to "1.0" (100%) according to the number of pixels whose luminance A is equal to or greater than the threshold value VT0.

[0127] The correction degree calculation unit 260F brings the correction degree α closer to "1.0" as the number of pixels whose brightness A is equal to or greater than the threshold value VT0 increases, and fixes the correction degree α to "1.0" when the number of pixels whose brightness A is equal to or greater than the threshold value VT0 is equal to or greater than a first predetermined number. Furthermore, the correction degree calculation unit 260F fixes the correction degree α to "0" when the number of pixels whose brightness A is less than the threshold value VT0 is equal to or less than a second predetermined number that is smaller than the first predetermined number.

[0128] In method 2, correction degree calculation unit 260F sets, for example, pixels in a point light source area to a first pixel value (e.g., "1.0") and pixels not in a point light source area to a second pixel value (e.g., "0") in the image represented by the first corrected image data. Correction degree calculation unit 260F applies a Gaussian filter to each pixel of the image represented by the first corrected image data as the first pixel value or the second pixel value, and performs image smoothing processing. Then, correction degree calculation unit 260F calculates pixel value luminance A of each pixel of the smoothed image, and calculates correction degree α according to the calculated luminance A.

[0129] In Method 3, correction degree calculation unit 260F sets each pixel of the image represented by the first corrected image data to a first pixel value or a second pixel value, as in Method 2. Then, correction degree calculation unit 260F applies a guided filter, performs edge-preserving smoothing processing on the image, calculates luminance A of each pixel of the smoothed image, and calculates correction degree α according to the calculated luminance A.

[0130] As described above, this embodiment can also achieve the same effects as the above-described embodiments. Furthermore, in this embodiment, even if the first corrected image data includes an area where pixels from point light source areas and pixels from non-point light source areas are mixed, any of methods 1 to 3 can be used to smooth the change in correction degree α at the boundary between the point light source area and the non-point light source area. This makes it possible to prevent unnatural images, such as mottled colors in the image of the corrected image data in which the color information of the pixels is corrected using the correction degree α.

[0131] On the other hand, when pixels from point light source areas and non-point light source areas are mixed, as in the image after point light source area detection processing shown in Fig. 20, if the correction degree α is calculated as is, problems such as dot marks being visible in the image after color information correction may occur. Also, the color around the point light source area may be changed from the original color due to illumination light from the light source. Therefore, when pixels from point light source areas and non-point light source areas are mixed, if the correction degree α is calculated as is, the image after color information correction may appear unnatural.

[0132] In Method 1, an example is shown in which the image represented by the first corrected image data is divided into nine blocks, but the number of blocks is not limited to nine. The more blocks there are, the higher the correction accuracy becomes, but the greater the calculation load becomes. The fewer blocks there are, the lower the correction accuracy becomes, but the smaller the calculation load becomes. In addition, the filters applied to Methods 2 and 3 are not limited to Gaussian filters or guided filters.

[0133] (Eighth embodiment) Fig. 21 shows an example of the functional configuration of an image processing device according to the eighth embodiment. Elements similar to those in Fig. 8 are given the same reference numerals, and detailed description thereof will be omitted. The image processing device 200G shown in Fig. 21 is similar to the image processing device 200 shown in Figs. 1 to 3, and is installed in the image processing system 100 together with the information processing device 300 and the display device 400.

[0134] Each functional unit of the image processing device 200G shown in FIG. 21 may be realized by software such as an image processing program executed by the image processing device 200G, may be realized by hardware, or may be realized by software and hardware working together.

[0135] Image processing device 200G has image correction unit 240G instead of image correction unit 240A of Fig. 8, and does not have WB correction unit 230 of Fig. 8. Image correction unit 240G has point light source area detection unit 250G, correction degree calculation unit 260G, and point light source area correction unit 270G instead of point light source area detection unit 250, correction degree calculation unit 260, and point light source area correction unit 270A of Fig. 8.

[0136] The point light source area correction unit 270G has a WB correction unit 275 similar to the WB correction unit 230 of FIG. 8, but does not have the inverse WB conversion unit 271 of FIG. 8. The point light source area correction unit 270G is an example of a color information correction unit. Other configurations of the image processing device 200G are similar to those of the image processing device 200A of FIG. 8. Because the point light source area correction unit 270G does not have the inverse WB conversion unit 271, it is possible to suppress deterioration in image quality that occurs when, for example, the first corrected image data generated by the WB correction unit 230 of FIG. 8 is converted into the original image data by the inverse WB conversion unit 271.

[0137] The point light source area detection unit 250G performs a point light source area detection process according to the image data (input image) acquired by the image data acquisition unit 210 and the WB value calculated by the WB value calculation unit 220, and outputs the point light source area information to the correction degree calculation unit 260G.

[0138] In this embodiment, the point light source area detection unit 250G detects a point light source area using image data before auto white balance processing is performed, rather than first-corrected image data on which auto white balance processing is performed. Therefore, it is not possible to detect a point light source area using the distance ΔC from the origin in the color difference space CbCr in the YCbCr color space described in FIG. 6. This is because if image data before auto white balance processing is used, the main light source is detected as a point light source area. An example of the point light source area detection process performed by the point light source area detection unit 250G is described in FIG. 22. The point light source area detection unit 250G is an example of an area detection unit.

[0139] Correction degree calculation unit 260G sets the correction degree α of the point light source region to "1.0", and sets the correction degree α of regions other than the point light source region to "0". Note that correction degree calculation unit 260G may gradually change the correction degree α from "0" to "1.0" from the inside to the outside at the boundary portion of the point light source region (the outer periphery of the cylindrical region in FIG. 22).

[0140] Similar to WB correction section 230 in Fig. 8, WB correction section 275 performs white balance processing on each pixel of image data according to the WB value calculated by WB value calculation section 220, and generates first corrected image data. The functions of WB correction section 272 and blending processing section 273 are similar to those of WB correction section 272 and blending processing section 273 in Fig. 8. Therefore, similar to point light source area correction section 270A in Fig. 8, point light source area correction section 270G can generate corrected image data by blending the first corrected image data and the second corrected image data for each pixel according to correction degree α.

[0141] Fig. 22 shows an example of processing by point light source area detection unit 250G of Fig. 21. Point light source area detection unit 250G performs XYZ conversion on an input image expressed in RGB color space to obtain coordinates of the input image expressed in XYZ color space. Furthermore, point light source area detection unit 250G performs xyY conversion on the coordinates of the input image expressed in XYZ color space to obtain coordinates of the input image expressed in xyY color space.

[0142] Point light source area detection unit 250G also performs XYZ conversion on the WB value of the main light source, such as a street lamp, calculated by WB value calculation unit 220, to determine the coordinates of the WB value of the main light source expressed in the XYZ color space. Point light source area detection unit 250G then performs xyY conversion on the coordinates of the WB value of the main light source expressed in the XYZ color space, to determine reference coordinates, which are the coordinates of the WB value of the main light source expressed in the xyY color space.

[0143] The coefficients used in the XYZ conversion shown in Fig. 22 are calculated in advance according to the characteristics of the image capture device 507. The x, y, and Y values ​​obtained by the xyY conversion are calculated using the formulas shown in Fig. 22. Here, the color of a pixel is represented by the x and y values, and the brightness of the pixel is represented by brightness Y, which corresponds to luminance.

[0144] The point light source area detection unit 250G performs a process of detecting a point light source area included in the first-corrected image data using the coordinates of the input image converted into the xyY color space and the reference coordinates of the WB value of the main light source. First, the point light source area detection unit 250G determines, in the xyY color space, the color within a range of a first distance ΔC from the color (x, y) at the reference coordinates of the WB value of the main light source, such as a street lamp, as the color of the subject illuminated by the illumination light from the main light source. The color (x, y) is represented by the chromaticity coordinates (x, y) in the xyY color space.

[0145] Next, point light source region detection unit 250G detects, as a point light source region, a first region which is a pixel region having a pixel value (x, y, Y) outside the range of the first distance ΔC and a brightness Y that is greater than or equal to a first threshold thL relative to the brightness Y at the reference coordinates of the WB value of the main light source. In the example shown in Fig. 22, point light source region detection unit 250G detects, as a point light source region, a region (first region) outside in the cross-sectional direction of a cylindrical region whose brightness Y is greater than or equal to the threshold thL and whose color (x, y) is indicated within the range of the first distance ΔC.

[0146] Then, point light source region detection unit 250G outputs point light source region information indicating the determined point light source region to correction degree calculation unit 260G in Fig. 21. As described above, correction degree calculation unit 260G sets correction degree α to, for example, "1.0" or "0" depending on whether or not the region is a point light source region.

[0147] After this, the point light source area correction unit 270G in FIG. 21, similar to the point light source area correction unit 270A in FIG. 8, blends the first corrected image data and the second corrected image data for each pixel according to the correction degree α to generate corrected image data.

[0148] As described above, this embodiment can also achieve the same effects as the above-described embodiments. Furthermore, in this embodiment, the point light source area correction unit 270G does not have the inverse WB conversion unit 271, and therefore the processing load on the point light source area correction unit 270G can be reduced compared to when the inverse WB conversion unit 271 is included. Furthermore, since the point light source area correction unit 270G does not have the inverse WB conversion unit 271, it is possible to suppress deterioration in image quality when, for example, the first corrected image data generated by the WB correction unit 230 in FIG. 8 is converted into the original image data by the inverse WB conversion unit 271.

[0149] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These requirements can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0150] 21 Interface Device 22 Drive device 23 Auxiliary storage device 24 Memory Device 30 Recording media 100 Image Processing System 200, 200A, 200B, 200C Image Processing Device 200D, 200E, 200F, 200G Image Processing Device 210 Image data acquisition unit 220 WB value calculation unit 230 WB correction section 240, 240A, 240B, 240D Image correction unit 250, 250A, 250B, 250D Point light source area detection unit 250E, 250F, 250G Point light source area detection unit 260, 260D, 260E, 260F, 260G Correction degree calculation section 270, 270A, 270B, 270C, 270E, 270G Point light source area correction section 271 Inverse WB conversion unit 272 WB correction section 273 Blending Processing Unit 275 WB correction section 280D point light source area correction section 290D Image selection section 300 Information processing device 400 display device 500 Mobile 502 Wireless communication device 503 Sensors 504 Drive Unit 505 Lamp unit 506 Navigation equipment 507 (507A, 507B, 507C, 507D, 507E) Imaging device

Claims

1. an image data acquisition unit that acquires first image data; a white balance value calculation unit that calculates a white balance value to be used in auto white balance processing in accordance with the first image data; a first white balance correction unit that performs auto white balance processing on the first image data in accordance with the white balance value and generates first corrected image data; a region detection unit that calculates a first luminance, which is a luminance obtained by adding a color difference component amount to a luminance, in the first corrected image data, and detects a first region including pixels whose first luminance is greater than a first threshold; a correction degree calculation unit that calculates a correction degree of color information of at least pixels included in the first region in the first corrected image data according to the first luminance; a color information correction unit that corrects color information of pixels included in at least the first region in the first corrected image data according to the correction degree; An image processing device having:

2. The area detection unit The first luminance is calculated by adding, for at least one pixel included in the first corrected image data, a luminance in a color space including luminance information and a color difference component amount in the color space. The image processing device according to claim 1 .

3. The correction degree calculation unit sets the correction degree of a pixel where the first luminance is equal to or greater than the first threshold and equal to or less than a second threshold that is greater than the first threshold, from 0% to 100% as the first luminance increases, and sets the correction degree of a pixel where the first luminance is greater than the second threshold to 100%.

3. The image processing device according to claim 1.

4. the area detection unit outputs information indicating pixels of the first area to the correction degree calculation unit; The correction degree calculation unit calculates a correction degree in accordance with the number of pixels of the first region included in each block when an image represented by the first corrected image data is divided into a plurality of blocks.

3. The image processing device according to claim 1.

5. the area detection unit outputs information indicating pixels of the first area to the correction degree calculation unit; The correction degree calculation unit calculates the correction degree by performing a smoothing process on the image represented by the first corrected image data, with pixels in the first region set to a first pixel value and pixels outside the first region set to a second pixel value.

3. The image processing device according to claim 1.

6. The color information correction unit a second white balance correction unit that performs white balance processing on the first image data using a white balance value of a standard light source to generate second corrected image data; a blending processing unit that blends pixel values ​​of the first corrected image data and the second corrected image data based on the correction degree; 6. The image processing device according to claim 1, further comprising:

7. The color information correcting unit adjusts the color of a pixel closer to an achromatic color as the degree of correction of the pixel increases. The image processing device according to claim 3 .

8. an image data acquisition unit that acquires first image data; a white balance value calculation unit that calculates a white balance value to be used in auto white balance processing in accordance with the first image data; a first white balance correction unit that performs auto white balance processing on the first image data in accordance with the white balance value and generates first corrected image data; a region detection unit that calculates a first luminance, which is a luminance obtained by adding a color difference component amount to a luminance, for each of a plurality of types of light source in the first corrected image data, and detects a first region including pixels whose first luminance is greater than a first threshold; a correction degree calculation unit that calculates a first correction degree for a specific pixel including specific color information in the first region and calculates a second correction degree for a normal pixel other than the specific pixel; a first color information correction unit that corrects color information of the specific pixel included at least in the first region according to the first correction degree; a second color information correction unit that corrects color information of the normal pixels included in at least the first region in accordance with the second correction degree; An image processing device having:

9. The correction degree calculation unit sets, in the first region, pixels whose distance r in polar coordinates (r, θ) in the color difference space CbCr is greater than a first threshold as the specific pixels, and sets pixels whose distance r is equal to or less than the first threshold as the normal pixels. The image processing device according to claim 8 .

10. The first color information correction unit a second white balance correction unit that performs white balance processing on the first image data using a white balance value of a standard light source to generate second corrected image data; a blending processing unit that blends pixel values ​​of the first corrected image data and the second corrected image data based on the first correction degree, The second color information correcting unit adjusts the color of a pixel closer to an achromatic color as the degree of second correction of the pixel increases.

10. The image processing device according to claim 8 or claim 9.

11. The correction degree calculation unit sets a pixel including color information of a red light of a traffic light as the specific pixel.

11. The image processing device according to claim 8.

12. an image data acquisition unit that acquires first image data; a white balance value calculation unit that calculates a white balance value to be used in auto white balance processing in accordance with the first image data; a first white balance correction unit that performs auto white balance processing on the first image data in accordance with the white balance value and generates first corrected image data; a region detection unit that calculates a first luminance, which is a luminance obtained by adding a color difference component amount to a luminance, in the first corrected image data, and detects a first region including pixels whose first luminance is greater than a first threshold; a correction degree calculation unit that sets correction degrees for pixels included in the first region, pixels included in a second region that is a predetermined range outside the first region, and pixels in a region outside the second region so that the correction degrees for the first region are large and the correction degrees for the region outside the second region are small; a color information correction unit that corrects color information of pixels included in at least the first region in the first corrected image data according to the correction degree; An image processing device having:

13. The correction degree calculation unit sets the correction degree of pixels included in the first region to 100%, sets the correction degree of pixels included in a second region within a predetermined range outside the first region to 0% from 100% as the distance from the first region increases, and sets the correction degree of pixels outside the second region to 0%. The image processing device according to claim 12.

14. The color information correction unit a second white balance correction unit that performs white balance processing on the first image data using a white balance value of a standard light source to generate second corrected image data, setting pixel values ​​of pixels included in the first region to pixel values ​​of the second corrected image data; setting pixel values ​​of pixels included in the second region by blending pixel values ​​of the first corrected image data and the second corrected image data based on the correction degree; setting pixel values ​​of pixels included outside the second region to pixel values ​​of the first corrected image data; The correction degree is a blending ratio of pixel values ​​of the first corrected image data and pixel values ​​of the second corrected image data. The image processing device according to claim 12 or 13.

15. an image data acquisition unit that acquires first image data; a white balance value calculation unit that calculates a white balance value to be used in auto white balance processing in accordance with the first image data; a first white balance correction unit that performs auto white balance processing on the first image data in accordance with the white balance value and generates first corrected image data; an area detection unit that calculates reference coordinates, which are coordinates in an xyY color space of the white balance value, and detects, in the first image data represented in the xyY color space, a first area including a pixel whose lightness Y is greater than the lightness Y of the reference coordinates by a first threshold value or more and whose chromaticity coordinates (x, y) are away from the chromaticity coordinates (x, y) of the reference coordinates by a first distance or more; a correction degree calculation unit that sets a correction degree of color information of pixels included in the first region in the first image data to be greater than a correction degree of color information of pixels not included in the first region; a color information correction unit that corrects color information of pixels included in at least the first region in the first corrected image data according to the correction degree; An image processing device having:

16. The color information correction unit a second white balance correction unit that performs white balance processing on the first image data using a white balance value of a standard light source to generate second corrected image data; a blending processing unit that blends pixel values ​​of the first corrected image data and the second corrected image data based on the correction degree; The image processing device according to claim 15, further comprising:

17. an image data acquisition process for acquiring first image data; a white balance value calculation process for calculating a white balance value to be used in auto white balance processing in accordance with the first image data; a first white balance correction process for performing auto white balance processing on the first image data in accordance with the white balance value to generate first corrected image data; a region detection process for calculating a first luminance, which is a luminance obtained by adding a color difference component amount to a luminance, in the first corrected image data, and detecting a first region including pixels whose first luminance is greater than a first threshold value; a correction degree calculation process for calculating a correction degree of color information of at least pixels included in the first region in the first corrected image data according to the first luminance; a color information correction process for correcting color information of pixels included in at least the first region in the first corrected image data according to the correction degree; An image processing method that performs the above.

Images