Image processing device and image processing program

Abstract

To improve night scene determination accuracy.SOLUTION: An image processing device of the present invention has a controller. The controller configured to: extract a traffic light region where a traffic light exists, from a camera image captured by an in-vehicle camera; determine a luminous region and a non-luminous region within the traffic light region; and determine the camera image as a night scene when brightness and chroma of the luminous region and the non-luminous region correspond to nighttime brightness and chroma characteristics.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing program. [Background technology]

[0002] There is a conventional technique for determining the shooting environment from image data captured by an in-vehicle camera. Patent Document 1 discloses a technique for determining a nighttime scene based on a histogram of the brightness values ​​of a sky region and a ground region set in the image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-165743 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, when street lights are present in the surrounding area at night, the surrounding light sources may become noise depending on the conditions of the surrounding light sources, which may reduce the accuracy of nighttime scene determination. Therefore, the conventional technology has room for improvement in improving the accuracy of nighttime scene determination.

[0005] The present invention has been made in view of the above, and has an object to provide an image processing device and an image processing program that can improve the accuracy of determining nighttime scenes. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objectives, the image processing device of the present invention has a controller, which extracts a signal area where a traffic light is present from a camera image captured by an on-board camera, determines luminous areas and non-luminous areas in the signal area, and determines that it is a nighttime scene if the brightness and saturation of each of the luminous and non-luminous areas correspond to the brightness and saturation characteristics of nighttime. [Effects of the Invention]

[0007] According to the present invention, a nighttime scene is determined based on a traffic light that emits its own light, so the influence of noise caused by surrounding light sources can be suppressed, thereby improving the accuracy of determining a nighttime scene. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of installation of an image processing device. [Figure 2] FIG. 2 is a diagram illustrating a processing flow performed by the image processing device. [Figure 3] FIG. 3 is an explanatory diagram of a light-emitting region and a non-light-emitting region. [Figure 4] FIG. 4 is an explanatory diagram of the determination logic of the image processing device. [Figure 5] FIG. 5 is a block diagram of the image processing device. [Figure 6] FIG. 6 is a schematic diagram of the setting image. [Figure 7] FIG. 7 is a schematic diagram of the setting image. [Figure 8] FIG. 8 is a diagram showing an example of the color determination threshold value. [Figure 9] FIG. 9 is a schematic diagram of light-emitting regions and non-light-emitting regions for each signal color. [Figure 10] FIG. 10 is an explanatory diagram of the average luminance and the average saturation. [Figure 11] FIG. 11 is a diagram showing an example of the conditions for separating the light-emitting regions. [Figure 12] FIG. 12 is a diagram showing an example of the classification conditions for the non-luminescent region. [Figure 13]FIG. 13 is a flowchart showing a processing procedure executed by the image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an image processing apparatus and an image processing program disclosed in the present application will be described with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[0010] First, an overview of processing by an image processing device according to an embodiment will be described with reference to Figs. 1 to 4. Fig. 1 is a diagram showing an example of installation of an image processing device. Fig. 2 is a diagram outlining the processing flow by the image processing device. Fig. 3 is an explanatory diagram of luminous regions and non-luminous regions. Fig. 4 is an explanatory diagram of the determination logic by the image processing device.

[0011] 1, an image processing device 10 according to the embodiment is a drive recorder equipped with an on-board camera that captures an image ahead of the vehicle V, and is mounted on the vehicle V. The on-board camera may capture an image around the vehicle V, including an image ahead of the vehicle V.

[0012] In the present disclosure, the image processing device 10 performs day / night determination from a camera image captured by a camera. The image processing device 10 performs day / night determination by utilizing the day / night characteristics of a traffic light 300 captured in a camera image 201. Specifically, as shown in FIG. 2, the image processing device 10 first extracts a rectangular area BB in which the traffic light 300 exists from the camera image 201 captured by the camera (step S1). The image processing device 10 extracts a bounding box in which the traffic light 300 has a score equal to or greater than a certain value, obtained by a deep learning method such as YOLO, as the rectangular area BB. The rectangular area BB corresponds to an example of a traffic light area.

[0013] Next, the image processing device 10 estimates the light emitting state of the traffic light 300 by performing image processing on the rectangular area BB (step S2). The light emitting state is estimated by estimating the light emitting areas and non-light emitting areas of the rectangular area BB.

[0014] Specifically, the estimation of luminous and non-luminous regions is performed by comparing the color information of the rectangular region BB with color determination thresholds set for each signal color. The color information includes the YUV components of each pixel, including information about luminance and saturation. In this disclosure, of the YUV components, the Y component is defined as the luminance value, and the norms of the U and V components are defined as the saturation value. Details of the color determination thresholds will be described later using FIG. 8.

[0015] In the example shown in Fig. 1, the result of estimating the light emission state indicates that the red light of traffic light 300 shown in rectangular area BB is on, and the area corresponding to the red light is light emission area 221. In other words, the green light and yellow light are not emitting light, and therefore are non-light emission areas.

[0016] Then, the image processing device 10 uses the color information of the luminous area and the non-luminous area to determine whether it is day or night (step S3). Here, an overview of the day or night determination will be described with reference to Figs.

[0017] As mentioned above, when a red light is on, the area corresponding to the red light becomes a light-emitting area that emits red light, and the areas corresponding to the green light and yellow light become non-light-emitting areas. Also, as shown in Figure 4, a daytime traffic light, which is a traffic light during the day, becomes bright overall, including the area around the traffic light, due to the influence of sunlight, etc.

[0018] Therefore, in daytime signals, the average luminance of the luminous area is "slightly high" and the average saturation is "high," while the average luminance and average saturation of the non-luminous area are "low." As will be described later, the determination of "high," "slightly high," and "low" is based on a threshold value.

[0019] In addition, night traffic lights, which are used at night, are generally dark, and the brightness of the traffic light itself dominates the area around the traffic light. Therefore, halation occurs in the light-emitting area of ​​night traffic lights, and the light-emitting area that is originally emitting red light appears white in the camera image, reducing the saturation of the light-emitting area.

[0020] Furthermore, in nighttime traffic signals, halation causes the center of the luminous area to turn white, and the area around the luminous area is prone to a color gradation, increasing saturation. As a result, nighttime traffic signals have the characteristic that the average luminance of the luminous area is "high" and the average saturation is "low," while the average luminance of the non-luminous area is "low" and the average saturation is "slightly high."

[0021] The image processing device 10 determines whether it is day or night based on these characteristics. Specifically, the image processing device 10 calculates the average brightness and average saturation for each luminous area and non-luminous area, and classifies each average brightness and average saturation as "high," "slightly high," or "low" based on the classification conditions in Figure 4.

[0022] The image processing device 10 determines a scene as a nighttime scene when the luminance and saturation of the luminous and non-luminous regions correspond to the luminance and saturation characteristics of a nighttime signal. The image processing device 10 also determines a scene as a daytime scene when the luminance and saturation of the luminous and non-luminous regions correspond to the luminance and saturation characteristics of a daytime signal.

[0023] In this way, the image processing device 10 determines the luminous and non-luminous areas of the traffic light from the camera image, and determines the nighttime scene based on the luminance and saturation of the luminous and non-luminous areas.

[0024] Therefore, the image processing device 10 determines a nighttime scene based on the traffic lights that it itself emits, and therefore can reduce the influence of noise caused by surrounding light sources, thereby improving the accuracy of determining a nighttime scene.

[0025] Next, an example of the configuration of the image processing device 10 will be described with reference to Fig. 5. Fig. 5 is a block diagram of the image processing device 10. As shown in Fig. 5, the image processing device 10 is connected to a camera 3 and an output unit 5.

[0026] The camera 3 is a drive recorder mounted on the vehicle. The camera 3 outputs a camera image of the area ahead of the vehicle to the image processing device 10 each time the camera image is captured. The output unit 5 outputs the processing results of the image processing device 10 to various ECUs (Electronic Control Units) of the vehicle V via an in-vehicle network such as a CAN (Controller Area Network). The output unit 5 may be provided inside the image processing device 10.

[0027] 5, the image processing device 10 includes a storage unit 11 and a controller 12. The storage unit 11 is realized by a storage device such as a read-only memory (ROM), a random access memory (RAM), or a flash memory. The storage unit 11 stores an image processing program according to the embodiment, color determination threshold information related to color determination thresholds, and the like.

[0028] Controller 12 extracts a signal area where a traffic light exists from a camera image captured by camera 3, which is an on-board camera, and determines luminous areas and non-luminous areas of the signal area. Furthermore, controller 12 determines that the scene is a nighttime scene when the luminance and saturation of the luminous area and non-luminous area correspond to the luminance characteristics and saturation characteristics of nighttime.

[0029] The controller 12 corresponds to a so-called processor. The controller 12 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like. The controller 12 executes a program according to an embodiment (not shown) stored in the storage unit 11, using RAM as a work area. The controller 12 can also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0030] First, when a camera image is input from the camera 3, the controller 12 extracts a signal area where a traffic light is present from the camera image. Note that, hereinafter, the signal area may be referred to as a rectangular area BB.

[0031] The controller 12 extracts a bounding box, obtained by a deep learning method such as YOLO, in which the traffic light score is equal to or greater than a certain value, as a rectangular area BB. If the controller 12 can extract the rectangular area BB from the camera image, the controller 12 executes the subsequent processing, and if the controller 12 cannot extract the rectangular area BB from the camera image, the controller 12 skips the subsequent processing.

[0032] Next, the controller 12 determines the luminous and non-luminous regions of the rectangular region BB. First, the controller 12 sets the luminous center of the rectangular region BB based on the color information of each pixel of the rectangular region BB. The luminous center is set based on the high luminance region and the high saturation region.

[0033] The controller 12 designates the pixel with the highest brightness in the rectangular region BB as the high brightness pixel Yh, and sets a group of pixels within a certain brightness range (for example, 10%) of the brightness value of the high brightness pixel Yh as the high brightness region.

[0034] Furthermore, the controller 12 designates the pixel in the rectangular region BB with the highest saturation as the high saturation pixel Sh, and sets a group of pixels within a certain saturation range (for example, 10%) from the saturation value of the high saturation pixel Sh as the high saturation region.

[0035] Then, the controller 12 generates a setting image in which high-brightness areas and high-saturation areas are set based on the color information of the rectangular area BB. Figures 6 and 7 are schematic diagrams of setting images. Note that Figure 6 shows an example of a setting image generated from the rectangular area BB of a camera image captured during the day, and Figure 7 shows an example of a setting image generated from the rectangular area BB of a camera image captured at night. In both Figures 6 and 7, it is assumed that the traffic light is flashing red.

[0036] As shown in Figure 6, during daytime traffic lights, the traffic light appears bright overall due to the influence of sunlight, etc., so in the daytime traffic light setting image Ba, high-brightness areas are widely distributed throughout the traffic light, and high-saturation areas are concentrated around the luminous surface (red light) of the traffic light.

[0037] Furthermore, as shown in Fig. 7, the light emitted by the traffic light is dominant in the case of a nighttime traffic light. Therefore, in the setting image Ba for a nighttime traffic light, high-brightness areas are concentrated on the light-emitting surface (red light) of the traffic light, causing halation on the light-emitting surface of the traffic light, which tends to cause gradations in color and disperses high-saturation areas compared to daytime traffic lights.

[0038] The controller 12 determines the center of gravity Yp of the high-luminance region, and calculates the variance of the high-luminance region from the difference between the center of gravity Yp and each pixel position in the high-luminance region, which is taken as the high-luminance region variation Yv. Similarly to the high-luminance region, the controller 12 also determines the center of gravity Sp of the high-saturation region, and calculates the variance of the high-saturation region from the difference between the center of gravity Sp and each pixel position in the high-saturation region, which is taken as the high-saturation region variation Sv.

[0039] Then, the controller 12 sets the center of gravity of the high luminance region variation Yv or the high saturation region variation Sv, whichever has the smaller variation, as the light-emitting center. That is, if the high luminance region variation Yv is smaller than the high luminance region variation Sv, the controller 12 sets the center of gravity Yp of the high luminance region as the light-emitting center, and if the high saturation region variation Sv is smaller, the controller 12 sets the center of gravity Sp of the high saturation region as the light-emitting center.

[0040] When the light emitting center is set to the center of gravity Yp of the high brightness region, the controller 12 can appropriately set the light emitting center of the nighttime signal, and when the light emitting center is set to the center of gravity Sp of the high saturation region, the controller 12 can appropriately set the light emitting center of the daytime signal.

[0041] That is, the controller 12 can appropriately set the light emission centers of the nighttime signal and the daytime signal by setting the light emission centers based on the high luminance region variation Yv and the high saturation region variation Sv.

[0042] Next, the controller 12 determines the signal color of the traffic light based on the set luminous center. Prior to determining the signal color, the controller 12 calculates a rectangular area that includes the above-mentioned high saturation area, and determines whether the calculated rectangular area includes the luminous center.

[0043] If the rectangular area includes a light-emitting center, the controller 12 performs a signal color determination, and if the rectangular area does not include a light-emitting center, the controller 12 omits subsequent processing including the signal color determination. In determining the signal color of a traffic light, the controller 12 first calculates the average values ​​for the Y, U, and V components of the YUV components in the high saturation area.

[0044] The controller 12 then compares the average values ​​of the Y, U, and V components with the color determination thresholds for YUV to determine the signal color. Fig. 8 is a diagram showing an example of the color determination thresholds.

[0045] 8, the color determination threshold is information in which the corresponding conditions for each of the green, yellow, red, and black color components of a traffic light are specified by predetermined lower and upper limits of Y, U, and V. Controller 12 determines the color of the traffic light based on whether the average values ​​of the Y, U, and V components in the high saturation region fall within the ranges of the Y threshold, U threshold, and V threshold set for each color component.

[0046] That is, if the average value of each of the Y, U, and V components falls within the corresponding color component threshold, the controller 12 determines the corresponding color component as the signal color of the traffic light. In other words, if any one of the average values ​​of the Y, U, and V components does not fall within the corresponding color component threshold, the controller 12 determines the signal color of the traffic light as "no color." Then, based on the signal color determination result, the controller 12 sets a flag for the corresponding signal color for each pixel.

[0047] Next, the controller 12 estimates the luminous regions and non-luminous regions based on the determined signal color. The controller 12 sets the luminous regions and non-luminous regions based on the arrangement pattern set for each signal color, with the luminous center as the starting point.

[0048] Fig. 9 is a schematic diagram of luminous regions and non-luminous regions for each signal color. As shown in Fig. 9, for a green signal, the luminous regions and non-luminous regions are determined from the left in a pattern of blue luminous regions, non-luminous regions, and non-luminous regions. For a yellow signal, the luminous regions and non-luminous regions are determined from the right in a pattern of non-luminous regions, yellow luminous regions, and non-luminous regions.

[0049] In addition, in the case of a red light, the light emitting and non-light emitting areas are determined from the left as a non-light emitting area, a non-light emitting area, and a red light emitting area pattern. Note that in the case of a non-light emitting signal where no light is emitted from any signal, the non-light emitting area is determined from the left as a non-light emitting area, a non-light emitting area, and a non-light emitting area.

[0050] The controller 12 determines a certain region including the luminous center as the luminous region, and determines a non-luminous region based on the arrangement pattern of the signal color of the corresponding luminous center. Specifically, if the luminous center is a blue component (i.e., a blue signal), the controller 12 determines the certain region including the luminous center as the luminous region, and the two certain regions arranged at a predetermined interval to the right of it as non-luminous regions.

[0051] Next, the controller 12 determines whether it is day or night based on the brightness and saturation of each of the light-emitting and non-light-emitting regions. First, the controller 12 calculates the average brightness and average saturation of each of the light-emitting and non-light-emitting regions.

[0052] FIG. 10 is an explanatory diagram of average brightness and average saturation. Note that FIG. 10 shows luminous areas and non-luminous areas when the traffic light is red. As shown in FIG. 10, controller 12 calculates the average brightness YL and average saturation SL of the luminous areas and the average brightness YN and average saturation SN of the non-luminous areas. Note that the average brightness YN and average saturation SN of the non-luminous areas may be determined by using the higher value of the average brightness and average saturation of the two non-luminous areas.

[0053] Next, the controller 12 distinguishes between luminous and non-luminous regions based on the high brightness pixels Yh, high saturation pixels Sh, average brightness YL and average saturation SL of the luminous regions, and average brightness YN and average saturation SN of the non-luminous regions in the rectangular region BB.

[0054] Specifically, as explained in FIG. 4, the controller 12 classifies the brightness and saturation of each of the luminous and non-luminous areas into "high," "slightly high," or "low," and determines whether it is day or night based on the results of this classification.

[0055] Fig. 11 is a diagram showing an example of the classification conditions for luminous regions. Fig. 12 is a diagram showing an example of the classification conditions for non-luminous regions. As shown in Fig. 11, classification condition 1 is a condition for determining that the average luminance of a luminous region is "high," and indicates that the average luminance YL of the luminous region is equal to or greater than a threshold value which is the product of a coefficient Yr1 and a high luminance value Ymax, which is the luminance value of the high luminance pixel Yh.

[0056] Furthermore, classification condition 2 is a condition for determining that the average brightness of a luminous region is "slightly high," and indicates that the average brightness YL of the luminous region is equal to or greater than a threshold value, which is the product of coefficient Yr2 and high brightness value Ymax, which is the brightness value of high brightness pixel Yh. Note that coefficient Yr1 is a value greater than coefficient Yr2; for example, coefficient Yr1 can be "0.6" and coefficient Yr2 can be "0.3." Furthermore, classification condition 3 is a condition for determining that the average brightness of a luminous region is "low," and requires that neither classification condition 1 nor classification condition 2 is met.

[0057] Note that the same classification conditions as those shown in Figure 11 can also be applied to the average saturation of the luminous area. In this case, the average luminance YL is replaced with the average saturation SL, and the high luminance value Ymax, which is the luminance value of the high luminance pixel Yh, is replaced with the high saturation value Smax, which is the saturation value of the high saturation pixel Sh. The average saturation SL of the luminous area is then classified as either "high," "slightly high," or "low."

[0058] Next, the separation conditions for non-luminescent regions will be described with reference to Fig. 12. As shown in Fig. 12, the separation conditions for non-luminescent regions also include separation conditions 1 to 3, similar to the separation conditions for luminescent regions.

[0059] Classification condition 1 is a condition for determining that the average brightness of a non-luminous area is "high," and indicates that the average brightness YN of the non-luminous area is greater than or equal to a threshold value which is the product of the coefficient Yr1 and the high brightness value Ymax, which is the brightness value of the high brightness pixel Yh.

[0060] Furthermore, classification condition 2 is a condition for determining the average brightness of a non-luminous region as "slightly high," and indicates that the average brightness YN of the non-luminous region is equal to or greater than a threshold value, which is the product of coefficient Yr2 and high brightness value Ymax, which is the brightness value of high brightness pixel Yh. Note that coefficients Yr1 and Yr2 are the same as those described for luminous regions. Furthermore, classification condition 3 is a condition for determining the average brightness of a non-luminous region as "low," and requires that neither classification condition 1 nor classification condition 2 is met.

[0061] Note that the same classification conditions as those shown in Fig. 12 can also be applied to the average saturation of non-luminous regions. In this case, the average brightness YN is replaced with the average saturation SN, and the high brightness value Ymax, which is the brightness value of the high brightness pixel Yh, is replaced with the high saturation value Smax, which is the saturation value of the high saturation pixel Sh. The average saturation SN of the non-luminous region is then classified as "high," "slightly high," or "low."

[0062] Controller 12 classifies the average brightness and average saturation of the luminous areas and the average brightness and average saturation of the non-luminous areas into either "high," "slightly high," or "low." Then, as shown in Figure 4, controller 12 determines whether the average brightness and average saturation of the luminous areas and the average brightness and average saturation of the non-luminous areas correspond to the brightness characteristics and saturation characteristics of a daytime signal or a nighttime signal.

[0063] Specifically, if the average luminance and average saturation of the non-luminous areas are "low" and "low," and the average luminance and average saturation of the luminous areas are "slightly high," and "high," the controller 12 determines that the scene corresponds to the luminance and saturation characteristics of daytime. In this case, the controller 12 determines that the scene is a daytime scene.

[0064] Furthermore, if the average luminance of the non-luminous areas is "low" and the average saturation is "slightly high," and the average luminance of the luminous areas is "high" and the average saturation is "low," controller 12 determines that the luminance and saturation characteristics are those of nighttime. In this case, controller 12 determines that the scene is a nighttime scene.

[0065] In this way, the controller 12 determines the luminance and saturation characteristics of the luminous and non-luminous regions based on the high luminance value Ymax, which is the luminance value of the high luminance pixel Yh, and the high saturation value Smax, which is the saturation value of the high saturation pixel Sh.

[0066] In other words, the controller 12 changes the nighttime luminance and saturation characteristics and the daytime luminance and saturation characteristics for each frame. This allows appropriate day / night determination even for signals that do not emit sufficient light. Note that the nighttime luminance and saturation characteristics and the daytime luminance and saturation characteristics may be constant values.

[0067] In addition, if the average brightness and average saturation of the luminous area and the average brightness and average saturation of the non-luminous area do not correspond to either the nighttime brightness and saturation characteristics or the daytime brightness and saturation characteristics, the controller 12 determines that they do not match.

[0068] In this way, when the scene does not correspond to either a night scene or a day scene, the controller 12 determines that the scene does not correspond, thereby making it possible to output only a highly accurate determination result.

[0069] Furthermore, the controller 12 holds the current determination result until the next day / night determination is performed. This is because the day / night determination is performed only on frames in which a traffic light is detected. In other words, by holding the current determination result until the next traffic light is detected, the controller 12 can appropriately manage the determination result.

[0070] When multiple traffic lights are detected in one frame, the controller 12 may perform day / night determination for the traffic light closest to the vehicle V. The controller 12 may also add different luminance characteristics and saturation characteristics for determining whether a nighttime signal is present, depending on whether halation is present or absent. That is, in such a case, the controller 12 may determine whether the scene is a nighttime scene by using different nighttime luminance characteristics and saturation characteristics, depending on whether halation is present or absent in the nighttime signal.

[0071] Next, a processing procedure executed by the image processing device 10 according to the embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the processing procedure executed by the image processing device 10. Note that the processing procedure shown below is repeatedly executed by the controller 12 every time a camera image is input.

[0072] 13, first, upon acquiring a camera image (step S101), the controller 12 determines whether or not a rectangular area BB in which a traffic light exists has been extracted (step S102). The rectangular area BB is extracted by extracting a bounding box, obtained by a deep learning method such as YOLO, in which the traffic light score is equal to or greater than a certain value, as the rectangular area BB.

[0073] If the controller 12 determines that the rectangular area BB has been extracted (step S102; Yes), the process proceeds to step S103, and if the controller 12 determines that the rectangular area BB has not been extracted (step S102; No), the process ends.

[0074] Next, the controller 12 determines the light-emitting center of the rectangular area BB (step S103). The controller 12 sets a high-luminance area and a high-saturation area for the rectangular area BB, compares the variations in the high-luminance area and the high-saturation area, and determines the light-emitting center.

[0075] Next, the controller 12 determines the light emitting regions and non-light emitting regions (step S104). The controller 12 determines the light emitting regions and non-light emitting regions based on the arrangement pattern set for each signal color, starting from the light emitting center.

[0076] Next, controller 12 calculates the average brightness and average saturation of the luminous and non-luminous regions (step S105), and classifies the luminous and non-luminous regions (step S106). Specifically, controller 12 classifies the average brightness and average saturation of each of the luminous and non-luminous regions into "high," "slightly high," or "low."

[0077] Next, controller 12 performs day / night determination based on the classification results (step S107). Specifically, controller 12 determines whether the classification results of the average luminance and average saturation of the luminous and non-luminous regions correspond to the luminance and saturation characteristics of nighttime or the luminance and saturation characteristics of daytime.

[0078] Then, the controller 12 outputs the day / night determination result (step S108), and ends the process.

[0079] In the above embodiment, the luminance and saturation of the luminous and non-luminous regions are classified into three values, "high," "slightly high," and "low," but the present invention is not limited to this. In other words, the luminance and saturation of the luminous and non-luminous regions may be classified into two values, or may be classified into four or more values.

[0080] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0081] 3 Camera 5 Output section 10 Image processing device 11 Storage section 12 Controllers V vehicle

Claims

1. a controller; The controller Extracting traffic light areas from the camera images taken by the in-vehicle camera, determining a light-emitting region and a non-light-emitting region of the signal region; If the luminance and saturation of the light-emitting area and the non-light-emitting area correspond to the luminance and saturation characteristics of nighttime, the scene is determined to be a nighttime scene. Image processing device.

2. The controller extracting a high luminance region where the luminance is equal to or greater than a threshold value and a high saturation region where the saturation is equal to or greater than a threshold value from the signal region; setting a light emitting center of the signal region based on a region with smaller variation between the high luminance region and the high saturation region; The light-emitting region and the non-light-emitting region are determined based on the light-emitting center. The image processing device according to claim 1 .

3. The controller If the average luminance of the light emitting area is higher than a threshold and the average luminance of the non-light emitting area is lower than a threshold, the scene is determined to be a nighttime scene. The image processing device according to claim 1 .

4. The controller A threshold value for the average brightness is set according to the maximum brightness value of the light emitting area. The image processing device according to claim 3 .

5. The controller If the average saturation of the luminous region is lower than a threshold and the average saturation of the non-luminous region is higher than a threshold, the scene is determined to be a night scene. The image processing device according to claim 1 .

6. The controller A threshold value for the average saturation is set according to the maximum saturation value of the light-emitting area. The image processing device according to claim 5 .

7. The controller If the luminance and saturation of the luminous area and the non-luminous area correspond to the luminance characteristics and saturation characteristics of daytime, the scene is determined to be a daytime scene, and if the scene does not correspond to either the nighttime scene or the daytime scene, the scene is determined to be non-applicable. The image processing device according to claim 1 .

8. The controller The current judgment result is retained until the next night scene judgment is performed. The image processing device according to claim 1 .

9. An extraction step of extracting a signal area where a traffic light is present from a camera image captured by an in-vehicle camera; a determination step for determining a light-emitting region and a non-light-emitting region of the signal region; a determination step of determining that the scene is a night scene when the luminance and saturation of the luminous area and the non-luminous area correspond to specific luminance and saturation characteristics of nighttime; An image processing program that causes a computer to execute the following.

Images