Driving assistance device and driving assistance method

Abstract

A driving assistance device (3) is provided with: an acquisition unit (321) that acquires an infrared image in which an image capture region including a driver's seat of a vehicle is captured; a determination unit (322) that, in a case where a driver cannot be recognized in the infrared image, determines whether or not a blaze highlight has occurred in the infrared image on the basis of whether or not a pixel value of a pixel of an image in which a region irradiated by sunlight in the image capture region is captured in the infrared image is equal to or greater than a predetermined value; and a setting unit (323) that, when it is determined that highlight has occurred in the infrared image, makes a waiting time longer than a reference time, the waiting time being a point in time at which a warning is issued.

Description

Technical Field

[0001] The present invention relates to a driving assistance device and a driving assistance method for assisting vehicle driving. Background Art

[0002] Techniques for determining whether there is an abnormality in a vehicle driver are known. JP-2017-16568-A discloses a technique for determining whether there is an abnormality in a driver state, in which the driver state is recognized from an infrared image captured in a state where infrared rays are irradiated within the image capture range of a camera. Summary of the Invention

[0003] Problems to be Solved by the Invention

[0004] However, when sunlight irradiates the driver's seat, a glare highlight appears in the infrared image, and in some cases, the driver cannot be recognized. In the case where the driver cannot be recognized, in some cases, even if the driver is driving normally, it is determined that there is an abnormality in the driver and a warning is undesirably issued.

[0005] In view of these problems, the present invention has been made, and an object of the present invention is to reduce false warnings.

[0006] Means for Solving the Problems

[0007] A first aspect of the present invention provides a driving assistance device having: an acquisition unit that acquires an infrared image captured when a first region is irradiated with infrared rays, the first region including a driver's seat on which a driver of a vehicle is seated; a determination unit that, in a case where the driver cannot be recognized in the infrared image, determines whether a glare highlight has appeared in the infrared image based on whether a pixel value of a pixel in a determined portion in the infrared image is equal to or greater than a predetermined value, the determined portion capturing an image of a second region that is in the first region and is irradiated with sunlight; and a setting unit that, in a case where it is determined that a glare highlight has occurred in the infrared image, makes a waiting time longer than a reference time, the waiting time being a time between a time point when the driver cannot be recognized in the infrared image and a time point when a warning is issued.

[0008] The acquisition unit may acquire a captured image captured in a state where no infrared rays are irradiated in a case where the determination unit determines that the pixel value of the pixel in the determined portion is equal to or greater than the predetermined value, and the determination unit may determine that a glare highlight has appeared in the infrared image in a case where the pixel value of the pixel in the determined portion is equal to or greater than the predetermined value and the driver can be recognized in the captured image.

[0009] The determination unit may determine whether a glare highlight has occurred in the infrared image in a case where an image capture time of the infrared image includes a time period during which the second region is irradiated with sunlight.

[0010] The time period may include: a first time period from the sunrise time on the image capture date when the infrared image is captured until a time that is a predetermined time after the sunrise time; and a second time period from a time that is a predetermined time before the sunset time on the image capture date until the sunset time.

[0011] The determination unit may increase the predetermined time as the time from the sunrise time on the image capture date until the sunset time decreases.

[0012] The acquisition unit may acquire an infrared image captured after the image capture device that captures the infrared image has performed exposure correction to change at least one of the aperture value or the exposure time.

[0013] The determination unit may determine that no glare highlight appears in the infrared image when the pixel value of the pixels in the determination part of the infrared image is less than a predetermined value, and the setting unit may set the waiting time to the reference time when the determination unit determines that no glare highlight appears in the infrared image.

[0014] The determination unit may determine that a glare highlight has appeared in the infrared image when the statistic of the pixel values of the plurality of pixels included in the determination part is equal to or greater than a predetermined value, and determine that no glare highlight appears in the infrared image when the statistic is less than the predetermined value.

[0015] The waiting time may be the abnormal determination time between the time point when the driver cannot be recognized in the infrared image and the time point when an abnormality in which the driver's normal driving posture deviates is determined to have occurred.

[0016] The determination unit may determine that a glare highlight has appeared in the infrared image when the minimum value of the pixel values of the plurality of pixels included in the determination part is greater than half of the maximum value of the signal that can be processed by the image capture device that captures the infrared image and is equal to or greater than a predetermined value smaller than the maximum value.

[0017] The determination unit may determine that no glare highlight appears in the infrared image when the maximum value of the pixel values of the plurality of pixels included in the determination part is greater than half of the maximum value of the signal that can be processed by the image capture device that captures the infrared image and is less than a predetermined value smaller than the maximum value.

[0018] The driving assistance device may have a notification control unit that issues a warning. The acquisition unit may acquire operation information related to the driving operation performed by the driver. The determination unit may determine whether the driving operation indicated by the operation information is abnormal, and the notification control unit may issue a warning when the driving operation is abnormal and the time elapsed since the driver cannot be recognized in the infrared image is equal to or longer than the waiting time set by the setting unit.

[0019] A second aspect of the present invention provides a driving assistance method executed by a computer of a vehicle. The driving assistance method includes the following steps: obtaining an infrared image captured when a first area is irradiated with infrared rays, where the first area includes a driver's seat on which the driver of the vehicle is seated; in a case where the driver cannot be recognized in the infrared image, determining whether a glare highlight has occurred in the infrared image based on whether the pixel values of the pixels in a determined portion of the infrared image are equal to or greater than a predetermined value, where the determined portion captures an image of a second area that is in the first area and is irradiated with sunlight; and in a case where it is determined that a glare highlight has occurred in the infrared image, making a waiting time longer than a reference time, where the waiting time is the time between the time point when the driver becomes unrecognizable in the infrared image and the time point when a warning is issued.

[0020] Effects of the present invention

[0021] The present invention achieves the effect of reducing false warnings. Description of the drawings

[0022] Figure 1 is a diagram for explaining the configuration of the driving assistance system.

[0023] Figure 2 is a diagram for explaining the image capture area of the image capture device.

[0024] Figure 3 is a schematic diagram of an infrared image captured by the image capture device.

[0025] Figure 4 is a diagram for explaining an infrared image that has lost gray scale.

[0026] Figure 5 is a flowchart depicting an example of the process of setting the waiting time. Detailed implementation manners

[0027] [Configuration of the driving assistance system]

[0028] The driving assistance system S assists the driver of the vehicle in driving. Use Figure 1 and Figure 2 to explain the configuration of the driving assistance system S. Figure 1 is a diagram for explaining the configuration of the driving assistance system S. Figure 2 is a diagram for explaining the image capture area 21 of the image capture device 2. The driving assistance system S includes an infrared lamp 1, an image capture device 2, a driving assistance device 3, and an output unit 4.

[0029] The infrared lamp 1 irradiates the first area with infrared rays 11, and the first area is the image capture area 21 including the driver's seat of the vehicle in which the driver 5 is seated. The infrared lamp 1 irradiates an area that is larger than the image capture area 21 and includes the entire image capture area 21 with infrared rays 11, so that the entire image capture area 21 can be irradiated with infrared rays 11. The infrared lamp 1 is arranged at a position where the infrared lamp 1 can irradiate the image capture area 21 with infrared rays 11. For example, the infrared lamp 1 is arranged on the instrument panel of the vehicle, but it is sufficient if the installation position of the infrared lamp 1 is such that the infrared lamp 1 can irradiate the image capture area 21 with infrared rays 11.

[0030] The image capture device 2 is arranged at a position where the image capture device 2 can capture an image of the image capture area 21 inside the vehicle. For example, the image capture device 2 is arranged on the vehicle instrument panel in front of the driver's seat and captures an image of the image capture area 21. In a specific example, the image capture device 2 is arranged on the instrument panel and between the windshield and the steering wheel. Note that the installation position of the image capture device 2 is not limited to this, but it is sufficient if the image capture device 2 is at a position where the image capture device 2 can capture an image of the image capture area 21.

[0031] The image capture device 2 captures an image of the image capture area 21 and generates an image. For example, the image capture device 2 captures an image of the image capture area 21 in a state where the infrared lamp 1 irradiates the image capture area 21 with infrared rays 11 and generates an infrared image 71. Figure 3 is a schematic diagram of the infrared image 71 captured by the image capture device 2. Specifically, after performing exposure correction in a state where the image capture area 21 is irradiated with infrared rays 11, the image capture device 2 captures an image of the image capture area 21 and generates an infrared image 71. More specifically, after performing exposure correction, the image capture device 2 captures an image of the image capture area 21 and generates an infrared image 71. Exposure correction is a process of changing at least one of the aperture value or the exposure time so that the gray level of the infrared image of the captured image capture area 21 becomes appropriate.

[0032] By analyzing the infrared image 71, the driving assistance device 3 extracts features of the driver 5, such as the positions of the facial contour, eyes, nose, and mouth, and identifies the driver 5 in the infrared image 71. However, in the case where the posture of the driver 5 is distorted abnormally and is not a normal driving posture, the driver 5 will not be included in the image capture area 21. In the following description, an abnormality in which the posture of the driver 5 is no longer a normal driving posture is referred to as posture distortion.

[0033] For example, the posture distortions include a forward falling posture, a downward looking posture, a backward tilting posture, a shrimp-like backward arching posture, a lateral head falling posture, a lateral falling posture, and a lateral tilting posture. In the state of the forward falling posture, the driver has fallen forward, and the driver's face is close to the steering wheel. In the state of the downward looking posture, the driver's face is facing downwards. In the state of the backward tilting posture, the upper half of the driver's body tilts backward and the face is facing upwards. In the state of the shrimp-like backward arching posture, the upper half of the driver's body arches backward and the face is facing upwards. In the state of the lateral head falling posture, the driver's head tilts to the left or right. In the state of the lateral falling posture, the upper half of the driver's body tilts to the left or right, and the face tilts in the same direction as the tilt of the upper half of the body. In the state of the lateral tilting posture, the upper half of the driver's body tilts to the left or right.

[0034] When the driver 5 has the above-mentioned posture distortion, the driving assistance device 3 cannot extract the facial features of the driver 5 from the infrared image 71, and thus cannot recognize the driver 5 in the infrared image 71. When the driver 5 cannot be recognized in the infrared image 71, the driving assistance device 3 determines that the driver 5 has had an abnormal posture distortion. Specifically, when the state of the abnormal posture distortion in which the driver 5 cannot be recognized in the infrared image 71 continues, the driving assistance device 3 determines that a posture distortion has occurred and causes the output unit 4 to issue a warning. The output unit 4 includes a speaker or a buzzer that issues a warning. When there is a posture distortion of the driver 5, the driving assistance device 3 controls the output unit 4 to issue a message notifying the abnormality.

[0035] At the same time, in some cases, a part of the image capture area 21 is irradiated by sunlight 6 (see Figure 2 ). For example, when the sun is in front of the vehicle or in front of the left side of the vehicle, the sunlight 6 passing through the windshield or the doors and windows of the vehicle irradiates the determination area 22, which is the second area including the left shoulder and chest of the driver 5. The determination area 22 becomes brighter than the area in the image capture area 21 that is not irradiated by sunlight 6.

[0036] When an infrared image is captured in a state where the determination area 22 is irradiated with sunlight 6, the determination part that captures the image of the determination area 22 in the infrared image becomes brighter than the part that captures the images of the areas other than the determination area 22 in the image capture area 21. In particular, when the image capture device 2 is provided on the vehicle dashboard, the distance between the determination area 22 and the image capture device 2 is shorter than the distance between the areas other than the determination area 22 and the image capture device 2. Therefore, the brightness of the determination area 22 undesirably becomes higher than that of the areas other than the determination area 22. In such a case, even if the image capture device 2 performs exposure correction to prevent the loss of gray scale of the determination part, in some cases, the gray scale of the infrared image cannot be made appropriate, and unwanted clipping appears in the infrared image.

[0037] Clipping is a phenomenon of gray scale loss in an infrared image. Specific examples of clipping include flare highlights where the pixel values of the pixels in the infrared image are saturated and the gray scale of the infrared image is lost, and occlusion shadows where the pixel values of the pixels are too small due to insufficient exposure and the gray scale of the infrared image is lost. For example, when the determination area 22 is too bright, even if the aperture value and exposure time are decreased, the pixel values of the pixels in the captured infrared image are saturated, and flare highlights appear. In addition, as a result of decreasing the aperture value and exposure time to prevent the loss of gray scale of the determination area 22, in some cases, the exposure amount of the part other than the determination area 22 in the image capture area 21 becomes insufficient, and occlusion shadows appear.

[0038] Figure 4 is a diagram for explaining an infrared image 72 that has lost its gray scale. The determination part 73 of the infrared image 72 is the part that captures the image of the determination area 22. Figure 4 In the infrared image 72, the determination part 73 has lost its gray scale because the pixel values are saturated, and flare highlights appear. In addition, due to insufficient exposure, the gray scale of the part other than the determination part 73 in the infrared image 72 is lost, and occlusion shadows have appeared. In such a case, the driving assistance device 3 cannot extract the features of the driver 5's face, for example, cannot extract the contour of the jaw, and thus cannot identify the driver 5 in the infrared image 72. In the case where the driver 5 cannot be identified in the infrared image 72, in some cases, although the driver 5 is driving in a normal driving posture, it is determined that the driver 5 has a posture distortion, and a warning is undesirably issued.

[0039] Meanwhile, it is determined whether the determination area 22 is irradiated by sunlight 6, which is affected by the relative positions of the sun and nearby buildings, etc. with respect to the vehicle. Then, since the relative positions of the sun and nearby buildings with respect to the vehicle have changed before the warning is issued, in the case where the determination area 22 is no longer irradiated by sunlight 6, the image capturing device 2 can perform exposure correction to prevent the loss of grayscale of the infrared image. As a result, the flare highlight of the infrared image is solved. Since the driver 5 can be recognized in the infrared image in this case, no warning is issued.

[0040] Taking this into account, in the case where the driver 5 cannot be recognized in the infrared image and flare highlights or occluding shadows appear in the infrared image, the driving assistance device 3 makes the time until the warning is issued longer. In the case where the relative positions of the sun and nearby buildings with respect to the vehicle have changed before the warning is issued and the determination area 22 is no longer irradiated by sunlight 6, the flare highlight of the infrared image is solved. In the case where the flare highlight of the infrared image is solved, the driving assistance device 3 can recognize the driver 5 in the infrared image and thus does not give a warning about posture distortion. That is, the driving assistance device 3 reduces false warnings by making the time until the warning is issued longer when flare highlights appear in the infrared image. Hereinafter, the processing performed by the driving assistance device 3 will be specifically described.

[0041] The driving assistance device 3 has a storage unit 31 and a control unit 32. The storage unit 31 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, etc. The storage unit 31 stores programs to be executed by the control unit 32.

[0042] For example, the control unit 32 is a computing resource including a processor such as a CPU (Central Processing Unit). By executing the programs stored on the storage unit 31, the control unit 32 functions as an acquisition unit 321, a determination unit 322, a setting unit 323, and a notification control unit 324.

[0043] The acquisition unit 321 acquires an infrared image from the image capturing device 2. For example, the acquisition unit 321 acquires the infrared image captured by the image capturing device 2 in a state where the image capturing area 21 is irradiated by infrared rays 11. Specifically, the acquisition unit 321 acquires the infrared image captured by the image capturing device 2 after the image capturing device 2 has performed exposure correction. In addition, the acquisition unit 321 acquires the image capturing time from the image capturing device 2, at which the infrared image is acquired. Specifically, the acquisition unit 321 acquires image information from the image capturing device 2, in which the infrared image is associated with the image capturing time at which the captured image is captured.

[0044] The determination unit 322 determines whether the driver 5 can be recognized in the infrared image. The determination unit 322 determines that the driver 5 can be recognized in the infrared image when the facial features of the driver 5 can be extracted by analyzing the infrared image. When the facial features of the driver 5 cannot be extracted even by analyzing the infrared image, the determination unit 322 determines that the driver 5 cannot be recognized in the infrared image. When the driver 5 cannot be recognized, the determination unit 322 determines whether a glare highlight has appeared in the infrared image.

[0045] Meanwhile, the time period during which the determination area 22 is irradiated by the sunlight 6 and the time period during which the determination area 22 is not irradiated by the sunlight 6 are determined in advance. For example, the determination area 22 is irradiated by the sunlight 6 in the first time period from sunrise until a predetermined time after sunrise and in the second time period from a time that is a predetermined time before sunset until sunset. However, since the top surface inside the vehicle blocks the sunlight 6, the determination area 22 is not irradiated by the sunlight 6 in the time period including the time when the sun is in the south. In addition, the determination area 22 is not irradiated by the sunlight 6 in the time period after sunset and before sunrise. In this way, in the time period when the determination area 22 is not irradiated by the sunlight 6, the possibility of a glare highlight appearing in the infrared image is low, and thus it is not necessary to determine whether a glare highlight has appeared.

[0046] In view of this, the determination unit 322 determines whether the image capture time when the infrared image is captured is included in the time period during which the determination area 22 is irradiated by the sunlight 6. For example, the time period during which the determination area 22 is irradiated by the sunlight 6 is the first time period from the sunrise time of the image capture date of the infrared image until a predetermined time after the sunrise time and the second time period from a predetermined time before the sunset time until the sunset time.

[0047] The determination unit 322 makes the predetermined times of the first time period and the second time period longer as the time from sunrise to sunset on the image capture date decreases. For example, the specific value of the predetermined time is two hours or three hours, but these are not the only examples. For example, when the sunrise time is 6:45 and the sunset time is 16:30, the determination unit 322 sets the first time period to three hours from 6:45 to 9:45, and sets the second time period to two hours from 14:30 to 16:30. When the sunrise time is 5:45 and the sunset time is 17:30, the determination unit 322 sets the first time period to two hours from 5:45 to 7:45, and sets the second time period to two hours from 15:30 to 17:30.

[0048] When the image capture time is included in the first time period or the second time period, the determination unit 322 determines whether a bright highlight has appeared in the infrared image. When the image capture time is neither included in the first time period nor in the second time period, the determination unit 322 does not determine whether a bright highlight has appeared in the infrared image. In other words, when the image capture time is included in the third time period, which is the entire time period of a day other than the first time period and the second time period, the determination unit 322 does not determine whether a bright highlight has appeared in the infrared image. In this way, in the time periods when the possibility of a bright highlight appearing in the infrared image is high, the determination unit 322 determines whether a bright highlight has appeared, and in the case where the possibility is low, it does not determine whether a bright highlight has appeared. As a result, the determination unit 322 can reduce unnecessary determinations and thus reduce waste of computing resources.

[0049] The determination unit 322 determines whether a bright highlight has appeared in the infrared image based on whether the pixel value of the pixels in the determination section 73 is equal to or greater than a predetermined value. The predetermined value is the maximum value of the signal that can be processed by the image capture device 2, but it can be a value less than this maximum value. For example, the predetermined value is greater than half of the maximum value of the signal that can be processed and less than the maximum value. In a specific example, the predetermined value is 90% of the maximum value of the signal that can be processed. The determination unit 322 determines that a bright highlight has appeared in the infrared image when the pixel value of the pixels in the determination section 73 is equal to or greater than the predetermined value, and determines that no bright highlight has appeared in the infrared image when the pixel value of the pixels in the determination section 73 is less than the predetermined value.

[0050] The determination unit 322 can determine whether a bright highlight has appeared in the infrared image based on whether a statistic of the pixel values of the plurality of pixels included in the determination section 73 is equal to or greater than a predetermined value. The statistic is any one of the average value, median value, sum, maximum value, and minimum value of the pixel values of the plurality of pixels included in the determination section 73. This statistic is preferably the average value, median value, or maximum value. In a specific example, when the maximum value of the pixel values of the pixels in the determination section 73 is less than the predetermined value, it can be considered that the pixel values of the pixels are not saturated and the gray level of the determination section 73 is not lost, and thus the determination unit 322 determines that no bright highlight has appeared in the infrared image.

[0051] When the statistic of the pixel value of the pixels in the determination section 73 is equal to or greater than a predetermined value, the determination section 322 determines that a flare highlight has appeared in the infrared image. For example, when the average value, median value, or minimum value of the pixel values of the pixels in the determination section 73 is equal to or greater than a predetermined value, the determination section 322 determines that a flare highlight has appeared in the infrared image. Specifically, when the minimum value of the pixel values of the pixels in the determination section 73 is equal to or greater than a predetermined value, it is considered that the pixel value of the pixel is saturated, and the gray level of the determination section 73 is lost, and thus the determination section 322 determines that a flare highlight has appeared in the infrared image.

[0052] In some cases, due to the influence of the infrared rays 11 radiated by the infrared lamp 1, a flare highlight appears in the infrared image. In this case, by re-determining whether a flare highlight has appeared in the infrared image based on the captured image captured in a state where no infrared rays are irradiated, the reliability of determining whether a flare highlight has appeared in the infrared image can be improved. Hereinafter, the determination process based on the captured image captured in a state where no infrared rays are irradiated will be described.

[0053] When it is determined that a flare highlight has appeared in the infrared image, the acquisition section 321 acquires the captured image captured in a state where no infrared rays 11 are irradiated. Specifically, when it is determined that a flare highlight has appeared in the infrared image, the acquisition section 321 causes the infrared lamp 1 to stop irradiating with the infrared rays 11. By causing the image capturing device 2 to generate a captured image after the infrared lamp 1 stops irradiating with the infrared rays 11, the acquisition section 321 acquires the captured image in which the image capturing area 21 is not irradiated with the infrared rays 11. Therefore, the acquisition section 321 can acquire the captured image acquired in a state where the image capturing area 21 is not irradiated with the infrared rays 11 and the image capturing area 21 is irradiated with the sunlight 6.

[0054] The determination unit 322 determines whether the driver 5 can be recognized in the captured image captured by the image capturing device 2 in a state where the infrared lamp has stopped irradiating with infrared rays 11. By using the captured image captured in a state where there is no irradiation with infrared rays 11 but there is irradiation with sunlight 6, the determination unit 322 can determine whether the driver 5 can be recognized in a situation where there is no influence of the infrared rays 11 on the image capturing area 21 but there is an influence of the sunlight 6 on the image capturing area 21. When the driver 5 can be recognized in the captured image, the determination unit 322 determines again that a flash highlight has appeared in the infrared image. Specifically, when the determination unit 322 determines that a flash highlight has appeared in the infrared image, it continues to determine that a flash highlight has appeared in the infrared image and can recognize the driver 5 in the captured image. In this way, by recognizing the driver 5 in the image captured in a state where there is no irradiation with infrared rays when a flash highlight has appeared in the infrared image, the determination unit 322 can enhance the reliability of determining that a flash highlight has appeared in the infrared image when the driver 5 is in a normal driving posture.

[0055] When the determination unit 322 cannot recognize the driver 5 in the captured image, it determines that there is an abnormal distortion in the posture of the driver 5. Specifically, the determination unit 322 cancels the determination that a flash highlight has appeared in the infrared image and determines that there is an abnormal distortion in the posture of the driver 5. When a flash highlight has appeared in the infrared image, the determination unit 322 can appropriately determine whether there is an abnormal distortion in the posture of the driver 5 based on whether the driver 5 can be recognized in the captured image captured in a state where there is no irradiation with infrared rays 11.

[0056] The setting unit 323 sets a waiting time between the time point when the driver 5 becomes unrecognizable and the time point when a predetermined control starts. For example, the predetermined control is at least a notification control for issuing a warning or an automatic braking control for stopping the vehicle. The setting unit 323 sets a first waiting time between the time point when the driver 5 becomes unrecognizable in the infrared image and the time point when the notification control starts. In addition, the setting unit 323 sets a second waiting time between the time point when the driver 5 becomes unrecognizable and the time point when the automatic braking control starts. The first waiting time and the second waiting time may be the same or different. Specifically, the setting unit 323 sets the waiting time between the time point when the notification control unit 324 causes the output unit 4 to issue a warning after the driver 5 becomes unrecognizable and the time point when the automatic braking control starts.

[0057] When it is determined that a flash highlight has appeared in the infrared image, the setting unit 323 sets the waiting time to be longer than the reference time. The waiting time is the abnormal determination time between the time point when the driver 5 can no longer be recognized in the infrared image and the time point when it is determined that a posture distortion has occurred. The reference time is the reference time for the abnormal determination time, and for example, it is 10 seconds, but this is not the only example. In a specific example, when a flash highlight appears in the infrared image, the setting unit 323 sets the waiting time to 15 seconds, which is longer than the reference time (10 seconds).

[0058] The setting unit 323 sets the waiting time to the reference time when no flash highlight appears in the infrared image. The setting unit 323 sets the waiting time back to the reference time when it is determined that no flash highlight appears in the infrared image after it has been determined that a flash highlight has appeared in the infrared image. That is, the setting unit 323 sets the waiting time, which has become longer than the reference time, back to the reference time when the flash highlight in the infrared image has been resolved. Therefore, the setting unit 323 can appropriately set the waiting time according to whether a flash highlight appears in the infrared image.

[0059] In addition, the setting unit 323 can make the waiting time shorter than the reference time when the determination unit 322 cancels the determination that a flash highlight has appeared in the infrared image and determines that there is an abnormal posture distortion of the driver 5. In this case, the setting unit 323 sets the waiting time to 5 seconds, which is shorter than the reference time (10 seconds). By doing so, in the case of an abnormality of the driver 5, the setting unit 323 can issue a warning more quickly.

[0060] The notification control unit 324 causes the output unit 4 to issue a warning when the elapsed time since the driver 5 can no longer be recognized in the infrared image becomes equal to or longer than the waiting time set by the setting unit 323. Specifically, when the elapsed time is equal to or longer than the waiting time, the notification control unit 324 causes a message notifying of an abnormality to be issued as a warning. For example, the message notifying of an abnormality is a message such as "An abnormal driving posture has been observed. Please drive in a normal driving posture", but this is not the only example. When the driver 5 has performed an operation to stop the warning, the notification control unit 324 stops the warning.

[0061] When the notification control unit 324 can recognize the driver 5 in the infrared image before the elapsed time becomes equal to or longer than the waiting time, the notification control unit 324 does not cause the output unit 4 to issue a warning. Specifically, when the determination unit 322 determines that no flash highlight appears in the infrared image before the elapsed time becomes equal to or longer than the waiting time, the notification control unit 324 sets the elapsed time to zero and does not cause the output unit 4 to issue a warning before the elapsed time becomes equal to or longer than the waiting time again.

[0062] As described above, the notification control unit 324 issues a warning when the time elapsed since the driver 5 becomes unrecognizable in the infrared image is equal to or longer than the waiting time set by the setting unit 323. That is, since the waiting time is set longer than the reference time when there is a shining highlight in the infrared image, the time between the point in time when the driver 5 becomes unrecognizable in the infrared image and the point in time when the warning is issued becomes longer. If the positions of the sun and the building near the vehicle relative to the vehicle have changed during the waiting time before the warning is issued, and it is determined that the determination area 22 is no longer irradiated by the sunlight 6, the shining highlight in the infrared image is resolved. Since the driver 5 can be recognized when the shining highlight in the infrared image is resolved, the notification control unit 324 does not cause the output unit 4 to issue a warning. In this way, by making the time before the warning is issued longer when there is a shining highlight in the infrared image, the driving assistance device 3 can reduce false warnings.

[0063] In the case where an abnormality occurs to the driver 5, it is desired to quickly issue a warning even when there is a shining highlight in the infrared image. In view of this, if there is an abnormality in the driving operation performed by the driver 5, the driving assistance device 3 issues a warning. Hereinafter, the process of issuing a warning in the case where there is an abnormality in the driving operation will be described.

[0064] The acquisition unit 321 acquires operation information related to the driving operation performed by the driver 5. For example, the driving operation is at least any one of a steering operation, an accelerator operation, and a braking operation. As the operation information, the acquisition unit 321 sequentially acquires the sensed values sensed by each of a sensor that senses the rotation angle of the steering, a sensor that senses the depression amount of the accelerator pedal, and a sensor that senses the depression amount of the braking pedal.

[0065] Next, the determination unit 322 determines whether there is an abnormality in the driving operation represented by the operation information. For example, when the driving operation represented by the operation information is different from the driving operation performed by the driver 5 during normal times, the determination unit 322 determines that there is an abnormality in the driving operation represented by the operation information, where the operation information is obtained from the time point of a predetermined time before the current time point until the current time point. Specifically, first, the determination unit 322 determines the similarity that represents the degree of similarity between the driving operation represented by the acquired operation information and the driving operation performed by the driver 5 during normal times. Next, the determination unit 322 determines whether there is an abnormality in the driver 5 based on the similarity. The determination unit 322 determines that the driver 5 is normal when the determined similarity is within the allowable range, and the allowable range represents the driving operation that can be allowed as the driving operation performed by the driver 5 during normal times. The determination unit 322 determines that there is an abnormality in the driver 5 when the determined similarity is outside the allowable range.

[0066] The notification control unit 324 causes the output unit 4 to issue a warning when there is an abnormality in the driver 5. Specifically, the setting unit 323 causes the output unit 4 to issue a warning when it is determined that there is an abnormality in the driver 5 and the time elapsed since the driver 5 became unidentifiable in the infrared image is equal to or longer than the waiting time. Therefore, even when the driver 5 cannot be identified in the infrared image, the notification control unit 324 can issue a warning when the possibility of an abnormality in the current driver 5 is high.

[0067] [Processing for setting the waiting time]

[0068] Figure 5 is a flowchart showing an example of the processing for setting the waiting time. The processing for setting the waiting time is executed at a predetermined time interval while the vehicle engine is running. For example, the predetermined interval is 100 milliseconds, but this is not the only example.

[0069] The acquisition unit 321 acquires an infrared image (step S1). Specifically, the acquisition unit 321 acquires the infrared image captured after performing exposure correction from the image capture device 2 and the image capture time of the captured image.

[0070] The determination unit 322 determines whether the infrared image was captured during the period when the sunlight 6 irradiates the determination area 22 (step S2). For example, the determination unit 322 determines that the infrared image was captured during the period when the determination area 22 is irradiated by the sunlight 6 when the image capture time of the captured infrared image is included within a predetermined time period. The determination unit 322 determines that the infrared image was captured during the period when the determination area 22 is irradiated by the sunlight 6 when the image capture time is included within a first time period from sunrise until a predetermined time after sunrise or within a second time period from a predetermined time before sunset until sunset. The determination unit 322 determines that the infrared image was captured during the period when the determination area 22 is not irradiated by the sunlight 6 when the image capture time is included within a time period that is neither the first time period nor the second time period.

[0071] When it is determined that the infrared image was captured during the period when the determination area 22 is irradiated by the sunlight 6 (Yes at step S2), the determination unit 322 determines whether the pixel value of the pixels capturing the determination part 73 of the determination area 22 is equal to or greater than a predetermined value (step S3). For example, the determination unit 322 determines whether the statistic of the pixel values of a plurality of pixels included in the determination part 73 is equal to or greater than a predetermined value. Specifically, the determination unit 322 determines whether the average value of the pixel values of the plurality of pixels is equal to or greater than a predetermined value.

[0072] When it is determined that the pixel value of a pixel is equal to or greater than a predetermined value (Yes in step S3), the acquisition unit 321 acquires a captured image captured in a state where the infrared ray 11 is not irradiated (step S4). Specifically, the acquisition unit 321 stops irradiating the infrared lamp 1 with the infrared ray 11, and causes the image capturing device 2 to capture an image of the image capturing area 21 and generate the captured image.

[0073] The determination unit 322 determines whether the driver 5 can be recognized in the captured image (step S5). When the determination unit 322 cannot extract the facial features of the driver 5 from the captured image, it determines that the driver 5 cannot be recognized in the captured image. When the determination unit 322 can extract the facial features of the driver 5 from the captured image, it determines that the driver 5 can be recognized in the captured image.

[0074] When the driver 5 can be recognized in the captured image (Yes at step S5), the setting unit 323 makes the waiting time longer than the reference time (step S6). Specifically, when a bright highlight has appeared in the infrared image and the driver 5 can be recognized in the captured image, the setting unit 323 makes the waiting time longer than the reference time.

[0075] When it is determined that the infrared image is captured during a period when the determination area 22 is not irradiated by the sunlight 6 (No in step S2), the setting unit 323 sets the waiting time to the reference time (step S7). Further, when the pixel value of the pixels in the determination part 73 of the infrared image is less than the predetermined value (No at step S3) and the driver 5 cannot be recognized in the captured image (No at step S5), the setting unit 323 sets the waiting time to the reference time (step S7).

[0076] Note that clipping of the infrared image occurs not only when the determination area 22 is irradiated by the sunlight 6, but also when the hand of the driver 5 approaches the image capturing device 2. If the hand of the driver 5 approaches the image capturing device 2, the infrared ray 11 of the infrared lamp 1 is reflected by the hand of the driver 5, and the reflected light enters the image capturing device 2. In this case, the image capturing device 2 performs exposure correction to prevent the gray level of the hand of the driver 5 from being lost; as a result, an occlusion shadow in which the exposure amount of a part other than the hand of the driver 5 is undesirably reduced and the gray level of this part is lost appears. Even in such a case, the driving assistance device 3 according to the present embodiment can reduce false warnings by making the waiting time longer.

[0077] [Effect of the driving assistance device 3]

[0078] As described above, in a case where the driver 5 of the vehicle cannot be recognized in the infrared image, the driving assistance device 3 determines whether a glare highlight has occurred in the infrared image based on whether the pixel value of the pixels in the determined part irradiated by the sun 6 in the image capture area 21 of the infrared image is equal to or greater than a predetermined value, where the infrared image is captured when irradiating the image capture area 21 including the driver's seat in which the driver 5 is sitting with infrared rays 11. Then, in a case where it is determined that a glare highlight has occurred in the infrared image, the driving assistance device 3 makes the waiting time longer than the reference time, and the waiting time is the time between the time point when the driver 5 becomes unrecognizable in the infrared image and the time point when a warning is issued.

[0079] Since the waiting time is set to be longer than the reference time in a case where a glare highlight occurs in the infrared image, the time between the time point when the driver 5 becomes unrecognizable in the infrared image and the time point when a warning is issued becomes longer. If, before a warning is issued during the waiting time, the position of the sun relative to the vehicle and the influence of buildings and the like near the vehicle have changed, and the determination area 22 is no longer irradiated by the sunlight 6, then the glare highlight of the infrared image is resolved. Since the driver 5 can be recognized in a case where the glare highlight of the infrared image is resolved, a warning is not issued. In this way, by making the time until a warning is issued longer in a case where a glare highlight occurs in the infrared image, the driving assistance device 3 can reduce false warnings.

[0080] Although the present invention has been described so far using embodiments, the technical scope of the present invention is not limited to the description scope of the above-described embodiments, but various modifications and changes can be made within the scope of the gist. For example, all or part of the devices can be configured in any unit in a functionally or physically distributed / integrated manner. In addition, embodiments of the present invention also include new embodiments generated by any combination of a plurality of embodiments. The effects of the new embodiments generated by the combination have the effects of the original embodiments both.

[0081] Description of Reference Numerals

[0082] S Driving Assistance System

[0083] 1 Infrared Lamp

[0084] 2 Image Capture Device

[0085] 3 Driving Assistance Device

[0086] 4 Output Unit

[0087] 5 Driver

[0088] 31 Storage Unit

[0089] 32 Control Unit

[0090] 321 Acquisition Unit

[0091] 322 Determination Unit

[0092] 323 Setting Unit

[0093] 324 Notification Control Unit

Claims

1. A driving assistance device, comprising: an acquisition unit that acquires an infrared image captured when a first region is irradiated with infrared rays, the first region including a driver's seat on which a driver of a vehicle is seated; a determination unit that, when the driver cannot be recognized in the infrared image, determines whether a flash highlight has occurred in the infrared image based on whether a pixel value of a pixel in a determined portion in the infrared image is equal to or greater than a predetermined value, the determined portion capturing an image of a second region that is in the first region and is irradiated with sunlight; and a setting unit that, when it is determined that the flash highlight has occurred in the infrared image, makes a waiting time longer than a reference time, the waiting time being a time between a time point when the driver cannot be recognized in the infrared image and a time point when a warning is issued.

2. The driving assistance device according to claim 1, wherein the acquisition unit acquires a captured image captured in a state where the first region is not irradiated with the infrared rays when the determination unit determines that the pixel value of the pixel in the determined portion is equal to or greater than the predetermined value, and the determination unit determines that a flash highlight has occurred in the infrared image when the pixel value of the pixel in the determined portion is equal to or greater than the predetermined value and the driver can be recognized in the captured image.

3. The driving assistance device according to claim 1 or 2, wherein, The determination unit determines whether a flash highlight has occurred in the infrared image when an image capture time of the infrared image includes a time period during which the second region is irradiated with sunlight.

4. The driving assistance device according to claim 3, wherein The time period includes: a first time period from a sunrise time of an image capture date when the infrared image is captured until a time that is a predetermined time after the sunrise time; and a second time period from a time that is a predetermined time before a sunset time of the image capture date until the sunset time.

5. The driving assistance device according to claim 4, wherein, The determination unit increases the predetermined time as a time from the sunrise time of the image capture date until the sunset time decreases.

6. The driving assistance device according to claim 1 or 2, wherein, The acquisition unit acquires the infrared image captured after an image capture device that captures the infrared image has performed exposure correction to change at least one of an aperture value or an exposure time.

7. The driving assistance device according to claim 6, wherein the determination unit determines that no flash highlight has occurred in the infrared image when the pixel value of the pixel in the determined portion of the infrared image is less than the predetermined value, and the setting unit sets the waiting time to the reference time when the determination unit determines that no flash highlight has occurred in the infrared image.

8. The driving assistance device according to claim 1 or 2, wherein, The determination unit determines that a flash highlight has occurred in the infrared image when a statistic of pixel values of a plurality of pixels included in the determined portion is equal to or greater than the predetermined value, and determines that no flash highlight has occurred in the infrared image when the statistic is less than the predetermined value.

9. The driving assistance device according to claim 1 or 2, wherein, The waiting time is an abnormality determination time between a time point when the driver becomes unrecognizable in the infrared image and a time point when it is determined that an abnormality has occurred in which there is a deviation from the normal driving posture of the driver.

10. The driving assistance device according to claim 1 or 2, wherein, The determination unit determines that a flare highlight has occurred in the infrared image when a minimum value of pixel values of a plurality of pixels included in the determination part is greater than half of a maximum value of a signal that can be processed by an image capturing device that captures the infrared image and is equal to or greater than the predetermined value that is smaller than the maximum value.

11. The driving assistance device according to claim 1 or 2, wherein, The determination unit determines that no flare highlight has occurred in the infrared image when a maximum value of pixel values of a plurality of pixels included in the determination part is greater than half of a maximum value of a signal that can be processed by an image capturing device that captures the infrared image and is smaller than the predetermined value that is smaller than the maximum value.

12. The driving assistance device according to claim 1 or 2, wherein, the driving assistance device includes a notification control unit that issues a warning, the acquisition unit acquires operation information related to a driving operation performed by the driver, the determination unit determines whether the driving operation indicated by the operation information is abnormal, and the notification control unit issues a warning when the driving operation is abnormal and a time elapsed from when the driver becomes unrecognizable in the infrared image is equal to or longer than the waiting time set by the setting unit.

13. A driving assistance method executed by a computer of a vehicle, the driving assistance method including the following steps: acquiring an infrared image captured when a first area is irradiated with infrared rays, the first area including a driver's seat on which a driver of the vehicle is seated; when the driver cannot be recognized in the infrared image, determining whether a flare highlight has occurred in the infrared image based on whether pixel values of pixels in a determination part in the infrared image are equal to or greater than a predetermined value, the determination part capturing an image of a second area that is in the first area and is irradiated with sunlight; and when it is determined that the flare highlight has occurred in the infrared image, making the waiting time longer than a reference time, the waiting time being a time between a time point when the driver becomes unrecognizable in the infrared image and a time point when a warning is issued.

Images