Driving assistance devices, driving assistance methods

Abstract

To provide a driving support device and a driving support method that more effectively estimate eye fatigue and support safe driving.SOLUTION: The driving support device 10 according to the present disclosure comprises an acquisition unit 11, an image specification unit 12, a line of sight detection unit 13, an accumulation unit 14, and a notification unit 15. The acquisition unit 11 acquires an environmental image including a surrounding environment in a traveling direction of a vehicle and a facial image including the face of a driver. The image specification unit 12 specifies a position of an attention target on the environmental image. The line of sight detection unit 13 detects a line of sight direction of the driver. The accumulation unit 14 measures a gaze time of the driver and accumulates the measured gaze time when a position of the attention target on the environmental image overlaps with a gaze direction of the driver. When the accumulated gaze time exceeds a threshold value, the notification unit 15 notifies the driver of this fact.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a driving support device and a driving support method.

Background Art

[0002] In recent years, technologies for detecting the state of a driver during vehicle driving have been developed. For example, Patent Document 1 describes an information processing device that determines a driver's conscious state, such as distracted driving, concentration level, and fatigue level of the driver, based on the driver's line-of-sight direction. The information processing device disclosed in Patent Document 1 provides a reference for the line-of-sight direction for the driver in order to take into account individual differences in the driver's line-of-sight direction and specifies the line-of-sight direction.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The information processing device disclosed in Patent Document 1 described above enhances the detection accuracy of the driver's line-of-sight direction to perform risk prediction. However, such risk prediction alone is not sufficient, and a technology for more accurately estimating eye fatigue and supporting safe driving is desired.

[0005] The present disclosure has been made in view of such circumstances, and an object thereof is to provide a driving support device and a driving support method that more effectively estimate a driver's eye fatigue and support safe driving.

Means for Solving the Problems

[0006] The driver assistance device according to this disclosure is characterized by comprising: an acquisition unit that acquires an environmental image including the surrounding environment in the direction of travel of the vehicle and a face image including the face of the driver; an image identification unit that identifies a traffic safety sign or a person that is a target of attention based on the environmental image and identifies the position of the target of attention on the environmental image; a gaze detection unit that detects the direction of the driver's gaze based on the face image; an accumulation unit that measures the gazing time the driver gazes at the target of attention when the position of the target of attention on the environmental image and the direction of the driver's gaze overlap, and accumulates the measured gazing time; and a notification unit that notifies the driver when the accumulated gazing time exceeds a threshold.

[0007] The driving assistance device described herein can estimate eye fatigue caused by driving and support safe driving by accumulating the time during which the position of the object of attention in the image and the direction of the driver's gaze in the image overlap.

[0008] The driving assistance method according to this disclosure is characterized by acquiring an environmental image including the surrounding environment in the direction of travel of the vehicle and a facial image including the driver's face, identifying a traffic safety sign or a person or other object of attention based on the environmental image, determining the position of the traffic safety sign on the environmental image, detecting the driver's gaze direction based on the facial image, measuring the gazing time the driver gazes at the object of attention if the position of the object of attention on the environmental image and the driver's gaze direction overlap, accumulating the measured gazing time, and notifying the driver if the accumulated gazing time exceeds a threshold.

[0009] The driving assistance method described herein can support safe driving by accumulating the time during which the position of the object of attention in the image and the direction of the driver's gaze in the image overlap. [Effects of the Invention]

[0010] This disclosure makes it possible to provide a driving assistance device and driving assistance method that can more effectively estimate eye fatigue and support safe driving. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram illustrating the driver assistance device 10 according to Embodiment 1. [Figure 2] This figure shows the environmental image 105 acquired by the acquisition unit 11 according to Embodiment 1, divided into four sections. [Figure 3] This figure shows an example illustrating the gaze time accumulated by the accumulation unit 14 according to Embodiment 1. [Figure 4] This is a flowchart illustrating the driving assistance method according to Embodiment 1. [Figure 5] This is a block diagram illustrating the driver assistance device 20 according to Embodiment 2. [Figure 6] This figure shows the change in the coefficient corresponding to the degree of tension associated with the object of attention according to Embodiment 2. [Figure 7] This figure shows the change in the coefficient corresponding to the degree of tension associated with the object of attention according to Embodiment 2. [Figure 8] This figure shows the change in the coefficient according to the number of characters in the traffic safety display according to Embodiment 2. [Figure 9] This figure shows the change in the coefficient according to the number of characters in the traffic safety display according to Embodiment 2. [Figure 10] This figure shows the change in the amount of eye movement according to Embodiment 2. [Figure 11] This figure shows the change in the coefficient according to the amount of eye movement of the driver in Embodiment 2. [Figure 12] This figure shows the change in the coefficient according to the amount of eye movement of the driver in Embodiment 2. [Figure 13] This figure shows the change in the amount of eye movement according to Embodiment 2. [Figure 14] This figure shows the change in the coefficient corresponding to the degree of tension associated with the object of attention in Embodiment 2, and the change in the coefficient corresponding to the amount of eye movement of the driver. [Figure 15] This is a flowchart illustrating the driving assistance method according to Embodiment 2. [Figure 16] It is a block diagram illustrating the driving support device 30 according to Embodiment 3. [Figure 17] It is a diagram showing the change in the coefficient according to the driving state of the vehicle according to Embodiment 3. [Figure 18] It is a diagram showing the change in the coefficient according to the driving state of the vehicle according to Embodiment 3. [Figure 19] It is a flowchart illustrating the driving support method according to Embodiment 3.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant descriptions may be omitted as necessary for clarity of explanation. Also, some reference numerals are omitted so that the drawings do not become complicated.

[0013] (Embodiment 1) <Driving Support Device> First, the driving support device 10 according to Embodiment 1 will be described. FIG. 1 is a block diagram illustrating the driving support device 10 according to Embodiment 1. The driving support device 10 includes an acquisition unit 11, an image identification unit 12, a gaze detection unit 13, an accumulation unit 14, and a notification unit 15. The driving support device 10 is, for example, a drive recorder, an in-vehicle camera, or a portable device with a camera mounted inside the vehicle. The acquisition unit 11, the image identification unit 12, the gaze detection unit 13, and the accumulation unit 14 may be built into the driving support device 10, or may be a database placed in a communicable server (not shown).

[0014] The acquisition unit 11 acquires an environmental image, including the surrounding environment in the direction of the vehicle's movement, and a facial image, including the driver's face, from a camera (not shown). The environmental image includes, for example, road signs, traffic lights, bus stop signs, lane markings, roadways, sidewalks, road shoulders, buildings, moving objects such as vehicles ahead or oncoming vehicles, and images including pedestrians or people riding bicycles. The acquisition unit 11 may also acquire a still image, which may be a scene from a video. Furthermore, the image acquired using the acquisition unit 11 may be a single image containing both the environmental image and the facial image, or it may be multiple images containing both the environmental image and the facial image. A single image containing both the environmental image and the facial image is, for example, a 360-degree image. In addition, the acquisition unit 11 may acquire images from a camera installed outside the vehicle.

[0015] The image identification unit 12 uses a machine learning-based image recognition dictionary to identify that the object of attention is included in the environmental image and to identify the location of the object of attention on the environmental image. Here, the object of attention refers to an object, including traffic safety signs or people, that a driver should pay attention to when getting into a vehicle. Traffic safety signs are signs and equipment that clearly indicate traffic rules or warnings that drivers must strictly observe when driving. Typical examples of traffic safety signs are road signs, traffic lights, and bus stop signs installed on roadways or sidewalks. Other examples of traffic safety signs include road markings and lane markings displayed on roadways. The image identification unit 12 may also use other known techniques, such as pattern matching, to identify that the object of attention is included in the environmental image.

[0016] The method by which the image identification unit 12 identifies the location of the object of attention on the environmental image will be explained. Based on the environmental image, the image identification unit 12 identifies the location of the object of attention on the environmental image on a pixel-by-pixel basis or on a block-by-block basis of multiple pixels. The higher the number of pixels in the environmental image acquired by the acquisition unit 11, the more accurately the image identification unit 12 can identify the location of the object of attention on the environmental image.

[0017] Furthermore, the image identification unit 12 may divide the environmental image into multiple images and identify which of the divided images the object of attention belongs to. Figure 2 is a diagram showing the environmental image 105 acquired by the acquisition unit 11 according to Embodiment 1 divided into four parts. In Figure 2, the environmental image 105 acquired by the acquisition unit 11 is divided into four parts: environmental image 101, environmental image 102, environmental image 103, and environmental image 104. Traffic light D1 belongs to environmental image 104. In other words, the image identification unit 12 identifies that traffic light D1 is located in the upper right of environmental image 105. On the other hand, sign D2 belongs to environmental image 102. In other words, the image identification unit 12 identifies that sign D2 is located in the lower left of environmental image 105. In Figure 2, the environmental image 105 is divided into four parts, but it is not limited to this, and it is sufficient if it can be divided into at least two or more images.

[0018] The gaze detection unit 13 detects the driver's gaze direction based on the facial image. For example, the gaze detection unit 13 detects the gaze direction based on the positional relationship between the reference point (the inner corner of the eye) and the moving point (the iris), with the reference point being the inner corner of the eye and the moving point being the iris. As another example, the gaze detection unit 13 detects the gaze direction based on the positional relationship between the reference point and the moving point, with the reference point being the position of corneal reflection and the moving point being the pupil.

[0019] The accumulation unit 14 measures the gazing time of the driver when the position of the object of attention on the environmental image coincides with the driver's line of sight, and accumulates the measured gazing time. The accuracy of the determination of when the position of the object of attention on the environmental image coincides with the driver's line of sight can be set in advance, and a high accuracy is desirable. The driver gazing at the object of attention refers to the case where the position of the object of attention on the environmental image coincides with the driver's line of sight for a predetermined time, such as 0.2 seconds to 0.5 seconds or more, and the gazing time is measured by a timer or the like. Furthermore, it is preferable for the accumulation unit 14 to reset the accumulated gazing time when the driver finishes driving or when the driver takes a break. For example, the accumulation unit 14 may reset the accumulated time when power is not supplied to the driving support device 10. The accumulated time may also be reset when the driver performs an operation to instruct a reset on an input unit (not shown), or when a vehicle information acquisition unit (not shown) acquires the parking state of the vehicle for a predetermined time or longer.

[0020] Figure 3 is a diagram illustrating an example of the gaze time accumulated by the accumulation unit 14 according to Embodiment 1. Here, traffic safety signs d1 and d2 are traffic safety signs installed on the sidewalk 110. On the other hand, traffic safety signs d3, d4, and d5 are traffic safety signs written on the roadway 111. If the time that the driver of vehicle 100 gazes at traffic safety signs d1, d2, d3, d4, and d5 is time ta, time tb, time tc, time td, and time te, respectively, the accumulation unit 14 accumulates time ta, time tb, time tc, time td, and time te. That is, the accumulated time accumulated by the accumulation unit 14 is ta + tb + tc + td + te.

[0021] The notification unit 15 estimates that the cumulative gaze time accumulated by the accumulation unit 14 exceeds a threshold, and notifies the driver if the accumulated value of the driver's eye fatigue reaches a level requiring notification. The notification by the notification unit 15 is not limited to methods that the driver can perceive visually, but may also be methods that the driver can perceive auditorily. The notification by the notification unit 15 may also be a combination of methods that the driver can perceive visually and auditorily. The notification by the notification unit 15 may be, for example, a display and voice on a monitor or speaker suggesting a break, or it may be setting a rest point and setting it as a waypoint in the car navigation system. In addition, the notification by the notification unit 15 may also be a buzzer, flashing LED, or vibration. Furthermore, the output destination of the notification unit 15 is assumed to be a terminal such as a monitor or speaker, but it may also be a database or other device. The notification unit 15 may also notify a server of a management company that manages the operation of vehicles or a server of an insurance company via a communication unit (not shown).

[0022] Preferably, the threshold in the notification unit 15 is determined for each driver by managing the staring time and subjective evaluation of eye fatigue as a history. The notification unit 15 may also perform a five-level evaluation of the staring time accumulated by the accumulation unit 14. In this case, the notification unit 15 may set different notification methods for each level.

[0023] <Driving assistance methods> Next, the driving assistance method according to Embodiment 1 will be described. Figure 4 is a flowchart illustrating the driving assistance method according to Embodiment 1.

[0024] First, the acquisition unit 11 acquires an environmental image including the surrounding environment in the direction of vehicle travel and a face image including the driver's face (step ST1). Next, the image identification unit 12 identifies the object of attention based on the environmental image acquired by the acquisition unit 11 and identifies the position of the object of attention on the environmental image (step ST2). Next, the gaze detection unit 13 detects the direction of the driver's gaze based on the face image acquired by the acquisition unit 11 (step ST3). Next, the accumulation unit 14 determines whether the position of the object of attention on the environmental image identified by the image identification unit 12 and the direction of the driver's gaze identified by the gaze detection unit 13 overlap (step ST4). If they overlap (step ST4 YES), the accumulation unit 14 accumulates the gazing time of the driver (step ST5). On the other hand, if there is no overlap (step ST4NO), the process from step ST1 is repeated.

[0025] Next, the notification unit 15 determines whether the accumulated gaze time of the driver recorded by the accumulation unit 14 is equal to or greater than a threshold. If the accumulated gaze time of the driver recorded by the accumulation unit 14 is equal to or greater than the threshold (step ST6YES), the notification unit 15 notifies the driver (step ST7). On the other hand, if the accumulated driver's gaze time in the accumulation unit 14 is not equal to or greater than the threshold (step ST6NO), the process from step ST1 is repeated.

[0026] As described above, by using the driving assistance device according to Embodiment 1, the time during which the position of the object of attention on the environmental image and the direction of the driver's gaze on the facial image overlap can be accumulated, and eye fatigue due to driving can be estimated from the accumulated value of the time the driver was gazing at the object of attention. In this way, eye fatigue can be estimated more effectively, and safe driving can be supported.

[0027] (Embodiment 2) <Driving support system> Next, a driver assistance device 20 according to Embodiment 2 will be described. Figure 5 is a block diagram illustrating a driver assistance device 20 according to Embodiment 2. The driver assistance device 20 includes an acquisition unit 21, an image identification unit 22, a gaze detection unit 23, an accumulation unit 24, and a notification unit 25. The driver assistance device 20 is, for example, a drive recorder or in-vehicle camera installed in a vehicle, or a portable device with a camera. The acquisition unit 21, image identification unit 22, gaze detection unit 23, and accumulation unit 24 may be built into the driver assistance device 20, or they may be a database located on a communication-enabled server (not shown).

[0028] The acquisition unit 21 and notification unit 25 in the driver assistance device 20 are the same as those in the driver assistance device 10 according to Embodiment 1, so their description will be omitted. Here, the image identification unit 22, gaze detection unit 23, and accumulation unit 24 will be described.

[0029] In addition to identifying the object of attention, the image identification unit 22 further identifies the degree of tension associated with the object of attention, which is set in advance. The degree of tension associated with the object of attention is the degree to which a driver in a vehicle needs to concentrate when looking at the object of attention, and the degree of eye fatigue increases because the driver is more likely to stare at the object of attention. For example, a supplementary sign that specifies the day of the week or time of day requires a higher degree of concentration from the driver when looking at it compared to the main sign. As another example, a person riding a bicycle in front of a vehicle driven by a driver requires a higher degree of concentration from the driver when looking at it compared to a person walking on the sidewalk. The degree of tension associated with the object of attention may also be expressed using a score set up in hierarchical order. For example, a score set up in hierarchical order could be level 1 for the degree of tension associated with a traffic light, level 2 for the degree of tension associated with a stop sign, level 3 for the degree of tension associated with a speed limit sign, level 4 for the degree of tension associated with a no-entry sign, and level 5 for the degree of tension associated with a sign that specifies the day of the week or time of day. Furthermore, the level of tension associated with the object of attention may be expressed using a 10-point score. In addition, since the level of tension associated with the object of attention changes depending on the driver's familiarity with driving or the driver's eyesight, different settings may be used for each driver.

[0030] At this time, the accumulation unit 24 accumulates the time the driver gazes at the object of attention multiplied by a coefficient corresponding to the degree of tension associated with the object of attention. The coefficient corresponding to the degree of tension associated with the object of attention is a positive value and can be changed at the driver's discretion. For example, the driver may change the coefficient corresponding to the degree of tension associated with the object of attention once they become more accustomed to driving.

[0031] Next, we will explain the coefficient corresponding to the degree of tension associated with the object of attention. Figure 6 is a diagram showing the change in the coefficient corresponding to the degree of tension associated with the object of attention according to Embodiment 2. Here, the degree of tension associated with traffic lights is defined as Level 1, the degree of tension associated with traffic safety signs other than traffic lights is defined as Level 2, and the degree of tension associated with people is defined as Level 3. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 1 is set to 1.0. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 2 is set to 1.2. This means that even if the time spent gazing at a traffic light corresponding to Level 1 is the same as the time spent gazing at a traffic safety sign corresponding to Level 2, the degree of eye fatigue is estimated to be 1.2 times greater. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 3 is set to 1.5. Similarly, even if the time spent gazing at a traffic light corresponding to Level 1 is the same as the time spent gazing at a person corresponding to Level 3, the degree of eye fatigue is estimated to be 1.5 times greater.

[0032] At this time, assume that the vehicle driver gazed at each level of attention object in Figure 6 for the same amount of time t1. The cumulative time accumulated in the cumulative unit 24 is 1.0 × t1 + 1.2 × t1 + 1.5 × t1.

[0033] Figure 7 shows the change in the coefficient corresponding to the degree of tension associated with the object of attention according to Embodiment 2. Figure 7 shows the change in the coefficient corresponding to the degree of tension associated with the object of attention when the state of the traffic light is different, even though it is the same traffic light. Here, the degree of tension associated with the left-turn / straight-ahead traffic light is Level 1, the degree of tension associated with the right-turn traffic light is Level 2, and the degree of tension associated with the traffic light when the display changes is Level 3. Furthermore, the coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 1 is set to 1.0. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 2 is set to 1.2. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 3 is set to 1.5.

[0034] In this case, assume that the vehicle driver gazed at each level of attention object in Figure 7 for the same amount of time t2. The cumulative time accumulated in the cumulative unit 24 will be 1.0 × t2 + 1.2 × t2 + 1.5 × t2.

[0035] Thus, even if the object of attention is the same, if the state of that object differs, the vehicle driver can arbitrarily set a coefficient that corresponds to the degree of tension associated with that object of attention.

[0036] In other words, the coefficient corresponding to the degree of tension associated with the object of attention may be a value set in stages for each object of attention, or it may be a value set for each object of attention that has different states, even if it is the same object of attention.

[0037] Here, the level of tension associated with the object of attention may vary depending on the number of characters in the traffic safety sign. Examples of traffic safety signs include signs prohibiting entry by day of the week or time of day, construction signs, and traffic congestion signs on electronic billboards. Methods for identifying characters from an image may include using OCR (Optical Character Recognition) or by substituting similar characters with those present in the image. The number of characters in a traffic safety sign is determined by the number of characters identified. In this case, lowercase letters, diacritics, punctuation marks, and middle dots may be counted as one character or 0.5 characters.

[0038] Referring to Figures 8 and 9, we will explain a coefficient corresponding to the number of characters in a traffic safety sign as an example of a coefficient corresponding to the degree of tension associated with the object of attention. Figure 8 is a diagram showing the change in the coefficient corresponding to the number of characters in a traffic safety sign according to Embodiment 2. In the example shown in Figure 8, the coefficient corresponding to the number of characters in a traffic safety sign is set to increase in stages. Since the more characters a vehicle driver has in a traffic safety sign, the longer they will stare at the sign, the coefficient corresponding to the number of characters in a traffic safety sign also increases as the number of characters increases. Here, the coefficient corresponding to the number of characters in a traffic safety sign corresponding to less than 5 characters is set to 1.0. The coefficient corresponding to the number of characters in a traffic safety sign corresponding to 5 characters or more but less than 10 characters is set to 1.2. The coefficient corresponding to the number of characters in a traffic safety sign corresponding to 10 characters or more but less than 15 characters is set to 1.5.

[0039] In this case, suppose the vehicle driver stares at a traffic safety sign with 5 characters, a traffic safety sign with 10 characters, and a traffic safety sign with 15 characters for the same amount of time t2. The cumulative time accumulated by the cumulative unit 24 will be 1.0 × t2 + 1.2 × t2 + 1.5 × t2.

[0040] Here, the coefficients corresponding to the number of characters in the traffic safety display (5, 10, and 15 characters) are set to 1.0, 1.2, and 1.5, respectively. However, these coefficients can be arbitrarily changed as long as they are positive values. Furthermore, the upper and lower limits for the number of characters in the traffic safety display, which change in stages according to the coefficient, can be arbitrarily changed according to the driver's familiarity with driving or the driver's eyesight.

[0041] Figure 9 shows the change in the coefficient depending on the number of characters in the traffic safety display according to Embodiment 2. In the example shown in Figure 9, the coefficient depending on the number of characters in the traffic safety display is a value that increases in proportion to the number of characters in the traffic safety display. Here, when the number of characters in the traffic safety display increases by 5, the coefficient depending on the number of characters in the traffic safety display increases by 0.1.

[0042] In other words, the coefficient corresponding to the number of characters in the traffic safety display may be a value set in stages for each predetermined character count range, or it may be a value set to increase in proportion to the number of characters in the traffic safety display. Alternatively, the coefficient corresponding to the number of characters in the traffic safety display may be set to a constant value for some character count ranges, and the coefficient corresponding to the number of characters in the traffic safety display for other character count ranges may be set to a value that increases in proportion to the number of characters in the traffic safety display. For example, the coefficient corresponding to the number of characters in the traffic safety display may be set to a constant value for numbers of less than 5 characters, and the coefficient corresponding to the number of characters in the traffic safety display for numbers of 5 or more may be set to a value that increases in proportion to the number of characters in the traffic safety display.

[0043] The gaze detection unit 23 detects the driver's gaze direction based on the facial image, and further detects the amount of eye movement of the driver. The amount of eye movement is the amount of eye movement, and the maximum value of the eye movement occurs when the driver's gaze direction moves from the normal driving state, where the driver's gaze is directed in the direction of vehicle travel, to the left or right edge of the driver's field of view. The amount of eye movement can be measured using the positional relationship between the inner corner of the eye and the iris, or the positional relationship between the corneal reflection and the pupil. In addition, the amount of eye movement may be measured using a method that utilizes the difference in reflectivity between the cornea and the sclera, or by measuring the amount of eye movement using a television camera.

[0044] At this time, the accumulation unit 24 accumulates the time the driver stares at the vehicle multiplied by a coefficient corresponding to the amount of eye movement. The coefficient corresponding to the amount of eye movement is a positive value and can be changed at the driver's discretion. For example, the driver may change the coefficient corresponding to the amount of eye movement once they become accustomed to driving.

[0045] Figure 10 shows the change in eye movement according to Embodiment 2. Line C1 is a line parallel to the direction of travel of the vehicle 200 and passing through the driver's seat DS1 of the vehicle 200. Areas A2 and A4 are areas that are symmetrical with respect to line C1 as the axis of symmetry. Areas A3 and A5 are also areas that are symmetrical with respect to line C1 as the axis of symmetry. Traffic safety sign d11 is located in area A1. Traffic safety sign d12 is located in area A2. Traffic safety sign d13 is located in area A3.

[0046] In the example shown in Figure 10, the amount of eye movement when the driver of vehicle 200 looks at traffic safety sign d11 is the angle between the line connecting seat DS1 and traffic safety sign d11 and line C1. The amount of eye movement when the driver of vehicle 200 looks at traffic safety sign d12 is the angle between the line connecting seat DS1 and traffic safety sign d12 and line C1. The amount of eye movement when the driver of vehicle 200 looks at traffic safety sign d13 is the angle between the line connecting seat DS1 and traffic safety sign d13 and line C1. Here, the amount of eye movement of the driver of vehicle 200 increases in the order of eye movement when looking at traffic safety sign d11, eye movement when looking at traffic safety sign d12, and eye movement when looking at traffic safety sign d13. In other words, the amount of eye movement of the driver of vehicle 200 increases in the following order: when looking at the object of attention located in area A1, when looking at the objects of attention located in areas A2 and A4, and when looking at the objects of attention located in areas A3 and A5.

[0047] Figure 11 shows the change in the coefficient corresponding to the driver's eye movement amount according to Embodiment 2. In the example shown in Figure 11, the coefficient corresponding to the driver's eye movement amount is a value set in stages for each area. The driver's eye movement amount of vehicle 200 increases in the following order: eye movement amount when looking at the attention object located in area A1, eye movement amount when looking at the attention objects located in areas A2 and A4, and eye movement amount when looking at the attention objects located in areas A3 and A5. Therefore, the coefficient corresponding to the driver's eye movement amount also increases in the following order: area A1, area A2 and area A4, and area A3 and area A5. Here, the coefficient corresponding to the eye movement amount corresponding to area A1 is set to 1.0. The coefficient corresponding to the eye movement amount corresponding to areas A2 and A4 is set to 1.2. The coefficient corresponding to the eye movement amount corresponding to areas A3 and A5 is set to 1.5.

[0048] At this time, assume that the vehicle driver gazed at traffic safety signs d11, d12, and d13 in Figure 10 for the same amount of time t3. The cumulative time accumulated in the accumulation unit 24 will be 1.0 × t3 + 1.2 × t3 + 1.5 × t3.

[0049] Figure 12 shows the change in the coefficient corresponding to the driver's eye movement amount according to Embodiment 2. In the example shown in Figure 12, the coefficient corresponding to the driver's eye movement amount is a value that increases in proportion to the driver's eye movement amount. In the example shown in Figure 12, unlike the example shown in Figure 11, the coefficient corresponding to the driver's eye movement amount within the same area becomes larger as the distance from line C1 increases.

[0050] In other words, the coefficient corresponding to the driver's eye movement may be set in stages for each area, or it may be set to increase in proportion to the driver's eye movement. Alternatively, the coefficient corresponding to the driver's eye movement may be set to a constant value for some areas, while the coefficient corresponding to the driver's eye movement for other areas may increase in proportion to the driver's eye movement. For example, the coefficient corresponding to the driver's eye movement for area A1 may be set to a constant value, while the coefficients corresponding to the driver's eye movement for areas A2 and A4, and areas A3 and A5 may increase in proportion to the driver's eye movement. Furthermore, the coefficient corresponding to the driver's eye movement may be determined by the amount of change in the angle of the line of sight relative to the direction of vehicle travel, and the value of the coefficient may be determined exponentially.

[0051] Referring to Figures 13 and 14, an example will be described in which the accumulation unit 24 accumulates the driver's gazing time by multiplying it by both a coefficient corresponding to the degree of tension associated with the object of attention and a coefficient corresponding to the amount of eye movement of the driver. Figure 13 is a diagram showing the change in eye movement amount according to Embodiment 2. Line C11 is a line that is parallel to the direction of travel of vehicle 300 and passes through the driver's seat DS11 of vehicle 200. Area A22 and Area A33 are areas that are symmetrical with respect to line C11 as the axis of symmetry. The left-turn / straight signal d21 is located in Area A11. The signal d22 when the display changes is located in Area A22.

[0052] Figure 14 shows the changes in the coefficient corresponding to the degree of tension associated with the object of attention and the change in the coefficient corresponding to the amount of eye movement of the driver, according to Embodiment 2. Here, the degree of tension associated with the left-turn / straight signal d21 is defined as Level 1, and the degree of tension associated with the signal d22 when the display changes is defined as Level 2. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 1 is set to 1.0. The coefficient corresponding to the degree of tension associated with the object of attention corresponding to Level 2 is set to 1.5. The coefficient corresponding to the amount of eye movement corresponding to Area A11 is set to 1.0. The coefficient corresponding to the amount of eye movement corresponding to Areas A22 and A33 is set to 1.5.

[0053] At this time, assume that the vehicle driver stared at the traffic safety indicators for the same amount of time t4, both the left-turn / straight signal d21 and the signal d22 when the indicator changed, as shown in Figure 13. The cumulative time accumulated by the accumulation unit 24 is 1.0 × 1.0 × t4 + 1.5 × 1.5 × t4.

[0054] Thus, the accumulation unit 24 can also accumulate the driver's gazing time by multiplying it by both a coefficient corresponding to the degree of tension associated with the object of attention and a coefficient corresponding to the amount of eye movement of the driver.

[0055] <Driving assistance methods> Next, a driving assistance method according to Embodiment 2 will be described. Figure 15 is a flowchart illustrating the driving assistance method according to Embodiment 2.

[0056] First, the acquisition unit 21 acquires an environmental image including the surrounding environment in the direction of vehicle travel and a face image including the driver's face (step ST01). Next, the image identification unit 22, based on the environmental image acquired by the acquisition unit 21, identifies the object of attention and further identifies the degree of tension associated with the object of attention, which is set in advance, and identifies the position of the object of attention on the environmental image (step ST02). Next, the gaze detection unit 23, based on the face image acquired by the acquisition unit 21, detects the driver's gaze direction and further detects the amount of gaze movement of the driver (step ST03). Next, the accumulation unit 24 determines whether the position of the object of attention on the environmental image identified by the image identification unit 22 and the driver's gaze direction identified by the gaze detection unit 23 overlap (step ST04). If they overlap (step ST04YES), the accumulation unit 24 accumulates the gazing time of the driver (step ST05). At this time, the accumulation unit 24 accumulates the driver's gazing time by multiplying it by both a coefficient corresponding to the degree of tension associated with the object of attention and a coefficient corresponding to the amount of eye movement of the driver. On the other hand, if there is no overlap (step ST04NO), the process from step ST01 is repeated.

[0057] Next, the notification unit 25 determines whether the accumulated staring time of the driver recorded by the accumulation unit 24 is equal to or greater than a threshold. If the accumulated staring time of the driver recorded by the accumulation unit 24 is equal to or greater than the threshold (step ST06YES), the notification unit 25 notifies the driver (step ST07). On the other hand, if the accumulated driver's gaze time in the accumulation unit 24 is not equal to or greater than the threshold (step ST06NO), the process from step ST01 is repeated.

[0058] In the example shown in Figure 15, the image identification unit 22 identifies the object of attention based on the environmental image, and further identifies the degree of tension associated with the object of attention, which is set in advance. The gaze detection unit 23 detects the driver's gaze direction and further detects the amount of the driver's gaze movement based on the face image. At this time, the accumulation unit 24 accumulates the driver's gaze time by multiplying the driver's gaze time by both a coefficient corresponding to the degree of tension associated with the object of attention and a coefficient corresponding to the amount of the driver's gaze movement. The image identification unit 22 identifies the object of attention based on the environmental image, and further identifies the degree of tension associated with the object of attention, which is set in advance, to determine its position on the image of the traffic safety display. The gaze detection unit 23 may then detect the driver's gaze direction based on the face image. At this time, the accumulation unit 24 accumulates the driver's gaze time by multiplying the driver's gaze time by a coefficient corresponding to the degree of tension associated with the object of attention. Furthermore, the image identification unit 22 identifies the object of attention based on the environmental image and determines the position of the object of attention on the environmental image. The gaze detection unit 23 then detects the driver's gaze direction based on the face image and may further detect the amount of the driver's gaze movement. At this time, the accumulation unit 24 accumulates the driver's gaze time by multiplying the driver's gaze time by a coefficient corresponding to the amount of the driver's gaze movement.

[0059] Thus, by using the driving assistance device according to Embodiment 2, it is possible to estimate eye fatigue caused by the amount of eye movement during driving or the degree of tension associated with the object of attention. In this way, eye fatigue of the driver can be estimated more effectively, and safe driving can be supported.

[0060] (Embodiment 3) <Driving support system> Next, a driver assistance device 30 according to Embodiment 3 will be described. Figure 16 is a block diagram illustrating a driver assistance device 30 according to Embodiment 3. The driver assistance device 30 includes a vehicle status acquisition unit 36, an acquisition unit 31, an image identification unit 32, a gaze detection unit 33, an accumulation unit 34, and a notification unit 35. The driver assistance device 30 is, for example, a drive recorder or in-vehicle camera installed in a vehicle, or a portable device with a camera. The vehicle status acquisition unit 36, the acquisition unit 31, the image identification unit 32, the gaze detection unit 33, and the accumulation unit 34 may be built into the driver assistance device 30, or they may be a database located on a communication-enabled server (not shown).

[0061] The acquisition unit 31, image identification unit 32, gaze detection unit 33, accumulation unit 34, and notification unit 35 in the driver assistance device 30 are the same as those in the driver assistance device 10 according to Embodiment 1 and the driver assistance device 20 according to Embodiment 2, so their description will be omitted. Here, we will describe the vehicle status acquisition unit 36.

[0062] The vehicle status acquisition unit 36 ​​acquires the vehicle's driving status. The vehicle's driving status includes the state of the vehicle itself and the state outside the vehicle. The state of the vehicle itself is, for example, when parking, when turning right, when turning left, when driving at high speed, and when driving at low speed. The state outside the vehicle is, for example, weather and road conditions. Road conditions are, for example, road surface condition and lane width. The vehicle's driving status may also be represented using a hierarchical scoring system. For example, the hierarchical scoring system might assign Level 1 to the vehicle's driving status when driving at low speed, Level 2 to the vehicle's driving status when turning left, Level 3 to the vehicle's driving status when turning right, Level 4 to the vehicle's driving status when parking, and Level 5 to the vehicle's driving status when driving at high speed. Alternatively, the vehicle's driving status may also be represented using a 10-level scoring system.

[0063] At this time, the accumulation unit 34 accumulates the gazing time, which is calculated by multiplying the time the driver gazes at the object of attention by a coefficient that corresponds to the vehicle's driving state. The coefficient that corresponds to the vehicle's driving state is a positive value and can be changed at the driver's discretion. For example, the driver may change the coefficient that corresponds to the vehicle's driving state once they become more accustomed to driving. In addition, the driver of the vehicle can decrease the coefficient for driving conditions in which they are good at driving and increase the coefficient for driving conditions in which they are not good at driving.

[0064] Next, we will explain the coefficients that correspond to the vehicle's driving conditions. Figure 17 is a diagram showing the change in the coefficients that correspond to the vehicle's driving conditions according to Embodiment 3. Here, the coefficient corresponding to the vehicle's driving conditions, corresponding to parking, is set to 1.5. The coefficient corresponding to the vehicle's driving conditions, corresponding to left turns, is set to 1.0. The coefficient corresponding to high driving speeds, corresponding to high driving speeds, is set to 1.5.

[0065] In this case, assume that the vehicle driver gazed at the traffic safety sign for the same amount of time t5 during the parking maneuver, left turn, and high-speed driving shown in Figure 17. The cumulative time accumulated in the cumulative unit 34 will be 1.5 × t5 + 1.0 × t5 + 1.5 × t5.

[0066] Figure 18 shows the change in the coefficient according to the vehicle's driving conditions according to Embodiment 3. Figure 18 shows the change in the coefficient according to the vehicle's driving conditions when the parking situation is different, even during the same parking operation. In the example shown in Figure 18, the coefficient according to the vehicle's driving conditions when parked on the street is set to 1.5. The coefficient according to the vehicle's driving conditions when parked in a home garage is set to 1.0. The coefficient according to the vehicle's driving conditions when parked parallel is set to 1.5.

[0067] At this time, the vehicle driver ensures that traffic safety is maintained for the same amount of time t6 in each state shown in Figure 18. Assume the display is stared at intently. The cumulative time accumulated by the accumulation unit 34 is 1.5 × t6 + 1.0 × t6 + 1.5 × t6.

[0068] Thus, even when the same vehicle is in motion, the situation can differ, and the driver of the vehicle can arbitrarily set a coefficient that corresponds to the vehicle's motion.

[0069] In other words, the coefficient corresponding to the vehicle's driving conditions may be a value set in stages for each driving condition of the vehicle, or it may be a value set for each different situation even for the same vehicle's driving conditions.

[0070] In the driver assistance device 30 according to Embodiment 3, the gaze detection unit 33 detects the driver's gaze direction based on a facial image and may further detect the amount of the driver's gaze movement. In this case, the accumulation unit 34 can also accumulate the driver's gaze time by multiplying the driver's gaze time by both a coefficient corresponding to the vehicle's driving state and a coefficient corresponding to the amount of the driver's gaze movement.

[0071] <Driving assistance methods> Next, a driving assistance method according to Embodiment 3 will be described. Figure 19 is a flowchart illustrating the driving assistance method according to Embodiment 3.

[0072] First, the vehicle status acquisition unit 36 ​​acquires the vehicle's driving status (step ST001). The acquisition unit 31 acquires an environmental image including the surrounding environment in the direction of travel of the vehicle, and a face image including the driver's face (step ST002). Next, the image identification unit 32 identifies the object of attention based on the environmental image acquired by the acquisition unit 31 and determines the position of the object of attention on the environmental image (step ST003). Next, the gaze detection unit 33 further detects the amount of the driver's gaze movement based on the face image acquired by the acquisition unit 31 (step ST004). Next, the accumulation unit 34 determines whether the position of the object of attention on the environmental image determined by the image identification unit 32 and the direction of the driver's gaze determined by the gaze detection unit 33 overlap (step ST005). If they overlap (step ST005 YES), the accumulation unit 34 accumulates the gazing time of the driver (step ST006). At this time, the accumulation unit 34 accumulates the driver's gaze time, which is calculated by multiplying the gaze time the driver has spent looking at the vehicle by a coefficient that corresponds to the vehicle's driving conditions. On the other hand, if there is no overlap (step ST005NO), the process from step ST001 is repeated.

[0073] Next, the notification unit 25 determines whether the accumulated staring time of the driver recorded by the accumulation unit 34 is equal to or greater than a threshold. If the accumulated staring time of the driver recorded by the accumulation unit 34 is equal to or greater than a threshold (step ST007YES), the notification unit 25 notifies the driver (step ST008). On the other hand, if the accumulated gaze time of the driver recorded by the accumulation unit 34 is not equal to or greater than the threshold (step ST007NO), the process from step ST01 is repeated.

[0074] Thus, by using the driver assistance device according to Embodiment 3, it is possible to estimate eye fatigue caused by the vehicle's driving conditions during operation. In this way, eye fatigue of the driver can be estimated more effectively, and safe driving is supported.

[0075] The driver assistance devices according to Embodiments 1 to 3 are not limited to a single physical device. In other words, the driver assistance device may be distributed across multiple devices. For example, the acquisition unit 11, image identification unit 12, gaze detection unit 13, accumulation unit 14, etc., may be composed of a single physical personal computer, and the notification unit 15 may be composed of a tablet terminal.

[0076] Furthermore, some or all of the processing in the aforementioned driver assistance devices 10, 20, and 30 can be implemented as computer programs. Such programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0077] Although the present disclosure has been described in accordance with the above embodiments, the present disclosure is not limited to the configuration of the above embodiments, and of course includes various modifications, alterations, and combinations that a person skilled in the art could make within the scope of the claims of the present patent application.

[0078] This disclosure includes matters that contribute to achieving the Sustainable Development Goal (SDG) of "Ensure healthy lives and promote well-being for all" and to creating value through healthcare products and services. [Explanation of Symbols]

[0079] 10, 20, 30 Driving assistance devices 11, 21, 31 Acquisition part 12, 22, 32 Image Identification Section 13, 23, 33 Eye-line detection unit 14, 24, 34 Cumulative section 15, 25, 35 Hochi Department 36 Vehicle status acquisition unit 100, 200, 300 vehicles 101, 102, 103, 104, 105 Environmental images 110 Sidewalk 111 Roadway Areas A1, A2, A3, A4, A5, A11, A22, A33 D1 signal D2 sign d1, d2, d3, d4, d5, d11, d12, d13 traffic safety display d21 Left turn / straight signal d22 Traffic lights when the display changes DS1, DS11 driver's seat C1, C11 line

Claims

1. An acquisition unit that acquires environmental images including the surrounding environment in the direction of vehicle travel and facial images including the driver's face, An image identification unit identifies traffic safety signs or objects of attention, including people, based on the aforementioned environmental image, and identifies the location of the objects of attention on the environmental image. A gaze detection unit detects the driver's gaze direction based on the aforementioned facial image, The accumulation unit measures the time during which the position of the object of attention on the environmental image and the direction of the driver's gaze overlap, and if the measured time is longer than a predetermined time, it is determined that the driver gazed at the object of attention, and the measured time is accumulated as the gazing time. The system includes a notification unit that notifies the driver that fatigue is accumulating when the accumulated staring time exceeds a threshold. Driving assistance system.

2. The gaze detection unit further detects the amount of the driver's gaze movement, The accumulation unit accumulates the time obtained by multiplying the time the driver stares by a coefficient corresponding to the amount of eye movement. The driving support device according to claim 1.

3. The image identification unit further identifies the tension level associated with the pre-set attention target, The accumulation unit accumulates the time obtained by multiplying the time the driver stares at the object of attention by a coefficient corresponding to the degree of tension associated with the object of attention. The driving support device according to claim 1 or 2.

4. The vehicle further comprises a vehicle status acquisition unit that acquires the driving status of the aforementioned vehicle, The accumulation unit accumulates the time obtained by multiplying the time the driver gazes at the vehicle by a coefficient corresponding to the vehicle's driving state. The driving support device according to claim 1 or 2.

5. The system acquires environmental images including the surrounding environment in the direction of vehicle travel, and facial images including the driver's face. Based on the aforementioned environmental image, traffic safety signs or objects of attention including people are identified, and the location of the objects of attention on the environmental image is determined. Based on the aforementioned facial image, the direction of the driver's gaze is detected, The time during which the position of the object of attention on the environmental image and the driver's line of sight overlap is measured, and if the measured time is longer than a predetermined time, the measured time is accumulated as the gazing time. If the accumulated staring time exceeds a threshold, the driver is notified that fatigue is accumulating. Driving assistance methods.

Images