Driving assistance device, driving assistance processing method, and recording medium

Abstract

This driving assistance device: detects driving scenes in which there is a predetermined risk object that obstructs vehicle movement; sets a reference visual recognition pattern, which includes visual recognition points to be visually recognized by a driver, a visual recognition sequence, and visual recognition timings, according to the driving scenes; determines the driver's attention and awareness state by comparing the driver's visual recognition characteristics, which include the driver's visual recognition points, visual recognition sequence, and visual recognition timings as learned for each driving scene in which the risk object exists, with the reference visual recognition pattern; and warns the driver.

Description

Driving assistance device, driving assistance processing method, and recording medium

[0001] The present disclosure relates to a driving assistance device, a driving assistance processing method, and a recording medium.

[0002] A vehicle is equipped with various sensors for detecting obstacles present around the vehicle. When a risk object requiring attention, such as an obstacle, is detected by these sensors, a technology is known that notifies the driver of the presence of the risk object.

[0003] For example, Patent Document 1 discloses an object notification system that includes a peripheral object detection device that detects objects present around a vehicle, an alarm device that alerts the driver to the presence of an object detected by the peripheral object detection device, and a control unit, wherein the control unit functions as a gaze detection unit that sequentially detects the line of sight of the vehicle driver, and an alarm control unit that sets an alarm range by excluding the recognition direction range determined based on the line of sight detected by the gaze detection unit from a standard alarm range determined based on the object detection range of the peripheral object detection device, and that causes the alarm device to alert the driver of the presence of an object based on the presence of an object detected by the peripheral object detection device within the alarm range.

[0004] Furthermore, Patent Document 2 discloses a driving assistance system that determines the driver's state by creating a frequency distribution of the driver's gaze time during a judgment time based on the results of sequentially detecting the gaze direction of the vehicle driver, and comparing the frequency distribution of the driver's gaze time during the judgment time with models of the frequency distribution of gaze time corresponding to each of a concentration state, a absent-minded state, and a state of impatience.

[0005] JP 2019-117668 A JP 2018-49486 A

[0006] However, the technology described in Patent Document 1 has a configuration that alerts the driver to the presence of an obstacle when the driver's gaze is not directed in the direction of the obstacle, but because the presence or absence of an alert is determined based on whether the driver is looking in the direction of the obstacle or not, there is a risk that it will not be possible to reduce both annoyance and the possibility of a collision.

[0007] Furthermore, the technology described in Patent Document 2 has a configuration for determining whether the driver is in a concentrated state, absent-minded state, or impatient state based on the frequency distribution of the driver's gaze time, but it does not have a configuration for determining at what timing and in what order the driver moves their eyes and pays attention to their surroundings. Therefore, even if the deviation between the frequency distribution of gaze time and the frequency distribution model is small, it cannot be determined that the driver is looking at the gaze target at the appropriate timing and in the appropriate order.

[0008] The present disclosure has been made in consideration of the above-mentioned problems, and an object of the present disclosure is to provide a driving assistance device, a driving assistance processing method, and a recording medium that can assist a driver in visually checking the visual points that should be checked at an appropriate time and in an appropriate order according to a driving scene in which a risk object is present.

[0009] In order to solve the above problem, according to an aspect of the present disclosure, there is provided a driving assistance device that assists in driving a vehicle, the driving assistance device including one or more processors, one or more memories communicably connected to the one or more processors, and a visual characteristic storage unit that records information on visual characteristics including a driver's viewing location, a viewing order, and a viewing timing, learned for each driving scene in which a predetermined risk object that will hinder movement of the vehicle is present, wherein the one or more processors execute the following: a driving scene detection process that detects a driving scene in which the risk object is present; a reference visual pattern setting process that sets a reference visual pattern including a viewing location, a viewing order, and a viewing timing that the driver should view according to the driving scene; an attention awareness state determination process that compares the visual characteristics of the driver in the driving scene, which are identified from the driver's visual characteristics recorded in the visual characteristic storage unit, with the reference visual pattern, to determine the attention awareness state of the driver; and a warning process that issues a warning regarding the risk object or the viewing location.

[0010] Furthermore, in order to solve the above-mentioned problem, according to another aspect of the present disclosure, there is provided a driving assistance processing method for assisting driving of a vehicle by a computer, wherein the computer executes a driving scene detection process for detecting a driving scene in which a predetermined risk object that will hinder movement of the vehicle is present; a reference visual pattern setting process for setting a reference visual pattern including a visual location, a visual order, and a visual timing that the driver should look at according to the driving scene; an attention awareness state determination process for determining the attention awareness state of the driver by comparing the reference visual pattern with the visual characteristics of the driver in the driving scene identified from information on the driver's visual characteristics including the driver's visual location, the visual order, and the visual timing that is learned for each driving scene in which the risk object is present; and a warning process for issuing a warning to the driver based on the attention awareness state.

[0011] Furthermore, according to another aspect of the present disclosure to solve the above problem, there is provided a non-transitory tangible recording medium having recorded thereon a computer program that causes a computer to execute the following: a driving scene detection process that detects a driving scene in which a predetermined risk object that will hinder the movement of the vehicle is present; a reference visual pattern setting process that sets a reference visual pattern including a visual location, a visual order, and a visual timing that the driver should look at according to the driving scene; an attention awareness state determination process that determines the attention awareness state of the driver by comparing the visual characteristics of the driver in the driving scene, which are identified from information on the driver's visual characteristics including the driver's visual location, the visual order, and the visual timing that is learned for each driving scene in which the risk object is present, with the reference visual pattern; and a warning process that issues a warning to the driver based on the attention awareness state.

[0012] As described above, according to the present disclosure, the driver can be made to view the areas that need to be viewed at the appropriate time and in the appropriate order according to the driving scene in which a risk object is present, thereby appropriately allocating the driver's attention resources.

[0013] 1 is an explanatory diagram showing an example of a transition (visual recognition pattern) of a driver's visual recognition location; FIG. 2 is an explanatory diagram showing an example of a transition (visual recognition pattern) of a driver's visual recognition location of another driver; FIG. 3 is a schematic diagram showing an example of a configuration of a vehicle equipped with a driving assistance device according to an embodiment of the present disclosure; FIG. 4 is a block diagram showing an example of a configuration of the driving assistance device according to the same embodiment; FIG. 5 is an explanatory diagram illustrating a reference visual recognition pattern used in the same embodiment; FIG. 6 is an explanatory diagram illustrating a visual recognition characteristic of a driver; FIG. 7 is a flowchart showing a routine of driving assistance processing by the driving assistance device according to the same embodiment; FIG. 8 is an explanatory diagram showing an example of a determination process of a driver's attention and awareness state by the driving assistance device according to the same embodiment; FIG. 9 is an explanatory diagram showing an example of a determination process of a driver's attention and awareness state by the driving assistance device according to the same embodiment; FIG. 10 is an explanatory diagram showing an example of a warning process based on a transition of a driver's actual visual recognition location by the driving assistance device according to the same embodiment; FIG. 11 is an explanatory diagram showing a driving scene of another application example of the same embodiment; FIG. 12 is an explanatory diagram showing an example of a determination process of a driver's attention and awareness state by another application example of the same embodiment;

[0014] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0015] In this specification, the term "risk object" includes not only obstacles that may collide with a vehicle, but also objects or events that may be obstacles to the vehicle or the driver of the vehicle.

[0016] <1. Overview of the Present Disclosure> First, an overview of the present disclosure will be described. When passing through driving situations where there is a risk of pedestrians or other vehicles jumping out, or contact with an obstacle, such as when passing beside a blind spot, when passing through an intersection without traffic lights, or when parking a vehicle in a position where there are other vehicles or side walls nearby, the driver needs to drive while paying attention to their surroundings. However, depending on the driver's driving skill, there are differences in the driver's understanding of where to pay attention and direct their gaze in each driving situation, and in what order and timing to direct their gaze to each visual location.

[0017] 1 and 2 are explanatory diagrams showing an example of a transition (visual pattern) of a driver's visual location. FIGS. 1 and 2 show the visual pattern of the driver in a driving scene in which a vehicle passes by a parked vehicle 90 that is located at the edge of the road and extends into the driving lane. The example of the driver's visual pattern shown in FIGS. 1 and 2 shows the visual location of the driver according to a change in the distance from the vehicle to the parked vehicle 90. A driver is present in the parked vehicle 90, and possible risks include "the driver opening the door," "the parked vehicle 90 starting," and "a pedestrian or the like jumping out from the shadow (blind spot) of the parked vehicle 90."

[0018] In the example shown in FIG. 1 , when the distance D from the vehicle to the parked vehicle 90 is 47 m and 41 m, the driver visually recognizes the periphery A1 of the window on the driver's side of the parked vehicle 90 for 0.5 seconds. Furthermore, when the distance D is 34 m, the driver visually recognizes the periphery A2 of the taillight of the parked vehicle 90 for 0.05 seconds. Furthermore, when the distance D is 23 m, the driver visually recognizes the rear of the parked vehicle 90 through a gap A3 under the floor of the parked vehicle 90 for 0.1 seconds. Furthermore, when the distance D is 17 m, the driver again visually recognizes the periphery A1 of the window on the driver's side of the parked vehicle 90 for 0.5 seconds. Furthermore, when the distance D is 10 m, the driver visually recognizes the vicinity A4 of the right corner A4 of the front bumper of the parked vehicle 90 for 0.1 seconds. Furthermore, when the distance D is 5 m, the driver again visually recognizes the periphery A1 of the window on the driver's side of the parked vehicle 90 for 0.5 seconds.

[0019] 1, the driver repeatedly looks at the area around the driver's seat of the parked vehicle 90, and also looks at the area around the driver's seat of the parked vehicle 90 again when the vehicle 1 is closest to the parked vehicle 90. In addition, the driver pays attention to the lighting status of the tail lamps and blind spots, and it can be said that the driver is transitioning the area he looks at in an appropriate order and at an appropriate timing.

[0020] On the other hand, in the example shown in FIG. 2 , when the distance D from the vehicle to the parked vehicle 90 is 47 m and 41 m, the driver visually recognizes the periphery A1 of the window on the driver's side door of the parked vehicle 90. Furthermore, when the distance D is 34 m, the driver visually recognizes the periphery A2 of the tail lamp of the parked vehicle 90. Furthermore, when the distance D is 22 m, the driver visually recognizes the rear of the parked vehicle 90 through a gap A3 under the floor of the parked vehicle 90. Furthermore, when the distance D is 13 m, the driver visually recognizes the vicinity A4 of the right corner of the front bumper of the parked vehicle 90. Furthermore, when the distance D is 7 m, the driver visually recognizes the periphery A1 of the window on the driver's side of the parked vehicle 90 again.

[0021] 2, the driver visually checks the area around the driver's seat of the parked vehicle 90, but the time spent visually checking the area around the driver's seat is short, and the timing of the visual check is too early, so the driver does not visually check the area around the driver's seat of the parked vehicle 90 when the vehicle is closest to the parked vehicle 90. Therefore, if the driver of the parked vehicle 90 opens the driver's seat door, the driver of the vehicle may be late in noticing this, which could result in a collision.

[0022] In this way, it is considered that there are viewing locations, viewing orders, and viewing timings that should be viewed for each driving scene where a risk object exists. In the present disclosure, the driving assistance device sets a standard viewing pattern including viewing locations, viewing orders, and viewing timings that should be viewed according to the driving scene, and assists the driver of the vehicle to transition between viewing locations in accordance with the standard viewing pattern. This allows the driver's attention resources to be appropriately allocated, reducing the driver's workload and the risk of a collision, etc.

[0023] 2. Overall Configuration of Vehicle First, an example of the overall configuration of a vehicle equipped with a driving assistance device according to an embodiment of the present disclosure will be described.

[0024] 3 is a schematic diagram showing an example of the configuration of the vehicle 1. The vehicle 1 is equipped with front imaging cameras 31LF and 31RF, a vehicle state sensor 33, an interior imaging camera 37, and a notification device 43.

[0025] The front imaging cameras 31LF, 31RF constitute ambient environment sensors for acquiring information about the ambient environment of the vehicle 1. The front imaging cameras 31LF, 31RF capture images of the area ahead of the vehicle 1 and generate image data. The front imaging cameras 31LF, 31RF are equipped with imaging elements such as CCDs (Charged Coupled Devices) or CMOS (Complementary Metal Oxide Semiconductors), and transmit the generated image data to the driving assistance device 50. In the vehicle 1 shown in FIG. 3 , the front imaging cameras 31LF, 31RF are configured as stereo cameras including a pair of left and right cameras, but the front imaging cameras may also be monocular cameras.

[0026] In addition to the front-facing cameras 31LF, 31RF, the surrounding environment sensor may include, for example, a camera that captures an image behind the vehicle 1, or a camera that is provided on a side mirror and captures an image behind the left or right of the vehicle 1. In addition, the surrounding environment sensor may include one or more sensors selected from the group consisting of a radar sensor such as a light detection and ranging (LiDAR) or a millimeter-wave radar, and an ultrasonic sensor.

[0027] The vehicle state sensor 33 is composed of at least one sensor that detects the operating state and behavior of the vehicle 1. The vehicle state sensor 33 includes, for example, at least one of a steering angle sensor, an accelerator position sensor, a brake stroke sensor, a brake pressure sensor, or an engine rotation speed sensor. The vehicle state sensor 33 also includes, for example, at least one of a vehicle speed sensor, an acceleration sensor, or an angular velocity sensor. The vehicle state sensor 33 transmits a sensor signal indicating the detected information to the driving assistance device 50.

[0028] The interior camera 37 captures images of the interior of the vehicle and generates image data. The interior camera 37 is equipped with an imaging element such as a CCD or CMOS, and transmits the generated image data to the driving assistance device 50. The interior camera 37 is installed in a position where it can capture an image of at least the face of the driver.

[0029] The notification device 43 is driven by the driving assistance device 50 and notifies the driver of various information by means of image display, audio output, etc. The notification device 43 includes, for example, a display device provided in the instrument panel and a speaker provided in the vehicle 1. The display device may be a display device that displays information from a navigation system. The notification device 43 may also include a HUD (head-up display) that displays information on the front window superimposed on the scenery around the vehicle 1.

[0030] 3. Driving Assistance Device Next, the driving assistance device 50 according to this embodiment will be described in detail.

[0031] (3-1. Configuration Example) The driving assistance device 50 functions as a device that assists in driving a vehicle by having one or more processors, such as CPUs (Central Processing Units), execute a computer program. The computer program is a computer program that causes the processor to execute the operations, described below, that should be performed by the driving assistance device 50. The computer program executed by the processor may be recorded on a recording medium that functions as a storage unit (memory) 53 provided in the driving assistance device 50, or may be recorded on a recording medium built into the driving assistance device 50 or any recording medium that can be externally attached to the driving assistance device 50.

[0032] The recording medium for recording a computer program may be a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape; an optical recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), or a Blu-ray (registered trademark); a magneto-optical medium such as a floptical disk; a memory element such as a RAM or a ROM; a flash memory such as a USB (Universal Serial Bus) memory or an SSD (Solid State Drive); or any other medium capable of storing a program.

[0033] 4 is a block diagram showing an example of the configuration of a driving assistance device 50 according to this embodiment. A surrounding environment sensor 31, a vehicle state sensor 33, and an in-vehicle camera 37 are connected to the driving assistance device 50 via a dedicated line or communication means such as a controller area network (CAN) or local internet (LIN). A notification device 43 is also connected to the driving assistance device 50.

[0034] The driving assistance device 50 is not limited to an electronic control device mounted on the vehicle 1, but may also be a terminal device such as a smartphone or a wearable device.

[0035] The driving assistance device 50 includes a processing unit 51, a storage unit 53, a visual characteristic storage unit 55, and a reference visual pattern storage unit 57. The processing unit 51 is configured to include one or more processors such as a CPU and various peripheral components. A part or all of the processing unit 51 may be configured with updatable firmware or the like, or may be a program module or the like executed by instructions from the CPU or the like.

[0036] (Storage Unit) The storage unit 53 is composed of one or more storage elements such as RAM or ROM connected to the processing unit 51 so as to be able to communicate with it. However, the type and number of storage units 53 are not particularly limited. The storage unit 53 stores information such as computer programs executed by the processing unit 51, various parameters used in arithmetic processing, detection data, and calculation results. A part of the storage unit 53 is used as a work area for the processing unit 51.

[0037] (Reference visual pattern storage unit) The reference visual pattern storage unit 57 is configured by a storage element such as a RAM or a ROM communicably connected to the processing unit 51, or a storage medium such as an HDD, a CD, a DVD, an SSD, a USB flash drive, or a storage device. The information of the reference visual pattern stored in the reference visual pattern storage unit 57 includes a visual location, a visual order, and a visual timing that the driver should visually recognize according to a driving scene in which a predetermined risk object that may hinder the movement of the vehicle 1 is present.

[0038] The standard visual recognition patterns are set according to the driving scene, for example, as follows: - Obvious risks such as parked vehicles themselves and pedestrians about to cross the road are set as targets that should be visually recognized at an early stage. - Potential risks such as blind spots caused by intersections or three-dimensional objects, and the shadow of a person inside a vehicle are set as targets that should be visually recognized immediately before the vehicle passes the side of the potential risk. - Targets that are easy to see (obvious targets) and lights that are easily turned on or off, which make it easy to determine the risk, are set as targets that should be visually recognized for a short period of time. - Targets that are difficult to see, such as the shadow of a person inside a vehicle, are set as targets that should be visually recognized for a long period of time. - Targets that are not completely visible, such as blind spots caused by intersections or three-dimensional objects, are set as targets that should be visually recognized frequently.

[0039] Fig. 5 is an explanatory diagram illustrating an example of a reference visual pattern in a driving scene in which a vehicle 1 passes by a parked vehicle that is parked at the edge of a road and extends into the driving lane, as illustrated in Figs.

[0040] 5, the viewing locations, viewing order, and viewing timing to be viewed are set according to the change in the distance from the vehicle 1 to the parked vehicle. In the reference viewing pattern, three periods C1 to C3 are set according to the distance D from the vehicle 1 to the parked vehicle, and the viewing locations and viewing times to be viewed are set in each of the periods C1 to C3. The order of the viewing locations and viewing times set in each of the periods C1 to C3 may be interchanged as long as they are within the periods C1 to C3.

[0041] In other words, during a period C1 in which the distance D from vehicle 1 to the parked vehicle is between 50 and 18 m, the following visual confirmation locations (i) to (iii) are set to be visually confirmed for a predetermined period of time or more. During period C1, the order in which (i) to (iii) are visually confirmed is not limited. (i) The entire parked vehicle for 0.5 seconds or more (ii) The area around the tail lamp of the parked vehicle for 0.5 seconds or more (iii) The gap under the floor of the parked vehicle (the back of the gap) for 1.0 second or more

[0042] Furthermore, during a period C2 in which the distance D from vehicle 1 to the parked vehicle is between 18 and 6 m, the following visual confirmation locations (iv) to (v) are set to be visually confirmed for a predetermined period of time or more. During period C2, the order in which (iv) to (v) are visually confirmed is not limited. (iv) The front (rear), near the right corner, and under the floor of the parked vehicle for 1.0 second or more. (v) The area near the driver's seat of the parked vehicle for 1.0 second or more.

[0043] In addition, during a period C3 in which the distance D from vehicle 1 to the parked vehicle is 6 to 0 m, the following visual confirmation location (vi) is set to be visually confirmed for a predetermined time or longer: (vi) The vicinity of the driver's seat and the vicinity of the right corner of the parked vehicle for 1.0 second or longer.

[0044] 5, until the vehicle 1 approaches the parked vehicle, the attention resources of the driver of the vehicle 1 are allocated to whether a pedestrian or the like is present in the rear of the parked vehicle (blind spot) and whether the parked vehicle is preparing to depart. Furthermore, when the vehicle 1 approaches the parked vehicle, the attention resources of the driver of the vehicle 1 are allocated to whether a pedestrian or the like is present in the rear of the parked vehicle (blind spot), whether a pedestrian or the like will suddenly appear from the rear of the parked vehicle, and whether an occupant is present in the parked vehicle. Furthermore, when the vehicle 1 is closest to the parked vehicle, the attention resources of the driver of the vehicle 1 are allocated to whether a pedestrian or the like will suddenly appear from the rear of the parked vehicle and whether an occupant is present in the parked vehicle.

[0045] In this way, the reference visual confirmation pattern is set so that the driver can visually confirm the visual confirmation locations to be visually confirmed in an appropriate order and at an appropriate timing according to the driving scene. The reference visual confirmation pattern is set for each driving scene in which a predetermined risk object exists, and is stored in the reference visual confirmation pattern storage unit 57. The reference visual confirmation pattern illustrated in Fig. 5 is set according to a change in the distance from the vehicle 1 to the risk object (parked vehicle), but may be set according to a change over time after the time when the distance D from the vehicle 1 to the risk object becomes a predetermined distance.

[0046] (Visual characteristic storage unit) The visual characteristic storage unit 55 is configured by a storage element such as a RAM or a ROM connected to the processing unit 51 so as to be able to communicate with the processing unit 51, or a storage medium such as an HDD, a CD, a DVD, an SSD, a USB flash drive, or a storage device. The information on the visual characteristic of the driver stored in the visual characteristic storage unit 55 includes the driver's gaze locations, gaze order, and gaze timing, which are learned for each driving scene in which a predetermined risk object that may be an obstacle to the movement of the vehicle 1 is present.

[0047] Fig. 6 is an explanatory diagram illustrating an example of the driver's visual characteristics in a driving scene in which vehicle 1 passes by a parked vehicle that is located at the edge of the road and extends into the driving lane, as illustrated in Fig. 1 and Fig. 2. The driver's visual characteristics shown in Fig. 6 are information obtained by learning and recording the driver's visual locations and visual times during periods C1 to C3 corresponding to the reference visual pattern shown in Fig. 5.

[0048] In the example of the driver's visual characteristics shown in Figure 6, during a period C1 when the distance D from vehicle 1 to the parked vehicle is 50 to 18 m, the driver sequentially views the following visual locations (i), (iii), and (ii): (i) the entire parked vehicle for 0.5 seconds, (iii) the gap under the floor of the parked vehicle (the back of the gap) for 1.0 seconds, and (ii) the area around the tail lamp of the parked vehicle for 0.5 seconds.

[0049] During the period C2 when the distance D from vehicle 1 to the parked vehicle is between 18 and 6 m, the driver sequentially looks at the following visual points (iv) to (?): (iv) The front (rear) of the parked vehicle, the area around the right corner, and the gap under the floor (back of the gap) for 0.5 seconds; (?) The front (rear) of the parked vehicle, the left side, and the gap under the floor (back of the gap) for 1.0 seconds.

[0050] In addition, during a period C3 in which the distance D from vehicle 1 to the parked vehicle is 6 to 0 m, the driver is set to view the following (???) viewing points: (???) Front (rear) and right corner of the parked vehicle for 1.0 second

[0051] The driver's visual characteristics are stored in the visual characteristics storage unit 55 for each driving scene by specifying the order and timing of visual recognition of the visual locations to be recognized when the driver of the vehicle 1 passes through a driving scene in which a predetermined risk object is present. The driver's visual characteristics in each driving scene may be the most frequently occurring visual pattern or the most recent visual pattern. The driver's visual characteristics illustrated in FIG. 6 are recorded according to changes in the distance from the vehicle 1 to the risk object (parked vehicle), but may also be recorded according to changes over time after the distance D from the vehicle 1 to the risk object becomes a predetermined distance.

[0052] (3-2. Functional Configuration of Processing Unit) Next, the functional configuration of the processing unit 51 of the driving assistance device 50 will be described. The processing unit 51 includes an acquisition unit 61, a surrounding environment detection unit 63, a gaze detection unit 65, a driving scene detection unit 67, a reference visual pattern setting unit 69, an attention awareness state determination unit 71, and a warning processing unit 73. Each of these units is a function realized by execution of a computer program by one or more processors such as a CPU. However, part of the acquisition unit 61, the surrounding environment detection unit 63, the driving scene detection unit 67, the reference visual pattern setting unit 69, the attention awareness state determination unit 71, the gaze detection unit 65, and the warning processing unit 73 may be configured using analog circuits.

[0053] (Acquisition Unit) The acquisition unit 61 acquires information or messages transmitted from the surrounding environment sensor 31 , the vehicle state sensor 33 , and the interior camera 37 .

[0054] (Surrounding environment detection unit) The surrounding environment detection unit 63 detects the surrounding environment of the vehicle 1. Based on image data or detection information transmitted from the surrounding environment sensor 31, the surrounding environment detection unit 63 detects the surrounding environment such as other vehicles and obstacles present around the vehicle 1, road boundaries, etc.

[0055] (Gaze Detection Unit) The gaze detection unit 65 detects the driver's gaze point based on the gaze direction of the driver of the vehicle 1. Specifically, the gaze detection unit 65 detects the gaze direction of the driver based on an image of the driver's face captured by the in-vehicle camera 37, and detects the driver's gaze point based on the captured image captured by the front-facing cameras 31LF, 31RF and the gaze direction of the driver.

[0056] (Driving Scene Detection Unit) Based on the information on the surrounding environment detected by the surrounding environment detection unit 63, the driving scene detection unit 67 detects that the driving scene in which the vehicle 1 is traveling is a predetermined driving scene in which there is a predetermined risk object that will hinder the movement of the vehicle 1. The predetermined driving scenes are set in advance, such as a scene of passing beside a parked vehicle at the edge of the road, a scene of merging at a junction of two driving lanes, a scene of passing beside a blind spot such as a building, a scene of passing through an intersection without traffic lights, a scene of passing through a road with a pedestrian crossing without traffic lights, etc., and the driving scene detection unit 67 determines whether the surrounding environment of the vehicle 1 corresponds to any of the driving scenes.

[0057] (Reference Visual Pattern Setting Unit) The reference visual pattern setting unit 69 sets a reference visual pattern corresponding to the driving scene detected by the driving scene detection unit 67. In the present embodiment, the reference visual pattern setting unit 69 selects a reference visual pattern corresponding to the driving scene from the reference visual patterns stored in the reference visual pattern storage unit 57, and sets the selected reference visual pattern as the reference visual pattern used for determining the attention consciousness state.

[0058] (Attention Awareness State Determination Unit) The attention awareness state determination unit 71 reads out the driver's visual characteristics in a driving scene in which the vehicle 1 is traveling from the driver's visual characteristics stored in the visual characteristic storage unit 55, and compares the driver's visual characteristics with a reference visual pattern corresponding to the driving scene to determine the driver's attention awareness state. For example, the attention awareness state determination unit 71 compares the driver's visual characteristics with the reference visual pattern and determines whether or not there is a visual location that the driver has not recognized in the driver's visual characteristics among the visual locations set in the reference visual pattern. If there is a visual location that the driver has not recognized, the attention awareness state determination unit 71 determines that the driver's attention awareness is low and designates the visual location as a visual location requiring attention.

[0059] Furthermore, the attention consciousness state determination unit 71 determines whether or not there is a visual location where the visual timing assumed by the driver's visual characteristics differs from the visual timing of the reference visual pattern while the driver visually recognizes all visual locations set in the reference visual pattern. That is, when the driver visually recognizes all visual locations that should be visually recognized but there is a visual location where the visual timing differs from the reference visual pattern, the attention consciousness state determination unit 71 determines that the driver's attention consciousness is low and designates the visual location as an attention-requiring visual location.

[0060] Furthermore, the attention awareness state determination unit 71 may determine whether or not the driver is moving the viewing position along the reference viewing pattern while the vehicle 1 is passing through a driving scene, based on information about the driver's viewing position detected by the line-of-sight detection unit 65. That is, the attention awareness state determination unit 71 may determine that the driver's attention awareness is low when the driver is not moving the viewing position to be viewed along the reference viewing pattern.

[0061] (Warning Processing Unit) The warning processing unit 73 issues a warning regarding a risk object and a visual location. For example, when it is determined that the driver's visual characteristics do not match the reference visual pattern, the warning processing unit 73 notifies the driver of a risk object that the driver should pay attention to or a visual location that the driver should visually recognize before the timing when the risk object or visual location should be recognized. Alternatively, when it is determined that the driver's visual characteristics do not match the reference visual pattern, the warning processing unit 73 may notify the driver of a risk object that the driver should pay attention to or a visual location that the driver should visually recognize at the timing when the risk object or visual location should be recognized.

[0062] (3-3. Operation of Driving Assistance Device) Next, an example of the processing operation by the processing unit 51 of the driving assistance device 50 according to this embodiment will be described with reference to a flowchart.

[0063] 7 is a flowchart showing the driving assistance process performed by the processing unit 51. First, the processing unit 51 activates (turns on) the driving assistance function (step S1). The processing unit 51 may activate the driving assistance function in response to an input operation by the driver, or may activate the driving assistance function together with activation of the system of the vehicle 1.

[0064] Next, the processing unit 51 identifies the driver of the vehicle 1 (step S3). Specifically, the processing unit 51 extracts facial feature points of the driver based on image data transmitted from the in-vehicle camera 37 and identifies the driver. The processing unit 51 assigns identification information to each identified driver and records the assigned identification information. The identification of the driver is not limited to the example in which it is performed based on image data. For example, the processing unit 51 may identify the driver based on the driver's name, nickname, etc., input by the driver himself / herself via a touch panel or microphone.

[0065] Next, the line-of-sight detection unit 65 of the processing unit 51 starts a process of detecting the direction of the driver's line of sight based on the captured image output from the in-vehicle camera 37 (step S5).

[0066] Next, the surrounding environment detection unit 63 of the processing unit 51 detects the surrounding environment of the vehicle 1 based on the information output from the surrounding environment sensor 31 (step S7). Specifically, the surrounding environment detection unit 63 detects the surrounding environment of the vehicle 1 based on the sensor signal or image data output from the surrounding environment sensor 31. For example, the surrounding environment detection unit 63 recognizes objects by extracting feature points from the image data and matching the feature point patterns with pre-prepared reference data. The surrounding environment detection unit 63 recognizes various objects, including moving objects such as people, bicycles, motorcycles, and four-wheeled automobiles, as well as man-made or natural stationary objects, and lines painted on roads such as white lines and crosswalks. The surrounding environment detection unit 63 also calculates the position and speed of the recognized objects, as well as the distance to the objects.

[0067] Next, the driving scene detection unit 67 of the processing unit 51 determines whether a predetermined driving scene in which a risk object exists has been detected based on the detected information about the surrounding environment (step S9). Specifically, the driving scene detection unit 67 determines whether the surrounding environment of the vehicle 1 corresponds to any of the predetermined driving scenes set in advance based on information such as the type and position of one or more obstacles around the vehicle 1, the shape of the road, boundary lines and crosswalks, and the presence or absence of traffic lights.

[0068] If the driving scene detection unit 67 does not determine that a predetermined driving scene in which a risk object exists has been detected (S9 / No), the process returns to step S7. On the other hand, if the driving scene detection unit 67 determines that a predetermined driving scene in which a risk object exists has been detected (S9 / Yes), the reference visual pattern setting unit 69 of the processing unit 51 sets a reference visual pattern corresponding to the detected driving scene (step S11). Specifically, the reference visual pattern setting unit 69 selects a reference visual pattern for the detected driving scene from the reference visual patterns stored in the reference visual pattern storage unit 57, and sets the reference visual pattern as the reference visual pattern to be viewed by the driver.

[0069] Next, the attention consciousness state determination unit 71 of the processing unit 51 calculates the visual characteristics of the driver of the vehicle 1 in the detected driving scene (step S13). Specifically, the attention consciousness state determination unit 71 selects a visual characteristic associated with the identification information of the driver in the detected driving scene from the visual characteristic information stored in the visual characteristic storage unit 55. The attention consciousness state determination unit 71 may determine the most frequently occurring visual pattern among the visual characteristics (visual patterns) of the driver in the detected driving scene as the visual characteristic of the driver, or may determine the most recent visual pattern as the visual characteristic of the driver.

[0070] Next, the attention and consciousness state determination unit 71 compares the driver's visual characteristics with the reference visual pattern to determine the driver's attention and consciousness state (step S15).

[0071] 8 and 9 are explanatory diagrams showing an example of a process for determining the driver's attention and consciousness state. Both of these diagrams show the reference visual pattern and the visual characteristics of drivers A and B in a driving scene in which the vehicle 1 passes by a parked vehicle that is parked at the edge of the road and extends into the driving lane, as shown in FIG. 5.

[0072] In the example shown in Figure 8, when comparing the visual characteristics of driver A during a period C1 in which the distance from vehicle 1 to the parked vehicle is 50 to 18 m with the reference visual pattern, the order of the visual locations is different, but the visual locations and the visual duration within the period C1 are consistent. On the other hand, when comparing the visual characteristics of driver A during a period C2 in which the distance from vehicle 1 to the parked vehicle is 18 to 6 m with the reference visual pattern, driver A does not visually recognize the area near the driver's seat of the parked vehicle (v), but visually recognizes the area in front of (behind), the left side, and the gap under the floor (behind the gap) (?). Furthermore, during the period C2, driver A visually recognizes the area in front of (behind), the right corner, and the gap under the floor (behind the gap) (iv) of the parked vehicle, but the visual duration (0.5 seconds) is shorter than the visual duration (1.0 seconds) of the reference visual pattern. Furthermore, when comparing the visual characteristics of driver A during period C3 when the distance from vehicle 1 to the parked vehicle is 6 to 0 m with the reference visual pattern, driver A does not see the area near the driver's seat or near the right corner (vi) of the parked vehicle, but sees the area in front (rear) and near the right corner (??) of the parked vehicle.

[0073] Since there are three differences in the viewing locations or viewing times, and in particular, driver A has not been viewing the area around the driver's seat of the parked vehicle (v) after period C2, the attention awareness state determination unit 71 determines that driver A's attention awareness state in the driving scene is low. In this case, the attention awareness state determination unit 71 sets the area around the driver's seat of the parked vehicle (v), which driver A has not viewed, as an attention-requiring viewing location.

[0074] 9 , when the visual recognition characteristics of driver B during a period C1 in which the distance from vehicle 1 to the parked vehicle is 18 to 50 m are compared with the reference visual recognition pattern, driver B does not visually recognize the area around the tail lamp of the parked vehicle (ii), but visually recognizes the gap under the floor of the parked vehicle (the back of the gap) (iii) twice. Furthermore, when the visual recognition characteristics of driver A during a period C2 in which the distance from vehicle 1 to the parked vehicle is 6 to 18 m are compared with the reference visual recognition pattern, the visual recognition locations within the period C2 are the same, but the visual recognition time (0.5 seconds) of the front (back), the area around the right corner, and the gap under the floor of the parked vehicle (the back of the gap) (iv) is shorter than the visual recognition time (1.0 second) of the reference visual recognition pattern. Furthermore, when comparing the visual recognition characteristics of driver A with the reference visual recognition pattern during period C3 when the distance from vehicle 1 to the parked vehicle is 6 to 0 m, driver A not only recognizes the vicinity of the driver's seat and the vicinity of the right corner of the parked vehicle (vi), but also the vicinity of the tail lamp of the parked vehicle (ii). Also, during period C3, the time that driver B spends viewing the vicinity of the driver's seat and the vicinity of the right corner of the parked vehicle (vi) is shorter than that of the reference visual recognition pattern.

[0075] Although driver B views all of the viewing points (i) to (vi), there are four differences in the viewing order and timing, so the attention awareness state determination unit 71 determines that driver B's attention awareness state in the driving scene is low.

[0076] Returning to FIG. 7 , the warning processing unit 73 of the processing unit 51 then executes a warning process to issue a warning regarding a risk object or a visually recognized location (step S17). For example, the warning processing unit 73 notifies the driver of a visually recognized location that requires attention, identified by comparing the driver's visual characteristics with a reference visual pattern, before the timing for visually recognizing the visually recognized location. For example, in the example of FIG. 8 , the warning processing unit 73 may notify the driver of the presence of a visually recognized location that requires attention immediately after the attention consciousness state determination process is completed. The warning processing unit 73 may notify the driver of the presence of a visually recognized location by voice, such as, "Please check whether there is an occupant in the driver's seat of the parked vehicle." The warning processing unit 73 may also notify the driver of the presence of a risk object by voice, such as, "Please be careful whether an occupant is getting out of the parked vehicle."

[0077] The warning processing unit 73 may also notify the driver of the timing when the driver should visually check the caution-requiring visual location or the risk object, such as "When you approach the parked vehicle, please check whether there is an occupant in the driver's seat of the parked vehicle." This allows the driver's attention resources to be appropriately allocated, thereby reducing the driver's burden.

[0078] In addition to or in addition to audio notification, the warning processor 73 may temporarily or continuously display an icon or text indicating a visually-requiring area or a risk object on the captured image or animation image generated by the front-view cameras 31LF, 31RF. This allows the driver to recognize the visually-requiring area or the risk object and pay attention to it before passing through the driving scene. The warning processor 73 may display the risk object or the visually-requiring area on a display panel, or may display the risk object or the visually-requiring area on the front window superimposed on the scenery outside the vehicle.

[0079] Furthermore, the warning processing unit 73 may notify the driver of the presence of a caution-requiring visual location at the timing when the driver should view the caution-requiring visual location as the vehicle 1 travels. This allows the driver to view the caution-requiring visual location at the optimal timing, thereby appropriately allocating the driver's attention resources and reducing the driver's burden.

[0080] Furthermore, the warning processing unit 73 may determine whether the driver is moving the viewing point along the reference viewing pattern while the vehicle 1 is passing through the driving scene, and if the driver is not moving the viewing point along the reference viewing pattern, may notify the driver of the existence of a viewing point that is not being viewed.

[0081] 10 is an explanatory diagram showing an example of performing a warning process based on a transition of a driver's actual viewing position, which shows the transition of the driver's actual viewing position together with the driver's visual characteristics and reference visual pattern shown in FIG.

[0082] In the example shown in FIG. 10 , during a period C1, the driver transitions the visual location in the same manner as the learned driver's visual characteristics. Furthermore, during a period C2, the driver does not visually recognize the area near the driver's seat (v) of the parked vehicle, which is recognized according to the learned driver's visual characteristics. Furthermore, during a period C3, the driver visually recognizes the area in front (back) and near the right corner (??) of the parked vehicle in the same manner as the learned driver's visual characteristics. In this example, the warning processor 73 notifies the driver of the presence of a visual location requiring attention identified based on a comparison between the driver's visual characteristics and the reference visual pattern before the time when the driver should visually recognize the visual location requiring attention. In addition to this advance notification, when the driver's actual visual location differs from both the driver's visual characteristics and the reference visual pattern (when the driver's actual visual location in FIG. 10 is (?)), the warning processor 73 notifies the driver of the visual location that should be visually recognized according to the reference visual pattern.

[0083] This makes it possible to make the driver view the viewing location that the driver should view at an appropriate timing, even if the driver transitions the viewing location using a viewing pattern that differs from the driver's visual characteristics.

[0084] However, the above-mentioned attention-requiring visual location is information obtained by comparing the driver's visual characteristics learned from the driver's past driving behavior data with the reference visual pattern, and the driver may actually direct his or her gaze toward the attention-requiring visual location or risk target while driving in a predetermined driving scene. Therefore, when the driver's gaze direction detected by the gaze detection unit 65 is directed toward the attention-requiring visual location or risk target at a predetermined timing set in the reference visual pattern, the warning processing unit 73 terminates the warning processing regarding the attention-requiring visual location or risk target. This prevents the driver from feeling annoyed by the warning notification.

[0085] 7 , after executing the warning process, the warning processing unit 73 associates the data of the driver's viewing location acquired in the current routine with the driving scene information and records it in the visual characteristic storage unit 55 (step S19). As a result, the data of the driver's viewing location is accumulated in the visual characteristic storage unit 55 together with the driving scene information.

[0086] Next, it is determined whether the driving assistance function has been turned off (step S21). If the warning processing unit 73 does not determine that the driving assistance function has been turned off (S21 / No), the process returns to step S7 and repeats the processing of each of the above steps. On the other hand, if the warning processing unit 73 determines that the driving assistance function has been turned off (S21 / Yes), the series of processes ends.

[0087] As described above, the driving assistance device 50 according to this embodiment is configured to detect a driving scene in which a predetermined risk object is present, compare a reference visual pattern set for the driving scene with the driver's visual characteristics learned for each driving scene to determine the driver's attentional awareness state, and issue a warning to the driver based on the attentional awareness state. Therefore, the driver can be made to visually recognize the visual locations that should be visually recognized at an appropriate timing and in an appropriate order according to the driving scene in which a risk object is present. As a result, the driver's attentional resources are appropriately allocated, the driver's workload is reduced, and accidents such as collisions can be prevented.

[0088] Furthermore, when the driver's visual characteristics do not match the reference visual pattern, the driving assistance device 50 according to this embodiment notifies the driver of a risk object that the driver should pay attention to or a visual location that the driver should view before the viewing timing of the visual location. Specifically, the driving assistance device 50 notifies the driver in advance of a visual location that is not visible in the driver's visual characteristics or a visual location whose viewing timing differs from the reference visual pattern as a visual location requiring caution. Therefore, when passing through a predetermined driving scene, the driver can be made aware of a visual location that the driver should view in advance, thereby urging the driver to be careful.

[0089] Furthermore, the driving support device 50 according to this embodiment detects a transition of the driver's actual viewing location based on the driver's line of sight, and if the viewing location differs from both the driver's visual characteristics and the reference visual pattern, notifies the driver of the timing at which the driver should view the viewing location. Therefore, even if the driver transitions the viewing location using a visual pattern that differs from the driver's visual characteristics, the driver can be made to view the viewing location that the driver should view at the appropriate timing.

[0090] 4. Other Application Examples In the above embodiment, the operation of the driving assistance device of the present disclosure has been described using an example of a driving scene in which a vehicle passes by a parked vehicle at the edge of a road. Below, examples in which the technology of the present disclosure is applied to other driving scenes will be described.

[0091] 11 to 13 are diagrams illustrating another application example. As shown in FIG. 11, this application example illustrates an application example in which a driver parks vehicle 1 in a parking space between another vehicle 93 and a side wall 95. FIGS. 12 and 13 respectively show the viewing locations (gaze direction and field of view) of the driver of vehicle 1 at the positions of vehicle 1 shown at times t1 to t4 in FIG. 11. FIGS. 12 and 13 show the visual characteristics and reference visual patterns of drivers A and B, respectively. Differences in the density of hatching in FIGS. 12 and 13 represent differences in viewing time, with higher hatching density indicating longer viewing time.

[0092] At time t1, driver A shown in FIG. 12 looks at the rearview mirror (back mirror) and looks behind the vehicle 1, similar to the viewing location in the reference visual pattern. At time t2, driver A looks at the rearview mirror without looking at the right door mirror (right side mirror), which is a viewing location in the reference visual pattern. At time t2, driver A looks at the rearview mirror, similar to the reference visual pattern, but the viewing time is shorter than in the reference visual pattern. At time t3, driver A looks at the left door mirror (left side mirror), similar to the viewing location in the reference visual pattern, and looks behind the vehicle 1. At time t4, driver A looks at the rearview mirror and looks behind the vehicle 1, similar to the reference visual pattern.

[0093] 12, the visual recognition characteristics of the driver A do not match the reference visual recognition pattern, so the driver A's attention consciousness state is determined to be low. Also, at time t2, the right door mirror is set as the visual recognition area requiring caution.

[0094] At time t1, driver B shown in FIG. 13 looks at the rearview mirror (back mirror) and looks behind the vehicle 1, similar to the viewing location in the reference visual pattern. At time t2, driver B looks at the rearview mirror without looking at the right door mirror (right side mirror), which is a viewing location in the reference visual pattern. At time t2, driver B looks at the rearview mirror, similar to the reference visual pattern, but the time spent looking behind the vehicle 1 is shorter than in the reference visual pattern. At time t3, driver B looks at the left door mirror (left side mirror), similar to the viewing location in the reference visual pattern, and looks behind the vehicle 1. At time t4, driver B looks at the rearview mirror and looks behind the vehicle 1, similar to the reference visual pattern, but the time spent looking behind the vehicle 1 is shorter than in the reference visual pattern.

[0095] 13, the visual characteristics of driver B do not match the reference visual pattern, and therefore it is determined that driver B's attention consciousness state is low. Furthermore, at time t2, the right door mirror is set as the visual area requiring attention, and at time t4, the rear of the vehicle 1 is set as the visual area requiring attention.

[0096] In a driving scene in which the vehicle 1 is parked in a parking space, the driver's visual characteristics are compared with the reference visual pattern as described above, and if the driver's visual characteristics do not match the reference visual pattern, the warning processing unit 73 issues a warning regarding a risk object (another vehicle 93 and the side wall 95) or a visual location (such as the rearview mirror or the rear of the vehicle 1). This makes it possible to appropriately allocate the driver's attention resources during the parking operation, reduce the driver's burden, and prevent accidents such as a collision of the vehicle 1. In a driving scene in which the vehicle 1 is parked in a parking space, a transition of the driver's actual visual location may be detected, and a warning may be issued if the driver's visual characteristics differ from the reference visual pattern.

[0097] 13 has more differences from the reference visual pattern than the example of driver A shown in FIG. 12. Therefore, the warning processing unit 73 may increase the strength of the warning as the number of differences from the reference visual pattern increases compared to the example of driver A. For example, the warning processing unit 73 may provide a notice such as advice when there are few differences between the driver's visual characteristics and the reference visual pattern, and may generate an alarm or a warning sound when there are many differences between the driver's visual characteristics and the reference visual pattern.

[0098] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art to which the present disclosure pertains can conceive of various modifications or alterations within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0099] For example, in the above embodiment, the process of assisting the driving of the vehicle is executed by a driving assistance device mounted on the vehicle 1, but some or all of the functions may be provided by an external management server.

[0100] In addition, the technology of the present disclosure can also be realized as a vehicle equipped with the driving assistance device described in the above embodiment, a driving assistance processing method using the driving assistance device, a computer program that causes a computer to function as the above driving assistance system, and a non-transitory tangible recording medium on which the computer program is recorded.

[0101] 1: Vehicle 43: Notification device 50: Driving assistance device 51: Processing unit 53: Storage unit 55: Visibility characteristic storage unit 57: Reference visual pattern storage unit 61: Acquisition unit 63: Surrounding environment detection unit 65: Gaze detection unit 67: Driving scene detection unit 69: Reference visual pattern setting unit 71: Attention awareness state determination unit 73: Warning processing unit 90: Parked vehicle

Claims

1. A driving assistance device that assists in driving a vehicle, comprising: one or more processors; one or more memories communicatively connected to the one or more processors; and a visual characteristic storage unit that records information on visual characteristics including a driver's viewing location, viewing order, and viewing timing learned for each driving scene in which a predetermined risk object that is an obstacle to movement of the vehicle is present, wherein the one or more processors execute a driving scene detection process that detects the driving scene in which the risk object is present; a reference visual pattern setting process that sets a reference visual pattern including the viewing locations, viewing order, and viewing timing that the driver should view in accordance with the driving scene; an attention awareness state determination process that determines the driver's attention awareness state by comparing the visual characteristics of the driver in the driving scene identified from the visual characteristics of the driver recorded in the visual characteristic storage unit with the reference visual pattern; and a warning process that issues a warning regarding the risk object or the viewing location.

2. A driving assistance device as described in claim 1, wherein the reference visual pattern includes information on the visual location and the visual timing that the driver should visually recognize, which are set according to the passage of time or a change in the distance from the vehicle to the risk object.

3. The driving assistance device of claim 2, wherein, when the attention awareness state determination process determines that the driver's visual characteristics do not match the reference visual pattern, the one or more processors, in the warning process, notify the driver of the risk object to which the driver should pay attention or the visual location that the driver should view before the viewing timing of the visual location.

4. The driving assistance device of claim 2, wherein the one or more processors, in the attention consciousness state determination process, compare the visual characteristics of the driver with the reference visual pattern, and if it is determined that the driver views all of the visual locations set in the reference visual pattern, but there is a visual location requiring attention whose viewing timing assumed by the visual characteristics of the driver differs from the viewing timing of the reference visual pattern, in the warning process, notify of the existence of at least the visual location requiring attention before the viewing timing of the visual location requiring attention.

5. The driving assistance device of claim 2, wherein the one or more processors, in the attention consciousness state determination process, compare the visual characteristics of the driver with the reference visual pattern, and if it is determined that the driver views all of the visual points set in the reference visual pattern, but there is a visual point requiring attention where the viewing timing assumed by the visual characteristics of the driver is different from the viewing timing of the reference visual pattern, in the warning process, notify the driver of the visual point requiring attention at the viewing timing at which the visual point requiring attention should be viewed.

6. The driving assistance device of claim 2, wherein the one or more processors further execute a gaze detection process to detect the driver's gaze point, and in the warning process, determine whether the driver is moving the gaze point along the reference viewing pattern while the vehicle is passing through the driving scene, and if the driver is not moving the gaze point along the reference viewing pattern, notify the driver of the existence of a gaze point that is not being viewed.

7. A driving assistance processing method for assisting driving of a vehicle by a computer, wherein the computer executes the following steps: a driving scene detection process for detecting a driving scene in which a predetermined risk object that is an obstacle to movement of the vehicle is present; a reference visual pattern setting process for setting a reference visual pattern including a visual location, a visual order, and a visual timing that the driver should look at according to the driving scene; an attention awareness state determination process for determining the attention awareness state of the driver by comparing the visual characteristics of the driver in the driving scene identified from information on the driver's visual characteristics including the visual locations, the visual order, and the visual timing of the driver learned for each driving scene in which the risk object is present with the reference visual pattern; and a warning process for issuing a warning to the driver based on the attention awareness state.

8. A non-transitory tangible recording medium having recorded thereon a computer program that causes a computer to execute the following: a driving scene detection process that detects driving scenes in which a specific risk object that is an obstacle to the movement of the vehicle is present; a reference visual pattern setting process that sets a reference visual pattern including the visual locations, visual order, and visual timing that the driver should look at according to the driving scene; an attention awareness state determination process that determines the driver's attention awareness state by comparing the visual characteristics of the driver in the driving scene identified from information on the driver's visual characteristics including the driver's visual locations, visual order, and visual timing learned for each driving scene in which the risk object is present with the reference visual pattern; and a warning process that issues a warning to the driver based on the attention awareness state.

Images