Driving assistance device, driving assistance method, and program

Abstract

To flexibly provide driving assistance in accordance with the direction and the time of an occupant's line of sight.SOLUTION: Provided is a driving assistance device comprising a target recognition unit that recognizes a target existing in the surrounding of a moving vehicle; a line of sight estimation unit that estimates which object of interest among a plurality of objects of interest including an object of interest existing on the structure of the moving vehicle and the target, the line of sight of an occupant of the moving vehicle is directed to; a visual recognition determination unit that, when the occupant's line of sight is directed to one of the plurality of objects of interest for a first prescribed period or more, determines that the occupant is recognizing the object of interest; and a driving assistance unit that provides assistance in driving the moving vehicle, on the basis of the determination result of the visual recognition determination unit. The visual recognition determination unit maintains the determination result that the occupant has been visually recognizing the object of interest that is found to be visually recognized, for a second prescribed period from a time when the occupant is assumed to have missed the object of interest.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device, a driving assistance method, and a program. [Background technology]

[0002] Conventionally, there is known a technology for detecting the line of sight of a vehicle occupant and assisting the vehicle occupant in driving the vehicle based on the detected line of sight. For example, Patent Document 1 discloses a technology for restricting reception of commands to an operating unit of the vehicle when the vehicle is in a traveling state and the detected line of sight is not directed toward a predetermined area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-33570 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technologies, it has sometimes been impossible to flexibly provide driving assistance depending on the direction of the driver's gaze and the time of day of the vehicle's occupant.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide a driving assistance device, a driving assistance method, and a program that can flexibly perform driving assistance depending on the direction of the occupant's line of sight and the time. [Means for solving the problem]

[0006] A driving assistance device, a driving assistance method, and a program according to the present invention employ the following configuration. (1): A driving assistance device according to one embodiment of the present invention includes a target recognition unit that recognizes targets in the vicinity of a moving body; a gaze estimation unit that estimates which of a plurality of gaze targets, including a gaze target present on a structure of the moving body and the target, an occupant of the moving body is looking at; a visibility determination unit that, when it is estimated that the occupant is looking at one of the plurality of gaze targets for a first predetermined period or longer, determines that the occupant is looking at the gaze target; and a driving assistance unit that performs driving assistance for the moving body based on the determination result of the visibility determination unit. When the visibility determination unit determines that the occupant is looking at a certain gaze target and then estimates that the occupant has looked away from the gaze target, the visibility determination unit maintains the determination result that the occupant is looking at the gaze target that was determined to be looked at for a second predetermined period from the time it is estimated that the occupant has looked away from the gaze target.

[0007] (2): In the above aspect (1), the gaze estimation unit estimates which of the multiple gaze targets the occupant is directing his or her gaze at based on the degree of coincidence between the gaze and a gaze probability distribution that represents the probability of viewing the gaze target.

[0008] (3): In the above aspect (2), the gaze estimation unit calculates the degree of coincidence based on the gaze represented by polar coordinates centered on the head of the occupant of the moving body and the central angle and angle width of the gaze target represented by polar coordinates centered on the head, and estimates which of the multiple gaze targets the occupant is directing his or her gaze at.

[0009] (4) In the above aspect (1), the visual recognition determination unit changes the length of the second predetermined period depending on the types of the plurality of gaze targets.

[0010] (5): In the aspect (1) above, the visibility determination unit is configured to lengthen the second predetermined period the longer the period during which the gaze estimation unit determines that the occupant is viewing the gaze target.

[0011] (6) In the above aspect (1), the visibility determination unit extends the second predetermined period as the number of times the occupant has visually recognized the target recognized by the target recognition unit increases.

[0012] (7): In the above aspect (2), the visual recognition determination unit changes the length of the second predetermined period in accordance with the magnitude of the degree of coincidence calculated by the line-of-sight estimation unit.

[0013] (8): In the above aspect (1), when the visibility determination unit determines that the occupant is viewing the gaze target and that by viewing the gaze target, it is possible to recognize a target included in the gaze target, the driving assistance unit performs the driving assistance with a tendency to lower the level of vigilance toward the target.

[0014] (9): In another aspect of the present invention, a driving assistance method is provided in which a computer recognizes targets in the vicinity of a moving body, estimates which of a plurality of gaze targets, including a gaze target present on a structure of the moving body and the target, an occupant of the moving body is looking at, and if it is estimated that the occupant has been looking at one of the plurality of gaze targets for a first predetermined period or longer, it determines that the occupant is looking at the gaze target, and performs driving assistance for the moving body based on the result of the gaze determination. If it is determined that the occupant is looking at a certain gaze target and then it is estimated that the occupant has looked away from the gaze target, the determination result that the occupant is looking at the gaze target that was determined to be looked at is maintained for a second predetermined period from the time it is estimated that the occupant has looked away from the gaze target.

[0015] (10): Another aspect of the present invention provides a program that causes a computer to recognize targets in the vicinity of a moving body, estimate which of a plurality of gaze targets, including a gaze target present on a structure of the moving body and the target, an occupant of the moving body is looking at, and if it is estimated that the occupant is looking at one of the plurality of gaze targets for a first predetermined period or longer, determine that the occupant is looking at the gaze target, and perform driving assistance for the moving body based on the result of the gaze determination. If it is determined that the occupant is looking at a certain gaze target and then it is estimated that the occupant has looked away from the gaze target, maintain the determination result that the occupant is looking at the gaze target that was determined to be looked at for a second predetermined period from the time it is estimated that the occupant has looked away from the gaze target. [Effects of the Invention]

[0016] According to aspects (1) to (10), driving assistance can be flexibly performed according to the direction of the passenger's line of sight and the time. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a configuration diagram of a vehicle M on which a driving assistance device 100 according to an embodiment is mounted. [Figure 2] 1 is a diagram showing an example of the position of a driver monitor camera 60 in a vehicle M on which a driving assistance device 100 is mounted. [Figure 3] 10 is a diagram for explaining a method in which the line-of-sight estimation unit 120 detects the line of sight LS of the driver. FIG. [Figure 4] 10 is a diagram for explaining a method by which the gaze estimation unit 120 calculates the central angle and angle width of the gaze target. FIG. [Figure 5] 10 is a diagram for explaining a method by which the visual recognition determination unit 130 determines whether the driver is visually recognizing an object to be watched. FIG. [Figure 6] 10 is a diagram for explaining a method in which the visual recognition determination unit 130 calculates a second predetermined period in consideration of an overlap rate. FIG. [Figure 7]2 is a diagram for explaining driving assistance performed by a driving assistance unit 140. FIG. [Figure 8] 10 is another diagram for explaining the driving assistance performed by the driving assistance unit 140. FIG. [Figure 9] 4 is a flowchart showing an example of the flow of operations executed by the driving assistance device 100. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of a driving assistance device, a driving assistance method, and a program according to the present invention will be described with reference to the drawings.

[0019] [Overall configuration] FIG. 1 is a configuration diagram of a vehicle M on which a driving assistance device 100 according to an embodiment is mounted. The vehicle M may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell. The vehicle M is an example of a "mobile body."

[0020] The vehicle M is equipped with, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, a driver monitor camera 60, a driving operator 70, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other by multiplexed communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added.

[0021] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location of a vehicle (hereinafter referred to as vehicle M) in which the vehicle system 1 is installed. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically captures images of the periphery of the vehicle M. The camera 10 may be a stereo camera.

[0022] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method.

[0023] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 may be attached to any location on the vehicle M.

[0024] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, the radar device 12, and the LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1.

[0025] The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, a speaker, a buzzer, a vibration generator (vibrator), a touch panel, switches, keys, and the like.

[0026] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the vehicle M, and the like.

[0027] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver, a guidance control unit, and a storage unit storing map information. The GNSS receiver identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an INS (Inertial Navigation System) that uses the output of the vehicle sensors 40. The guidance control unit, for example, determines a route from the position of the vehicle M identified by the GNSS receiver (or any input position) to a destination input by the occupant by referring to map information, and causes the HMI 30 to output guidance information so that the vehicle M travels along the route. The map information is, for example, information that represents road shapes using links indicating roads and nodes connected by the links. The map information may include road curvature, POI (Point of Interest) information, and the like. The navigation device 50 may transmit the current position and destination of the vehicle M to a navigation server via a communication device and acquire the route from the navigation server.

[0028] The driver monitor camera 60 is a digital camera that uses a solid-state imaging element such as a CCD or CMOS. The driver monitor camera 60 is attached to any location on the vehicle M in a position and orientation that allows it to capture an image of the head of an occupant (hereinafter, driver) seated in the driver's seat of the vehicle M from the front (in an orientation that captures an image of the face). The driver monitor camera 60 captures an image of the interior of the vehicle M, including the driver, from its installed position, and outputs the image to the driving assistance device 100.

[0029] Fig. 2 is a diagram showing an example of the position of the driver monitor camera 60 in the vehicle M equipped with the driving assistance device 100. In Fig. 2, RVM represents the rearview mirror, FWS represents the windshield, LSM represents the left side mirror, LWS represents the left side glass, RSM represents the right side mirror, RWS represents the right side glass, DB represents the dashboard, SW represents the steering wheel, and OB represents an object (pedestrian) outside the vehicle recognized by the object recognition device 16. As shown in Fig. 2, the driver monitor camera 60 is attached, for example, to the bottom of a display device (HMI 30) provided in the center of the instrument panel of the vehicle M, and captures an image of the driver's head from the front. The rearview mirror RVM, windshield FWS, left side mirror LSM, left side window LWS, right side mirror RSM, right side window RWS, dashboard DB, and object OB are examples of "objects of gaze" by the driver, and the gaze estimation unit 120, which will be described later, determines which object of gaze the driver is directing his or her gaze at.

[0030] The driving operators 70 include, for example, an accelerator pedal, a brake pedal, a steering wheel, a shift lever, and other operators. The driving operators 70 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to some or all of the driving force output device 200, the braking device 210, and the steering device 220.

[0031] The driving force output device 200 outputs a driving force (torque) for the vehicle to travel to the driving wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components according to information input from the driving assistance device 100 or information input from the driving operator 70.

[0032] Braking device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and an ECU. The ECU controls the electric motor according to information input from driving assistance device 100 or information input from driving operator 70, so that a brake torque corresponding to the braking operation is output to each wheel. Braking device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in driving operator 70 to the cylinder via a master cylinder. Note that braking device 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake device that controls an actuator according to information input from driving assistance device 100 to transmit hydraulic pressure from a master cylinder to the cylinder.

[0033] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor applies a force to a rack and pinion mechanism to change the direction of the steered wheels. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the driving operator 70.

[0034] [Driving assistance devices] The driving assistance device 100 includes, for example, a target recognition unit 110, a line-of-sight estimation unit 120, a visibility determination unit 130, and a driving assistance unit 140. These functional units are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a HDD or flash memory of the driving assistance device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0035] The target recognition unit 110 recognizes targets present in the vicinity of the vehicle M based on the recognition result by the object recognition device 16. The target recognition unit 110 may recognize all targets present in the vicinity of the vehicle M, or may recognize only moving targets (risk targets) present in the vicinity of the vehicle M using, for example, the radar device 12 or the LIDAR 14.

[0036] Based on the output images of the camera 10 and the driver monitor camera 60, the gaze estimation unit 120 estimates which of multiple gaze targets, including gaze targets present on the structure of the vehicle M and targets recognized by the target recognition unit 110, the driver of the vehicle M is directing his or her gaze towards.

[0037] FIG. 3 is a diagram for explaining a method by which the gaze estimation unit 120 detects the driver's gaze LS. First, the gaze estimation unit 120 acquires an image including the driver of the vehicle M captured by the driver monitor camera 60, and then detects the driver's gaze using any algorithm (for example, corneal reflex method). The gaze estimation unit 120 expresses the detected gaze, for example, using polar coordinates with the driver's head as the center O. Alternatively, the gaze estimation unit 120 may express the gaze using polar coordinates with the center O being between the driver's eyebrows, for example, or more generally, the center O may be set near both of the driver's eyes. Hereinafter, the polar coordinates of the gaze LS detected by the gaze estimation unit 120 at time t will be referred to as s. t (φ s_t ,θ s_t ) is expressed as

[0038] Furthermore, when an image including a target captured by the camera 10 is acquired, the gaze estimation unit 120 calculates the central angle and the angular width of a plurality of gaze targets including the target, which are expressed by polar coordinates with the driver's head as the center O. Here, the gaze target represents a candidate that the driver of the vehicle M gazes at, as described above, and includes, for example, the rearview mirror RVM, the windshield FWS, the left side mirror LSM, the left side window LWS, the right side mirror RSM, the right side window RWS, the dashboard DB, and the target OB.

[0039] FIG. 4 is a diagram for explaining a method by which the gaze estimation unit 120 calculates the central angle and angular width of a gaze target. FIG. 4 shows, as an example, an example in which the gaze estimation unit 120 calculates the angle and angular width of a target OB as a gaze target. In FIG. 4, RP represents a reference point of the target OB (for example, a point representing the target OB, such as the center of gravity). The gaze estimation unit 120 calculates the reference point RP using polar coordinates, and hereinafter, the polar coordinates of the reference point RP of the nth gaze target calculated by the gaze estimation unit 120 at time t are referred to as a central angle (φ obj_n,t ,θ obj_n,t) Furthermore, the line-of-sight estimation unit 120 calculates the angular width of the target OB (in other words, the spread of the angle (φ, θ) of the target OB in the up, down, left, and right directions) with respect to the center O. Hereinafter, the angular width of the n-th gaze target calculated by the line-of-sight estimation unit 120 at time t is referred to as the angular width (Φ obj_n,t ,Θ obj_n,t ) where Φ represents the angular width in the φ direction, and Θ represents the angular width in the θ direction. Similarly, the gaze estimation unit 120 calculates the central angles and angular widths with respect to the center O for the rearview mirror RVM, the windshield FWS, the left side mirror LSM, the left side glass LWS, the right side mirror RSM, the right side glass RWS, and the dashboard DB, but the central angles and angular widths of these gaze targets, which have fixed positional relationships from the driver's seat, may be calculated and stored in advance.

[0040] The line of sight estimation unit 120 calculates the polar coordinates s of the line of sight LS at time t. t (φ s_t ,θ s_t ) and the central angle (φ obj_n,t ,θ obj_n,t ) and angular width (Φ obj_n,t ,Θ obj_n,t ), these values ​​are input into a probability model expressed by the following equation (1) to calculate the probability value that the driver is directing his / her gaze at each gaze target.

[0041]

number

[0042] The left side of equation (1) P(targ=n|s 0:t ,obj 1:N,0:t ) represents the probability that the driver is directing his / her gaze at the nth gaze target, given the polar coordinates of the gaze LS from time 0 to time t and the central angles and angular widths of the first to Nth gaze targets from time 0 to time t.

[0043] On the right side of equation (1), p(s t |targ=n,obj n,t) is the line of sight s when the driver is looking at object n. t When the gaze target is one of the rearview mirror RVM, windshield FWS, left side mirror LSM, left side window LWS, right side mirror RSM, right side window RWS, and dashboard DB, the probability density p(s t |targ=n,obj n,t ) is defined by a uniform distribution. On the other hand, when the gaze target is the target OB, the probability density p(s t |targ=n,obj n,t ) is defined by the following equation (2) which represents the multidimensional normal distribution.

[0044]

number

[0045] Equation (2) is the central angle of the target OB (φ obj_n,t ,θ obj_n,t ) is the average, and the angular width (Φ obj_n,t ,Θ obj_n,t ) represents a normal distribution with standard deviation. In other words, equation (2) expresses the direction of the line of sight LS as t (φ s_t ,θ s_t ) is the center direction of the target OB (φ obj_n,t ,θ obj_n,t ), the closer the variance σ is to the target OB, the higher the probability that the driver is looking at the target OB. φ (Φ obj_n,t ) and variance σ θ (Θ obj_n,t ) is a parameter Φ that is determined as a fixed value in advance. base and Θ base are defined by the following equations (3) and (4), respectively.

[0046]

number

[0047]

number

[0048] Furthermore, on the right side of equation (1), P(targ=n|s 0:t-1 ,obj 1:N,0:t-1 ) represents the previous probability that the driver is directing his / her gaze at the nth gaze target, and its initial value is a fixed value or is designed in advance depending on the number of objects present around the vehicle M. In this way, the previous probability value P(targ=n|s 0:t-1 ,obj 1:N,0:t-1 ) as the probability density p(s t |targ=n,obj n,t ), the current probability value can be calculated robustly by taking into account the previous probability value. Furthermore, on the right side of equation (1), p(s t |s 0:t-1 ,obj 1:N,0:t ) is a normalization parameter for adjusting the probability value, and is determined so that the following equation (5) holds.

[0049]

number

[0050] When it is estimated that the driver has directed his / her gaze at any one of a plurality of gaze targets for a first predetermined period or more, the visual confirmation determination unit 130 determines that the driver is visually recognizing the gaze target. Fig. 5 is a diagram for explaining a method by which the visual confirmation determination unit 130 determines whether the driver is visually recognizing a gaze target. As shown in Fig. 5, first, the visual confirmation determination unit 130 determines whether a probability value corresponding to a certain gaze target is equal to or greater than a threshold value p th Thereafter, until time t2 when the first predetermined period T1 has elapsed, the probability value is recognized as being equal to or greater than the threshold value p th If the probability is greater than or equal to the threshold value p thFor the above reasons, the visual confirmation determination unit 130 continuously determines that the driver is visually confirming the gaze target.

[0051] After that, after time t3, the probability value exceeds the threshold p th However, the probability value is less than the threshold p th Even if the gaze target is less than the predetermined time T2, the visibility determination unit 130 maintains the determination result that the driver is viewing the gaze target for the second predetermined time T2. This is because, if the driver has been directing his / her gaze toward the gaze target for at least the first predetermined time T1, it is assumed that the driver will be aware of the situation related to the gaze target for a certain period of time, even after he / she moves his / her gaze away from the gaze target. By maintaining the determination result for a certain period of time even after the driver moves his / her gaze away from the gaze target, the driving assistance unit 140 (described later) can provide driving assistance that is more in line with the driver's actual situation. More specifically, by maintaining the determination result for a certain period of time, the driving assistance unit 140 can provide driving assistance that reduces the discrepancy between the driver's awareness and the driving assistance strength, and the annoyance of the driving assistance warning that accompanies the discrepancy.

[0052] At this time, the visibility determination unit 130 may change the lengths of the first and second predetermined periods used to determine visibility depending on the type of gaze target. For example, when the gaze target is an object used to check the front or side of the vehicle M, such as the windshield FWS, the left side window LWS, the right side window RWS, or the dashboard DB, the visibility determination unit 130 may shorten the first predetermined period or lengthen the second predetermined period compared to when the gaze target is an object used to check the rear of the vehicle M, such as the rearview mirror RVM, the left side mirror LSM, or the right side mirror RSM.

[0053] Furthermore, for example, the visual recognition determination unit 130 determines the period during which the driver directs his / her gaze at the gaze target, that is, the period during which the probability value is greater than or equal to the threshold value p thThe longer the period in which the probability value is equal to or greater than the threshold value p th over a period of time that is greater than or equal to the probability value (or probability value and threshold p th The larger the integral value, the longer the second predetermined period may be set.

[0054] Furthermore, for example, the visibility determination unit 130 may set the second predetermined period longer the more targets (number of times) the driver has visually recognized among a plurality of targets recognized by the target recognition unit 110 within a certain period in the past (this indicates that the driver's driving skill is high). In this case, the visibility determination unit 130 may set the second predetermined period longer the more the driver visually recognizes a high-risk target among a plurality of targets recognized within a certain period in the past. For example, the visibility determination unit 130 may calculate a TTC (time to collision) between the vehicle M and the target, and determine that the risk of the visually recognized target is higher the shorter the TTC of the target visually recognized by the driver, and may set the second predetermined period longer.

[0055] Furthermore, for example, the visibility determination unit 130 may calculate the overlap ratio between the vehicle M and the target, and determine that the greater the overlap ratio of the target visually recognized by the driver, the higher the risk of the visually recognized target, and lengthen the second predetermined period. FIG. 6 is a diagram for explaining a method in which the visibility determination unit 130 calculates the second predetermined period in consideration of the overlap ratio. For example, the visibility determination unit 130 derives the overlap amount β as the amount of overlap between the area obtained by extending the vehicle width α of the vehicle M in the traveling direction and the area of ​​the pedestrian who is the target OB (the distance in the vehicle width direction in the example of FIG. 6). The visibility determination unit 130 derives the overlap ratio [%] by multiplying the value obtained by dividing the overlap amount β by the vehicle width α by 100 ((β / α)×100). The visibility determination unit 130 may determine that the greater the overlap ratio of the target visually recognized by the driver, the higher the risk of the visually recognized target, and lengthen the second predetermined period. By performing such processing, the period for which the determination result is maintained can be determined in accordance with the actual situation of the driver.

[0056] The driving assistance unit 140 performs driving assistance for the vehicle M based on the determination result by the visual identification determination unit 130. More specifically, when the visual identification determination unit 130 determines that the driver is visually identifying a certain gaze target, and determines that a target included in the gaze target can be recognized by visually identifying the gaze target, the driving assistance unit 140 performs driving assistance with a tendency to lower the alertness level for the target.

[0057] FIG. 7 is a diagram for explaining driving assistance performed by the driving assistance unit 140. FIG. 7 shows, as an example, a scene in which the driving assistance unit 140 performs driving assistance in a situation in which the driver visually recognizes the left side mirror LSM. In FIG. 7, symbols M1 and M2 represent other vehicles, and symbol LSM_R represents the range of targets that can be recognized by visually recognizing the left side mirror LSM. Here, the recognition range LSM_R includes the other vehicle M1 but does not include the other vehicle M2. In other words, it is assumed that the driver recognizes the other vehicle M1 by visually recognizing the left side mirror LSM, but does not recognize the other vehicle M2.

[0058] Therefore, the driving assistance unit 140 performs driving assistance with a tendency to lower the alert level for the other vehicle M1 (and increase the alert level for the other vehicle M2). FIG. 8 is another diagram for explaining driving assistance performed by the driving assistance unit 140. As shown in FIG. 8, the driving assistance unit 140, for example, displays a message on the HMI 30 warning the driver to drive the vehicle M while paying attention to the other vehicle M2 (motorcycle) that is not included in the recognition range LSM_R. Alternatively, for example, the driving assistance unit 140 may output a message warning the driver to drive the vehicle M while paying attention to the other vehicle M2 from a speaker. Alternatively, for example, the driving assistance unit 140 may output a command value to the steering device 220 to drive the vehicle M away from the other vehicle M2 that is not included in the recognition range LSM_R. In this way, by lowering the alert level of the driving assistance only for targets included in the range recognized by visually recognizing the gaze target, driving assistance that is more in line with the driver's actual situation can be performed.

[0059] [Operation flow] Next, the flow of operations executed by the driving assistance device 100 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the flow of operations executed by the driving assistance device 100. The processing of the flowchart shown in Fig. 9 is repeatedly executed by the driving assistance device 100 in a predetermined control cycle while the vehicle M is traveling.

[0060] First, the target recognition unit 110 recognizes targets present around the vehicle M based on the recognition result by the object recognition device 16 (step S100). Next, the gaze estimation unit 120 calculates the probability value of the driver's gaze pointing at each of a plurality of gaze targets including the recognized targets, and compares the calculated probability value with a threshold value p th The gaze target thus determined is estimated as the gaze target at which the driver is directing his or her gaze (step S102).

[0061] Next, the visual recognition determination unit 130 determines whether the calculated probability value of the gaze target estimated by the gaze estimation unit 120 is equal to or exceeds a threshold value p th It is determined whether the period in which the probability is equal to or greater than the first predetermined period T1 or greater (step S104). th If it is determined that the period equal to or greater than this is less than the first predetermined period T1, the visual confirmation determination unit 130 returns the process to step S102, and the visual confirmation determination unit 130 again estimates the gaze target at which the driver is directing his or her gaze.

[0062] On the other hand, the calculated probability value is the threshold p th If it is determined that the period in which the gaze is equal to or greater than the first predetermined period T1 is equal to or greater than the first predetermined period T2, the visual confirmation determination unit 130 determines that the driver is visually confirming the gaze target (step S106). Next, the visual confirmation determination unit 130 determines whether the driver has averted their gaze from the gaze target (in other words, whether the probability value is greater than or equal to the threshold value p th If it is determined that the driver has not taken his / her eyes off the gaze target, the visual confirmation determination unit 130 returns the process to step S106, and determines that the driver is continuously gazing at the gaze target.

[0063] If it is determined that the driver has averted their gaze from the gaze target, the visual confirmation determination unit 130 maintains the determination result that the driver is visually confirming the gaze target for a second predetermined period T2 after the driver averted their gaze (step S110). Next, the driving assistance unit 140 lowers the alert level for the gaze target (more specifically, the alert level for targets included in the range recognized by visually confirming the gaze target) for the second predetermined period T2, and performs driving assistance for the vehicle M (step S112). This ends the processing of this flowchart.

[0064] According to the present embodiment described above, targets in the vicinity of a moving body are recognized, and it is estimated which of a plurality of gaze targets including the recognized target the occupant of the moving body is looking at. If it is estimated that the occupant is looking at the gaze target for a first predetermined period or more, it is determined that the occupant is looking at the gaze target. After the occupant turns their gaze away from the gaze target, the determination result that the occupant is looking at the gaze target is maintained for a second predetermined period, and driving assistance for the moving body is performed with a tendency to lower the alertness level for the gaze target that is determined to be looked at by the occupant. This allows driving assistance to be performed flexibly according to the direction and time of the occupant's gaze.

[0065] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Recognizes targets around the moving object, estimating which of a plurality of gaze targets, including a gaze target present on a structure of the moving body and the target, the occupant of the moving body is directing his / her gaze to; When it is estimated that the occupant is directing his / her gaze at any one of the plurality of gaze targets for a first predetermined period or more, it is determined that the occupant is visually recognizing the gaze target; Execute driving assistance for the moving body based on the result of the visual recognition determination; When it is determined that the occupant is gazing at a certain gaze target and then it is estimated that the occupant has moved away from the gaze target, the determination result that the occupant is gazing at the gaze target that was determined to be gazed at is maintained for a second predetermined period from the time when it is estimated that the occupant has moved away from the gaze target. The driving assistance device is configured as follows.

[0066] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0067] 10 Camera 12 Radar equipment 14 LIDAR 16 Object recognition device 30 HMI 40 Vehicle Sensors 50 Navigation equipment 60 Driver monitor camera 70 Driving controls 100 Driving assistance device 110 Target recognition unit 120 Gaze estimation section 130 Visibility Judgment Section 140 Driving Support Department 200 Driving force output device 210 Brake equipment 220 Steering device

Claims

1. a target recognition unit that recognizes targets in the vicinity of the moving object; a gaze estimation unit that estimates to which gaze target an occupant of the moving body is directing his / her gaze, among a plurality of gaze targets including a gaze target present on a structure of the moving body and the target; a viewing determination unit that determines that the occupant is viewing one of the plurality of gaze targets when it is estimated that the occupant is directing his or her gaze at the gaze target for a first predetermined period or more; a driving assistance unit that performs driving assistance for the moving object based on a determination result of the visibility determination unit; Equipped with When it is determined that the occupant is visually recognizing a certain gaze target and then it is estimated that the occupant has moved away from the gaze target, the visual recognition determination unit maintains the determination result that the occupant is visually recognizing the gaze target that was determined to be visually recognized for a second predetermined period from the time when it is estimated that the occupant has moved away from the gaze target, When the visibility determination unit determines that the occupant is viewing the gaze target and that a target included in the gaze target can be recognized by viewing the gaze target, the driving assistance unit performs the driving assistance with a tendency to lower a vigilance level for the target during the second predetermined period. Driving assistance device.

2. the gaze estimation unit calculates a probability value by inputting the gaze value expressed by polar coordinates centered on the head of the occupant of the moving body into a gaze probability distribution that is defined by a central angle and an angle width of the gaze target, expressed by polar coordinates centered on the head, and that indicates a probability of viewing the gaze target, and estimates which of the plurality of gaze targets the occupant is directing his / her gaze towards based on the probability value; The angular width represents the spread of the angle of the gaze target in the up, down, left, and right directions with the center as the reference. The driving assistance device according to claim 1 .

3. the visual recognition determination unit changes the length of the second predetermined period depending on the types of the plurality of gaze targets. The driving assistance device according to claim 1 .

4. The viewing determination unit is configured to lengthen the second predetermined period as the period during which the line-of-sight estimation unit determines that the occupant is viewing the gaze target becomes longer. The driving assistance device according to claim 1 .

5. the visibility determination unit extends the second predetermined period as the number of times the occupant has visually recognized the target recognized by the target recognition unit increases; The driving assistance device according to claim 1 .

6. the visibility determination unit changes the length of the second predetermined period in accordance with the magnitude of an accumulated value of the probability values ​​calculated by the line-of-sight estimation unit. The driving assistance device according to claim 2 .

7. The computer Recognizes targets around the moving object, estimating which of a plurality of gaze targets, including a gaze target present on a structure of the moving body and the target, the occupant of the moving body is directing his / her gaze to; When it is estimated that the occupant is directing his / her gaze at any one of the plurality of gaze targets for a first predetermined period or more, it is determined that the occupant is visually recognizing the gaze target; Execute driving assistance for the moving body based on the result of the visual recognition determination; When it is determined that the occupant is gazing at a certain gaze target and then it is estimated that the occupant has moved away from the gaze target, the determination result that the occupant is gazing at the gaze target that was determined to be gazed at is maintained for a second predetermined period from the time when it is estimated that the occupant has moved away from the gaze target, When it is determined that the occupant is viewing the gaze target and that a target included in the gaze target can be recognized by viewing the gaze target, the driving assistance is performed with a tendency to lower a vigilance level for the target during the second predetermined period. Driving assistance methods.

8. On the computer, Recognizes targets around the moving object, estimates which of a plurality of gaze targets, including a gaze target present on a structure of the moving body and the target, the occupant of the moving body is directing his / her gaze to; When it is estimated that the occupant is directing his / her gaze at any one of the plurality of gaze targets for a first predetermined period or more, it is determined that the occupant is visually recognizing the gaze target; Execute driving assistance for the moving body based on the result of the visual recognition determination; When it is determined that the occupant is visually recognizing a certain gaze target and then it is estimated that the occupant has moved away from the gaze target, the determination result that the occupant is visually recognizing the gaze target that was determined to be visually recognized is maintained for a second predetermined period from the time when it is estimated that the occupant has moved away from the gaze target, When it is determined that the occupant is viewing the gaze target and that a target included in the gaze target can be recognized by viewing the gaze target, the driving assistance is performed with a tendency to lower a vigilance level for the target during the second predetermined period. program.

Images