Driving support device, driving support method, and storage medium

Abstract

A driving support device, a driving support method, and a storage medium that can flexibly perform driving support in accordance with the direction of the line of sight of an occupant, the time, and the like. The driving support device includes: an object target recognition unit that recognizes an object target in the vicinity of a mobile body; a line-of-sight estimation unit that estimates which of a plurality of gaze objects, including a gaze object present on a structure of the mobile body and the object target, the line of sight of an occupant of the mobile body is directed; a visual recognition determination unit that determines that the occupant visually recognizes an arbitrary gaze object of the plurality of gaze objects when it is estimated that the line of sight of the occupant is directed toward the gaze object for a first prescribed period or more; and a driving support unit that performs driving support of the mobile body based on the determination result of the visual recognition determination unit, the visual recognition determination unit maintaining the determination result that the occupant visually recognizes the gaze object determined to be visually recognized for a second prescribed period from the point in time at which it is estimated that the line of sight of the occupant has left the gaze object.

Description

Technical Field

[0001] This invention relates to a driving support device, a driving support method, and a storage medium. Background Technology

[0002] Previously, it was known to detect the gaze of an occupant of a moving vehicle and to support driving the vehicle by that occupant based on the detected gaze. For example, Japanese Patent Application Publication No. 2021-33570 discloses a technique for restricting the accessibility of the vehicle's operating unit when the moving vehicle is in motion and the detected gaze is not directed toward a designated area.

[0003] However, in previous technologies, driving support could not always be flexibly implemented based on the direction and time of the occupants' line of sight. Summary of the Invention

[0004] The present invention was made in consideration of such circumstances, and one of its objectives is to provide a driving support device, driving support method, and storage medium capable of flexibly performing driving support based on the direction and time of the occupant's gaze.

[0005] The driving support device, driving support method and storage medium involved in this invention adopt the following structure.

[0006] (1): A driving support device according to one aspect of the present invention includes: an object target recognition unit that identifies an object target located around a mobile body; a gaze estimation unit that estimates which of a plurality of gaze targets, including a gaze target existing on the structure of the mobile body and the object target, is being gazed at by an occupant of the mobile body; a vision recognition determination unit that determines that the occupant is visually recognizing the gaze target if it is estimated that the occupant's gaze is being gazed at any of the plurality of gaze targets for a first predetermined period or more; and a driving support unit that performs driving support for the mobile body based on the determination result of the vision recognition determination unit, wherein if the vision recognition determination unit determines that the occupant is visually recognizing a certain gaze target and then estimates that the occupant's gaze has left the gaze target, the determination result that the occupant is visually recognizing the gaze target is maintained for a second predetermined period from the time point at which it is estimated that the occupant's gaze has left the gaze target.

[0007] (2): Based on the above (1) scheme, the gaze estimation unit estimates which of the plurality of gaze objects the occupant's gaze is directed toward based on the degree of consistency between the gaze and the probability distribution of the gaze that represents the probability of visually recognizing the gaze object.

[0008] (3): Based on the above (2) scheme, the gaze estimation unit calculates the degree of consistency based on the gaze expressed in polar coordinates centered on the head of the occupant of the moving body, and the center angle and the width of the gaze object expressed in polar coordinates centered on the head, and estimates which of the multiple gaze objects the occupant's gaze is directed towards.

[0009] (4): Based on the above (1) scheme, the visual recognition determination unit changes the length of the second specified period according to the types of the multiple gaze objects.

[0010] (5): Based on the above (1) scheme, the longer the period during which the gaze estimation unit determines that the occupant is visually recognizing the object of gaze, the longer the second predetermined period is extended by the visual recognition determination unit.

[0011] (6): Based on the above (1) scheme, the more times the occupant visually recognizes the object target identified by the object target recognition unit, the longer the second specified period is extended by the visual recognition determination unit.

[0012] (7): Based on the above (2) scheme, the visual recognition determination unit changes the length of the second specified period according to the degree of consistency calculated by the gaze estimation unit.

[0013] (8): Based on the above (1) scheme, the driving support unit performs driving support by reducing the tendency to be alert to the object target when the visual recognition determination unit determines that the occupant is visually recognizing the object being gazed at and determines that the object target being gazed at can be identified by visually recognizing the object being gazed at.

[0014] (9): Another aspect of the present invention relates to a driving support method in which a computer performs the following processing: identifying an object target in the vicinity of a mobile body; presuming which of a plurality of objects of gaze, including a gaze object existing on the structure of the mobile body and the object target, is being gazed upon by an occupant of the mobile body; determining that the occupant is visually recognizing the gaze object if the presumed gaze object is being gazed upon by any of the plurality of gaze objects for a first predetermined period or more; performing driving support for the mobile body based on the determination result of the visual recognition; and maintaining the determination result that the occupant is visually recognizing the gaze object for a second predetermined period from the time point when the presumed gaze object is being gazed upon after the determination that the occupant is visually recognizing a certain gaze object.

[0015] (10): In another aspect of the present invention, the storage medium stores a program that causes a computer to perform the following processing: identify an object target in the vicinity of the mobile body; presume which of a plurality of objects of gaze, including a gaze object present on the structure of the mobile body and the object target, is being gazed upon by the occupant of the mobile body; if it is presumed that the occupant's gaze is being gazed upon by any of the plurality of objects of gaze for a first predetermined period or more, determine that the occupant is visually recognizing the object of gaze; perform driving support for the mobile body based on the determination result of the visual recognition; and if it is presumed that the occupant's gaze has left the object of gaze after it is determined that the occupant is visually recognizing it, maintain the determination result that the occupant is visually recognizing the object of gaze for a second predetermined period from the time point when it is presumed that the occupant's gaze has left the object of gaze.

[0016] According to the schemes (1) to (10), driving support can be flexibly implemented based on the direction and time of the occupant's line of sight. Attached Figure Description

[0017] Figure 1 This is a structural diagram of a vehicle M equipped with the driving support device 100 according to the implementation method.

[0018] Figure 2 This is a diagram showing an example of the placement of the driver monitoring camera 60 in a vehicle M equipped with a driver support device 100.

[0019] Figure 3 This is a diagram illustrating the method by which the line-of-sight estimation unit 120 detects the driver's line of sight LS.

[0020] Figure 4 This diagram illustrates the method by which the line-of-sight estimation unit 120 calculates the center angle and the width of the angle of the object being gazed upon.

[0021] Figure 5 This diagram illustrates the method by which the visual recognition determination unit 130 determines the visual recognition of the object being viewed by the driver.

[0022] Figure 6 This is a diagram illustrating the method by which the visual recognition determination unit 130 calculates the second specified period by considering the overlap rate.

[0023] Figure 7 This is a diagram used to illustrate the driving support performed by the driving support unit 140.

[0024] Figure 8 This is another diagram used to illustrate the driving support performed by the driving support unit 140.

[0025] Figure 9 This is a flowchart illustrating an example of the process of actions performed by the driving support device 100. Detailed Implementation

[0026] Hereinafter, embodiments of the driving support device, driving support method, and storage medium of the present invention will be described with reference to the accompanying drawings.

[0027] [Overall Structure]

[0028] Figure 1 This is a structural diagram of a vehicle M equipped with the driving support device 100 of the embodiment. The vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a secondary battery or fuel cell. The vehicle M is an example of a "mobile body".

[0029] The vehicle M is equipped with, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) system 14, an object recognition device 16, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, a driver monitoring camera 60, driving controls 70, a driving support device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected through multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. It should be noted that... Figure 1 The structure shown is just one example; you can omit part of the structure or add other structures.

[0030] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). Camera 10 is mounted anywhere on the vehicle equipped with vehicle system 1 (hereinafter referred to as vehicle M). When taking pictures of the front, camera 10 is mounted on the upper part of the windshield, behind the rearview mirror inside the vehicle, etc. Camera 10, for example, periodically and repeatedly takes pictures of the surroundings of vehicle M. Camera 10 can also be a stereo camera.

[0031] Radar device 12 radiates millimeter-wave or other radio waves around the vehicle M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and orientation) of the objects. Radar device 12 can be installed at any location on the vehicle M. Radar device 12 can also detect the position and speed of objects using FM-CW (Frequency Modulated Continuous Wave) method.

[0032] The LIDAR14 illuminates the periphery of vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR14 determines the distance to the object based on the time from emission to reception. The illuminating light can be, for example, a pulsed laser. The LIDAR14 can be mounted at any location on vehicle M.

[0033] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the cameras 10, radar device 12, and LIDAR 14 to identify the position, type, speed, etc. of objects. The object recognition device 16 outputs the recognition results to the driver support device 100. Alternatively, the object recognition device 16 can directly output the detection results from the cameras 10, radar device 12, and LIDAR 14 to the driver support device 100. The object recognition device 16 can also be omitted from the vehicle system 1.

[0034] The HMI30 provides various information to the occupants of vehicle M and accepts input operations performed by the occupants. The HMI30 includes various display devices, speakers, buzzers, vibration generating devices (vibrators), touch panels, switches, buttons, etc.

[0035] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity about a vertical axis, and an orientation sensor for detecting the orientation of the vehicle M.

[0036] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver, a guidance control unit, and a storage unit containing map information. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented using INS (Inertial Navigation System) output from the vehicle sensors 40. The guidance control unit, for example, refers to map information to determine the path from the position of the vehicle M determined by the GNSS receiver (or any input position) to the destination input by the occupant, and causes the HMI 30 to output guidance information to guide the vehicle M along the path. The map information, for example, represents the shape of a road by indicating road segments and nodes connecting those segments. The map information may also include road curvature, POI (Point of Interest) information, etc. The navigation device 50 can also send the current position and destination of the vehicle M to a navigation server via a communication device and obtain the path from the navigation server.

[0037] The driver monitoring camera 60 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS. The driver monitoring camera 60 is installed at any location within the vehicle M, positioned and facing the driver's seat from the front (with the face facing outwards), capturing the head of the occupant (hereinafter referred to as the driver). The driver monitoring camera 60 outputs an image obtained from capturing the interior of the vehicle M, including the driver, from its designated location to the driver support device 100.

[0038] Figure 2 This diagram illustrates an example of the placement of the driver monitoring camera 60 in a vehicle M equipped with the driver support device 100. Figure 2 In the diagram, RVM represents the interior rearview mirror, FWS represents the front windshield, LSM represents the left-side rearview mirror, LWS represents the left-side windshield, RSM represents the right-side rearview mirror, RWS represents the right-side windshield, DB represents the front bulkhead, SW represents the steering wheel, and OB represents an external object (pedestrian) detected by the object recognition device 16. Figure 2 As shown, the driver monitoring camera 60 is mounted, for example, on the lower part of the display device (HMI30) located in the center of the dashboard of this vehicle M, and captures the driver's head from the front. The interior rearview mirror RVM, front windshield FWS, left side rearview mirror LSM, left side windshield LWS, right side rearview mirror RSM, right side windshield RWS, front bulkhead DB, and object target OB are examples of "objects of gaze" that the driver is looking at. The gaze estimation unit 120, described later, determines which object the driver's gaze is directed towards.

[0039] The driving control unit 70 includes, for example, an accelerator pedal, a brake pedal, a steering wheel, a gear shift lever, and other control components. Sensors are installed on the driving control unit 70 to detect the amount of operation or whether operation has occurred, and the detection results are output to some or all of the driving force output device 200, the braking device 210, and the steering device 220.

[0040] The driving force output device 200 outputs driving force (torque) to the drive wheels to propel the vehicle. The driving force output device 200 may include, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the aforementioned structure based on information input from the driving support device 100 or from the driving operation device 70.

[0041] The braking device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and an ECU. The ECU controls the electric motor according to information input from the driving support device 100 or from the driving operation unit 70, so that braking torque corresponding to the braking operation is output to each wheel. The braking device 210 may have a mechanism as a backup for transmitting hydraulic pressure generated by the operation of the brake pedal included in the driving operation unit 70 to the hydraulic cylinder via the master hydraulic cylinder. It should be noted that the braking device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that transmits hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder by controlling the actuator according to information input from the driving support device 100.

[0042] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving support device 100 or from the driving operation device 70.

[0043] [Driver Support Device]

[0044] The driving support device 100 includes, for example, an object recognition unit 110, a gaze estimation unit 120, a vision recognition determination unit 130, and a driving support unit 140. These functional units are implemented by executing programs (software) through hardware processors such as CPUs (Central Processing Units). Furthermore, some or all of these components can be implemented using hardware (including circuitry) such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units), or through the coordinated use of software and hardware. The program can be pre-stored in a storage device such as an HDD or flash memory (a storage device with a non-transitory storage medium) of the driving support device 100, or stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving support device 100 by mounting the storage medium (a non-transitory storage medium) to a drive unit.

[0045] The object target recognition unit 110 identifies object targets existing around the vehicle M based on the recognition results identified by the object recognition device 16. The object target recognition unit 110 can identify all object targets existing around the vehicle M, or it can use, for example, a radar device 12 or a LIDAR 14 to identify only moving object targets (risk object targets) existing around the vehicle M.

[0046] The gaze estimation unit 120 estimates which of a plurality of gaze objects the driver of the vehicle M is looking at, including the gaze object existing on the structure of the vehicle M and the object target identified by the object target recognition unit 110, based on the output images of the camera 10 and the driver monitoring camera 60.

[0047] Figure 3 This diagram illustrates the method by which the gaze estimation unit 120 detects the driver's gaze LS. The gaze estimation unit 120 first acquires an image including the driver of the vehicle M captured by the driver monitoring camera 60, and then detects the driver's gaze using an arbitrary algorithm (e.g., corneal reflectance method). The gaze estimation unit 120 represents the detected gaze, for example, using polar coordinates centered O on the driver's head. Alternatively, the gaze estimation unit 120 may represent it using polar coordinates centered O on the driver's brow; more generally, the center O can be set near the driver's eyes. Hereinafter, the polar coordinates of the gaze LS detected by the gaze estimation unit 120 at time point t will be represented as follows:

[0048] When the gaze estimation unit 120 acquires an image containing an object target captured by the camera 10, it calculates the center angle and the width of the angle of multiple objects to be gazed upon, including the object target, expressed in polar coordinates centered on the driver's head O. Here, the objects to be gazed upon, as described above, represent candidates to be gazed upon by the driver of the vehicle M, such as the interior rearview mirror RVM, the front windshield FWS, the left side rearview mirror LSM, the left side windshield LWS, the right side rearview mirror RSM, the right side windshield RWS, the front bulkhead DB, and the object target OB.

[0049] Figure 4 This diagram illustrates the method by which the line-of-sight estimation unit 120 calculates the center angle and the width of the angle of the object being gazed upon. Figure 4 This is an example illustrating how the gaze estimation unit 120 calculates the angle and width of the object target OB as the object of gaze. Figure 4 In this context, RP represents the reference point of the object target OB (e.g., the center of gravity, or other point representing the object target OB). The line-of-sight estimation unit 120 calculates the reference point RP using polar coordinates. Hereinafter, the polar coordinates of the reference point RP of the nth gaze target calculated by the line-of-sight estimation unit 120 at time point t will be expressed as the center angle. Furthermore, the line-of-sight estimation unit 120 calculates the width of the angle of the object target OB with the center O as the reference (in other words, the angle of the object target OB). The width in the vertical and horizontal directions), hereafter, the width of the angle of the nth gaze object calculated by the gaze estimation unit 120 at time point t will be expressed as the angle width (Φ). obj_n，t Θ obj_n，t Φ represents The angular width in the direction Θ represents the angular width in the θ direction. Similarly, the line-of-sight estimation unit 120 calculates the center angle and angular width with the center O as a reference for the rearview mirror RVM inside the vehicle, the front glass FWS, the left rearview mirror LSM, the left glass LWS, the right rearview mirror RSM, the right glass RWS, and the front bulkhead DB. However, it can also calculate and store the center angle and angular width of these objects of gaze with fixed positional relationships from the driver's seat in advance.

[0050] Line of sight estimation unit 120 calculates the polar coordinates of line of sight LS at time t. And the central angle of each object of gaze. and angle width (Φ obj_n，t Θ obj_n，t When these values ​​are input into the probability model represented by the following equation (1), the probability value of the driver’s gaze being directed toward each object is calculated.

[0051]

[0052] The left side of equation (1) is P(targ=n|s 0：t obj 1：N，0：t ) represents the probability that the driver's gaze is directed toward the nth object, given the polar coordinates of the line of sight LS from time 0 to time t, and the center angle and angular width of the first to Nth objects of gaze from time 0 to time t.

[0053] On the right side of equation (1), p(s) t |targ=n,obj n，t () represents the line of sight s assuming the driver's line of sight is directed towards object n. t The probability density of the driver's gaze towards the object of gaze is as follows: Given that the object of gaze is one of the following: interior rearview mirror RVM, front windshield FWS, left side rearview mirror LSM, left side windshield LWS, right side rearview mirror RSM, right side windshield RWS, or front bulkhead DB, the probability density p(s) of the driver's gaze being directed towards the object of gaze is as follows. t |targ=n,obj n，t The probability density p(s) is defined by a uniform distribution. On the other hand, when the object of gaze is an object target OB, the probability density p(s) is... t |targ=n,obj n，t It is defined by the following equation (2) representing the multidimensional normal distribution.

[0054]

[0055] Equation (2) represents the angle of the center of the target object OB. To average and with angle width (Φ) obj_n，t Θ obj_n，t ) represents a normal distribution with standard deviation. In other words, in equation (2), the direction of the line of sight LS... The closer to the center direction of the object target OB The higher the probability that the driver's line of sight is directed towards the target OB, the higher the probability will be evaluated. In equation (2), variance... and variance σ θ (Θ obj_n，t ) through a parameter Φ that has been predetermined to a fixed value base and Θ base , is defined by the following equations (3) and (4) respectively.

[0056]

[0057]

[0058] Moreover, on the right side of equation (1), P(targ=n|s0：t-1 obj 1：N，0：t-1 Let P(targ = n|s) represent the probability of the driver's gaze being directed towards the nth object in the previous observation. Its initial value is pre-designed based on a fixed value or the number of objects surrounding the vehicle M. Thus, by using the previous probability value P(targ = n|s)... 0：t-1 obj 1：N，0：t-1 Multiply by probability density p(s) t |targ=n,ob jn，t This allows for a robust calculation of the current probability value, taking into account the previous probability value. Furthermore, on the right-hand side of equation (1), p(s) t |S 0：t-1 obj 1：N，0：t ) is the normalization parameter used to adjust the probability value, and is determined in such a way that the following equation (5) holds.

[0059]

[0060] If the visual recognition determination unit 130 presumes that the driver's gaze is directed toward any one of the multiple objects of gaze for a first predetermined period of time or more, it determines that the driver is visually recognizing that object of gaze. Figure 5 This diagram illustrates the method by which the visual recognition determination unit 130 determines the driver's visual recognition of the object being gazed upon. (See diagram for example.) Figure 5 As shown, firstly, the visual recognition determination unit 130 identifies the probability value corresponding to a certain gaze object at time point t1 as a threshold p. th That's all. Afterwards, during the period up to time t2 after the first specified period T1, this probability value is the threshold p. th In the above scenario, the visual recognition determination unit 130 determines that the driver is visually recognizing the object being looked at. Then, up to time point t3, the probability value is the threshold p. th Therefore, the visual recognition determination unit 130 continues to determine that the driver is visually recognizing the object being looked at.

[0061] Subsequently, when time point t3 is exceeded, the probability value is less than the threshold p. th However, even when the probability value is less than the threshold p... thIn this case, the visual recognition determination unit 130 also maintains the determination result that the driver is visually recognizing the object being gazed upon for a second predetermined period T2. This is because, even if the driver's gaze is directed towards the object being gazed upon for at least a first predetermined period T1, it is assumed that the driver will recognize the situation related to the object being gazed upon for a short period of time after the gaze leaves the object being gazed upon. In this way, the determination result is maintained for a certain period of time after the driver's gaze leaves the object being gazed upon, thereby enabling the driving support unit 140, described later, to perform driving support that is more consistent with the driver's actual state. More specifically, by maintaining the determination result for a certain period of time, the driving support unit 140 can provide driving support that reduces the deviation between the driver's self-awareness of recognition and the intensity of driving support, and the annoyance of driving support warnings caused by the deviation.

[0062] It should be noted that, at this time, the visual recognition determination unit 130 can also change the length of the first and second predetermined periods used for determining visual recognition according to the type of object being gazed upon. For example, when the object being gazed upon is such as the front windshield FWS, left side windshield LWS, right side windshield RWS, or front bulkhead DB, used to confirm the front or side of the vehicle M, the visual recognition determination unit 130 can shorten the first predetermined period or lengthen the second predetermined period compared to when the object being gazed upon is such as the interior rearview mirror RVM, left side rearview mirror LSM, or right side rearview mirror RSM, used to confirm the rear of the vehicle M.

[0063] Alternatively, for example, it could be the period during which the driver's gaze is directed towards the object being gazed upon, i.e., the probability value is a threshold p. th During the above period (in) Figure 5 In this context, the longer the period between time point t1 and time point t3, the longer the second predetermined period is for visual recognition determination unit 130. Alternatively, for example, visual recognition determination unit 130 may set the probability value to a threshold p. th The above period will be used to determine the probability value (or the probability value and the threshold p). th The difference is integrated, and the larger the integral value, the longer the second specified period is.

[0064] Alternatively, for example, the more object targets identified by the object target recognition unit 110 over a certain period of time, the more object targets the driver visually recognizes (meaning the driver has high driving skills), the longer the second predetermined period can be determined by the visual recognition determination unit 130. In this case, the more high-risk the object target visually recognized by the driver is among the multiple object targets identified over a certain period of time, the longer the second predetermined period can be determined by the visual recognition determination unit 130. For example, the visual recognition determination unit 130 can calculate the TTC (time to collision) between the vehicle M and the object target; the smaller the TTC of the object target visually recognized by the driver, the higher the risk assessment of the visually recognized object target, and the longer the second predetermined period can be determined.

[0065] Alternatively, for example, the visual recognition determination unit 130 may calculate the overlap rate between the vehicle M and the object target. The greater the overlap rate of the object target visually recognized by the driver, the higher the risk assessment of the visually recognized object target, and the longer the second prescribed period. Figure 6 This diagram illustrates the method by which the visual recognition determination unit 130 calculates the second specified period considering the overlap rate. For example, the visual recognition determination unit 130 calculates the overlap between the area extending from the vehicle width α of the vehicle M in the direction of travel and the area of ​​the pedestrian as the object target OB. Figure 6 In the example, the overlap amount β is the distance in the vehicle width direction. The visual recognition determination unit 130 derives the overlap rate [%] by multiplying the value obtained by dividing the overlap amount β by the vehicle width α by 100. Alternatively, the greater the overlap rate of the object target visually recognized by the driver, the higher the risk assessment of the visually recognized object target by the visual recognition determination unit 130, and the longer the second predetermined period. Through this processing, the period for maintaining the determination result can be determined in a way that better matches the driver's actual state.

[0066] The driving support unit 140 performs driving support for the vehicle M based on the determination result made by the vision recognition determination unit 130. More specifically, when the vision recognition determination unit 130 determines that the driver is visually recognizing a certain object and determines that the driver can identify the object contained in the object through visual recognition, the driving support unit 140 performs driving support by reducing the degree of vigilance towards that object.

[0067] Figure 7 This is a diagram used to illustrate the driving support performed by the driving support unit 140. Figure 7 As an example, this describes a scenario where the driver visually recognizes the left-side rearview mirror (LSM) and the driver support unit 140 performs driver support. Figure 7 In the attached diagram, reference numerals M1 and M2 represent other vehicles, and reference numeral LSM_R represents the range of objects that can be identified visually through the left-side rearview mirror (LSM). Here, the identification range LSM_R includes other vehicle M1 but excludes other vehicle M2. That is, it is assumed that the driver identifies other vehicle M1 visually through the left-side rearview mirror (LSM), but it is also assumed that the driver does not identify other vehicle M2.

[0068] Therefore, the driving support unit 140 provides driving support with a tendency to reduce the alertness of other vehicles M1 (increase the alertness of other vehicles M2). Figure 8 This is another diagram used to illustrate the driving support performed by the driving support unit 140. (Example) Figure 8 As shown, the driving support unit 140 displays a message on the HMI 30, for example, warning the driver to be aware of other vehicles M2 (motorcycles) not included in the recognition range LSM_R. Alternatively, the driving support unit 140 may output a message warning the driver to be aware of other vehicles M2 from a speaker. Alternatively, the driving support unit 140 may output a command value to the steering device 220 to cause the vehicle M to move away from other vehicles M2 not included in the recognition range LSM_R. In this way, by reducing the alertness of driving support only for objects within the range identified through visual observation, driving support that is more consistent with the driver's actual state can be implemented.

[0069] [Action Flow]

[0070] Next, refer to Figure 9 This will explain the process of the actions performed by the driving support device 100. Figure 9 This is a flowchart illustrating an example of the process of actions performed by the driving support device 100. Figure 9 The flowchart shown illustrates the process where the driving support device 100 repeatedly executes a prescribed control loop while the vehicle M is in motion.

[0071] First, the object target recognition unit 110 identifies objects existing around the vehicle M based on the recognition results identified by the object recognition device 16 (step S100). Next, the gaze estimation unit 120 calculates the probability value of the driver's gaze direction for each of the multiple objects being gazed upon, including the identified objects, and sets the calculated probability value as a threshold p. th The object of gaze mentioned above is presumed to be the object of gaze toward which the driver's line of sight is directed (step S102).

[0072] Next, the visual recognition determination unit 130 determines the calculated probability value of the object of gaze that the driver's gaze is presumed to be directed by the gaze estimation unit 120, based on a threshold p.th Whether the above period is greater than or equal to the first specified period T1 (step S104). The calculated probability value is determined to be the threshold p. th If the above period is less than the first specified period T1, the visual recognition determination unit 130 returns the processing to step S102, and the visual recognition determination unit 130 once again presumes the object that the driver's gaze is directed towards.

[0073] On the other hand, when the calculated probability value is determined to be a threshold p th If the above period is a first predetermined period T1 or longer, the visual recognition determination unit 130 determines that the driver is visually recognizing the object being gazed at (step S106). Next, the visual recognition determination unit 130 determines whether the driver's gaze has left the object being gazed at (in other words, whether the probability value has become less than the threshold p). th (Step S108). If it is determined that the driver's gaze has not left the object being gazed upon, the visual recognition determination unit 130 returns the process to step S106, and the visual recognition determination unit 130 continues to determine that the driver is visually recognizing the object being gazed upon.

[0074] If it is determined that the driver's gaze has left the object being gazed upon, the visual recognition determination unit 130 maintains the determination that the driver is visually recognizing the object being gazed upon for a second predetermined period T2 from the moment the driver's gaze leaves (step S110). Next, during the second predetermined period T2, the driving support unit 140 reduces the alert level for the object being gazed upon (more specifically, the alert level for objects within the range identified by visual recognition of the object being gazed upon) and performs driving support for the vehicle M (step S112). Thus, the processing of this flowchart ends.

[0075] According to this embodiment as described above, objects surrounding the mobile vehicle are identified, and it is estimated which of a plurality of objects the occupant's gaze is directed toward, including the identified objects. If the occupant's gaze is estimated to be directed toward the object for a first predetermined period or more, it is determined that the occupant is visually recognizing that object. After the occupant's gaze leaves the object, the determination that the occupant is visually recognizing the object is maintained for a second predetermined period, and driving support for the mobile vehicle is performed to reduce the occupant's tendency to lower their alertness toward the determined visually recognized object. Thus, driving support can be flexibly performed based on the direction and duration of the occupant's gaze.

[0076] The implementation methods described above can be performed as follows.

[0077] A driving support device, comprising:

[0078] Storage medium, which stores computer-readable instructions; and

[0079] The processor, which is connected to the storage medium,

[0080] The processor performs the following processing by executing computer-readable instructions:

[0081] Identify objects surrounding the moving object;

[0082] Predict which of a plurality of gaze objects the occupants of the moving body are looking toward, including the gaze object present on the structure of the moving body and the object target;

[0083] If it is presumed that the occupant's gaze is directed toward any of the plurality of objects of gaze for a first predetermined period of time or more, it is determined that the occupant is visually recognizing the object of gaze.

[0084] Based on the determination result of the visual recognition, driving support for the mobile body is performed;

[0085] If it is determined that the occupant is visually recognizing an object and it is presumed that the occupant's gaze has left the object, the determination that the occupant is visually recognizing the object will be maintained for a second specified period from the time point at which it is presumed that the occupant's gaze has left the object.

[0086] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A driving support device, wherein, The driving support device includes: The object target recognition unit identifies object targets located around the moving body; A gaze estimation unit that estimates which of a plurality of gaze objects the occupant of the moving body is looking toward, including a gaze object present on the structure of the moving body and the object target; The visual recognition determination unit determines that the occupant is visually recognizing the object being gazed at when it is presumed that the occupant's gaze is directed toward any one of the plurality of objects being gazed at for a first predetermined period of time or more. as well as The driving support unit performs driving support for the mobile body based on the determination result of the visual recognition determination unit. If the visual recognition determination unit determines that the occupant is visually recognizing an object and then presumes that the occupant's gaze has left the object, it will maintain the determination that the occupant is visually recognizing the object for a second predetermined period from the point in time when it presumes that the occupant's gaze has left the object. The gaze estimation unit estimates which of the plurality of gaze objects the occupant's gaze is directed toward based on the degree of consistency between the probability distributions of the gazes, where the probability distributions of the gazes represent the probability of visually recognizing the gaze object. The gaze estimation unit calculates the degree of consistency based on the gaze expressed in polar coordinates centered on the occupant's head, and the center angle and width of the gaze object expressed in polar coordinates centered on the head, and estimates which of the plurality of gaze objects the occupant's gaze is directed toward.

2. The driving support device according to claim 1, wherein, The visual recognition determination unit changes the length of the second specified period based on the types of the multiple objects being gazed at.

3. The driving support device according to claim 1, wherein, The longer the period during which the gaze estimation unit determines that the occupant is visually recognizing the object of gaze, the longer the visual recognition determination unit extends the second predetermined period.

4. The driving support device according to claim 1, wherein, The more times the occupant visually identifies an object target recognized by the object target recognition unit, the longer the second predetermined period is extended by the visual recognition determination unit.

5. The driving support device according to claim 1, wherein, The visual recognition determination unit changes the length of the second specified period based on the degree of consistency calculated by the gaze estimation unit.

6. The driving support device according to claim 1, wherein, When the visual recognition determination unit determines that the occupant is visually recognizing the object being gazed upon, and determines that the object being gazed upon can be identified by visually recognizing the object being gazed upon, the driving support unit performs driving support by tending to reduce the level of alertness towards the object being gazed upon.

7. A driving support method, wherein, The driving support method causes the computer to perform the following processing: Identify objects surrounding the moving object; Predict which of a plurality of gaze objects the occupants of the moving body are looking toward, including the gaze object present on the structure of the moving body and the object target; If it is presumed that the occupant's gaze is directed toward any of the plurality of objects of gaze for a first predetermined period of time or more, it is determined that the occupant is visually recognizing the object of gaze. Based on the determination result of the visual recognition, driving support for the mobile body is performed; If it is determined that the occupant is visually recognizing an object and it is presumed that the occupant's gaze has left the object, the determination that the occupant is visually recognizing the object will be maintained for a second specified period from the time point when it is presumed that the occupant's gaze has left the object. Based on the degree of consistency between the probability distributions of the gazes, it is inferred which of the multiple gaze objects the occupant's gaze is directed towards, where the probability distribution of the gazes represents the probability of visually recognizing the gaze object. The degree of consistency is calculated based on the line of sight expressed in polar coordinates centered on the occupant's head, and the center angle and width of the gaze object expressed in polar coordinates centered on the head, and the occupant's line of sight is inferred to be directed toward which of the plurality of gaze objects.

8. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Identify objects surrounding the moving object; Predict which of a plurality of gaze objects the occupants of the moving body are looking toward, including the gaze object present on the structure of the moving body and the object target; If it is presumed that the occupant's gaze is directed toward any of the plurality of objects of gaze for a first predetermined period of time or more, it is determined that the occupant is visually recognizing the object of gaze. Based on the determination result of the visual recognition, driving support for the mobile body is performed; If it is determined that the occupant is visually recognizing an object and it is presumed that the occupant's gaze has left the object, the determination that the occupant is visually recognizing the object will be maintained for a second specified period from the time point when it is presumed that the occupant's gaze has left the object. Based on the degree of consistency between the probability distributions of the gazes, it is inferred which of the multiple gaze objects the occupant's gaze is directed towards, where the probability distribution of the gazes represents the probability of visually recognizing the gaze object. The degree of consistency is calculated based on the line of sight expressed in polar coordinates centered on the occupant's head, and the center angle and width of the gaze object expressed in polar coordinates centered on the head, and the occupant's line of sight is inferred to be directed toward which of the plurality of gaze objects.

Images