Operation support device, vehicle, control method of operation support device, program and storage medium

Abstract

To provide a technique for determining easily whether or not an operator confirms a plurality of directional regions which contain a lateral direction of a vehicle.SOLUTION: The operation support device comprises detection means which detects a visual line direction or a face direction of an operator of a vehicle, first determination means which determines whether or not the operator visually sees a specific position by determining whether or not a prescribed area from the center of the visual line direction is contained in the specific position of a periphery confirmation device provided on the vehicle, second determination means which determines that a vehicle peripheral region capable of being confirmed by the periphery confirmation device is visually seen when it is determined that the specific position is visually seen, and third determination means which determines whether or not the operator visually sees the lateral region of the vehicle on the basis of the face direction.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a driving assistance device, a vehicle, a control method for a driving assistance device, a program, and a storage medium. [Background technology]

[0002] Patent Document 1 discloses a method for determining whether the driver's line of sight is directed toward the side mirror, evaluating whether the driver is looking at an obstacle in combination with other information other than the line of sight, and alerting the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-087117 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technique described in Patent Document 1 has a problem in that it is not possible to determine whether the side of the vehicle has been checked.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology for easily determining whether a driver has checked areas in multiple directions, including to the sides of the vehicle. [Means for solving the problem]

[0006] A driving assistance device according to one aspect of the present invention that achieves the above object includes: a detection means for detecting the line of sight or face direction of a driver of a vehicle; a first determination means for determining whether a predetermined range from the center of the line of sight is included in a specific position of a surrounding confirmation device provided in the vehicle, thereby determining whether the driver has visually checked the specific position; a second determination means for determining that a vehicle periphery area that can be confirmed by the periphery confirmation device has been visually confirmed when it is determined that the specific position has been visually confirmed; a third determination means for determining whether the driver has looked at a side area of ​​the vehicle based on the face direction; The present invention is characterized by comprising: [Effects of the Invention]

[0007] According to the present invention, it is possible to easily determine whether the driver has checked areas in multiple directions including the sides of the vehicle. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a vehicle and a control device according to an embodiment. [Figure 2] 4 is a flowchart showing a procedure of a process performed by a driving assistance device according to an embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a rear side area of ​​a vehicle according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a specific position set on a side mirror according to an embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a specific position set on a monitor according to an embodiment. [Figure 6] FIG. 2 is a diagram showing an example of left and right side regions of a vehicle according to an embodiment. [Figure 7] FIG. 10 is an explanatory diagram of a facial movement according to an embodiment. [Figure 8] FIG. 4 is an explanatory diagram of a driver's line of sight range in the up-down direction according to one embodiment. [Figure 9] FIG. 4 is an explanatory diagram of a driver's line of sight range in the left-right direction according to an embodiment. [Figure 10] 10A and 10B are diagrams showing examples of setting a forward area when turning right or left according to a modified example. [Figure 11] FIG. 10 is an explanatory diagram of notification conditions when a vehicle turns right or left according to an embodiment. [Figure 12] FIG. 6 is an explanatory diagram of a notification condition when a vehicle is ahead of a vehicle according to an embodiment. [Figure 13]FIG. 10 is an explanatory diagram of a notification condition when a pedestrian enters a predetermined area ahead of a vehicle according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0010] <Control device and its application examples> FIG. 1 is a block diagram of a control device CNT according to one embodiment of the present invention, and a schematic diagram of a vehicle V to which the control device CNT is applied. In FIG. 1, the vehicle V is shown in outline in plan view and side view. The vehicle V of this embodiment is, as an example, a four-wheeled sedan-type passenger vehicle, and may be, for example, a parallel hybrid vehicle. Note that the vehicle V is not limited to a four-wheeled passenger vehicle, and may be a saddle-type vehicle (motorcycle, motor tricycle), or a large vehicle such as a truck or bus.

[0011] The control device CNT includes a controller 1, which is an electronic circuit that controls the vehicle V, including driving assistance for the vehicle V. The controller 1 is equipped with multiple ECUs (Electronic Control Units). An ECU is provided, for example, for each function of the control device CNT. Each ECU includes a processor represented by a CPU (Central Processing Unit), a storage device such as a semiconductor memory, an interface with an external device, etc. The storage device stores programs executed by the processor and data used by the processor for processing, etc. The interface includes an input / output interface and a communication interface. Each ECU may be equipped with multiple processors, multiple storage devices, and multiple interfaces. The programs stored in the storage device may be stored in the storage device by being installed in the control device CNT using a storage medium such as a CD-ROM.

[0012] The controller 1 controls the drive (acceleration) of the vehicle V by controlling a power unit (power plant) 2. The power unit 2 is a traveling drive unit that outputs drive force to rotate the drive wheels of the vehicle V, and may include an internal combustion engine, a motor, and an automatic transmission. The motor can be used as a drive source to accelerate the vehicle V, and can also be used as a generator during deceleration, etc. (regenerative braking).

[0013] In this embodiment, the controller 1 controls the output of the internal combustion engine and the motor and controls the shifting of the gears of the automatic transmission in response to the driving operation of the driver detected by the operation detection sensor 2a provided on the accelerator pedal AP and the operation detection sensor 2b provided on the brake pedal BP, and the vehicle speed of the vehicle V detected by the rotation speed sensor 2c. The automatic transmission is provided with a rotation speed sensor 2c that detects the rotation speed of the output shaft of the automatic transmission as a sensor that detects the running state of the vehicle V. The vehicle speed of the vehicle V can be calculated from the detection result of the rotation speed sensor 2c.

[0014] The controller 1 controls the braking (deceleration) of the vehicle V by controlling the hydraulic device 3. The driver's braking operation on the brake pedal BP is converted into hydraulic pressure in the brake master cylinder BM and transmitted to the hydraulic device 3. The hydraulic device 3 is an actuator that can control the hydraulic pressure of the hydraulic oil supplied to the brake devices 3a (e.g., disc brake devices) provided on each of the four wheels based on the hydraulic pressure transmitted from the brake master cylinder BM.

[0015] The controller 1 can control the braking of the vehicle V by controlling the driving of the solenoid valves and the like provided in the hydraulic device 3. The controller 1 can also configure an electric servo brake system by controlling the distribution of braking force by the brake device 3a and braking force by regenerative braking of the motor provided in the power unit 2. The controller 1 may also turn on the brake lamps 3b during braking.

[0016] The controller 1 controls the steering of the vehicle V by controlling the electric power steering device 4. The electric power steering device 4 includes a mechanism for steering the front wheels in response to the driver's driving operation (steering operation) with respect to the steering wheel ST. The electric power steering device 4 includes a drive unit 4a that generates a driving force (sometimes referred to as steering assist torque) for assisting the steering operation or for automatically steering the front wheels of the vehicle V. The drive unit 4a includes a motor as a drive source. The electric power steering device 4 also includes a steering angle sensor 4b that detects the steering angle, a torque sensor 4c that detects the steering torque borne by the driver (called steering burden torque, to be distinguished from steering assist torque), and the like.

[0017] The controller 1 controls an electric parking brake device 3c provided on the rear wheels of the vehicle V. The electric parking brake device 3c has a mechanism for locking the rear wheels. The controller 1 can control the electric parking brake device 3c to lock and unlock the rear wheels.

[0018] The controller 1 controls an information output device 5 that notifies the driver of information inside the vehicle. The information output device 5 includes, for example, a display device 5a that notifies the driver of information by image and / or an audio output device 5b that notifies the driver of information by audio. The display device 5a includes, for example, a display device provided on the instrument panel or a display device provided on the steering wheel ST. The display device 5a may also include a head-up display. The information output device 5 may notify the occupants of information by vibration or light.

[0019] The controller 1 receives instruction inputs from a passenger (e.g., the driver) via the input device 6. The input device 6 is arranged in a position operable by the driver, and includes, for example, a group of switches 6a through which the driver issues instructions to the vehicle V, and / or a turn signal lever 6b that activates a turn signal (blinker).

[0020] The controller 1 recognizes and determines the current position and course (attitude) of the vehicle V. In this embodiment, the vehicle V is provided with a gyro sensor 7a, a GNSS (Global Navigation Satellite System) sensor 7b, and a communication device 7c. The gyro sensor 7a detects the rotational motion (yaw rate) of the vehicle V. The GNSS sensor 7b detects the current position of the vehicle V. The communication device 7c wirelessly communicates with a server that provides map information and traffic information to acquire this information. In this embodiment, the controller 1 determines the course of the vehicle V based on the detection results of the gyro sensor 7a and the GNSS sensor 7b, and sequentially acquires map information related to the course from the server via the communication device 7c and stores it in a database 7d (storage device). The vehicle V may be provided with other sensors for detecting the state of the vehicle V, such as an acceleration sensor that detects the acceleration of the vehicle V.

[0021] The controller 1 performs driving assistance for the vehicle V based on the detection results of various detection units provided in the vehicle V. The vehicle V is provided with surrounding detection units 8a to 8b, which are external sensors that detect the outside of the vehicle V (surrounding conditions), and interior detection units 9a to 9b, which are interior sensors that detect the conditions inside the vehicle (the conditions of the occupants (particularly the driver)). The controller 1 is able to grasp the surrounding conditions of the vehicle V based on the detection results of the surrounding detection units 8a to 8b, and perform driving assistance in accordance with the surrounding conditions. Furthermore, the controller 1 is able to determine, based on the detection results of the interior detection units 9a to 9b, whether the driver is performing the predetermined operational obligations imposed on the driver when driving assistance is performed.

[0022] The surroundings detection unit 8a is an imaging device that captures images in front of the vehicle V (hereinafter, sometimes referred to as the front camera 8a), and is attached, for example, to the inside of the passenger compartment of the windshield at the front of the roof of the vehicle V. The controller 1 can extract the contours of targets and lane markings (white lines, etc.) on the road by analyzing the images captured by the front camera 8a.

[0023] The surroundings detection unit 8b is a millimeter wave radar (hereinafter, may be referred to as radar 8b), and uses radio waves to detect targets around the vehicle V, and detect (measure) the distance to the target and the direction (azimuth) of the target relative to the vehicle V. In the example shown in FIG. 1, five radars 8b are provided: one in the center of the front of the vehicle V, one at each of the left and right corners of the front, and one at each of the left and right corners of the rear.

[0024] The surrounding detection unit installed in the vehicle V is not limited to the above configuration, and the number of cameras and the number of radars may be changed, or a lidar (Light Detection and Ranging: LIDAR) may be installed to detect targets around the vehicle V.

[0025] The interior detection unit 9a is an imaging device that captures images of the interior of the vehicle (hereinafter, sometimes referred to as the interior camera 9a), and is attached, for example, to the interior side of the vehicle cabin at the front roof of the vehicle V. In this embodiment, the interior camera 9a is a driver monitor camera that captures images of the driver (for example, the driver's eyes and face). The controller 1 can determine the driver's line of sight and facial direction (the direction of the line of sight and the direction of the face) by analyzing the image (the driver's facial image) captured by the interior camera 9a.

[0026] The in-vehicle detection unit 9b is a grip sensor that detects the driver's grip of the steering wheel ST (hereinafter, may be referred to as grip sensor 9b), and is provided, for example, on at least a part of the steering wheel ST. Note that the torque sensor 4c that detects the driver's steering torque may be used as the in-vehicle detection unit.

[0027] The right side mirror 10a is a mirror having a mirror surface area and is located on the right side of the vehicle V. The left side mirror 10b is a mirror having a mirror surface area and is located on the left side of the vehicle. The rearview mirror 11 is a mirror having a mirror surface area and is attached to the inside of the passenger compartment at the front of the roof of the vehicle V.

[0028] It should be noted that electronic mirrors may be provided instead of the right side mirror 10a, the left side mirror 10b, and the rearview mirror 11. That is, instead of the right side mirror 10a and the left side mirror, monitors may be provided that display images captured by cameras (not shown) that are disposed on the left and right sides of the vehicle and capture images of the area behind the vehicle. The monitors may be located inside the vehicle. Similarly, instead of the rearview mirror, a monitor may be provided that displays images captured by a camera (not shown) that captures images of the area behind the vehicle.

[0029] The right side mirror 10a, the left side mirror 10b, and the rearview mirror 11 (or each monitor) each function as a surroundings confirmation device provided in the vehicle. By using the surroundings confirmation device, various surrounding areas of the vehicle (right rear side area, left rear side area, rear area) can be confirmed. The surrounding areas of the vehicle will be described in detail later.

[0030] <Driving assistance> 2 is a flowchart showing an example of a driving assistance control process executed by the ECU included in the controller 1 according to this embodiment. In this embodiment, the controller 1 operates as a driving assistance device.

[0031] In S201, the ECU acquires information about the surroundings of the vehicle V using the surrounding detection units 8a to 8b, which are external sensors that detect the outside (surrounding conditions) of the vehicle V. The surrounding information is continuously acquired.

[0032] In S202, the ECU detects targets present around the vehicle V based on the surrounding information acquired in S201. Here, the targets may include various objects, such as traffic lights, road signs, vehicles ahead, vehicles behind, vehicles to the side, bicycles, pedestrians, running people (runners), etc.

[0033] In S203, the ECU detects (identifies) a notification target object that is to be notified to the driver from among the targets detected in S202. The notification to the driver is a notification to alert the driver, and can be made by voice and / or display via the information output device 5. The notification is made when a notification condition is satisfied. For example, a case where the notification target object suddenly approaches the vehicle V, or a case where the relative position or trajectory (movement trajectory) with respect to the vehicle V changes significantly, etc. Details of the notification condition will be described later. The notification target object is, for example, a traffic participant present in the vicinity of the vehicle V, such as a four-wheeled vehicle, a two-wheeled vehicle, a pedestrian, a bicycle, or a runner in the vicinity of the vehicle V.

[0034] In S204, the ECU detects the direction of the driver's line of sight by analyzing the image (image of the driver's face) captured by the in-vehicle camera 9a.

[0035] In S205, the ECU determines whether the driver has visually checked the specific position of the periphery confirmation device based on the detected line of sight. If this step is Yes, proceed to S206. On the other hand, if this step is No, proceed to S208.

[0036] For example, the ECU determines whether the driver has visually checked the specific position by determining whether a specific range from the center of the detected line of sight is included in the specific position of the surroundings confirmation device installed in the vehicle. Here, the specific range from the center of the line of sight can be, for example, an area (central vision area) within a range of a specific angle (e.g., 2°) from the center of the line of sight.

[0037] In this embodiment, the right side mirror 10a is mainly used as the periphery confirmation device, but the processing according to this embodiment is similarly performed for periphery confirmation devices other than the right side mirror 10a. The specific position of the right side mirror 10a may be, for example, at least one of a plurality of points 411-417 set in a mirror surface area 401 of the right side mirror 10a shown in FIG. 4. The plurality of points 411-417 may include end points 416 and 417 of the mirror surface area 401 on the side farther from the vehicle V. The plurality of points 411-417 may further include a center point 411 of the mirror surface area 401 and one or more points 412-415 within the mirror surface area around the center point 411.

[0038] Since the body of the vehicle V may be reflected in the mirror surface area of ​​the right side mirror 10a close to the vehicle V, even if a point on the mirror surface area close to the vehicle V is viewed, it is possible that the right rear lateral area is not viewed. On the other hand, if a point on the mirror surface area far from the vehicle V or its vicinity is viewed, it may be determined that the right rear lateral area is viewed.

[0039] Therefore, by including end points 416 and 417 of the mirror surface area 401 on the side farther from the vehicle V in addition to the center point 411 and multiple points 412 to 415 around it as the specific positions for the right side mirror 10a, control that is more in line with the actual driving situation becomes possible. The specific positions for the left side mirror 10b may be set symmetrically to the specific positions for the right side mirror 10a.

[0040] Note that, in the case where monitors 51, which are arranged on the left and right sides of the vehicle and display images captured by cameras (not shown) capturing images of the area behind the vehicle, are used as the periphery confirmation device instead of side mirrors, the specific position can be as shown in FIG. 5. In the case of monitor 51, the cameras (not shown) can be arranged so that the body of vehicle V is hardly visible. Therefore, unlike the case of side mirrors, the specific position can be at least one of a plurality of points 511 to 515 set in the image display area 501 of monitor 51. In other words, it is not necessary to provide end points as specific positions at the ends of monitor 51. However, even in the case of a monitor, points corresponding to points 416 and 417 shown in FIG. 4 can be arranged at the ends of monitor 51, as in the case of side mirrors.

[0041] In the case where the surroundings confirmation device is a rearview mirror 11 for checking the area behind the vehicle V, the specific position may be at least one of a plurality of points set on the mirror surface area of ​​the rearview mirror 11. The arrangement of the plurality of points may be the same as that of the monitor 51.

[0042] In S206, the ECU determines that the driver has visually checked the vehicle periphery area that can be checked by the periphery confirmation device. Here, with reference to FIG. 3, an example of the vehicle periphery area according to this embodiment will be described. When the periphery confirmation device is the right side mirror 10a, the vehicle periphery area is, for example, the right rear side area 301. When the periphery confirmation device is the left side mirror 10b, the vehicle periphery area is, for example, the left rear side area 302. When the periphery confirmation device is the rearview mirror 11, the vehicle periphery area may be a rear area (not shown). For example, the rear area (not shown) is an area between the right rear side area 301 and the left rear side area 302.

[0043] In S207, the ECU suppresses notifications for notification targets present within the vehicle periphery. Since it is assumed that the driver has already confirmed notification targets present within the vehicle periphery, providing notifications for such notification targets may be bothersome to the driver. This process can suppress excessive notifications to the driver. The notification suppression may be canceled when the driver averts their gaze from the periphery confirmation device or after a predetermined time has elapsed since the driver averted their gaze.

[0044] In S208, the ECU detects the direction of the driver's face by analyzing the image (image of the driver's face) captured by the in-vehicle camera 9a.

[0045] In S209, the ECU determines whether the driver has looked at the side area of ​​the vehicle V based on the detected face direction. If this step is Yes, proceed to S210. On the other hand, if this step is No, proceed to S211. Here, the side area is the right side area or the left side area. Here, FIG. 6 shows an example of the side area according to this embodiment. In FIG. 6, a right side area 601 and a left side area 602 are shown. The right side area 601 is set to be larger than the left side area 602. This is because this is an example in which the driver is in the right seat; if the driver is in the left seat, the right side area and the left side area may be set left and right opposite to those in FIG. 6.

[0046] As a first determination method, when the gaze direction is no longer detected, the ECU may determine whether the driver has gazed toward a side area based on the amount of change over time in the face direction before the gaze direction is no longer detected. When the driver turns his or her face toward the side area of ​​the vehicle V to check the side area, the gaze direction cannot be detected after a certain time. This is because the position of the image captured by the in-vehicle camera 9a gradually moves from the front of the face to the side of the face, which places a limit on gaze detection. Therefore, in the first determination method, it is estimated that the driver gazed toward the side based on the movement of the face before the gaze direction is no longer detected. More specifically, it may be determined that the driver gazed toward a side area when the amount of change over time (e.g., the change over time in the angle of the face direction) is equal to or greater than a threshold. For example, as shown in FIG. 7, when the face moves to the left or right by 10° per frame or more, it may be determined that the driver gazed toward the side area in the direction of movement. In the illustrated example, it is determined that the driver gazed toward the left side area.

[0047] As a second determination method, when the gaze direction is no longer detected, the ECU may determine whether the driver has looked at the lateral area based on the gaze direction before the gaze direction is no longer detected.

[0048] More specifically, if the gaze direction before detection is no longer possible is within a first predetermined angle range in the vertical direction of the driver and is equal to or greater than a second predetermined angle in one of the horizontal directions of the driver, it may be determined that the driver has looked at a lateral area in that one direction. The first predetermined angle range may be, for example, a range of +30° to -30° in the vertical direction (pitch) as shown in Fig. 8. The second predetermined angle may be, for example, 15° or greater in the horizontal direction (yaw) as shown in Fig. 9.

[0049] That is, if the driver's line of sight is within the range of +30° to -30° in the vertical direction (pitch) and is +15° or more to the left, it may be determined that the driver has looked at the right side area. If the driver's line of sight is within the range of +30° to -30° in the vertical direction (pitch) and is -15° or less to the left, it may be determined that the driver has looked at the left side area.

[0050] In S210, the ECU suppresses notification to the driver of notification objects in the side area that are determined to have been visually observed by the driver. Since it is assumed that the driver has already confirmed notification objects that exist in the side area, providing notification of such notification objects may be bothersome to the driver. This process can suppress excessive notifications to the driver. The notification suppression may be canceled if it is determined that the driver has not visually observed the side area.

[0051] In S211, the ECU determines whether or not to continue the processing. For example, the processing may be continued while the driver is starting the vehicle V (the engine is running, and the power supply of the electric vehicle is on). If this step is Yes, the process returns to S201. On the other hand, if this step is No, the process ends. This is the end of the processing in FIG. 2.

[0052] <Example of notification conditions> Here, the notification condition may be that a notification object (e.g., a bicycle, pedestrian, or runner) is detected approaching vehicle V when vehicle V makes a right or left turn. For example, the notification condition may be that a notification object (e.g., a bicycle, pedestrian, or runner) is detected crossing a crosswalk when vehicle V makes a right turn. FIG. 11 is a diagram showing a situation in which vehicle V turns right at an intersection. A notification object 1103 is about to cross a crosswalk in the direction of the right turn. At this time, a warning area 1102 may be set outside a predetermined area 1101 in the traveling direction of vehicle V, and the notification condition may be satisfied when it is detected that notification object 1103 has entered warning area 1102.

[0053] The notification condition may also be that the time to collision (TTC) is equal to or less than a threshold. That is, notification may be performed when the value obtained by dividing the distance between vehicle V and an object to be notified (e.g., a vehicle ahead) present within a predetermined area in the traveling direction of the vehicle by the relative speed between the vehicle and the object to be notified is equal to or less than a threshold. FIG. 12 is a diagram showing the relationship between vehicle V and a preceding vehicle traveling ahead of vehicle V. A preceding vehicle 1201 is present within a predetermined area 1202 in the traveling direction of vehicle V. At this time, the TTC may be continuously calculated, and the notification condition may be satisfied when the TTC is equal to or less than the threshold. Alternatively, the moving average value of the TTC (the average value of TTC changes within a predetermined period) may be constantly calculated, and the notification condition may be satisfied when a second moving average value after a predetermined time (e.g., 1 second) becomes smaller than the first moving average value.

[0054] The notification condition may also be that an object to be notified (for example, a bicycle, pedestrian, or runner) has entered a predetermined area in the traveling direction of the vehicle V (for example, a predetermined area ahead of the vehicle). FIG. 13 is a diagram showing the relationship between the vehicle V and the object to be notified. FIG. 13 shows a state in which an object to be notified 1302 has entered a predetermined area 1301 in the traveling direction of the vehicle V.

[0055] In this way, when the driver fails to recognize an object to be notified and overlooks it, a warning is sent to the driver.

[0056] As described above, according to this embodiment, while central vision is important when the driver checks the side mirrors, checking the sides can be determined by the direction of the driver's face. By taking this into account, it is possible to appropriately determine whether the driver has looked to the sides and rear of the vehicle. Therefore, it is possible to easily determine whether the driver has checked areas in multiple directions, including the sides of the vehicle.

[0057] [Variations] In the above-described embodiment, an example was described in which it was determined whether the driver had visually inspected the area around the vehicle (rear area, rear area) or the area to the side of the vehicle that can be confirmed by a surrounding confirmation device (side mirror, rearview mirror, monitor, etc.), and if the driver had visually inspected the area, notification of objects to be notified within the area was suppressed.

[0058] However, notification suppression is not limited to the rearward, rearward, and sideward regions of the vehicle. In this modification, an example will be described in which notification suppression is performed by visually determining the front region of the vehicle.

[0059] The ECU detects a driving operation of the driver. When a right turn operation is detected as the driving operation, the ECU determines whether the driver has visually checked a first forward area corresponding to the right turn operation. When a left turn operation is detected as the driving operation, the ECU determines whether the driver has visually checked a second forward area corresponding to the left turn operation.

[0060] Here, the forward area will be described in detail with reference to FIG. 10. The first forward area corresponding to a right turn operation is a forward area indicated by angle range 1001. Angle range 1001 is a range of -55° to 60°. The second forward area corresponding to a left turn operation is a forward area indicated by angle range 1002. Angle range 1002 is a range of -60° to 45°. In this way, the range of the forward area set when turning right is different from that when turning left. Angle range 1003 is a range of -55° to 30°, which corresponds to the forward area when traveling straight.

[0061] When the ECU determines that the first forward area has been viewed, it suppresses notifications for notification objects present in the first forward area. Furthermore, when the ECU determines that the second forward area has been viewed, it suppresses notifications for notification objects present in the second forward area. By setting different ranges for the forward area when turning right and when turning left, notification suppression appropriate for each of right and left turns can be achieved. More specifically, the second forward area (when turning left) is narrower than the first forward area (when turning right), and an angle range greater than 45° is not subject to notification suppression for the second forward area. Because drivers tend to look mainly to the left when turning left, controlling the system so that notifications for notification objects present in an area greater than 45° to the right are not suppressed can provide appropriate attention.

[0062] In the above-described embodiment, an example was described in which the driver visually confirmed a notification object in various areas and suppressed notifications for notification objects within the areas. Notification suppression may be canceled when the driver averts their gaze from a periphery confirmation device (e.g., corresponding to the right side mirror). However, even when the driver averts their gaze, they may still be mentally aware of a notification object in the area corresponding to the periphery confirmation device (e.g., the right rear side area). In particular, if the driver averts their gaze from the periphery confirmation device for a short period of time, they may be able to imagine the notification object in the area. Therefore, notification suppression may be continued in a situation where the driver is mentally aware of the notification object even if they are not visually confirming the area. For example, notification suppression may be continued until a predetermined time has elapsed since the driver averted their gaze from the periphery confirmation device. Furthermore, if it is determined that the driver visually confirmed the same periphery confirmation device again within that predetermined time, notification suppression for notification objects within that area may be continued. This makes it possible to suppress excessive notifications in a situation where the driver is mentally aware of a notification object even if they are not visually confirming it.

[0063] <Summary of the embodiment> 1. The driving assistance device (1) according to the above embodiment is A driving assistance device (1), A detection means (1, 9a) for detecting the line of sight or face direction of a driver of a vehicle (V); a first determination means (1) for determining whether a predetermined range from the center of the line of sight direction is included in a specific position of a surrounding confirmation device (10a, 10b, 11) provided in the vehicle, thereby determining whether the driver has visually checked the specific position; a second determination means (1) for determining, when it is determined that the specific position has been visually confirmed, that a vehicle periphery area (301, 302) that can be confirmed by the periphery confirmation device has been visually confirmed; a third determination means (1) for determining whether the driver has looked at a side area (601, 602) of the vehicle based on the face direction; Equipped with.

[0064] While central vision is important when a driver checks the side mirrors, checking the sides can be determined by the direction of the face. By reflecting this characteristic, it is possible to appropriately determine whether the driver has looked to the sides and rear of the vehicle. This makes it easy to determine whether the driver has checked areas in multiple directions, including the sides of the vehicle.

[0065] 2. The driving assistance device (1) according to the above embodiment is An object detection means (1, 8a to 8b) for detecting an object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; and a suppression means (1) for suppressing notification of the notification target, The suppression means suppresses the notification of the object to be notified that is located within the vehicle surrounding area when the second determination means determines that the vehicle surrounding area has been visually observed, and suppresses the notification of the object to be notified that is located within the side area when the third determination means determines that the side area has been visually observed.

[0066] This makes it possible to prevent excessive notifications from being made about notification targets within the area that the driver is viewing.

[0067] 3. In the driving assistance device (1) according to the above embodiment, The surrounding confirmation device includes side mirrors (10a, 10b) arranged on the sides of the vehicle, the specific position is at least one of a plurality of points (411 to 417) set in the mirror surface area of ​​the side mirror, The vehicle surrounding area is the rear side area (301, 302) of the vehicle.

[0068] This makes it possible to appropriately determine whether the driver has checked the side mirror, and therefore to appropriately determine whether the driver has looked at the area behind the vehicle.

[0069] 4. In the driving assistance device (1) according to the above embodiment, The plurality of points includes the end points (416, 417) of the mirror surface area on the side remote from the vehicle.

[0070] This allows for appropriate determination of whether the driver has looked at the area behind the vehicle, taking into account the characteristics of the side mirror (the possibility that the vehicle body may be reflected in the mirror surface area close to the vehicle body).

[0071] 5. In the driving assistance device (1) according to the above embodiment, The plurality of points further includes a center point (411) of the mirror surface area and one or more points (412 to 415) within the mirror surface area around the center point.

[0072] This makes it possible to appropriately determine whether the driver has looked at the area behind the vehicle.

[0073] 6. In the driving assistance device (1) according to the above embodiment, The surroundings confirmation device includes a monitor (51) that displays an image of the rear side of the vehicle, The specific position is at least one of a plurality of points set in the image display area (501) of the monitor, The vehicle surrounding area is a rear side area of ​​the vehicle.

[0074] This makes it possible to apply the above embodiment to so-called mirrorless vehicles. That is, since it is possible to appropriately determine whether the driver has checked the monitor, it is possible to appropriately determine whether the driver has looked at the rear side area of ​​the vehicle.

[0075] 7. In the driving assistance device (1) according to the above embodiment, The surroundings confirmation device includes a rearview mirror (11) for checking the area behind the vehicle, the specific position is at least one of a plurality of points set on a mirror surface area of ​​the rearview mirror, The vehicle surrounding area is a rear area of ​​the vehicle.

[0076] This makes it possible to properly determine whether the driver has checked the rearview mirror, and therefore to properly determine whether the driver has looked at the area behind the vehicle.

[0077] 8. In the driving assistance device (1) according to the above embodiment, When the gaze direction is no longer detected, the third determination means determines whether the driver has looked at the side area based on the amount of change in the face direction over time before the gaze direction is no longer detected (Figures 7 to 9).

[0078] This makes it possible to estimate whether the driver has looked at the side area even when the driver's line of sight cannot be detected.

[0079] 9. In the driving assistance device (1) according to the above embodiment, The third determination means determines that the driver has looked at the side area when the amount of change over time is equal to or greater than a threshold value (FIG. 7).

[0080] This makes it possible to estimate that the driver has looked at the side area from the movement of the driver's face before the driver's line of sight direction becomes undetectable.

[0081] 10. The driving assistance device (1) according to the above embodiment is a fourth determination means (1) for determining, when the gaze direction is no longer detected, whether or not the driver has looked at the side area based on the gaze direction before the gaze direction is no longer detected, The fourth determination means determines that the lateral area in one of the directions has been viewed if the gaze direction before the gaze direction is no longer detected is within a first predetermined angle range in the vertical direction of the driver and is equal to or greater than a second predetermined angle in one of the horizontal directions of the driver (Figures 8 and 9).

[0082] This makes it possible to estimate that the driver has looked at the side area, taking into account the situation before the driver's line of sight direction became undetectable.

[0083] 11. The driving assistance device (1) according to the above embodiment is The vehicle further includes an operation detection means (1) for detecting a driving operation by the driver, The driving control device further includes a fifth determination means for determining whether a first forward area (1001) corresponding to the right turn operation has been visually inspected when a right turn operation is detected as the driving operation, and for determining whether a second forward area (1002) corresponding to the left turn operation has been visually inspected when a left turn operation is detected as the driving operation, The suppression means suppresses the notification of the notification object located within the first forward area when it is determined that the first forward area has been viewed, and suppresses the notification of the notification object located within the second forward area when it is determined that the second forward area has been viewed.

[0084] This makes it possible to prevent excessive notifications to the driver when turning right or left.

[0085] 12. The driving assistance device (1) according to the above embodiment is The second anterior region is narrower than the first anterior region.

[0086] In this way, by setting an appropriate area range depending on whether the vehicle is turning right or left, notification suppression appropriate for turning right or left can be achieved.

[0087] 13. The vehicle according to the above embodiment is A vehicle is characterized by including the driving assistance device according to the above embodiment.

[0088] This allows the functions of the driving assistance device according to the above embodiment to be realized in the vehicle.

[0089] 14. The driving assistance device (1) according to the above embodiment is A control method for a driving assistance device, comprising: A detection step (S204, S208) of detecting the gaze direction or face direction of the driver of the vehicle (V); a first determination step (S205) of determining whether a predetermined range from the center of the line of sight direction is included in a specific position of a surrounding confirmation device (10a, 10b, 11) provided in the vehicle, thereby determining whether the driver has visually checked the specific position; a second determination step (S206) of determining that the vehicle surrounding area (301, 302) that can be confirmed by the surrounding confirmation device has been visually confirmed when it is determined that the specific position has been visually confirmed; a third determination step (S209) of determining whether the driver has looked at the side areas (601, 602) of the vehicle based on the face direction; It has.

[0090] 15. The program according to the above embodiment 1 is a program for causing a computer to execute a control method for a driving assistance device according to the above embodiment.

[0091] This makes it possible to realize the processing of the driving assistance device by a computer.

[0092] 16. The storage medium according to the above embodiment is A storage medium storing a program for causing a computer to execute the control method for the driving assistance device according to the above embodiment.

[0093] This makes it possible to realize the processing of the driving assistance device using the storage medium.

[0094] <Other embodiments> Furthermore, a program for realizing one or more functions described in each embodiment can be supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device can read and execute the program. The present invention can also be realized in such an embodiment.

[0095] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0096] CNT: control device, V: vehicle, 1: controller

Claims

1. A driving assistance device, a detection means for detecting the line of sight or face direction of a driver of a vehicle; a first determination means for determining whether a predetermined range from the center of the line of sight is included in a specific position of a surrounding confirmation device provided in the vehicle, thereby determining whether the driver has visually checked the specific position; a second determination means for determining that a vehicle periphery area that can be confirmed by the periphery confirmation device has been visually confirmed when it is determined that the specific position has been visually confirmed; a third determination means for determining whether the driver has looked at a side area of ​​the vehicle based on the face direction; A driving assistance device comprising:

2. an object detection means for detecting a notification object to be notified to a driver of the vehicle based on information about the vicinity of the vehicle; a suppression means for suppressing a notification to the notification object, 2. The driving assistance device according to claim 1, wherein the suppression means suppresses the notification of the notification object that exists in the vehicle surrounding area when the second determination means determines that the vehicle surrounding area has been visually observed, and suppresses the notification of the notification object that exists in the side area when the third determination means determines that the side area has been visually observed.

3. the surroundings confirmation device includes a side mirror disposed on the side of the vehicle, the specific position is at least one of a plurality of points set on a mirror surface area of ​​the side mirror, 2. The driving assistance device according to claim 1, wherein the vehicle surrounding area is a rear side area of ​​the vehicle.

4. The driving assistance device according to claim 3 , wherein the plurality of points includes an end point of the mirror surface area on a side farther from the vehicle.

5. The driving assistance device according to claim 4 , wherein the plurality of points further includes a center point of the mirror surface area and one or more points within the mirror surface area surrounding the center point.

6. The surroundings confirmation device includes a monitor that displays an image of the rear side of the vehicle, the specific position is at least one of a plurality of points set in an image display area of ​​the monitor, 2. The driving assistance device according to claim 1, wherein the vehicle surrounding area is a rear side area of ​​the vehicle.

7. the surroundings confirmation device includes a rearview mirror for checking the rear of the vehicle, the specific position is at least one of a plurality of points set on a mirror surface area of ​​the rearview mirror, 2. The driving assistance device according to claim 1, wherein the vehicle surrounding area is a rear area of ​​the vehicle.

8. 2. The driving assistance device according to claim 1, wherein, when the gaze direction is no longer detected, the third determination means determines whether the driver has looked at the side area based on the amount of change over time in the face direction before the gaze direction is no longer detected.

9. 9. The driving support device according to claim 8, wherein the third determination means determines that the driver has looked at the side area when the amount of change over time is equal to or greater than a threshold value.

10. a fourth determination means for determining, when the gaze direction is no longer detected, whether or not the driver has looked at the side area based on the gaze direction before the gaze direction is no longer detected; 2. The driving assistance device according to claim 1, wherein the fourth determination means determines that the driver has visually inspected the lateral area in one of the directions when the gaze direction before the gaze direction is no longer detected is within a first predetermined angle range in an up-down direction of the driver and is equal to or greater than a second predetermined angle in one of the directions to the left or right of the driver.

11. further comprising an operation detection means for detecting a driving operation by the driver, a fifth determination means for determining whether a first forward area corresponding to the right turn operation has been visually inspected when a right turn operation has been detected as the driving operation, and for determining whether a second forward area corresponding to the left turn operation has been visually inspected when a left turn operation has been detected as the driving operation; 3. The driving assistance device according to claim 2, wherein the suppression means suppresses the notification for the notification object that exists in the first forward area when it is determined that the first forward area has been viewed, and suppresses the notification for the notification object that exists in the second forward area when it is determined that the second forward area has been viewed.

12. The driving assistance device according to claim 11, wherein the second forward area is narrower than the first forward area.

13. A vehicle comprising the driving assistance device according to any one of claims 1 to 12.

14. A control method for a driving assistance device, comprising: a detection step of detecting a gaze direction or a face direction of a driver of a vehicle; a first determination step of determining whether the driver has visually checked the specific position by determining whether a predetermined range from the center of the line of sight is included in a specific position of a periphery confirmation device provided in the vehicle; and a second determination step of determining, when it is determined that the specific position has been visually confirmed, that a vehicle periphery area that can be confirmed by the periphery confirmation device has been visually confirmed; a third determination step of determining whether the driver has looked at a side area of ​​the vehicle based on the face direction; A control method for a driving assistance device, comprising:

15. A program for causing a computer to execute the method for controlling a driving assistance device according to claim 14.

16. A storage medium storing a program for causing a computer to execute the method for controlling a driving assistance device according to claim 14.

Images