Notification control device, notification control method, and program

Abstract

To provide a notification control device, a notification control method, and a program that can provide more appropriate notifications to the driver. [Solution] The notification control device of the embodiment comprises a monitoring direction recognition unit that recognizes the monitoring direction of the driver of a moving object, a determination unit that determines whether the monitoring direction recognized by the monitoring direction recognition unit is appropriate, and a notification control unit that controls notification to the driver according to the determination result by the determination unit, wherein the notification control unit makes it easier to terminate the notification if the accuracy of the monitoring direction recognition by the monitoring direction recognition unit decreases during the notification.

Description

Technical Field

[0001] The present invention relates to an alert control device, an alert control method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development related to preventive safety technologies have been focused on further improving traffic safety and convenience through research and development. In this context, conventionally, there is a known technology that uses an image captured by a driver camera to recognize the driver's line of sight and the direction of the face, and provides support to the driver when it is determined from the recognition result that the driver is engaged in distracted driving (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in conventional preventive safety technologies, due to changes in the surrounding situation of a moving body or the like, the recognition accuracy of the monitoring direction of the driver recognized from an image or the like changes, so there is a possibility that the state of the driver such as distracted driving cannot be appropriately determined. Therefore, there has been a problem that alert for distracted driving may not be appropriately controlled.

[0005] One of the objectives of the present application is to provide an alert control device, an alert control method, and a program that can more appropriately alert the driver in order to solve the above problems. And by extension, it contributes to the development of a sustainable transportation system.

Means for Solving the Problems

[0006] The notification control device, notification control method, and program according to this invention employ the following configuration. (1) A notification control device according to one aspect of the present invention comprises: a monitoring direction recognition unit that recognizes the monitoring direction of the driver of a moving object; a determination unit that determines whether the monitoring direction recognized by the monitoring direction recognition unit is appropriate; and a notification control unit that controls notification to the driver according to the determination result by the determination unit, wherein the notification control unit makes it easier to terminate the notification if the accuracy of the monitoring direction recognition by the monitoring direction recognition unit decreases during the notification.

[0007] (2) In the embodiment of (1) above, if the accuracy of the recognition of the monitoring direction decreases during the notification, the notification control unit terminates the notification regardless of the determination result of the determination unit regarding whether the monitoring direction is appropriate or not.

[0008] (3) In the embodiment of (1) above, the monitoring direction recognition unit recognizes the monitoring direction using at least one of the driver's eye movements and face movements, and the determination unit determines that the driver's monitoring direction recognized by the monitoring direction recognition unit, at least based on the eye movements, is inappropriate, and that the accuracy of the recognition of the monitoring direction decreases during notification by the notification control unit, it determines whether the driver's monitoring direction is appropriate based on the monitoring direction recognized based on the driver's face movements without using the eye movements.

[0009] (4) In the embodiment of (1) above, the notification control unit starts notifying the driver when the determination unit determines that the monitoring direction recognized by the monitoring direction recognition unit, at least based on the movement of the driver's face, is inappropriate, and makes it easier to terminate the notification when the accuracy of the monitoring direction recognition unit's recognition of the monitoring direction decreases during the notification.

[0010] (5) In the embodiment of (1) above, the notification control unit, when the determination unit determines during the notification that the driver's monitoring direction is an appropriate monitoring direction, counts the time during which it is determined to be an appropriate monitoring direction, and terminates the notification if it continues for a predetermined time or longer. If it is determined that the accuracy of the recognition of the driver's monitoring direction has decreased while the time has been counted for the monitoring direction recognized based on the driver's eye movements and facial movements which has been determined to be an appropriate monitoring direction, and the monitoring direction recognized based on the orientation of the driver's face is determined to be an appropriate monitoring direction, the unit continues the count.

[0011] (6) Another aspect of the present invention is a notification control method in which a computer recognizes the direction of monitoring of the driver of a moving object, determines whether the recognized monitoring direction is appropriate or not, controls the notification to the driver according to the result of the determination, and makes it easier to terminate the notification if the accuracy of the recognition of the monitoring direction decreases during the notification.

[0012] (7) Another aspect of the present invention is a program that causes a computer to recognize the direction of monitoring of the driver of a moving object, to determine whether the recognized monitoring direction is appropriate, to control the notification to the driver according to the result of the determination, and to make it easier to terminate the notification if the accuracy of the recognition of the monitoring direction decreases during the notification. [Effects of the Invention]

[0013] According to the above-mentioned methods (1) to (7), the driver can be notified more appropriately. [Brief explanation of the drawing]

[0014] [Figure 1] This is a configuration diagram of a vehicle system 1 including a notification control device according to an embodiment. [Figure 2] This diagram shows the relationship between the driver's monitoring direction and the area being monitored. [Figure 3]This flowchart shows an example of a process performed by the driver assistance device 100 in the embodiment. [Modes for carrying out the invention]

[0015] The following describes embodiments of the notification control device, notification control method, and program of the present invention with reference to the drawings. Below, an example of the notification control device being applied to a mobile body will be described. A vehicle will be used as an example of a mobile body. In addition to vehicles, mobile bodies may include, for example, ships that can move on the ground (roads) such as hovercraft, flying vehicles that can travel on roads, stand-up vehicles with power units, and micromobility such as electric kick scooters.

[0016] [Overall structure] Figure 1 is a diagram showing the configuration of a vehicle system 1 including a notification control device according to an embodiment. The vehicle on which the vehicle system 1 is installed (hereinafter referred to as vehicle M) is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle or a micromobility, and its power source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a battery (storage battery) such as a secondary battery or a fuel cell.

[0017] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an in-cabin camera 70, a driver control unit 80, a driver assistance device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Note that the configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. Camera 10, radar device 12, and LiDAR 14 are examples of "detection devices DD". HMI 30 is an example of a "notification unit". HMI 30 and driver assistance device 100 are examples of "notification control devices".

[0018] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle M on which the vehicle system 1 is installed. When imaging the front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, or on the front of the vehicle body. When imaging the rear, camera 10 is mounted on the top of the rear windshield or on the tailgate. When imaging the sides, camera 10 is mounted on the left and right door mirrors, etc. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may also be a stereo camera.

[0019] The radar device 12 emits radio waves (radar) such as millimeter waves around the vehicle M, and detects radio waves (reflected waves) reflected by surrounding objects to detect at least the position (distance and azimuth) of the objects. The radar device 12 is attached to an arbitrary location of the vehicle M. The radar device 12 may detect the position and speed of an object by the FM-CW (Frequency Modulated Continuous Wave) method.

[0020] The LIDAR 14 irradiates light around the vehicle M and measures scattered light. The LIDAR 14 detects the distance to the target based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is attached to an arbitrary location of the vehicle M.

[0021] The communication device 20 communicates with, for example, other vehicles existing around the vehicle M, the terminal device of the user using the vehicle M, or various server devices by using networks such as a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet.

[0022] The HMI 30 outputs various types of information to the passengers (including the driver) of the vehicle M and accepts input operations by the passengers. The HMI 30 includes, for example, a display unit 32 and a speaker 34. The display unit 32 is, for example, an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display device, or the like. The display unit 32 displays various images (including videos) in the embodiment. The display unit 32 may be integrally configured with the input unit as a touch panel. The speaker 34 outputs a predetermined sound (such as a notification sound or a message sound). Further, the HMI 30 may include a microphone, a buzzer, a touch panel, a switch, a key, etc. The switch may include a switch for executing or ending a predetermined driving control or the like that can be executed by a driving control unit described later, a switch for approving (permitting) or rejecting a recommendation (proposal) for driving control from the system (vehicle system 1) side, etc. The switch may also include a switch for performing a direction indication operation (such as a turn signal switch).

[0023] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects acceleration, and a yaw rate sensor that detects a yaw rate (for example, the rotational angular velocity around a vertical axis passing through the center of gravity point of the vehicle M). Further, the vehicle sensor 40 may include a lateral acceleration sensor (lateral G sensor) that detects the lateral acceleration (lateral G) of the vehicle M, a steering angle sensor that detects the steering angle of the vehicle M (which may be the angle of the steering wheel or the operation angle of the steering wheel), a steering angular velocity sensor that detects the steering angular velocity, a direction sensor that detects the direction of the vehicle M, etc.

[0024] Furthermore, the vehicle sensor 40 may include a position sensor that detects the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using, for example, a GNSS (Global Navigation Satellite System) receiver of a navigation device 50. The vehicle sensor 40 may derive the speed of the vehicle M from the difference (i.e., distance) of position information at a predetermined time from the position sensor. The results detected by the vehicle sensor 40 are output to the driver assistance device 100.

[0025] The navigation device 50 includes, for example, a GNSS receiver, a navigation HMI, and a route determination unit. The navigation device 50 may store map information in a storage device such as an HDD (Hard Disk Drive) or flash memory, or it may acquire map information 192 stored in a storage unit 190, which will be described later. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI includes a display device, speaker, touch panel, keys, etc. The GNSS receiver may be provided on the vehicle sensor 40. The navigation HMI may be partially or completely shared with the HMI 30 described above. The route determination unit determines, for example, a route (hereinafter referred to as a route on a map) from the position of the vehicle M determined by the GNSS receiver (or any input position) to a destination input by the occupant using the navigation HMI, by referring to, for example, map information 192, etc. Furthermore, the navigation device 50 provides route guidance using the navigation HMI based on the determined route on the map. The navigation device 50 may also transmit its current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0026] Here, map information 192 is information in which the road shape is represented by links indicating roads (an example of a travel route) and nodes connected by those links. Map information 192 may also include POI (Point of Interest) information, etc. Map information 192 also includes, for example, the number of lanes (number of travel routes), the type and shape of road markings, information on the center of the lanes, or information on road boundaries. Map information 192 may also include information on whether the road boundary is a boundary (physical boundary) that includes structures that vehicles cannot pass through (including crossing and contact). Physical boundaries include, for example, guardrails, curbs, median strips, fences, etc. Map information 192 may also include road shape information, traffic regulation information, address information (address and postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, the curvature of the road (which may be rephrased as radius of curvature; the same applies below), width, road surface gradient, branching and merging points, intersections, T-junctions, etc. Furthermore, the map information 192 may include information on structures around roads, information indicating the location (section) of tunnels, and information indicating the location (section) of roads passing under bridges or overpasses. The map information 192 may be updated as needed by the communication device 20 communicating with an external device.

[0027] The in-vehicle camera 70 is, for example, a digital camera using a solid-state image sensor such as a CCD or CMOS. The in-vehicle camera 70 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the driver seated in the driver's seat of the vehicle M from the front. For example, the in-vehicle camera 70 is mounted near (for example, above or below) the display device located in the center of the instrument panel of the vehicle M. The in-vehicle camera 70 may also capture the interior of the vehicle, including the area of ​​the occupants (passengers) seated in the passenger seat or other seats of the vehicle M, in addition to the driver. The in-vehicle camera 70 may also capture the interior of the vehicle by irradiating it with infrared light. The in-vehicle camera 70 captures the interior of the vehicle repeatedly and periodically, for example.

[0028] The driver control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driver control unit 80 may also include a shift lever, a modified steering wheel, a joystick, or other controls. Each control of the driver control unit 80 is equipped with an operation detection unit that detects, for example, the amount of operation performed by the driver on the control unit or whether or not an operation has been performed. The operation detection unit detects, for example, the steering angle and steering torque of the steering wheel (for example, the amount of steering due to the driver's driving operation (steering input torque)), the rate of change of the steering torque, the amount of depression of the accelerator pedal and brake pedal, etc. The operation detection unit then outputs the detection results to the driver assistance device 100, or to one or both of the driving force output device 200, the brake device 210, and the steering device 220. The driver control unit 80 may also include a turn signal control unit (for example, a turn signal lever, a turn signal switch). When the turn signal control unit is operated, the turn signal of the vehicle M corresponding to the operation will flash, and the operation details (including, for example, the detection result that the operation was performed by the driver) will be output to the driver assistance device 100.

[0029] The driver assistance device 100 performs various controls to assist the driver of the vehicle M. The driver assistance device 100 includes, for example, a recognition unit 120, a determination unit 140, an HMI control unit 160, a driving control unit 180, and a storage unit 190. The recognition unit 120, the determination unit 140, the HMI control unit 160, and the driving control unit 180 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The above-mentioned program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the driver assistance device 100, or it may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driver assistance device 100 when the storage medium (non-transient storage medium) is inserted into a drive device or card slot. The driving control unit 180 is an example of a "mobility control unit". The HMI control unit 160 is an example of a "notification control unit".

[0030] The storage unit 190 may be implemented using the various storage devices described above, or an EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 190 stores, for example, map information 192, various information in the embodiment, programs, etc. The storage unit 190 may also store various setting information used in the processing in this embodiment.

[0031] The recognition unit 120 includes, for example, a surrounding recognition unit 122, a monitoring direction recognition unit 124, and a behavior recognition unit 126. The surrounding recognition unit 122 recognizes the surrounding conditions of the vehicle M based on, for example, the detection results of the detection device DD (information input from the camera 10, radar device 12, and LIDAR 14). For example, the surrounding recognition unit 122 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the position (relative position), size, speed (relative speed), acceleration, and other states of objects present around the vehicle M (within a predetermined distance). Objects recognized by the surrounding recognition unit 122 may include, for example, physical boundaries that demarcate roads (travel paths), as well as other vehicles, pedestrians, bicycles, and other traffic participants (examples of obstacles). The position of an object is recognized as a position on an absolute coordinate system with the vehicle M's representative point (such as the center of gravity or the drive axis center) as the origin, and is used for control. The position of an object may be represented by its center of gravity, a representative point such as a corner, or by the represented region. The "state" of an object may include, for example, the acceleration or jerk of another vehicle, or its "action state" (for example, whether the other vehicle is changing lanes or is about to change lanes).

[0032] Furthermore, the surrounding area recognition unit 122 may recognize, for example, stop lines, red lights, toll booths, other road events, road signs, and markings drawn on the road (e.g., speed limits). The surrounding area recognition unit 122 may also recognize the curvature of the vehicle M's lane (road) based on the detection results of the detection device DD or the map information 192. The surrounding area recognition unit 122 may also recognize the road surface conditions (e.g., whether the road surface is slippery, such as being frozen) based on the detection results of the detection device DD.

[0033] Furthermore, the surrounding recognition unit 122 recognizes, for example, the lane in which the vehicle M is traveling (driving lane) and other surrounding lanes (for example, adjacent lanes). For example, the surrounding recognition unit 122 recognizes road markings from images captured by the camera 10 and recognizes the driving lane and other lanes based on the positional relationship of the road markings as seen from the recognized vehicle M. Alternatively, the surrounding recognition unit 122 may refer to map information 192 based on the position information of the vehicle M obtained from the vehicle sensor 40, etc., to recognize the lane in which the vehicle M is traveling and other lanes. The surrounding recognition unit 122 may also recognize structures such as tunnels, elevated roads, and overpasses, as well as structures around the road.

[0034] The monitoring direction recognition unit 124 recognizes the state of the occupants of vehicle M using images captured by the in-vehicle camera 70. For example, the monitoring direction recognition unit 124 performs known image analysis processing on the images captured by the in-vehicle camera 70 (e.g., feature extraction of edges, shape, size, and color, pattern matching, etc.) and recognizes the monitoring direction of the driver of vehicle M based on the image analysis results. For example, the monitoring direction recognition unit 124 recognizes the monitoring direction using at least one of the driver's eye movements and face movements obtained from the image analysis results. Eye movements are, for example, the line of sight (the direction the driver is looking). Face movements are, for example, the orientation of the face.

[0035] For example, the monitoring direction recognition unit 124 uses methods such as pattern matching to detect a combination of a reference point (the stationary part of the eye) and a moving point (the moving part of the eye) of the driver's eye from the captured image. The combination of the reference point and the moving point is, for example, a combination of the inner corner of the eye and the iris, or a combination of the corneal reflection region and the pupil. The corneal reflection region is, for example, the region of infrared light reflection on the cornea when the in-vehicle camera 70 shines infrared light towards the driver. Then, based on the position of the moving point relative to the reference point, the monitoring direction recognition unit 124 performs coordinate transformations from the image plane to real space and recognizes the driver's line of sight.

[0036] Furthermore, the monitoring direction recognition unit 124 recognizes the orientation of the driver's face based on positional information of the eyes, nose, mouth, etc. within the face region (relative positional information of each part, etc.) obtained from the analysis results of the captured image.

[0037] The monitoring direction recognition unit 124 recognizes the monitoring direction using both the driver's gaze and the direction of their face when it can recognize both. This allows for more accurate recognition of the monitoring direction. If the monitoring direction recognition unit 124 recognizes only one of the gaze or the direction of the face, it uses only that one to recognize the monitoring direction.

[0038] Furthermore, the monitoring direction recognition unit 124 may recognize, based on the analysis results of the image from the in-vehicle camera 70, that the driver is wearing sunglasses, glasses, or a mask, etc., through template matching or the like. In this case, if the driver is wearing sunglasses or glasses, the monitoring direction recognition unit 124 recognizes the monitoring direction based only on the direction of the face. If the driver is wearing a mask, the accuracy of recognizing the direction of the face decreases because distinctive features such as the nose and mouth cannot be recognized from the image, so the monitoring direction is recognized based only on the gaze.

[0039] In recognizing each piece of information using the images described above, a pre-trained model, for example, that has been trained in advance by machine learning, may be used. In this case, the monitoring direction recognition unit 124 takes an image as input and outputs the monitoring direction of a person contained in the image (for example, gaze or face direction), and inputs the image captured by the in-vehicle camera 70 to obtain the driver's monitoring direction. Alternatively, the monitoring direction recognition unit 124 may recognize the monitoring direction using other known methods.

[0040] Furthermore, the recognition results of the monitoring direction recognition unit 124 may include information indicating whether the monitoring direction was recognized based on the driver's eye movements (e.g., gaze), the driver's face movements (e.g., head orientation), or both. The recognition results may also include information indicating whether the driver's eye movements were recognized from one eye or both eyes, and may include information indicating that eye movements or face movements could not be recognized.

[0041] The behavior recognition unit 126 recognizes the behavior of vehicle M based on the detection results of the vehicle sensor 40 and the control content executed by the driving control unit 180. The behavior of vehicle M includes behavior due to manual driving by the driver and behavior due to driving control executed by the driving control unit 180.

[0042] For example, the behavior recognition unit 126 recognizes the lateral position (position in the lane width direction) of the vehicle M relative to the driving lane and the attitude (orientation) of the vehicle M relative to the direction of extension of the driving lane, based on the positional relationship of the vehicle M with respect to the driving lane. For example, the behavior recognition unit 126 may recognize the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centers of the lanes in the direction of travel of the vehicle M, as the relative position and attitude of the vehicle M relative to the driving lane. Alternatively, the behavior recognition unit 126 may recognize the position of the vehicle M's reference point relative to any side edge of the driving lane (road marking or road boundary), etc., as the relative position (lateral position) of the vehicle M relative to the driving lane. Furthermore, the behavior recognition unit 126 may recognize the lateral behavior of the vehicle M (for example, whether or not it has moved laterally by a predetermined distance or more in a predetermined time) from the amount of change in the lateral position and orientation (yaw rate) of the vehicle M as described above, and may recognize that the behavior is erratic if the amount of change is greater than a predetermined amount.

[0043] Furthermore, the behavior recognition unit 126 detects the behavior of the vehicle M from the amount of steering wheel operation (e.g., steering angle, steering torque, and rate of change of steering torque) obtained by the operation detection unit when the vehicle M is being driven manually, the amount of depression of the accelerator pedal and brake pedal, etc., and recognizes the amount of change in the behavior over a predetermined period of time.

[0044] Furthermore, the behavior recognition unit 126 recognizes the behavior of the vehicle M based on the content of the driving control performed by the driving control unit 180. Driving control is a control that drives the vehicle M by controlling at least one of the steering and speed of the vehicle M, without relying on driving operations by the driver, or by accepting only partial operation instructions. Driving control includes, for example, ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), and ALC (Auto Lane Changing). Driving control may also include control to stop the vehicle M in a safe position such as the shoulder of the road, and control to control the steering and speed to avoid contact between the vehicle M and obstacles recognized by the surrounding recognition unit 122. For example, the behavior recognition unit 126 recognizes the behavior of the vehicle M as a result of the driving control unit 180 performing LKAS, ALC, and other driving control operations.

[0045] The determination unit 140 includes, for example, a recognition accuracy determination unit 142 and a monitoring determination unit 144. The recognition accuracy determination unit 142 determines whether the accuracy (recognition accuracy) of the recognition result of the driver's monitoring direction recognized by the monitoring direction recognition unit 124 has decreased. Recognition accuracy is an index value that indicates the degree to which the monitoring direction of the monitored driver is correctly recognized from the image captured by the in-vehicle camera 70. Low recognition accuracy means, for example, that there is a high possibility that the driver's monitoring direction is not correctly recognized from the image captured by the in-vehicle camera 70, or that it is not correctly recognized at all (this may include the monitoring direction itself not being recognized).

[0046] The monitoring and determination unit 144 determines, for example, whether the driver's monitoring direction is appropriate based on the monitoring direction recognized by the monitoring direction recognition unit 124 and a preset monitoring target area. Details of the functions of the recognition accuracy determination unit 142 and the monitoring and determination unit 144 will be described later.

[0047] The HMI control unit 160 notifies the occupants (including the driver) of predetermined information via the HMI 30 and receives information input by the HMI 30. The predetermined information includes, for example, information related to the driving of vehicle M, such as information regarding the status of vehicle M and information regarding driving control. Information regarding the status of vehicle M includes, for example, the speed of vehicle M, engine speed, and shift position. Information regarding driving control includes, for example, whether or not driving control is being performed by the driving control unit 180, information regarding the status of driving control, information regarding driving control recommendations (suggestions) from the system, and notification information to the driver (warnings, etc.). The predetermined information may also include information regarding the surrounding conditions recognized by the detection device DD. The predetermined information may also include information unrelated to the driving of vehicle M, such as content (e.g., video) stored on a storage medium such as a television program or DVD. The predetermined information may also include, for example, information regarding the current location and destination of vehicle M, and the remaining fuel level of vehicle M. The HMI control unit 160 may output the information received by the HMI 30 to the communication device 20, navigation device 50, recognition unit 120, determination unit 140, driving control unit 180, etc.

[0048] Furthermore, the HMI control unit 160 may generate inquiry information and recommendation information for the occupants, recognition results from the recognition unit 120, judgment results from the judgment unit 140, notification information, etc., and output the generated information to the HMI 30. The generated information may include images and sounds (including notification sounds, etc.). In addition, the HMI control unit 160 may transmit the various information to be output to the HMI 30 to terminal devices used by the occupants of the vehicle M via the communication device 20.

[0049] Furthermore, the HMI control unit 160 includes a counting unit 162. The counting unit 162, for example, when providing predetermined notification information to the occupants (driver, etc.) of the vehicle M via the HMI 30, counts the time from when the conditions for notification are met until the notification actually begins, or counts the time from when the notification begins until it stops (ends). The counting unit 162 may also count the duration of the driver's monitoring direction becoming the appropriate monitoring direction while the notification information is being provided.

[0050] The driving control unit 180 controls the movement of the vehicle M. For example, the driving control unit 180 performs driving control on the vehicle M based on the recognition results from the recognition unit 120, the determination results from the determination unit 140, etc. Driving control may be performed in response to instructions from the driver via the HMI 30, or it may be performed independently of the driver's instructions based on the recognition results from the recognition unit 120, etc. When performing driving control, the driving control unit 180 generates a future target trajectory for the vehicle M according to the content of the driving control based on the recognition results from the recognition unit 120, etc., and controls at least one of the steering and speed of the vehicle M so that the vehicle M moves along the generated target trajectory.

[0051] For example, the driving control unit 180 performs driving control such as ACC, LKAS, and ALC. Furthermore, if the driver's monitoring direction does not improve even after a predetermined time has elapsed since the HMI control unit 160 has outputted notification information indicating that the driver's monitoring direction is inappropriate, the driving control unit 180 will perform control to stop the vehicle M in a safe position such as the shoulder of the road, or control to avoid contact between the vehicle M and an obstacle.

[0052] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the driving control unit 180 or information input from the accelerator pedal of the driver control unit 80.

[0053] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 180 or from the brake pedal of the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 180 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0054] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 180 or from the steering wheel of the driver control unit 80.

[0055] [Recognition accuracy determination unit and monitoring determination unit] Next, the functions of the recognition accuracy determination unit 142 and the monitoring determination unit 144 will be described in detail. In the following, the driver's gaze will be used as an example of the driver's eye movement, and the direction of the driver's face will be used as an example of the driver's face movement. For example, the recognition accuracy determination unit 142, based on the recognition result of the monitoring direction recognition unit 124, determines that the recognition accuracy of the monitoring direction has decreased if the monitoring direction cannot be recognized from the driver's gaze, regardless of whether the monitoring direction can be recognized from the direction of the driver's face.

[0056] Furthermore, the recognition accuracy determination unit 142 may determine that the recognition accuracy of the monitoring direction has decreased if it determines, based on the image analysis results of the images captured by the in-vehicle camera 70, that the image is being affected by disturbances. For example, the recognition accuracy determination unit 142 determines that the recognition accuracy has decreased if the amount of change in the brightness value of the captured image over a predetermined time is greater than or equal to a predetermined amount, and determines that the recognition accuracy has not decreased if it is less than the predetermined amount. The brightness value of the captured image may be the average brightness value of the entire image, or it may be the average brightness value of the driver's face region extracted by edge extraction processing of the captured image. For example, if the brightness value of the image changes rapidly due to disturbances such as when the vehicle M is driving near the entrance or exit of a tunnel, shadows cast by structures around the road, sunlight shining into the vehicle interior, or headlights of oncoming vehicles, there is a high possibility that the driver's gaze and face direction cannot be correctly recognized by image analysis. Therefore, regardless of the recognition result by the monitoring direction recognition unit 124, the recognition accuracy determination unit 142 determines that the recognition accuracy has decreased due to the influence of disturbances if the amount of change in the brightness value of the image captured by the in-vehicle camera 70 (the amount of change over a predetermined time) is greater than or equal to a predetermined amount.

[0057] Furthermore, the recognition accuracy determination unit 142 may refer to map information 192 based on the location information of vehicle M detected by the vehicle sensor 40 and determine that the recognition accuracy in the monitoring direction has decreased if the location of vehicle M is within a predetermined distance from a place that is presumed to be susceptible to disturbances such as tunnels, bridges, or roads under elevated structures. In addition, if the in-vehicle camera 70 is equipped with an exposure control function that controls the amount of light according to the brightness of the shooting area, the recognition accuracy determination unit 142 may determine whether the recognition accuracy has decreased based on its parameters (for example, exposure control PWM (Pulse Width Modulation) value, shutter opening, shutter speed, exposure time), etc.

[0058] Furthermore, the recognition accuracy determination unit 142 may determine, based on the image analysis results of the captured image, that the accuracy of recognizing the direction of observation is reduced if it is determined that the driver is wearing sunglasses or glasses, etc., and that the gaze is not being correctly recognized due to the effect of reflection from the lenses. In addition, the recognition accuracy determination unit 142 may determine, based on the image analysis results of the captured image, that the accuracy of recognizing the direction of observation is reduced if at least a part of the area of ​​the driver's eyes in the image is hidden by an object (obstacle) such as a hand.

[0059] Furthermore, the recognition accuracy determination unit 142 may derive the driver's eye opening rate (an index value indicating how open the eyes are) based on the image analysis results of the captured images, and if the derived eye opening rate is below a threshold, it may determine that the recognition accuracy in the monitoring direction has decreased. For example, the recognition accuracy determination unit 142 detects the contour of the eye by fitting an eye curve model to the edge distribution obtained as a result of the image analysis. Subsequently, the recognition accuracy determination unit 142 calculates the vertical distance (distance between feature points) between the uppermost point and the lowermost point in the image among the multiple feature points that form the contour of the eye, and calculates the eye opening rate by dividing the calculated distance between feature points by a preset maximum distance. The maximum distance may be the maximum value of the distance between feature points obtained from a group of captured images taken for a predetermined time at the start of driving, or it may be a predetermined fixed distance.

[0060] The monitoring determination unit 144 determines, for example, whether the driver's monitoring direction is appropriate based on the driver's monitoring direction recognized by the monitoring direction recognition unit 124 and a preset monitoring target area.

[0061] Figure 2 is a diagram showing the relationship between the driver's monitoring direction and the monitored area. In the example in Figure 2, driver D is seated in the driver's seat ST1 of vehicle M and is performing manual driving of vehicle M by operating the driving controls 80 such as the steering wheel SW. In addition, in the example in Figure 2, the display units 32-1 and 32-2 included in the HMI 30 are shown.

[0062] For example, when the vehicle M is traveling in a straight line (X-axis direction in the figure), the monitoring and determination unit 144 sets a monitoring target area AR1 that has predetermined angles to the left and right, with respect to the direction of travel V of the vehicle M from the position of the driver D's head, as shown in Figure 2.

[0063] The monitored area AR1 may be adjusted according to the vehicle M's speed, road shape, recognition results from the recognition unit 120, etc. In this case, the monitoring and determination unit 144 may, for example, set the angle (radians) θ1 indicating the size of the arc of the monitored area AR1 to be smaller depending on the speed, or set the angle θ1 to be larger the wider the road.

[0064] Furthermore, the monitoring and determination unit 144 may adjust the monitored area AR1 based on the behavior of the vehicle M recognized by the behavior recognition unit 126. For example, if the vehicle M changes lanes from the driving lane to the adjacent lane on the right, the current monitored area AR1 is rotated to the right around the position of the driver D's head, and if the vehicle changes lanes from the driving lane to the adjacent lane on the left, the monitored area AR1 is rotated to the left around the position of the driver D's head. In this case, the monitoring and determination unit 144 may change the size of the angle θ1 in addition to (or instead of) the rotation. Also, when lane changes or right / left turns are performed by manual driving, the monitoring and determination unit 144 rotates the monitored area AR1 or changes the size of the angle θ1 according to the steering angle and steering amount of the vehicle M. Furthermore, if the surrounding area recognition unit 122 recognizes an obstacle around the vehicle M (for example, an obstacle on the road), the monitoring and determination unit 144 may change the size of the angle θ1 to include that obstacle.

[0065] For example, in the situation shown in Figure 2, the monitoring determination unit 144 determines that the monitoring direction of driver D is appropriate if the monitoring direction of driver D recognized by the monitoring direction recognition unit 124 is within the angle θ1 of the monitored area AR1 (or if the state of being within the angle θ1 continues for a predetermined time or longer). Conversely, the monitoring determination unit 144 determines that the monitoring direction of driver D is inappropriate if the monitoring direction is not within the angle θ1 (or if the state of not being within the angle θ1 continues for a predetermined time or longer). In the example in Figure 2, the monitoring determination unit 144 determines that the monitoring direction of driver D is appropriate when the monitoring direction is arrow A1, and determines that the monitoring direction of driver D is inappropriate when the direction is arrow A2.

[0066] In addition, the monitoring and determination unit 144 may determine whether or not driver D is distracted, instead of (or in addition to) determining whether or not driver D's monitoring direction is appropriate. In this case, the monitoring and determination unit 144 determines that driver D is not distracted if driver D's monitoring direction is arrow A1, and determines that driver D is distracted if it is arrow A2.

[0067] [Notification control] Next, the content of the notification control in the embodiment will be described. The HMI control unit 160 controls the notification to the driver D according to the determination result of the monitoring determination unit 144. For example, as an example of a notification condition, if the monitoring determination unit 144 determines that the driver D's monitoring direction is inappropriate, the HMI control unit 160 generates notification information regarding distracted driving (distracted driving notification information) and outputs the generated distracted driving notification information to the HMI 30 to notify the driver D. The distracted driving notification information may be information indicating that the driver D's monitoring direction is inappropriate (or that the driver is distracted), or it may be information prompting the driver to face the correct monitoring direction. In addition, the distracted driving notification information may include, for example, an image showing the driver D's monitoring direction or an image showing the monitored area AR1, as shown in Figure 2. This allows the driver D to more accurately understand that the monitoring direction is inappropriate or the direction that should be monitored. The distracted driving notification information may be an image, an audio (for example, a predetermined notification sound), or both.

[0068] Furthermore, the HMI control unit 160 terminates the notification (termination can also be rephrased as stopping or canceling) if the monitoring determination unit 144 determines that the driver D's monitoring direction is appropriate while the distraction warning information is being broadcast. The count unit 162 counts the time since the start of the notification, and even if the monitoring determination unit 144 determines that the driver D's monitoring direction is appropriate during the notification, the notification may not be terminated until a predetermined time has elapsed since the start of the notification (until the count value exceeds a threshold). This prevents the notification from terminating before the driver D has fully grasped the content of the notification.

[0069] Furthermore, the HMI control unit 160 makes it easier to terminate the notification if the recognition accuracy determination unit 142 determines that the recognition accuracy of the driver D's monitoring direction has decreased while the distraction warning information is being notified. This prevents the notification time from becoming longer due to a decrease in the recognition accuracy of the monitoring direction recognition unit 124 and the inability of the monitoring determination unit 144 to determine whether the monitoring direction is appropriate or not while the distraction warning information is being notified.

[0070] For example, if the HMI control unit 160 determines that the accuracy of recognizing the driver D's monitoring direction has decreased, it terminates the notification regardless of the determination result of the monitoring determination unit 144 regarding whether the driver D's monitoring direction is appropriate or not. This prevents the notification time from becoming longer due to the monitoring determination unit 144 being unable to determine whether the monitoring direction is appropriate or not.

[0071] In addition, in this embodiment, if the monitoring direction recognition unit 124 determines that the monitoring direction of driver D recognized based on at least the driver D's gaze is inappropriate, and the HMI control unit 160 has performed notification, and the accuracy of recognizing the driver D's monitoring direction has decreased, the monitoring determination unit 144 may determine whether the driver's monitoring direction is appropriate based on the direction of driver D's face, without using the driver D's gaze. This makes it easier to determine whether the monitoring direction is appropriate using only the direction of the face, which is easier to recognize than the gaze, thus making it easier to obtain a determination result. Therefore, it is possible to suppress the situation in which the notification time is prolonged due to an inability to make a determination.

[0072] Furthermore, the HMI control unit 160 may initiate notification to driver D only when the monitoring direction recognition unit 124 determines that the monitoring direction recognized based on at least the orientation of driver D's face is inappropriate, and the accuracy of the monitoring direction recognition unit 124 decreases during the notification. This makes it possible to prevent, for example, situations where, if a monitoring direction without face orientation (monitoring direction recognized only from gaze) is determined to be inappropriate, the monitoring direction is not determined to be appropriate even though the driver's face is turned towards the monitored area AR1, resulting in a prolonged notification time.

[0073] Furthermore, if the monitoring determination unit 144 determines that the driver D's monitoring direction is an appropriate monitoring direction while the HMI control unit 160 is broadcasting distraction warning information, the counting unit 162 may count the time during which the monitoring direction is determined to be an appropriate monitoring direction, and if this continues for a predetermined time or longer (when the count value exceeds a threshold), the notification may be terminated. This allows the notification to be terminated while the driver D is more reliably monitoring the target area AR1. Also, if the recognition accuracy determination unit 142 determines that the recognition accuracy of the driver D's monitoring direction has decreased while the HMI control unit 160 is counting the time during which the monitoring direction recognized based on the driver D's gaze and face direction is determined to be an appropriate monitoring direction, the HMI control unit 160 may continue counting if the monitoring direction recognized based only on the driver D's face direction is an appropriate monitoring direction. This makes it easier to terminate the notification even if the recognition accuracy of the monitoring direction decreases during counting, thereby suppressing the notification time from becoming too long.

[0074] Furthermore, the monitoring determination unit 144 may enlarge the monitored area AR1 if the accuracy of the monitoring direction recognition unit 124 decreases during notification. In this case, the monitoring determination unit 144 sets, for example, the angle θ1 indicating the size of the arc of the monitored area AR1 to 360 degrees. This allows the monitoring determination unit 144 to determine that the monitoring direction is appropriate even if the recognition accuracy decreases and the correct monitoring direction cannot be recognized, and as a result, it becomes easier to terminate the notification.

[0075] [Processing flow] The following describes the processes performed by the driver assistance device 100 of this embodiment. In the following description, we will mainly focus on the notification process based on the driver's monitoring direction among the processes performed by the driver assistance device 100.

[0076] Figure 3 is a flowchart showing an example of processing performed by the driver assistance device 100 in the embodiment. In the example in Figure 3, the monitoring direction recognition unit 124 recognizes the monitoring direction of the driver of the vehicle M (step S100). Next, the monitoring determination unit 144 determines whether the driver's monitoring direction is an appropriate monitoring direction (step S110). If it is determined that the monitoring direction is not appropriate, the HMI control unit 160 generates distraction warning information (step S120), and outputs the generated distraction warning information to the HMI 30 to notify the driver (step S130).

[0077] Next, the HMI control unit 160 determines whether a predetermined time has elapsed since the notification was made to the driver, based on the time count result from the count unit 162 since the notification started (step S140). If it is determined that the predetermined time has not elapsed, the output (notification) of the distraction warning information continues. If it is determined that a predetermined time has elapsed since the notification, the recognition accuracy determination unit 142 determines whether the recognition accuracy of the driver's monitoring direction during the notification has decreased (step S150). If it is determined that the recognition accuracy of the monitoring direction has not decreased, the monitoring determination unit 144 determines whether the driver's monitoring direction during the notification is an appropriate monitoring direction (step S160). If it is determined that it is not an appropriate monitoring direction, the process returns to step S150 while the notification continues.

[0078] Furthermore, in the process of step S160, if it is determined that the monitoring direction during notification is the appropriate monitoring direction, the HMI control unit 160 determines, based on the time count result from the count unit 162 since the monitoring direction became the appropriate monitoring direction, whether the state in which the driver's monitoring direction is the appropriate monitoring direction has continued for a predetermined time or longer (step S170). If it is determined that it has not continued for a predetermined time or longer, the process returns to step S160. If it is determined that it has continued for a predetermined time or longer, the HMI control unit 160 terminates the notification (step S180).

[0079] Furthermore, if it is determined in step S150 that the accuracy of recognizing the monitoring direction during notification has decreased, the HMI control unit 160 makes it easier to terminate the notification that is currently running. In the example in Figure 3, the notification is terminated without performing the determination process of whether the driver's monitoring direction is the appropriate monitoring direction (processing in step S160) and the determination process of whether the state of being the appropriate monitoring direction has continued for a predetermined time or longer (processing in step S170). As a result, the processing in this flowchart is terminated.

[0080] Furthermore, if the process in step S110 determines that the driver's monitoring direction is the appropriate monitoring direction, the process in this flowchart terminates. Note that if the accuracy of recognizing the monitoring direction during notification is determined to be reduced in step S150, either step S160 or step S170 may be omitted. In this case as well, the notification currently in progress is more likely to terminate.

[0081] [Differentiation] In this embodiment, the HMI control unit 160 may generate and output to the HMI 30 information indicating that the accuracy of recognizing the driver's monitoring direction has decreased, information indicating the reason for the decreased recognition accuracy, and information indicating that the notification is more likely to end due to the decreased recognition accuracy. This allows the driver to more accurately understand the status of the notification control regarding the monitoring direction (distraction). Furthermore, even if the notification ends while the driver's monitoring direction is still inappropriate, it is possible to suppress causing discomfort to the driver.

[0082] In addition, in this embodiment, if the HMI control unit 160 determines that the driver's monitoring direction recognized by the monitoring direction recognition unit 124 is not an appropriate monitoring direction, it may generate information (arrow image, etc.) to guide the driver's monitoring direction into the monitoring target area AR1 and output it to the display unit 32.

[0083] Furthermore, in this embodiment, the monitoring direction recognition unit 124 may, when it can recognize the driver's gaze, prioritize the gaze direction over the direction of the face to recognize the monitoring direction, and when it is providing distraction warning information, it may use the direction of the face to recognize the monitoring direction in order to facilitate the termination of the warning.

[0084] In addition, in this embodiment, the driving control unit 180 may terminate the ongoing driving control if the recognition accuracy determination unit 142 determines that the recognition accuracy of the driver's monitoring direction has decreased during the execution of driving control. This prevents the execution of incorrect driving control due to the recognition of a monitoring direction different from the driver's actual monitoring direction due to a decrease in recognition accuracy. Furthermore, if the HMI control unit 160 is broadcasting distraction warning information and determines that the recognition accuracy of the driver's monitoring direction has decreased during the execution of driving control, it may be made easier to terminate the broadcast after terminating the ongoing driving control.

[0085] According to the embodiments described above, the notification control device includes a monitoring direction recognition unit 124 that recognizes the monitoring direction of the driver of a vehicle M (an example of a moving object), a determination unit 140 that determines whether the monitoring direction recognized by the monitoring direction recognition unit 124 is appropriate, and a notification control unit (HMI 30, HMI control unit 160) that controls notification to the driver according to the determination result by the determination unit 140. The notification control unit makes it easier to terminate the notification if the accuracy of the monitoring direction recognition by the monitoring direction recognition unit 124 decreases during notification, thereby enabling more appropriate notification to the driver.

[0086] For example, according to the embodiment, if the driver's gaze based on the captured image cannot be recognized due to the influence of ambient light, etc., and the accuracy of recognizing the monitoring direction is reduced, the monitoring direction can be recognized solely by the direction of the driver's face, making it easier to terminate the notification. Furthermore, according to the embodiment, even if the recognition accuracy decreases due to ambient light, etc., while the system is counting until the monitoring direction recognized using the gaze and direction of the face is the appropriate monitoring direction and that state continues for a predetermined time or longer, if the system determines that the monitoring direction is appropriate using the monitoring direction based on the direction of the face, the count will continue (without resetting). This makes it possible to suppress the situation where the notification time is prolonged due to incorrect determination caused by a decrease in recognition accuracy.

[0087] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Recognizes the direction of the driver of the moving object, Determine whether the recognized monitoring direction is appropriate. The notification to the driver is controlled according to the result of the determination. If the accuracy of recognizing the monitoring direction decreases during the notification, the notification will be made easier to terminate. Notification control device.

[0088] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0089] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 70...In-cabin camera, 80...Driver's control panel, 100...Driver's assistance device, 120...Recognition unit, 122...Surroundings recognition unit, 124...Monitoring direction recognition unit, 126...Behavior recognition unit, 140...Determination unit, 142...Recognition accuracy determination unit, 144...Monitoring determination unit, 160...HMI control unit, 180...Driving control unit, 190...Storage unit, 200...Driving force output device, 210...Brake device, 220...Steering device, M...Vehicle

Claims

1. A monitoring direction recognition unit that recognizes the monitoring direction of the driver of a moving object, A determination unit that determines whether the monitoring direction recognized by the monitoring direction recognition unit is appropriate or not, The system includes a notification control unit that controls notification to the driver according to the determination result of the determination unit, The notification control unit makes it easier to terminate the notification if the accuracy of the recognition of the monitoring direction by the monitoring direction recognition unit decreases during the notification. Notification control device.

2. The notification control unit terminates the notification if the accuracy of the recognition of the monitoring direction decreases during the notification, regardless of the determination result of the determination unit regarding whether the monitoring direction is appropriate or not. The notification control device according to claim 1.

3. The monitoring direction recognition unit recognizes the monitoring direction using at least one of the driver's eye movements and facial movements. The determination unit determines, based on the monitoring direction recognition unit, that the driver's monitoring direction recognized at least based on the eye movements is inappropriate, and if the accuracy of the monitoring direction recognition decreases during notification by the notification control unit, it determines whether the driver's monitoring direction is appropriate based on the driver's facial movements without using the eye movements. The notification control device according to claim 1.

4. The notification control unit initiates notification to the driver when the determination unit determines that the monitoring direction recognized by the monitoring direction recognition unit, at least based on the driver's facial movements, is inappropriate. If the accuracy of the monitoring direction recognition unit's recognition of the monitoring direction decreases during the notification, the notification is made easier to terminate. The notification control device according to claim 1.

5. The notification control unit, If the determination unit determines during the notification that the driver's monitoring direction is an appropriate monitoring direction, the time during which the monitoring direction is determined to be appropriate is counted, and if this continues for a predetermined time or longer, the notification is terminated. If, while the time is being counted after the monitoring direction recognized based on the driver's eye movements and facial movements has been determined to be an appropriate monitoring direction, and if the accuracy of the recognition of the driver's monitoring direction has decreased, and the monitoring direction recognized based on the orientation of the driver's face has been determined to be an appropriate monitoring direction, the count will be continued. The notification control device according to claim 1.

6. Computers Recognizes the direction of the driver of the moving object, Determine whether the recognized monitoring direction is appropriate. The notification to the driver is controlled according to the result of the determination. If the accuracy of recognizing the monitoring direction decreases during the notification, the notification will be made easier to terminate. Notification control method.

7. On the computer, To recognize the direction of the driver of the moving object, Determine whether the recognized monitoring direction is appropriate. Depending on the determined result, the notification to the driver will be controlled. If the accuracy of recognizing the monitoring direction decreases during the notification, the notification will be made easier to terminate. program.

Images