Notification control device, notification control method, and program

Abstract

Provide more appropriate notifications depending on the driver's condition. [Solution] The notification control device of the embodiment includes a recognition unit that recognizes the monitoring direction of the driver of a moving vehicle and the state of the driver; a first determination unit that determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit; a second determination unit that determines whether the driver is performing an action other than driving the moving vehicle based on the recognition result; and a notification control unit that controls notification to the driver based on the determination result of the first determination unit and the determination result of the second determination unit. The notification control unit suppresses notification when the first determination unit determines that the driver's monitoring direction is appropriate, and the second determination unit determines that the driver is performing an action other than driving the moving vehicle, but the impact of the action other than driving on surrounding monitoring is less than a first threshold.

Description

Technical Field

[0001] The present invention relates to a notification control device, a notification control method, and a program.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development have focused on further improving traffic safety and convenience through research and development related to preventive safety technologies. In this regard, conventionally, there is a known technology that uses an image captured by a driver camera to recognize the driver's line of sight and face direction, and provides assistance to the driver when it is determined from the recognition result that the driver is engaged in distracted driving (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in conventional preventive safety technologies, there is also a known technology that, apart from distracted driving, considers that the driver is not concentrating on driving when performing an action other than driving and issues a notification. However, since there are various types of actions other than driving, there are cases where notification information is output even when the driver is in a concentrated state. Therefore, there is a problem that appropriate notification may not be possible depending on the driver's state.

[0005] One of the objectives of the present application is to provide a notification control device, a notification control method, and a program that can perform more appropriate notification according to the driver's state in order to solve the above problems. And by extension, it contributes to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The notification control device, notification control method, and program according to this invention employ the following configuration. (1) A notification control device according to one aspect of the present invention comprises: a recognition unit that recognizes the monitoring direction of the driver of a mobile vehicle and the state of the driver; a first determination unit that determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit; a second determination unit that determines whether the driver is performing an action other than driving the mobile vehicle based on the recognition result; and a notification control unit that controls notification to the driver based on the determination result of the first determination unit and the determination result of the second determination unit, wherein the notification control unit suppresses notification when the first determination unit determines that the driver's monitoring direction is appropriate and the second determination unit determines that the driver is performing an action other than driving the mobile vehicle, but the impact of the action other than driving on surrounding monitoring is less than a first threshold.

[0007] (2) In the embodiment of (1) above, actions that have less impact on the surrounding monitoring than the first threshold include actions that are expected to be completed within a predetermined time.

[0008] (3) In the embodiment of (1) above, the actions that have an impact on the surrounding area monitoring less than the first threshold include actions that the driver is expected to be able to perform with one hand.

[0009] (4) In the embodiment of (1) above, an act that has less impact on the surrounding monitoring than the first threshold includes the act of putting on or taking off items worn by the driver.

[0010] (5) In the embodiment of (1) above, the actions that have an impact on the surrounding monitoring that are less than the first threshold include the driver's eating and drinking activities.

[0011] (6): In the embodiment of (1) above, the notification control unit makes the notification regardless of the determination result of the second determination unit when the first determination unit determines that the driver's monitoring direction is inappropriate.

[0012] (7) In the embodiment of (1) above, the recognition unit recognizes the amount of change or the speed at the time of change in the direction monitored by the driver over a predetermined period of time, and the notification control unit provides notification regardless of the determination result by the first determination unit and the determination result by the second determination unit if the amount of change or speed is equal to or greater than the second threshold.

[0013] (8) A notification control method according to another aspect of the present invention is a notification control method in which a computer recognizes the monitoring direction of the driver of a mobile vehicle and the state of the driver, determines whether the driver's monitoring direction is appropriate based on the recognized result, determines whether the driver is performing an action other than driving the mobile vehicle based on the recognized result, controls the notification to the driver based on the determined result, and suppresses the notification if it is determined that the driver's monitoring direction is appropriate and it is determined that the driver is performing an action other than driving the mobile vehicle, but the degree of influence of the action other than driving on surrounding monitoring is less than a first threshold.

[0014] (9): A program according to another aspect of the present invention causes a computer to recognize the monitoring direction of the driver of a mobile vehicle and the state of the driver, to determine whether the driver's monitoring direction is appropriate based on the recognized result, to determine whether the driver is performing an action other than driving the mobile vehicle based on the recognized result, to control the notification to the driver based on the determined result, and to suppress the notification if it is determined that the driver's monitoring direction is appropriate and it is determined that the driver is performing an action other than driving the mobile vehicle, but the impact of the action other than driving on surrounding monitoring is less than a first threshold. [Effects of the Invention]

[0015] According to the above aspects (1) to (9), more appropriate notification can be performed according to the driver's state.

Brief Description of Drawings

[0016] [Figure 1] It is a configuration diagram of a vehicle system 1 including a notification control device according to an embodiment. [Figure 2] It is a diagram for explaining an example of the data content of driver state data 194. [Figure 3] It is a diagram showing the relationship between the driver's monitoring direction and the monitoring target area. [Figure 4] It is a diagram for explaining an example of a scene where the notification of the second notification information is suppressed. [Figure 5] It is a diagram for explaining an example of a scene where the notification of the second notification information is not suppressed. [Figure 6] It is a diagram for explaining the change in the monitoring direction of driver D. [Figure 7] It is a flowchart showing an example of the process executed by the driving support device 100 in the embodiment.

Mode for Carrying Out the Invention

[0017] Hereinafter, embodiments of the notification control device, notification control method, and program of the present invention will be described with reference to the drawings. Hereinafter, an example in which the notification control device is applied to a moving body will be described. Further, as an example of the moving body, a vehicle will be used. In addition to vehicles, the moving body may include, for example, ships that can move on the ground (road) like hovercrafts, flying bodies that can travel on roads, standing vehicles having a power unit, micromobility such as electric kick scooters, and the like.

[0018] [Overall Configuration] FIG. 1 is a configuration diagram of a vehicle system 1 including a notification control device according to an embodiment. The vehicle (hereinafter referred to as vehicle M) on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, four-wheeled vehicle, or micromobility, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or discharge electric power of a battery (storage battery) such as a secondary battery or a fuel cell.

[0019] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an in-vehicle camera 70, a driving operator 80, a driving support device 100, a traveling driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. Note that the configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted, or another configuration may be added. The camera 10, the radar device 12, and the LIDAR 14 are an example of a "detection device DD". The HMI 30 is an example of a "notification unit". The HMI 30 and the driving support device 100 are an example of a "notification control device".

[0020] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted at any location on the vehicle M on which the vehicle system 1 is installed. When imaging the front, camera 10 is mounted on the top of the front windshield, behind the rearview mirror, or on the front of the vehicle body. When imaging the rear, camera 10 is mounted on the top of the rear windshield or on the tailgate. When imaging the sides, camera 10 is mounted on the left and right door mirrors, etc. Camera 10 periodically and repeatedly images the area around vehicle M. Camera 10 may also be a stereo camera.

[0021] The radar device 12 emits radio waves (radar) such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by surrounding objects to detect at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0022] The LIDAR 14 illuminates the area around the vehicle M with light and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time from emission to reception. The emitted light is, for example, pulsed laser light. The LIDAR 14 can be mounted at any location on the vehicle M.

[0023] The communication device 20 communicates with other vehicles in the vicinity of vehicle M, terminal devices of users using vehicle M, or various server devices, for example, by utilizing networks such as cellular networks, Wi-Fi networks, Bluetooth®, DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet.

[0024] The HMI 30 outputs various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. The HMI 30 includes, for example, a display unit 32 and a speaker 34. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including video) in the embodiment. The display unit 32 may be configured integrally with the input unit as a touch panel. The speaker 34 outputs predetermined sounds (for example, notification sounds or message sounds). The HMI 30 may also include a microphone, buzzer, touch panel, switches, keys, etc. The switches may include switches that execute or terminate predetermined driving controls that can be executed by the driving control unit described later, and switches that approve (permit) or reject driving control recommendations (suggestions) from the system (vehicle system 1). The switches may also include switches for operating the turn signals (turn signal switches), etc.

[0025] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, and a yaw rate sensor for detecting yaw rate (for example, the angular velocity of rotation around the vertical axis passing through the center of gravity of the vehicle M). The vehicle sensor 40 may also include a lateral acceleration sensor (lateral G sensor) for detecting the lateral acceleration (lateral G) of the vehicle M, a steering angle sensor for detecting the steering angle of the vehicle M (which may be the angle of the steering wheels or the operating angle of the steering wheel), a steering angular velocity sensor for detecting the steering angular velocity, and a compass sensor for detecting the orientation of the vehicle M.

[0026] Furthermore, the vehicle sensor 40 may include a position sensor that detects the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that acquires position information using, for example, a GNSS (Global Navigation Satellite System) receiver of a navigation device 50. The vehicle sensor 40 may derive the speed of the vehicle M from the difference (i.e., distance) of position information at a predetermined time from the position sensor. The results detected by the vehicle sensor 40 are output to the driver assistance device 100.

[0027] The navigation device 50 includes, for example, a GNSS receiver, a navigation HMI, and a route determination unit. The navigation device 50 may store map information in a storage device such as an HDD (Hard Disk Drive) or flash memory, or it may acquire map information 192 stored in a storage unit 190, which will be described later. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI includes a display device, speaker, touch panel, keys, etc. The GNSS receiver may be provided on the vehicle sensor 40. The navigation HMI may be partially or completely shared with the HMI 30 described above. The route determination unit determines, for example, a route (hereinafter referred to as a route on a map) from the position of the vehicle M determined by the GNSS receiver (or any input position) to a destination input by the occupant using the navigation HMI, by referring to, for example, map information 192, etc. Furthermore, the navigation device 50 provides route guidance using the navigation HMI based on the determined route on the map. The navigation device 50 may also transmit its current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0028] Here, map information 192 is information in which the road shape is represented by links indicating roads (an example of a travel route) and nodes connected by those links. Map information 192 may also include POI (Point of Interest) information, etc. Map information 192 also includes, for example, the number of lanes (number of travel routes), the type and shape of road markings, information on the center of the lanes, or information on road boundaries. Map information 192 may also include information on whether the road boundary is a boundary (physical boundary) that includes structures that vehicles cannot pass through (including crossing and contact). Physical boundaries include, for example, guardrails, curbs, median strips, fences, etc. Map information 192 may also include road shape information, traffic regulation information, address information (address and postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, the curvature of the road (which may be rephrased as radius of curvature; the same applies below), width, road surface gradient, branching and merging points, intersections, T-junctions, etc. Map information 192 may be updated as needed by the communication device 20 communicating with an external device.

[0029] The in-vehicle camera 70 is, for example, a digital camera using a solid-state image sensor such as a CCD or CMOS. The in-vehicle camera 70 is mounted at any location in the vehicle M in a position and orientation that allows it to capture the head of the driver seated in the driver's seat of the vehicle M from the front. For example, the in-vehicle camera 70 is mounted near (for example, above or below) the display device located in the center of the instrument panel of the vehicle M. The in-vehicle camera 70 may also capture the interior of the vehicle, including the area of ​​the occupants (passengers) seated in the passenger seat or other seats of the vehicle M, in addition to the driver. The in-vehicle camera 70 may also capture the interior of the vehicle by irradiating it with infrared light. The in-vehicle camera 70 captures the interior of the vehicle repeatedly and periodically, for example.

[0030] The driver control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driver control unit 80 may also include a shift lever, a modified steering wheel, a joystick, or other controls. Each control of the driver control unit 80 is equipped with an operation detection unit that detects, for example, the amount of operation performed by the driver on the control unit or whether or not an operation has been performed. The operation detection unit detects, for example, the steering angle and steering torque of the steering wheel (for example, the amount of steering due to the driver's driving operation (steering input torque)), the rate of change of the steering torque, the amount of depression of the accelerator pedal and brake pedal, etc. The operation detection unit then outputs the detection results to the driver assistance device 100, or to one or both of the driving force output device 200, the brake device 210, and the steering device 220. The driver control unit 80 may also include a turn signal control unit (for example, a turn signal lever, a turn signal switch). When the turn signal control unit is operated, the turn signal of the vehicle M corresponding to the operation will flash, and the operation details (including, for example, the detection result that the operation was performed by the driver) will be output to the driver assistance device 100.

[0031] The driver assistance device 100 performs various controls to assist the driver of the vehicle M. The driver assistance device 100 includes, for example, a recognition unit 120, a determination unit 140, an HMI control unit 160, a driving control unit 180, and a storage unit 190. The recognition unit 120, the determination unit 140, the HMI control unit 160, and the driving control unit 180 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the cooperation of software and hardware. The above-mentioned program may be stored in advance in a storage device (a storage device equipped with a non-transient storage medium) such as the HDD or flash memory of the driver assistance device 100, or it may be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driver assistance device 100 when the storage medium (non-transient storage medium) is inserted into a drive device or card slot. The driving control unit 180 is an example of a "mobility control unit". The HMI control unit 160 is an example of a "notification control unit".

[0032] The memory unit 190 may be implemented using the various storage devices described above, or an EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The memory unit 190 stores, for example, map information 192, driver status data 194, various information in the embodiment, programs, etc. The memory unit 190 may also store various setting information used in the processing in this embodiment.

[0033] Figure 2 is a diagram illustrating an example of the data content of driver state data 194. The driver state data 194 shown in Figure 2 is data that associates, for example, the driver's actions other than driving, characteristic information, and the degree of impact on surrounding monitoring. Actions other than driving by the driver include, for example, putting on or taking off items (e.g., accessories, hats, glasses, masks, etc.), eating and drinking by the driver, operating terminal devices such as smartphones, and operating navigation devices. Actions related to eating and drinking may include, for example, drinking beverages, eating food (including chewing gum), as well as opening and closing drink lids and taking food out of bags. Characteristic information is characteristic information corresponding to each action obtained from images captured by the in-vehicle camera 70. Characteristic information may also include characteristic information obtained from images (videos) captured in time series (for example, information about the driver's movements). The degree of impact on surrounding monitoring is an index value that indicates, for example, the degree to which concentration on surrounding monitoring decreases; the larger the value, the lower (or less) the concentration on surrounding monitoring is. Driver status data 194 may be acquired from an external device via the communication device 20, or it may be pre-registered by the driver or other relevant party.

[0034] Furthermore, the degree of impact on surrounding area monitoring may be adjusted for each driver, or it may be adjusted based on the number of times non-driving activities are performed. For example, even for the same activity, some drivers may be able to concentrate on surrounding area monitoring while others cannot. Also, in the case of attaching or detaching items or eating and drinking, as drivers become accustomed to these activities, they will be able to perform them with less impact on surrounding area monitoring. Therefore, by adjusting the degree of impact for each driver, or by reducing the degree of impact according to the number of times the activity is performed, the judgment processing by the judgment unit 140 described later can be performed more appropriately.

[0035] The recognition unit 120 includes, for example, a surrounding recognition unit 122, a monitoring direction recognition unit 124, a state recognition unit 126, and a behavior recognition unit 128. The surrounding recognition unit 122 recognizes the surrounding conditions of the vehicle M based on, for example, the detection results of the detection device DD (information input from the camera 10, radar device 12, and LIDAR 14). For example, the surrounding recognition unit 122 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the state of objects present around the vehicle M (within a predetermined distance), such as their position (relative position), size, speed (relative speed), and acceleration. Objects recognized by the surrounding recognition unit 122 may include, for example, physical boundaries that demarcate roads (travel paths), as well as other vehicles, pedestrians, bicycles, and other traffic participants (examples of obstacles). The position of an object is recognized as a position on an absolute coordinate system with the vehicle M's representative point (such as the center of gravity or drive axis center) as the origin, and is used for control. The position of an object may be represented by its center of gravity, a representative point such as a corner, or by the represented region. The "state" of an object may include, for example, the acceleration or jerk of another vehicle, or its "action state" (for example, whether the other vehicle is changing lanes or is about to change lanes).

[0036] Furthermore, the surrounding area recognition unit 122 may recognize, for example, stop lines, red lights, toll booths, other road events, road signs, and markings drawn on the road (e.g., speed limits). The surrounding area recognition unit 122 may also recognize the curvature of the vehicle M's lane (road) based on the detection results of the detection device DD or the map information 192. The surrounding area recognition unit 122 may also recognize the road surface conditions (e.g., whether the road surface is slippery, such as being frozen) based on the detection results of the detection device DD.

[0037] Furthermore, the surrounding recognition unit 122 recognizes, for example, the lane in which the vehicle M is traveling (driving lane) and other surrounding lanes (for example, adjacent lanes). For example, the surrounding recognition unit 122 recognizes road markings from images captured by the camera 10 and recognizes the driving lane and other lanes based on the positional relationship of the road markings as seen from the recognized vehicle M. Alternatively, the surrounding recognition unit 122 may refer to map information 192 based on the position information of the vehicle M obtained from the vehicle sensor 40, etc., to recognize the lane in which the vehicle M is traveling and other lanes. The surrounding recognition unit 122 may also recognize structures such as tunnels, elevated roads, and overpasses, as well as structures around the road.

[0038] The monitoring direction recognition unit 124 performs known image analysis processing on images captured by the in-vehicle camera 70 (for example, feature extraction of edges, shape, size, and color, and matching processing such as pattern matching), and recognizes the monitoring direction of the driver of the vehicle M based on the image analysis results. For example, the monitoring direction recognition unit 124 recognizes the monitoring direction using at least one of the driver's gaze and face orientation obtained from the image analysis results.

[0039] For example, the monitoring direction recognition unit 124 uses methods such as pattern matching to detect a combination of a reference point (the stationary part of the eye) and a moving point (the moving part of the eye) of the driver's eye from the captured image. The combination of the reference point and the moving point is, for example, a combination of the inner corner of the eye and the iris, or a combination of the corneal reflection region and the pupil. The corneal reflection region is, for example, the region of infrared light reflection on the cornea when the in-vehicle camera 70 shines infrared light towards the driver. Then, based on the position of the moving point relative to the reference point, the monitoring direction recognition unit 124 performs coordinate transformations from the image plane to real space and recognizes the driver's line of sight.

[0040] Furthermore, the monitoring direction recognition unit 124 recognizes the orientation of the driver's face based on positional information of the eyes, nose, mouth, etc. within the face region (relative positional information of each part, etc.) obtained from the analysis results of the captured image.

[0041] The monitoring direction recognition unit 124 recognizes the monitoring direction using both the driver's gaze and the direction of their face when it can recognize both. This allows for more accurate recognition of the monitoring direction. If the monitoring direction recognition unit 124 recognizes only one of the gaze or the direction of the face, it uses only that one to recognize the monitoring direction.

[0042] Furthermore, the monitoring direction recognition unit 124 may recognize, based on the analysis results of the image from the in-vehicle camera 70, that the driver is wearing sunglasses, glasses, or a mask, etc., through template matching or the like. In this case, if the driver is wearing sunglasses or glasses, the monitoring direction recognition unit 124 recognizes the monitoring direction based only on the direction of the face. If the driver is wearing a mask, the accuracy of recognizing the direction of the face decreases because distinctive features such as the nose and mouth cannot be recognized from the image, so the monitoring direction is recognized based only on the gaze.

[0043] In recognizing each piece of information using the images described above, a pre-trained model, for example, that has been trained in advance by machine learning, may be used. In this case, the monitoring direction recognition unit 124 takes an image as input and outputs the monitoring direction of a person contained in the image (for example, gaze or face direction), and inputs the image captured by the in-vehicle camera 70 to obtain the driver's monitoring direction. Alternatively, the monitoring direction recognition unit 124 may recognize the monitoring direction using other known methods.

[0044] Furthermore, the recognition results of the monitoring direction recognition unit 124 may include information indicating whether the monitoring direction was recognized based on the driver's gaze, based on the direction of the driver's face, or using both. The recognition results may also include information indicating whether the driver's gaze was recognized from one eye or both eyes, and may include information indicating that the gaze could not be recognized or that the direction of the face could not be recognized.

[0045] Furthermore, the monitoring direction recognition unit 124 may recognize the amount of change in the driver's monitoring direction over a predetermined time or the speed at which the monitoring direction changes. The amount of change is, for example, the total amount of movement over the predetermined time (for example, the total amount of movement to the left and right if the driver moves left and right). The speed may be the maximum speed over the predetermined time or the average speed.

[0046] The state recognition unit 126 recognizes the driver's state. For example, the state recognition unit 126 performs known image analysis processing (e.g., edge, shape, size, and color feature extraction) on the image captured by the in-vehicle camera 70, and uses the feature information obtained as a result of the image analysis to refer to the feature information of the driver state data 194 stored in the storage unit 190. It then obtains the driver's non-driving actions and their impact on surrounding monitoring that correspond to the feature information with the greatest degree of matching (which can also be called similarity) and is greater than or equal to a predetermined value. The feature information may include feature information obtained from images (videos) captured in time series (e.g., information about the driver's actions). For example, the feature information may include feature information based on objects (items) held by the driver, hand movements, the driver's gaze, and the direction of their face.

[0047] The state recognition unit 126 may also acquire the impact of non-driving actions on surrounding monitoring by inputting the feature information obtained from the image into a trained model that takes feature information as input and outputs the impact of non-driving actions by the driver and the impact of surrounding monitoring. The trained model may be acquired from an external device via the communication device 20, for example, or stored in the storage unit 190.

[0048] The behavior recognition unit 128 recognizes the behavior of vehicle M based on the detection results of the vehicle sensor 40 and the control content executed by the driving control unit 180. The behavior of vehicle M includes behavior due to manual driving by the driver and behavior due to driving control executed by the driving control unit 180.

[0049] For example, the behavior recognition unit 128 recognizes the lateral position (position in the lane width direction) of the vehicle M relative to the driving lane and the attitude (orientation) of the vehicle M relative to the direction of extension of the driving lane, based on the positional relationship of the vehicle M with respect to the driving lane. For example, the behavior recognition unit 128 may recognize the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centers of the lanes in the direction of travel of the vehicle M, as the relative position and attitude of the vehicle M relative to the driving lane. Alternatively, the behavior recognition unit 128 may recognize the position of the vehicle M's reference point relative to any side edge of the driving lane (road marking or road boundary), etc., as the relative position (lateral position) of the vehicle M relative to the driving lane. Furthermore, the behavior recognition unit 128 may recognize the lateral behavior of the vehicle M (for example, whether or not it has moved laterally for a predetermined distance or more in a predetermined time) from the amount of change in the lateral position and orientation (yaw rate) of the vehicle M as described above, and may recognize that the behavior is erratic if the amount of change is greater than a predetermined amount.

[0050] Furthermore, the behavior recognition unit 128 detects the behavior of the vehicle M from the amount of steering wheel operation (e.g., steering angle, steering torque, steering torque change rate), the amount of depression of the accelerator pedal and brake pedal, etc., obtained by the operation detection unit when the vehicle M is being driven manually, and recognizes the amount of change in the behavior over a predetermined period of time.

[0051] Furthermore, the behavior recognition unit 128 recognizes the behavior of the vehicle M based on the content of the driving control performed by the driving control unit 180. Driving control is a control that drives the vehicle M by controlling at least one of the steering and speed of the vehicle M, without relying on driving operations by the driver, or by accepting only some of the operation instructions. Driving control includes, for example, ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), and ALC (Auto Lane Changing). Driving control may also include control to stop the vehicle M in a safe position such as the shoulder of the road, and control to control the steering and speed to avoid contact between the vehicle M and obstacles recognized by the surrounding recognition unit 122. For example, the behavior recognition unit 128 recognizes the behavior of the vehicle M as a result of the driving control unit 180 performing LKAS, ALC, and other driving control operations.

[0052] The determination unit 140 includes, for example, a first determination unit 142 and a second determination unit 144. The first determination unit 142 determines whether the driver's monitoring direction is appropriate based on the recognition result of the monitoring direction recognition unit 124.

[0053] The second determination unit 144 determines, based on the recognition result of the state recognition unit 126, whether or not the driver is performing an action other than driving the vehicle M. Details of the functions of the first determination unit 142 and the second determination unit 144 will be described later.

[0054] The HMI control unit 160 notifies the occupants (including the driver) of predetermined information via the HMI 30 and receives information input by the HMI 30. The predetermined information includes, for example, information related to the driving of vehicle M, such as information regarding the status of vehicle M and information regarding driving control. Information regarding the status of vehicle M includes, for example, the speed of vehicle M, engine speed, and shift position. Information regarding driving control includes, for example, whether or not driving control is being performed by the driving control unit 180, information regarding the status of driving control, information regarding driving control recommendations (suggestions) from the system, and notification information to the driver (warnings, etc.). The predetermined information may also include information regarding the surrounding conditions recognized by the detection device DD. The predetermined information may also include information unrelated to the driving of vehicle M, such as content (e.g., video) stored on a storage medium such as a television program or DVD. The predetermined information may also include, for example, information regarding the current location and destination of vehicle M, and the remaining fuel level of vehicle M. The HMI control unit 160 may output the information received by the HMI 30 to the communication device 20, navigation device 50, recognition unit 120, determination unit 140, driving control unit 180, etc.

[0055] Furthermore, the HMI control unit 160 may generate inquiry information and recommendation information for the occupants, recognition results from the recognition unit 120, judgment results from the judgment unit 140, notification information, etc., and output the generated information to the HMI 30. The generated information may include images and sounds (including notification sounds, etc.). In addition, the HMI control unit 160 may transmit the various information to be output to the HMI 30 to terminal devices used by the occupants of the vehicle M via the communication device 20.

[0056] The driving control unit 180 controls the driving of the vehicle M. For example, the driving control unit 180 performs driving control on the vehicle M based on the recognition results from the recognition unit 120, the determination results from the determination unit 140, etc. Driving control may be performed in response to instructions from the driver via the HMI 30, or it may be performed independently of the driver's instructions based on the recognition results from the recognition unit 120, etc. When performing driving control, the driving control unit 180 generates a future target trajectory for the vehicle M according to the content of the driving control based on the recognition results from the recognition unit 120, destination information set by the navigation device 50, detection results from the vehicle sensor 40, etc., and controls at least one of the steering and speed of the vehicle M so that the vehicle M travels along the generated target trajectory.

[0057] For example, the driving control unit 180 performs driving control such as ACC, LKAS, and ALC. Furthermore, if the driver's monitoring direction does not improve even after a predetermined time has elapsed since the HMI control unit 160 has outputted notification information indicating that the driver's monitoring direction is inappropriate, or if the driver has been performing actions other than driving that have a high impact on surrounding monitoring for a predetermined period of time, the driving control unit 180 will perform controls to stop the vehicle M in a safe position such as the shoulder of the road, or controls to avoid contact between the vehicle M and an obstacle.

[0058] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to information input from the driving control unit 180 or information input from the accelerator pedal of the driver control unit 80.

[0059] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 180 or from the brake pedal of the driver control unit 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may be equipped with a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 180 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0060] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 180 or from the steering wheel of the driver control unit 80.

[0061] [Judgment section] Next, the functions of the determination unit 140 (first determination unit 142, second determination unit 144) will be described in detail. The first determination unit 142 determines, for example, whether the driver's monitoring direction is appropriate based on the driver's monitoring direction recognized by the monitoring direction recognition unit 124 and a preset monitoring target area.

[0062] Figure 3 is a diagram showing the relationship between the driver's monitoring direction and the monitored area. In the example in Figure 3, driver D is seated in the driver's seat ST1 of vehicle M and is performing manual driving of vehicle M by operating the driving controls 80 such as the steering wheel SW. In addition, in the example in Figure 3, the display units 32-1 and 32-2 included in the HMI 30 are shown.

[0063] For example, when the vehicle M is traveling in a straight line (X-axis direction in the figure), the first determination unit 142 sets a monitoring target area AR1 that has predetermined angles to the left and right with respect to the direction of travel V of the vehicle M from the position of the driver D's head, as shown in Figure 3.

[0064] The monitored area AR1 may be adjusted according to the speed of the vehicle M and the recognition results of the recognition unit 120 (for example, road shape, road width). In this case, the first determination unit 142 may, for example, set the angle (radians) θ1 indicating the size of the arc of the monitored area AR1 to be smaller according to the speed, or set the angle θ1 to be larger the wider the road.

[0065] Furthermore, the first determination unit 142 may adjust the monitored area AR1 based on the behavior of the vehicle M recognized by the behavior recognition unit 128. For example, if the vehicle M changes lanes from the driving lane to the adjacent lane on the right, the behavior recognition unit 128 rotates the current monitored area AR1 to the right around the position of the driver D's head, and if the vehicle changes lanes from the driving lane to the adjacent lane on the left, the first determination unit 142 rotates the monitored area AR1 to the left around the position of the driver D's head. In this case, the first determination unit 142 may change the size of the angle θ1 in addition to (or instead of) the rotation. Also, when lane changes or right / left turns are performed by manual driving, the first determination unit 142 rotates the monitored area AR1 or changes the size of the angle θ1 according to the steering angle and steering amount of the vehicle M. Furthermore, if the surrounding area recognition unit 122 recognizes an obstacle around the vehicle M (for example, an obstacle on the road), the first determination unit 142 may change the size of the angle θ1 to include that obstacle.

[0066] For example, in the situation shown in Figure 3, the first determination unit 142 determines that the monitoring direction of driver D is appropriate if the monitoring direction of driver D recognized by the monitoring direction recognition unit 124 is within the angle θ1 of the monitored area AR1 (or if the state of being within the angle θ1 continues for a predetermined time or longer). The first determination unit 142 also determines that the monitoring direction of driver D is inappropriate if the monitoring direction is not within the angle θ1 (or if the state of not being within the angle θ1 continues for a predetermined time or longer). In the example in Figure 3, the first determination unit 142 determines that the monitoring direction of driver D is appropriate when the monitoring direction is arrow A1, and determines that the monitoring direction of driver D is inappropriate when the arrow is A2.

[0067] In addition, the first determination unit 142 may determine whether or not driver D is distracted, instead of (or in addition to) determining whether or not driver D's monitoring direction is appropriate. In this case, the first determination unit 142 determines that driver D is not distracted if driver D's monitoring direction is arrow A1, and determines that driver D is distracted if it is arrow A2, based on the situation of the monitored area AR1 shown in Figure 3.

[0068] The second determination unit 144, based on the recognition result of the state recognition unit 126, determines that driver D is performing an action other than driving the vehicle M if it can acquire information about actions other than driving from the feature information of the image captured by the in-vehicle camera 70 using driver state data 194 and a trained model. The second determination unit 144 also determines that driver D is not performing an action other than driving the vehicle M if it cannot acquire information about actions other than driving.

[0069] Furthermore, the second determination unit 144 may determine whether an action other than driving has little impact on surrounding monitoring, based on the degree of impact on surrounding monitoring associated with the action other than driving recognized by the state recognition unit 126. For example, the second determination unit 144 determines that the action has little impact on surrounding monitoring if the degree of impact is less than a threshold (first threshold), and determines that the action does not have little impact on surrounding monitoring if it is equal to or greater than the first threshold. Note that the determination process by the second determination unit 144 described above may be executed only when the first determination unit 142 determines that the monitoring direction of driver D is appropriate. This can reduce the load on the determination process.

[0070] [Notification control] Next, the content of the notification control in the embodiment will be described. The HMI control unit 160 controls the notification to the driver D according to the determination results of the first determination unit 142 and the second determination unit 144. For example, as an example of a notification condition, if the first determination unit 142 determines that the driver D's monitoring direction is inappropriate, the HMI control unit 160 generates notification information regarding distracted driving (first notification information) regardless of the determination result of the second determination unit 144, and outputs the generated first notification information to the HMI 30 to notify the driver D. The first notification information may be information indicating that the driver D's monitoring direction is inappropriate (or that the driver is distracted), or it may be information prompting the driver to face the correct monitoring direction. In addition, the first notification information may include, for example, an image showing the driver D's monitoring direction or an image showing the monitored area AR1, as shown in Figure 3. This allows the driver D to more accurately understand that the monitoring direction is inappropriate or the direction that should be monitored. The first notification information may be an image, an audio (for example, a predetermined notification sound), or both. In this way, when driver D's monitoring direction is inappropriate, the system prioritizes safety and reliably provides notification, enabling the driver to receive more appropriate notification.

[0071] Furthermore, even if the first determination unit 142 determines that the driver D's monitoring direction is appropriate, if the second determination unit 144 determines that the driver D is performing an action other than driving the vehicle M, the HMI control unit 160 may generate notification information regarding surrounding monitoring (second notification information) and output the generated second notification information to the HMI 30 to notify the driver D. The second notification information may be information indicating that surrounding monitoring may not be sufficient due to an action other than driving, or it may be information urging the driver to stop the action other than driving. The second notification information may also include information regarding an action other than driving recognized from the image of the in-vehicle camera 70. The second notification information may be an image, an audio (e.g., a notification sound), or both. By notifying the second notification information, for example, the driver D can be notified that it is necessary to concentrate on surrounding monitoring. The HMI control unit 160 may also notify the driver D of the first notification information regarding distracted driving instead of the second notification information.

[0072] Furthermore, even if the first determination unit 142 determines that the driver D's monitoring direction is appropriate, and the second determination unit 144 determines that driver D is performing an action other than driving vehicle M, the HMI control unit 160 suppresses the notification of the second notification information if the action other than driving has an impact on surrounding monitoring that is less than the first threshold (for example, less than 5). Suppressing notification may mean not notifying at all, or if the second notification information consists of an image and sound, it may mean outputting either one of them, such as reducing the volume or shortening the display time of the image. This makes it possible to suppress excessive notification for actions that have little impact on surrounding monitoring.

[0073] The first threshold may be a fixed value or may be adjusted for each driver D. Furthermore, the first threshold may be adjusted according to the surrounding conditions of vehicle M. For example, if vehicle M is traveling on a straight road and there are no other vehicles or objects nearby, the first threshold should be set higher than the standard value. Conversely, if vehicle M is traveling on a curved road or other non-straight road, or if there are other vehicles or objects around vehicle M, the first threshold should be set lower than the standard value. This allows for more appropriate notification depending on the driver and surrounding conditions.

[0074] Furthermore, actions that have less impact on surrounding monitoring than the first threshold may include, for example, actions that are expected to be completed within a predetermined time (first action). Since actions that are completed in a short time have little impact on surrounding monitoring, suppressing notifications in the case of such first actions can suppress excessive notifications to driver D and enable more appropriate notifications.

[0075] Furthermore, actions that have less impact on surrounding monitoring than the first threshold may include, in addition to (or instead of) the first action, actions that driver D is expected to be able to perform with one hand (the second action). For example, even if driver D is holding an object (e.g., a drink or food) with one hand to perform an action other than driving, they are still holding the steering wheel switch with the other hand, so even if surrounding monitoring is temporarily neglected, proper driving is still possible. By suppressing notifications in such situations, excessive notifications to driver D can be suppressed, and more appropriate notifications can be provided.

[0076] Furthermore, actions with an impact on surrounding monitoring below the first threshold may include, in addition to (or instead of) the first or second actions described above, actions that are expected to be performed without the driver D seeing them (third actions). For example, removing accessories or other decorative items or a mask worn by the driver can be done without seeing them. Therefore, in the case of the third action, the driver D's line of sight can be within the monitored area AR1, and by suppressing notification in the case of the third action, excessive notification to the driver D can be suppressed, and more appropriate notification can be provided.

[0077] Furthermore, actions that have less than the first threshold of impact on surrounding monitoring may include actions such as putting on or taking off items worn by driver D, or actions related to eating or drinking by driver D. In this case, the above two actions may be adjusted so that their impact on surrounding monitoring included in the driver status data 194 is less than the first threshold. In this way, excessive alerts can be suppressed by suppressing notifications for actions such as putting on or taking off easily detachable accessories, or for simple eating or drinking such as drinks or gum.

[0078] Figure 4 illustrates an example of a scenario in which the notification of the second notification information is suppressed. In the example in Figure 4, the driver D is shown in an image captured by the in-vehicle camera 70. In the example in Figure 4, driver D is holding the steering wheel SW with one hand and an object (item) OB1 such as a drink case with the other hand. In such a scenario, driver D does not need to keep looking at object OB1 and can perform actions other than driving while directing the monitoring direction towards the monitored area AR1. Furthermore, the act of drinking a drink in this manner is expected to be completed in a few seconds. Therefore, in such a scenario, the HMI control unit 160 can suppress excessive notification by suppressing the notification of the second notification information.

[0079] Figure 5 illustrates an example of a scenario in which the notification of the second notification information is not suppressed. The example in Figure 5 differs from the example in Figure 4 in that the driver is holding an object (terminal device) OB2, such as a smartphone on which content such as video can be viewed, instead of an object OB1, such as a drink case. In such a scenario, the driver D is likely to continue looking at the screen of object OB2, and the monitoring direction is not directed towards the monitored area AR1. Furthermore, the act of viewing content is expected to be carried out for a long period of time. Therefore, in such a scenario, the HMI control unit 160 can provide more appropriate notification to the driver D by notifying (not suppressing) the second notification information.

[0080] Furthermore, the HMI control unit 160 may provide notification regardless of the determination result by the first determination unit 142 and the second determination unit 144 if the amount of change or the speed at which the driver D's monitoring direction changes over a predetermined time, as recognized by the state recognition unit 126, is equal to or greater than a threshold (second threshold). Providing notification regardless of the determination result by the first determination unit 142 and the second determination unit 144 includes, for example, providing notification even if the first determination unit 142 determines that the driver D's monitoring direction is appropriate, and the second determination unit 144 determines that the driver D is not performing any actions other than driving the vehicle M, or if the driver D is performing an action other than driving but the degree of impact on surrounding monitoring is less than the first threshold.

[0081] Figure 6 is a diagram illustrating the change in the monitoring direction of driver D. In the example in Figure 6, driver D's monitoring direction moves from A3 to A4 over a predetermined time. The HMI control unit 160 provides notification if the amount of change (angle change shown in Figure 6) △θ over the predetermined time is greater than or equal to a second threshold corresponding to the amount of change, or if the speed HV at the time of the change in monitoring direction is greater than or equal to a second threshold corresponding to the speed. At this time, monitoring directions A3 and A4 may be within or outside the monitoring area AR1. This ensures that even if the monitoring direction is appropriate, if the amount of movement △θ in the monitoring direction or the speed HV at the time of change is large, and it is predicted that the driver may not be able to concentrate on monitoring the surroundings, notification can be provided reliably, prioritizing safety. Note that the amount of movement △θ and speed HV in the monitoring direction shown in Figure 6 represent the amount of movement and speed in the horizontal direction (on the XY plane), but in this embodiment, they may also be the amount of movement and speed in three dimensions (XYZ axes).

[0082] The second threshold may be changed depending on the vehicle M's condition (e.g., lane change, right or left turn, collision avoidance). For example, when the vehicle is changing lanes, turning right or left, or avoiding a collision due to driving control by the driving control unit 180 or manual driving by the driver D, it becomes necessary to monitor not only the direction of travel of the vehicle M but also its surroundings. As a result, the amount of change in the driver D's monitoring direction and the speed at which the change occurs will be larger than under normal circumstances. Therefore, in the case of the vehicle's driving conditions described above, increasing the second threshold compared to normal can suppress unnecessary notifications regarding distraction and surrounding area monitoring during lane changes, right or left turns, and collision avoidance.

[0083] Furthermore, if the HMI control unit 160 determines that the monitoring direction of driver D is inappropriate by the first determination unit 142 and that driver D is performing an action other than driving vehicle M, it may output both the first and second notification information, or it may output the notification information with the higher pre-set priority. Also, if the HMI control unit 160 determines that the monitoring direction of driver D is appropriate by the first determination unit 142 and that driver D is not performing an action other than driving vehicle M by the second determination unit 144, it terminates the notification.

[0084] [Processing flow] The following describes the processes performed by the driver assistance device 100 of this embodiment. In the following description, we will mainly focus on the notification processes based on the driver D's monitoring direction and actions among the processes performed by the driver assistance device 100.

[0085] Figure 7 is a flowchart showing an example of processing performed by the driver assistance device 100 in the embodiment. In the example in Figure 7, the monitoring direction recognition unit 124 recognizes the monitoring direction of the driver D of the vehicle M (step S100). Next, the state recognition unit 126 recognizes an action other than driving performed by the driver D (step S110). Next, the first determination unit 142 determines whether the monitoring direction of the driver D is an appropriate monitoring direction (step S120). If it is determined that the monitoring direction is appropriate, the second determination unit 144 determines whether the driver D is performing an action other than driving (step S130).

[0086] If the system determines that driver D is performing an action other than driving, the second determination unit 144 determines whether the impact on surrounding monitoring is less than the first threshold (step S140). If the system determines that the impact on surrounding monitoring is less than the first threshold, the HMI control unit 160 determines whether the amount of change in the monitoring direction or the speed at the time of change is greater than or equal to the second threshold (step S150). If the system determines that it is not greater than or equal to the second threshold (less than the second threshold), the HMI control unit 160 suppresses notification to driver D (step S160).

[0087] Furthermore, if it is determined in step S120 that the monitoring direction is inappropriate, if it is determined in step S140 that the impact on surrounding monitoring is not below the first threshold (i.e., above the first threshold), or if it is determined in step S150 that the amount or speed of change in the monitoring direction is above the second threshold, the HMI control unit 160 generates corresponding notification information (e.g., first notification information, second notification information) and outputs it to the HMI 30 to notify the driver D (step S170). This completes the processing of this flowchart. Also, if it is determined in step S130 that the driver D is not performing any actions other than driving, the processing of this flowchart is completed.

[0088] [Differentiation] In this embodiment, the HMI control unit 160 may generate information regarding driver D's actions other than driving, as recognized by the state recognition unit 126, and output it to the HMI 30. Alternatively, the HMI control unit 160 may generate information to inquire with driver D whether the recognized non-driving actions are correct, output it to the HMI 30, and update the driver state data 194 and other information based on the result of the inquiry (driver D's response). This allows for more accurate state recognition for each driver D.

[0089] Furthermore, if the HMI control unit 160 is suppressing the notification as described above while the driver D's monitoring direction is within the monitored area AR1, it may release the suppression and issue a notification if the driver D's monitoring direction moves outside the monitored area AR1. This allows the notification control to be immediately activated (restored).

[0090] Furthermore, in this embodiment, the driving control unit 180 may continue driving control if, during the execution of driving control, the first determination unit 142 determines that the direction of monitoring of driver D is appropriate, and the second determination unit 144 determines that driver D is performing an action other than driving the vehicle M, but the impact of the action other than driving is less than the first threshold. This allows driving control to be continued for actions other than driving that have little impact on surrounding monitoring, thereby suppressing changes in control and achieving stable driving control.

[0091] Furthermore, the driving control unit 180 terminates the driving control if the first determination unit 142 determines that the driver D's monitoring direction is inappropriate during the execution of driving control (or if this determination continues for a predetermined time or longer). The driving control unit 180 may also terminate the driving control if the second determination unit 144 determines that the driver D is performing an action other than driving the vehicle M during the execution of driving control, and the action other than driving has an impact on surrounding monitoring of a first threshold or higher (or if the above condition continues for a predetermined time or longer). In this case, the HMI control unit 160 may output information to the HMI 30 indicating that the ongoing driving control has ended, along with the first or second notification information, to notify the driver D.

[0092] According to the embodiments described above, the notification control device includes a recognition unit 120 that recognizes the monitoring direction and state of the driver of a vehicle M (an example of a moving object), a first determination unit 142 that determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit 120, a second determination unit 144 that determines whether the driver is performing an action other than driving the vehicle M based on the recognition result, and a notification control unit (HMI 30, HMI control unit 160) that controls notification to the driver D based on the determination result of the first determination unit 142 and the determination result of the second determination unit 144. The notification control unit suppresses notification when the first determination unit 142 determines that the driver's monitoring direction is appropriate and the second determination unit 144 determines that the driver is performing an action other than driving the vehicle M, but the impact of the action on surrounding monitoring is less than a first threshold, thereby enabling more appropriate notification according to the driver's state.

[0093] For example, according to one embodiment, even if the driver is performing an action (work) other than driving, if the monitoring direction is appropriate and the action other than driving has little impact on surrounding monitoring, excessive notifications can be suppressed, and as a result, more appropriate notifications can be provided according to the driver's condition.

[0094] Furthermore, according to the embodiment, excessive notifications can be suppressed by suppressing notifications for short-duration actions or actions that can be performed with one hand (for example, putting on or taking off accessories or eating and drinking). In addition, according to the embodiment, notifications can be given when it is determined that the monitoring direction is inappropriate or when the amount or speed of change in the monitoring direction exceeds a second threshold, thereby providing more appropriate notifications according to the driver's condition.

[0095] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: The monitoring direction of the driver of the moving object and the status of the driver are recognized. Based on the recognized results, it is determined whether the driver's monitoring direction is appropriate. Based on the results of the recognition, it is determined whether the driver is performing any action other than driving the mobile body. Based on the results of the determination, the notification to the driver is controlled. If it is determined that the driver's monitoring direction is appropriate, and that the driver is performing an action other than driving the moving object, but the impact of the action other than driving on surrounding monitoring is less than the first threshold, the notification is suppressed. Notification control device.

[0096] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0097] 1...Vehicle system, 10...Camera, 12...Radar device, 14...LIDAR, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 70...In-cabin camera, 80...Driver's control panel, 100...Driver's assistance device, 120...Recognition unit, 122...Surroundings recognition unit, 124...Monitoring direction recognition unit, 126...State recognition unit, 128...Behavior recognition unit, 140...Determination unit, 142...First determination unit, 144...Second determination unit, 160...HMI control unit, 180...Driving control unit, 190...Storage unit, 200...Driving force output device, 210...Brake device, 220...Steering device, M...Vehicle

Claims

1. A recognition unit that recognizes the monitoring direction of the driver of the moving object and the state of the driver, A first determination unit determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit, A second determination unit that determines whether the driver is performing an action other than driving the mobile body based on the recognition result, The system includes a notification control unit that controls notification to the driver based on the determination result from the first determination unit and the determination result from the second determination unit, The notification control unit suppresses the notification if the first determination unit determines that the driver's monitoring direction is appropriate, and the second determination unit determines that the driver is performing an action other than driving the mobile body, but the impact of the action other than driving on surrounding monitoring is less than the first threshold. Notification control device.

2. Actions whose impact on the surrounding monitoring is less than the first threshold include actions that are expected to be completed within a predetermined time. The notification control device according to claim 1.

3. Actions that have less impact on surrounding monitoring than the first threshold include actions that the driver is expected to be able to perform with one hand. The notification control device according to claim 1.

4. Actions that have an impact on surrounding monitoring that is less than the first threshold include actions of putting on or taking off items worn by the driver. The notification control device according to claim 1.

5. Actions that have an impact on surrounding monitoring that is below the first threshold include actions related to the driver eating and drinking. The notification control device according to claim 1.

6. The notification control unit shall, if the first determination unit determines that the driver's monitoring direction is inappropriate, perform the notification regardless of the determination result of the second determination unit. The notification control device according to claim 1.

7. The recognition unit recognizes the amount of change or the speed at the time of change in the direction monitored by the driver over a predetermined period of time. The notification control unit shall, when the amount of change or speed is equal to or greater than the second threshold, provide notification regardless of the determination result by the first determination unit and the determination result by the second determination unit. The notification control device according to claim 1.

8. Computers The monitoring direction of the driver of the moving object and the status of the driver are recognized. Based on the recognized results, it is determined whether the driver's monitoring direction is appropriate. Based on the results of the recognition, it is determined whether the driver is performing any action other than driving the moving object. Based on the results of the determination, the notification to the driver is controlled. If it is determined that the driver's monitoring direction is appropriate, and it is determined that the driver is performing an action other than driving the moving object, but the impact of that action on surrounding monitoring is less than the first threshold, the notification is suppressed. Notification control method.

9. On the computer, The system recognizes the monitoring direction of the driver of the moving object and the state of the driver. Based on the recognized results, the system determines whether the driver's monitoring direction is appropriate. Based on the recognized result, determine whether the driver is performing any action other than driving the moving object. Based on the determined result, the notification to the driver is controlled. If it is determined that the driver's monitoring direction is appropriate, and it is determined that the driver is performing an action other than driving the moving object, but the impact of that action on surrounding monitoring is less than the first threshold, the notification is suppressed. program.

Images