Report control device, report control method, and storage medium

Abstract

Provided are a report control device, a report control method, and a storage medium capable of performing a more appropriate report in accordance with the state of a driver. A notification control device according to an embodiment includes: a recognition unit that recognizes a monitoring direction of a driver of a moving body and a state of the driver; a first determination unit that determines whether or not the monitoring direction of the driver is appropriate on the basis of the recognition result of the recognition unit; a second determination unit that determines, on the basis of the recognition result, whether or not the driver performs a behavior other than the driving of the moving body; and a report control unit that controls a report to the driver on the basis of the determination result determined by the first determination unit and the determination result determined by the second determination unit. The notification control unit suppresses the notification when the first determination unit determines that the monitoring direction of the driver is appropriate and the second determination unit determines that the driver performs a behavior other than driving of the moving body but the behavior other than driving is a behavior in which the degree of influence on the surroundings monitoring is less than a first threshold value.

Description

Technical Field

[0001] This invention relates to a report control device, a report control method, and a storage medium. Background Technology

[0002] In recent years, efforts have been intensifying to provide sustainable transportation systems that also take into account vulnerable individuals among transportation participants. Research and development related to preventative safety technologies are being undertaken to further improve the safety and convenience of transportation in order to achieve this. In this context, technologies previously known include those that use images captured by a driver's camera to identify the driver's gaze and facial orientation, and provide driver support when the identification results indicate that the driver is looking sideways (e.g., Japanese Patent Application Publication No. 2019-91281). Summary of the Invention

[0003] However, in previous safety prevention technologies, it is known to report situations where the driver is not focused on driving, unlike when the driver is looking around and engaging in activities other than driving. But there are various types of non-driving behaviors, so sometimes reports are output even when the driver is capable of focusing on driving. Therefore, there is a problem that reports may not be appropriate for the driver's state.

[0004] One of the objectives of this application is to provide a reporting control device, reporting control method, and storage medium capable of making more appropriate reports based on the driver's condition, in order to solve the aforementioned problems. Furthermore, it contributes to the development of sustainable transportation systems.

[0005] The report control device, report control method and storage medium involved in this invention adopt the following structure.

[0006] (1): A report control device according to one aspect of the present invention includes: an identification unit that identifies the monitoring direction of the driver of the mobile body and the state of the driver; a first determination unit that determines whether the monitoring direction of the driver is appropriate based on the identification result of the identification unit; a second determination unit that determines whether the driver is performing an action other than driving the mobile body based on the identification result; and a report control unit that controls the reporting to the driver based on the determination result determined by the first determination unit and the determination result determined by the second determination unit, wherein the report control unit suppresses the reporting when the first determination unit determines that the monitoring direction of the driver is appropriate, and although the second determination unit determines that the driver is performing an action other than driving the mobile body, the action other than driving is an action whose influence on the surrounding monitoring is less than a first threshold.

[0007] (2): Based on the above (1) scheme, behaviors that have less than the first threshold impact on the surrounding surveillance include behaviors that are pre-planned to end within a specified time.

[0008] (3): Based on the above (1) scheme, behaviors that have less than the first threshold impact on the surrounding surveillance include behaviors that the driver can perform with one hand.

[0009] (4): Based on the above (1) scheme, behaviors that have less than the first threshold impact on the surrounding surveillance include the driver’s behavior of removing or wearing items.

[0010] (5): Based on the above (1) scheme, behaviors that have less than the first threshold impact on the surrounding surveillance include the driver’s diet-related behaviors.

[0011] (6): Based on the above (1) scheme, if the first determination unit determines that the driver's monitoring direction is inappropriate, the report control unit shall make the report regardless of the determination result of the second determination unit.

[0012] (7): Based on the above (1) scheme, the identification unit identifies the amount of change or the speed of change in the driver's monitoring direction within a specified time. When the amount of change or the speed is above the second threshold, the reporting control unit reports regardless of the determination result determined by the first determination unit and the determination result determined by the second determination unit.

[0013] (8): Another aspect of the present invention relates to a reporting control method that causes a computer to perform the following processing: identifying the monitoring direction of the driver of the mobile body and the state of the driver; determining whether the monitoring direction of the driver is appropriate based on the identification result; determining whether the driver is performing an action other than driving the mobile body based on the identification result; controlling the reporting to the driver based on the determination result; and suppressing the reporting if it is determined that the monitoring direction of the driver is appropriate, and although it is determined that the driver is performing an action other than driving the mobile body, the action other than driving is an action whose influence on the surrounding monitoring is less than a first threshold.

[0014] (9): Another aspect of the present invention involves a storage medium storing a program that causes a computer to perform the following processing: identifying the monitoring direction of the driver of the mobile body and the state of the driver; determining whether the monitoring direction of the driver is appropriate based on the identification result; determining whether the driver is performing an action other than driving the mobile body based on the identification result; controlling the reporting to the driver based on the determination result; and suppressing the reporting if it is determined that the monitoring direction of the driver is appropriate, and although it is determined that the driver is performing an action other than driving the mobile body, the action other than driving has an impact on the surrounding monitoring less than a first threshold.

[0015] Based on the above schemes (1) to (9), more appropriate reports can be made based on the driver's condition. Attached Figure Description

[0016] Figure 1 It is a structural diagram of the vehicle system, including the reporting control device involved in the implementation method.

[0017] Figure 2 This is a diagram illustrating an example of the data content used to explain driver status data.

[0018] Figure 3 It is a diagram showing the relationship between the driver's monitoring direction and the monitored area.

[0019] Figure 4 This is a diagram illustrating an example of a scenario where the reporting of second-report information is suppressed.

[0020] Figure 5 This is a diagram illustrating an example of a scenario where the reporting of second-report information is not suppressed.

[0021] Figure 6 This is a diagram used to illustrate changes in the driver's monitoring direction.

[0022] Figure 7 This is a flowchart illustrating an example of the processing performed by the driving support device in the implementation. Detailed Implementation

[0023] Hereinafter, embodiments of the report control device, report control method, and storage medium of the present invention will be described with reference to the accompanying drawings. Examples of applications of the report control device to mobile bodies will be described below. As an example of a mobile body, a vehicle is used. Besides vehicles, mobile bodies may include, for example, ships capable of moving on land (roads) such as hovercraft, flying vehicles capable of traveling on roads, standing vehicles with power units, and miniature mobile bodies such as electric scooters.

[0024] [Overall Structure]

[0025] Figure 1 This is a structural diagram of vehicle system 1, including the reporting control device involved in the implementation method. The vehicle equipped with vehicle system 1 (hereinafter referred to as vehicle M) is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, a micro-mobile body, etc., and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a battery such as a secondary battery or a fuel cell.

[0026] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) system 14, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an in-vehicle camera 70, driver control components 80, a driver support device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected via CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. Figure 1 The structure shown is only one example; some parts of the structure may be omitted, and other structures may be added. Camera 10, radar device 12, and LIDAR 14 are examples of a "detection device DD". HMI 30 is an example of a "reporting unit". HMI 30 and driver support device 100 are examples of a "reporting control device".

[0027] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). Camera 10 is mounted anywhere on the vehicle M equipped with vehicle system 1. When shooting forward, camera 10 is mounted on the upper part of the windshield, behind the interior rearview mirror, or at the front of the vehicle body. When shooting backward, camera 10 is mounted on the upper part of the rear windshield or the tailgate. When shooting to the side, camera 10 is mounted on the left and right door rearview mirrors. Camera 10 periodically and repeatedly captures images of the surrounding area of ​​vehicle M. Camera 10 can also be a stereo camera.

[0028] Radar device 12 radiates millimeter-wave and other radio waves (radar) to the periphery of vehicle M, and detects radio waves (reflected waves) reflected from surrounding objects to at least detect the position (distance and orientation) of the objects. Radar device 12 can be installed at any location on vehicle M. Radar device 12 can also detect the position and speed of objects using FM-CW (Frequency Modulated Continuous Wave) method.

[0029] The LIDAR14 illuminates the perimeter of vehicle M and measures the scattered light. The LIDAR14 determines the distance to the object based on the time from emission to reception. The illuminating light can be, for example, a pulsed laser. The LIDAR14 can be mounted at any location on vehicle M.

[0030] The communication device 20 may utilize networks such as cellular networks, Wi-Fi networks, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), LAN (Local Area Network), WAN (Wide Area Network), and the Internet to communicate with other vehicles in the vicinity of vehicle M, using the terminal devices of users of vehicle M, or various server devices.

[0031] HMI 30 outputs various information to the occupants (including the driver) of vehicle M and accepts input operations performed by the occupants. HMI 30 includes, for example, a display unit 32 and a speaker 34. The display unit 32 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The display unit 32 displays various images (including video) in the embodiment. The display unit 32 may also be integrated with the input unit as a touch panel. The speaker 34 outputs prescribed sounds (e.g., report tones or message tones). HMI 30 may also include a microphone, a buzzer, a touch panel, switches, buttons, etc. Switches may also include switches that enable or terminate prescribed driving controls that can be performed by the driving control unit described later, or switches that acknowledge (permit) or reject driving control suggestions (proposals) from the system (vehicle system 1). Switches may also include switches for directional indication operations (turn indicator light switches), etc.

[0032] Vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of vehicle M, an acceleration sensor for detecting acceleration, and a yaw rate sensor for detecting yaw rate (e.g., the rotational angular rate about a vertical axis passing through the center of gravity of vehicle M). Vehicle sensor 40 may also include a lateral acceleration sensor (lateral G sensor) for detecting the lateral acceleration (lateral G) of vehicle M, a steering angle sensor for detecting the steering angle of vehicle M (which may be the angle of the steering wheel or the operating angle of the steering wheel), a steering angular rate sensor for detecting the steering angular rate, and an orientation sensor for detecting the orientation of vehicle M, etc.

[0033] Vehicle sensor 40 may also include a position sensor that detects the position of vehicle M. The position sensor may be, for example, a sensor that obtains position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that obtains position information using, for example, a GNSS (Global Navigation Satellite System) receiver of navigation device 50. Vehicle sensor 40 may also derive the speed of vehicle M based on the difference (i.e., distance) in position information over a specified time from the position sensor. The result detected by vehicle sensor 40 is output to driving support device 100.

[0034] The navigation device 50 includes, for example, a GNSS receiver, a navigation HMI, and a route determination unit. The navigation device 50 can store map information in a storage device such as an HDD (Hard Disk Drive) or flash memory, and can also retrieve map information 192 stored in the storage unit 190 described later. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40. The navigation HMI includes a display device, a speaker, a touch panel, buttons, etc. The GNSS receiver can also be installed in the vehicle sensor 40. The navigation HMI can also be partially or entirely shared with the aforementioned HMI 30. The route determination unit refers to, for example, map information 192, to determine a path (hereinafter referred to as the map path) from the position of the vehicle M determined by the GNSS receiver (or any input position) to the destination input by the occupant using the navigation HMI. The navigation device 50 provides route guidance using the navigation HMI based on the determined map path. The navigation device 50 can also send its current location and destination to the navigation server via the communication device 20, and obtain the same path as the path on the map from the navigation server.

[0035] Here, map information 192 represents road shape information, for example, by showing road segments (an example of a moving road) and nodes connecting these segments. Map information 192 may also include POI (Point of Interest) information. Map information 192 may include, for example, the number of lanes (moving roads), the type and shape of road markings, information about the center of lanes, or information about road boundaries. Map information 192 may also include information such as whether the road boundary is a physical boundary, including structures that vehicles cannot pass through (including crossing or contacting). Physical boundaries include, for example, guardrails, curbs, median strips, fences, etc. Map information 192 may include road shape information, traffic restriction information, address information (address / postal code), facility information, parking information, telephone number information, etc. Road shape information includes, for example, the road curvature (also referred to as radius of curvature, as will be the case below), road width, road surface slope, branching points, merging points, intersections, T-junctions, etc. Map information 192 can be updated at any time by communicating with external devices through communication device 20.

[0036] The in-vehicle camera 70 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS sensor. The in-vehicle camera 70 can be installed at any location within the vehicle M, positioned and facing forward to capture images of the driver's head seated in the driver's seat. For example, the in-vehicle camera 70 may be installed near a display device located in the center of the dashboard of the vehicle M (e.g., above or below). In addition to the driver, the in-vehicle camera 70 can also capture images of the interior of the vehicle, including occupants such as those in the front passenger seat. The in-vehicle camera 70 can also capture images of the interior by emitting infrared light into the vehicle. For example, the in-vehicle camera 70 may periodically and repeatedly capture images of the interior.

[0037] The driving control unit 80 includes, for example, a steering wheel, an accelerator pedal, and a brake pedal. The driving control unit 80 may also include a gear shift lever, a swivel steering gear, a joystick, and other control components. Each control component of the driving control unit 80 is equipped with an operation detection unit that detects the amount of operation performed by the driver or the presence or absence of such operation. The operation detection unit may detect, for example, the steering angle of the steering wheel, steering torque (e.g., the amount of steering input torque based on the driver's driving operation), the rate of change of steering torque, and the amount of pressure applied to the accelerator and brake pedals. Furthermore, the operation detection unit outputs the detection results to one or both of the following: the driving support device 100, the driving force output device 200, the braking device 210, and the steering device 220. The driving control unit 80 may also include a direction indication control unit (e.g., a turn indicator control lever, a turn indicator switch). When the direction indication control unit is operated, the vehicle's turn indicator (direction indicator) corresponding to the operation flashes, and the operation content (e.g., the detection result of the driver performing the operation) is output to the driving support device 100.

[0038] The driving support device 100 performs various controls to support the driving of the driver of the vehicle M. The driving support device 100 includes, for example, an identification unit 120, a determination unit 140, an HMI control unit 160, a driving control unit 180, and a storage unit 190. The identification unit 120, determination unit 140, HMI control unit 160, and driving control unit 180 are each implemented by executing programs (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can also be implemented using hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System on Chip), or through the coordinated use of software and hardware. The aforementioned program can be pre-saved in a storage device such as an HDD or flash memory (a storage device with a non-temporary storage medium) of the driver support device 100, or it can be saved in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driver support device 100 by mounting the storage medium (non-temporary storage medium) to the drive unit, card slot, etc. The driving control unit 180 is an example of a "movement control unit." The HMI control unit 160 is an example of a "reporting control unit."

[0039] The storage unit 190 can also be implemented using various storage devices described above, or EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. For example, map information 192, driver status data 194, various information and programs in the embodiment, etc., are stored in the storage unit 190. Various setting information used in the processing of this embodiment can also be stored in the storage unit 190.

[0040] Figure 2 This is a diagram illustrating an example of the data content of driver status data 194. Figure 2 The driver status data 194 shown is, for example, data that establishes a correspondence between the driver's non-driving behaviors, characteristic information, and the degree of influence on surrounding surveillance. Non-driving behaviors include, for example, the driver's actions such as removing or putting on items (e.g., jewelry, hats, glasses, masks, etc.), eating-related behaviors, operation of terminal devices such as smartphones, and operation of navigation devices. Eating-related behaviors may include, in addition to drinking beverages and eating food (including chewing gum), opening and closing beverage lids and taking food out of bags. Characteristic information is feature information corresponding to each behavior obtained from images captured by the in-vehicle camera 70. Characteristic information may also include feature information obtained from images (animations) captured in a time sequence (e.g., information related to the driver's actions). The degree of influence on surrounding surveillance is, for example, an index value indicating the degree of decreased attention to surrounding surveillance; the higher the value, the lower the attention to surrounding surveillance (the less focused the driver). Driver status data 194 can be obtained from an external device via communication device 20, or it can be pre-registered by the driver or others.

[0041] The degree of impact on surrounding surveillance can be adjusted for each driver or based on the number of times non-driving actions are performed. For example, even for the same action, there are cases where the driver can focus on surrounding surveillance and cases where they cannot, depending on the driver. In cases such as taking off or putting on items, or eating, as the driver becomes more proficient, they can perform the action with less impact on surrounding surveillance. Therefore, by adjusting the degree of impact for each driver or reducing the degree of impact based on the number of times the action is performed, the decision processing based on the decision unit 140 described later can be performed more appropriately.

[0042] The identification unit 120 includes, for example, a perimeter identification unit 122, a monitoring direction identification unit 124, a state identification unit 126, and a behavior identification unit 128. The perimeter identification unit 122 identifies the surrounding conditions of the vehicle M based on the detection results of the detection device DD (information input from the camera 10, radar device 12, and LIDAR 14). For example, the perimeter identification unit 122 performs sensor fusion processing on some or all of the detection results from the camera 10, radar device 12, and LIDAR 14 to identify the position (relative position), size, speed (relative speed), acceleration, and other states of objects existing around the vehicle M (within a specified distance). Objects identified by the perimeter identification unit 122 may include, in addition to the physical boundaries dividing the road (moving path), other traffic participants such as vehicles, pedestrians, and bicycles (an example of an obstacle). The position of the object is identified, for example, as its position on absolute coordinates with a representative point of the vehicle M (center of gravity, drive shaft center, etc.) as the origin, and is used for control. The position of an object can be represented by representative points such as its center of gravity or corners, or by the area it represents. The "state" of an object, for example, in the case of another vehicle, can also include the acceleration, jerk, or "action state" of the other vehicle (e.g., whether the other vehicle is changing lanes or is about to change lanes).

[0043] The perimeter recognition unit 122 can also recognize, for example, temporary stop lines, red lights, toll booths, other road phenomena, road signs, and markings painted on the road (e.g., speed limits). The perimeter recognition unit 122 can also recognize the curvature of the driving lane (driving road) of the vehicle M based on the detection results of the detection device DD or map information 192. The perimeter recognition unit 122 can also recognize the road surface condition (e.g., whether the road surface is frozen or otherwise slippery) based on the detection results of the detection device DD.

[0044] The perimeter recognition unit 122 identifies, for example, the lane in which vehicle M is traveling (driving lane) and other lanes existing in the surrounding area (e.g., adjacent lanes). For instance, the perimeter recognition unit 122 identifies road markings from images captured by camera 10, and identifies the driving lane and other lanes based on the identified positional relationship of the road markings as viewed from vehicle M. The perimeter recognition unit 122 can also identify the driving lane and other lanes of vehicle M by referring to map information 192 based on the position information of vehicle M obtained from vehicle sensor 40, etc. The perimeter recognition unit 122 can also identify structures such as tunnels, elevated roads, viaducts, and structures surrounding the road.

[0045] The monitoring direction recognition unit 124 performs known image analysis processing (e.g., feature extraction of edges, shapes, sizes, and colors, pattern matching, etc.) on the image captured by the in-vehicle camera 70, and identifies the monitoring direction of the driver of the vehicle M based on the image analysis results. For example, the monitoring direction recognition unit 124 uses at least one of the driver's gaze and the direction of their face obtained from the image analysis results to identify the monitoring direction.

[0046] For example, the monitoring direction recognition unit 124 uses methods such as pattern matching to detect combinations of a reference point (the stationary part of the eye) and a moving point (the moving part of the eye) of the driver's eyes from the captured image. The combination of the reference point and the moving point can be, for example, the combination of the corner of the eye and the iris, or the combination of the corneal reflective area and the pupil. The corneal reflective area is, for example, the area in the cornea that reflects infrared light when the in-vehicle camera 70 shines infrared light towards the driver. Furthermore, the monitoring direction recognition unit 124 identifies the driver's gaze by performing coordinate transformations from the image plane to actual space based on the position of the moving point relative to the reference point.

[0047] The monitoring direction recognition unit 124 identifies the orientation of the driver's face based on the positional information (relative positional information of each part, etc.) of the eyes, nose, mouth, etc. in the face area obtained from the analysis results of the captured image.

[0048] When the monitoring direction recognition unit 124 successfully recognizes both the driver's gaze and the direction of their face, it uses both to identify the monitoring direction. This allows for more accurate identification of the monitoring direction. Alternatively, when only one of the gaze or the direction of the face is recognized, the monitoring direction recognition unit 124 uses that one to identify the monitoring direction.

[0049] The monitoring direction recognition unit 124 can also identify whether the driver is wearing sunglasses, glasses, or a mask by means of template matching or the analysis results of the image from the in-vehicle camera 70. In this case, when the driver is wearing sunglasses or glasses, the monitoring direction recognition unit 124 identifies the monitoring direction based solely on the orientation of the face. When the driver is wearing a mask, features such as the nose and mouth cannot be identified from the image, and the accuracy of facial orientation recognition is reduced. Therefore, the monitoring direction is identified solely based on the line of sight.

[0050] In the recognition of various information using the images described above, a pre-learned model, learned in advance through machine learning or the like, can also be used. In this case, the monitoring direction recognition unit 124 can also input the image captured by the in-vehicle camera 70 into the pre-learned model, which takes the image as input and outputs the monitoring direction (e.g., gaze, face orientation) of the person contained in the image, to obtain the driver's monitoring direction. The monitoring direction recognition unit 124 can also use other known methods to recognize the monitoring direction.

[0051] The recognition result of the monitoring direction recognition unit 124 may also include information indicating whether the monitoring direction was recognized based on the driver's line of sight, the driver's face orientation, or both. The recognition result may also include information indicating whether the driver's line of sight was recognized from one eye or both eyes, and may also include information indicating that no line of sight was recognized or that no face orientation was recognized.

[0052] The monitoring direction recognition unit 124 can also recognize the amount of change in the driver's monitoring direction over a specified time or the speed when the monitoring direction changes. The amount of change is, for example, the total amount of change over the specified time (e.g., the total amount of left and right movement in the case of left or right movement). The speed can be the maximum speed over the specified time or the average speed.

[0053] The state recognition unit 126 identifies the driver's state. For example, the state recognition unit 126 performs known image analysis processing (e.g., feature extraction of edges, shapes, sizes, and colors) on images captured by the in-vehicle camera 70, and uses the feature information obtained as a result of image analysis to refer to the feature information of driver state data 194 stored in the storage unit 190 to obtain the driver's behavior other than driving and the degree of influence on surrounding surveillance corresponding to the feature information with a consistency (or similarity) of more than a predetermined value and the highest consistency. The feature information may include feature information obtained from images (animations) captured in a time sequence (e.g., information related to the driver's actions). For example, the feature information may include feature information based on the object (item) held by the driver, hand movements, the driver's gaze, the orientation of the face, etc.

[0054] The state recognition unit 126 can also input feature information obtained from the image into the learned model, which takes feature information as input and outputs non-driving behaviors of the driver and the degree of influence on surrounding surveillance, thereby obtaining non-driving behaviors and the degree of influence on surrounding surveillance. The learned model can be obtained from an external device via the communication device 20, or it can be stored in the storage unit 190.

[0055] The behavior recognition unit 128 identifies the behavior of the vehicle M based on the detection results of the vehicle sensor 40 and the control content executed by the driving control unit 180. The behavior of the vehicle M includes behaviors generated by the driver's manual driving and behaviors generated by the driving control executed by the driving control unit 180.

[0056] For example, the behavior recognition unit 128 identifies the lateral position (position in the lane width direction) of vehicle M relative to the driving lane and the orientation (orientation) of vehicle M relative to the extending direction of the driving lane based on the positional relationship of vehicle M relative to the driving lane. For example, the behavior recognition unit 128 can also identify the deviation of the reference point of vehicle M from the center of the lane and the angle formed by the direction of travel of vehicle M relative to the line connecting the centers of the lane as the relative position and orientation of vehicle M relative to the driving lane. Alternatively, the behavior recognition unit 128 can identify the position of the reference point of vehicle M relative to any side end (road dividing line or road boundary) of the driving lane as the relative position (lateral position) of vehicle M relative to the driving lane. The behavior recognition unit 128 can identify the lateral behavior of vehicle M (e.g., whether it has moved a lateral distance or more within a specified time) based on the changes in the lateral position and orientation (yaw rate) of vehicle M, and can also identify it as disordered behavior if the change is greater than a specified amount.

[0057] When the vehicle M is being manually driven, the behavior recognition unit 128 detects the behavior of the vehicle M based on the amount of steering wheel operation (e.g., steering angle, steering torque, steering torque change rate), the amount of accelerator pedal and brake pedal being pressed, etc., obtained by the operation detection unit, or identifies the amount of change in behavior over a specified time.

[0058] The behavior recognition unit 128 recognizes the behavior of the vehicle M based on the driving control executed by the driving control unit 180. Driving control is control that moves the vehicle M by controlling at least one of its steering and speed, independent of driving operations performed by the driver or by receiving only a portion of the operation instructions. Driving control includes, for example, ACC (Adaptive Cruise Control System), LKAS (Lane Keeping Assistance System), and ALC (Auto Lane Changing). Driving control may include controls to stop the vehicle M in a safe position such as a curb, and controls to control steering and speed to prevent contact between the vehicle M and obstacles detected by the perimeter recognition unit 122. For example, the behavior recognition unit 128 recognizes the behavior of the vehicle M resulting from the execution of driving control unit functions such as LKAS and ALC by the driving control unit 180.

[0059] The determination unit 140 includes, for example, a first determination unit 142 and a second determination unit 144. The first determination unit 142 determines whether the driver's monitoring direction is appropriate based on the recognition result of the monitoring direction recognition unit 124.

[0060] The second determination unit 144 determines whether the driver is engaging in any behavior other than driving the vehicle M based on the recognition result of the state recognition unit 126. Details regarding the functions of the first determination unit 142 and the second determination unit 144 will be described later.

[0061] HMI control unit 160 notifies occupants (including the driver) of prescribed information via HMI 30, or receives information input from HMI 30. The prescribed information includes, for example, information related to the state of vehicle M, information related to driving control, and other information relevant to the movement of vehicle M. Information related to the state of vehicle M includes, for example, the vehicle M's speed, engine speed, and gear position. Information related to driving control includes, for example, information regarding the presence or absence of driving control execution based on driving control unit 180, information related to the execution status of driving control, information related to driving control suggestions (proposals) from the system side, and report information (alarms, etc.) to the driver. The prescribed information may also include information related to the surrounding conditions identified by the detection device DD. The prescribed information may also include information not related to the movement of vehicle M, such as television programs or items stored on storage media (e.g., animations). The prescribed information may include, for example, information related to the vehicle M's current location, destination, and remaining fuel level. The HMI control unit 160 can also output the information received by the HMI 30 to the communication device 20, navigation device 50, identification unit 120, determination unit 140, driving control unit 180, etc.

[0062] The HMI control unit 160 can also generate inquiry information, suggestion information, recognition results identified by the recognition unit 120, judgment results determined by the judgment unit 140, report information, etc., for the occupants, and cause the HMI 30 to output the generated information. The generated information includes images, sounds (including report tones, etc.). The HMI control unit 160 can also send the various information output by the HMI 30 to the terminal device used by the occupants of the vehicle M via the communication device 20.

[0063] The driving control unit 180 controls the movement of the vehicle M. For example, the driving control unit 180 performs driving control for the vehicle M based on the recognition result identified by the recognition unit 120, the determination result determined by the determination unit 140, etc. Regarding driving control, it can be performed based on instructions from the driver via the HMI 30, or it can be performed independently of driver instructions based on the recognition result identified by the recognition unit 120, etc. When performing driving control, the driving control unit 180 generates a future target trajectory for the vehicle M corresponding to the content of the driving control based on the recognition result identified by the recognition unit 120, the destination information set by the navigation device 50, the detection results detected by the vehicle sensor 40, etc., and controls at least one of the steering and speed of the vehicle M so that the vehicle M travels according to the generated target trajectory.

[0064] For example, the driving control unit 180 performs driving controls such as ACC, LKAS, and ALC. If, even after a predetermined time has elapsed since the HMI control unit 160 outputs a report indicating that the driver's monitoring direction is inappropriate, and no improvement has been observed in the driver's monitoring direction, the driving control unit 180 may, if the driver has engaged in driving-related behaviors that have a high impact on surrounding monitoring for a predetermined time, perform controls such as stopping the vehicle M in a safe position, such as on the shoulder, or prevent obstacles from contacting the vehicle M.

[0065] The driving force output device 200 outputs driving force (torque) for vehicle movement to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the above structure according to information input from the driving control unit 180 or from the accelerator pedal of the driving operation unit 80.

[0066] The braking device 210 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 180 or from the brake pedal of the driving control unit 80, so that braking torque corresponding to the braking operation is output to each wheel. The braking device 210 may have a backup mechanism for transmitting the hydraulic pressure generated by the operation of the brake pedal to the hydraulic cylinder via the master hydraulic cylinder. The braking device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that transmits hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder by controlling the actuator according to information input from the driving control unit 180.

[0067] The steering system 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor according to information input from the driving control unit 180 or from the steering wheel of the driving operation unit 80, thereby changing the direction of the steering wheels.

[0068] [Judgment Department]

[0069] Next, the functions of the determination unit 140 (first determination unit 142 and second determination unit 144) will be explained in detail. The first determination unit 142 determines, for example, whether the driver's monitoring direction is appropriate based on the driver's monitoring direction identified by the monitoring direction recognition unit 124 and a preset monitoring target area.

[0070] Figure 3 This is a diagram showing the relationship between the driver's monitoring direction and the monitored area. Figure 3 In the example, a simplified illustration shows a scenario where driver D is seated in the driver's seat ST1 of vehicle M and manually drives vehicle M by operating driving controls 80 such as the steering wheel SW. Figure 3 In the example, display units 32-1 and 32-2 included in HMI30 are shown.

[0071] For example, when the first determination unit 142 is moving the vehicle M along the straight direction (X-axis direction in the figure), such as Figure 3 As shown, the monitoring target area AR1 is set at a specified angle to the left and right from the position of the driver D's head, with the direction of travel V of the vehicle M as the reference.

[0072] The monitored area AR1 can also be adjusted based on the speed of the vehicle M, the recognition results of the recognition unit 120 (e.g., road shape, road width), etc. In this case, the first determination unit 142 sets the angle (radians) θ1, which represents the size of the arc of the monitored area AR1, to be small, for example, based on the speed, or sets the angle θ1 to be large if the road width is larger.

[0073] The first determination unit 142 can also adjust the monitored target area AR1 based on the behavior of the vehicle M identified by the behavior recognition unit 128. For example, if the behavior recognition unit 128 determines that the vehicle M is changing lanes from the driving lane to the adjacent lane on the right, the current monitored target area AR1 is rotated to the right around the position of the driver's head; if the vehicle M is changing lanes from the driving lane to the adjacent lane on the left, the current monitored target area AR1 is rotated to the left around the position of the driver's head. In this case, the first determination unit 142 can also change the size of the angle θ1 based on the rotation (or instead). When changing lanes or turning left or right by manual driving, the first determination unit 142 also rotates the monitored target area AR1 or changes the size of the angle θ1 based on the steering angle, steering amount, etc. of the vehicle M. The first determination unit 142 can also change the size of the angle θ1 to include the obstacle if the perimeter recognition unit 122 identifies an obstacle around the vehicle M (e.g., an obstacle on the road).

[0074] For example, in Figure 3 In the illustrated situation, the first determination unit 142 determines that the driver D's monitoring direction is appropriate when the monitoring direction recognition unit 124 identifies that the driver D's monitoring direction is within angle θ1 of the monitored object area AR1 (or when the state of being within angle θ1 has lasted for a predetermined time or more). The first determination unit 142 determines that the driver D's monitoring direction is inappropriate when the monitoring direction is not within angle θ1 (or when the state of being not within angle θ1 has lasted for a predetermined time or more). Figure 3 In the example, the first determination unit 142 determines that the driver D's monitoring direction is appropriate when the monitoring direction of the driver D is arrow A1, and determines that the monitoring direction of the driver D is inappropriate when the monitoring direction of the driver D is arrow A2.

[0075] The first determination unit 142 can also determine whether driver D is making a sideways glance, either in place of whether the driver D's monitoring direction is appropriate or based on this. In this case, the first determination unit 142... Figure 3 If the monitoring direction of driver D is arrow A1 when the monitored area AR1 is shown, it is determined that driver D is not looking to the side. If it is arrow A2, it is determined that driver D is looking to the side.

[0076] If the second determination unit 144 determines that the driver D is performing an action other than driving the vehicle M after successfully learning the model using the driver state data 194 from the feature information of the image captured by the in-vehicle camera 70 based on the recognition result of the state recognition unit 126, the second determination unit 144 determines that the driver D is performing an action other than driving the vehicle M. If the second determination unit 144 fails to perform an action other than driving, the second determination unit 144 determines that the driver D is not performing an action other than driving the vehicle M.

[0077] The second determination unit 144 can determine whether a non-driving behavior has a low impact on surrounding surveillance based on the degree of influence on surrounding surveillance that corresponds to the non-driving behavior identified by the state recognition unit 126. For example, if the degree of influence is less than a threshold (first threshold), the second determination unit 144 determines that the behavior has a low impact on surrounding surveillance; if the degree of influence is greater than or equal to the first threshold, it determines that the behavior does not have a low impact on surrounding surveillance. The determination process described above by the second determination unit 144 can also be performed only when the first determination unit 142 determines that the driver D's monitoring direction is appropriate. This reduces the load on the determination process.

[0078] [Reporting Control]

[0079] Next, the content of the report control in the embodiment will be explained. The HMI control unit 160 controls the reporting to the driver D based on the determination results determined by the first determination unit 142 and the second determination unit 144. For example, as a reporting condition, if the first determination unit 142 determines that the driver D's monitoring direction is inappropriate, the HMI control unit 160 generates report information related to looking around (first report information) regardless of the determination result determined by the second determination unit 144, and outputs the generated first report information to the driver D by the HMI 30. The first report information may be information indicating that the driver D's monitoring direction is inappropriate (or that he is looking around), or it may be information urging him to turn to the correct monitoring direction. The first report information may also include, for example... Figure 3 The image shown represents the monitoring direction of driver D, and the image represents the monitored area AR1. This allows driver D to more accurately determine if the monitoring direction is inappropriate and which direction should be monitored. The first report information can be an image, an audio message (e.g., a pre-determined report tone), or both. Thus, in cases where driver D's monitoring direction is inappropriate, a reliable report is given with safety as a priority, enabling a more appropriate report to the driver.

[0080] Even if the first determination unit 142 determines that the driver D's monitoring direction is appropriate, the HMI control unit 160 can still generate a report information related to surrounding monitoring (second report information) when the second determination unit 144 determines that the driver D is engaging in activities other than driving the vehicle M. The HMI control unit 160 can then output the generated second report information to the driver D. The second report information may indicate that there is a possibility that surrounding monitoring cannot be adequately performed due to activities other than driving, or it may be information urging the driver to stop engaging in activities other than driving. The second report information may also include information related to activities other than driving identified from the image of the in-vehicle camera 70. The second report information may be an image, an audio message (e.g., a report tone), or both. By reporting the second report information, the driver D can be notified, for example, that they need to focus on surrounding monitoring. The HMI control unit 160 may also report first report information related to side-viewing instead of the second report information.

[0081] When the HMI control unit 160 determines, by the first determination unit 142, that the monitoring direction of driver D is appropriate, and even if the second determination unit 144 determines, that driver D is engaging in behavior other than driving vehicle M, but that such behavior has an impact on surrounding surveillance that is less than a first threshold (e.g., less than 5), the HMI control unit 160 suppresses the reporting of the second report information. Suppressing the report can mean not reporting, or, if the second report information is an image or sound, outputting either one, or reducing the volume or shortening the image display time. This allows for the suppression of excessive reporting of behaviors that have little impact on surrounding surveillance.

[0082] The first threshold can be a fixed value or adjusted for each driver D. The first threshold can also be adjusted based on the surrounding conditions of vehicle M. For example, if vehicle M is traveling on a straight road and there are no other vehicles or objects nearby, the first threshold can be larger than a baseline value; if vehicle M is traveling on a curved road or other non-straight road, or if there are other vehicles or objects nearby, the first threshold can be smaller than the baseline value. This allows for more appropriate reporting based on each driver and the surrounding conditions.

[0083] Actions that have less impact on surrounding surveillance than a first threshold could be, for example, actions that are pre-planned to end within a specified time (the first action). If such actions end in a short time, the impact on surrounding surveillance is less, thus suppressing reports in the case of such first actions can suppress excessive reporting against driver D and allow for more appropriate reporting.

[0084] Actions that have less impact on surrounding surveillance than the first threshold can also include actions other than (or alternatives to) the first action, such as actions that driver D could perform with one hand (the second action). For example, even if one hand is holding an object (e.g., a drink or food) for an action other than driving, the other hand is holding the steering wheel SW. Therefore, even if surrounding surveillance is temporarily neglected, proper driving can be performed. Thus, suppressing reports in such situations can suppress excessive reporting against driver D and make more appropriate reports.

[0085] Actions with an impact on surrounding surveillance less than the first threshold can also be based on (or replace) the first or second actions described above, for example, an action that driver D can perform without looking at the action (third action). For example, a driver can remove jewelry, masks, or other accessories without looking at the action. Therefore, in the case of the third action, driver D's line of sight can be within the monitored area AR1, thus suppressing reporting and preventing excessive reporting of driver D, resulting in more appropriate reporting.

[0086] Behaviors with an impact on surrounding surveillance less than the first threshold can also include driver D's actions of removing or putting on wearable items, or driver D's eating-related behaviors. In this case, the two behaviors mentioned above can be adjusted so that the impact on surrounding surveillance contained in driver state data 194 is less than the first threshold. In this way, reporting can be suppressed in cases of easily removable accessories, or simple eating such as drinks or chewing gum, thereby suppressing excessive reporting.

[0087] Figure 4 This diagram illustrates an example of a scenario where reporting of second-report information is suppressed. Figure 4 In the example, the image captured by the in-vehicle camera 70 shows the situation of driver D. Figure 4 In the example, driver D holds the steering wheel SW with one hand (single-handed) and holds an object (item) OB1, such as a beverage container, with the other hand. In this scenario, driver D does not need to continuously look at object OB1, but can perform actions other than driving while keeping the monitoring direction within the monitored area AR1. The act of drinking the beverage is assumed to last for several seconds. Therefore, in such a scenario, the HMI control unit 160 can suppress excessive reporting by suppressing the reporting of second reporting information.

[0088] Figure 5 This diagram illustrates an example of a scenario where the reporting of second-report information is not suppressed. Figure 5 In the example, with Figure 4The difference between this example and the previous one is that the driver is holding an object (terminal device) OB2, such as a smartphone, which can display images and other content, instead of an object like a beverage container (OB1). In this scenario, it is highly likely that the driver's gaze will be continuously fixed on the screen of object OB2, and the monitoring direction will not be directed towards the monitored area AR1. The audiovisual activities, such as viewing images and other content, are assumed to be ongoing for an extended period. Therefore, in this scenario, the HMI control unit 160 can provide a more appropriate report to the driver by reporting a second report (without suppressing the report).

[0089] The HMI control unit 160 may also report if the amount of change or the speed of change in the monitoring direction of driver D identified by the status recognition unit 126 within a specified time is above a threshold (second threshold), regardless of the determination results determined by the first determination unit 142 and the second determination unit 144. Reporting regardless of the determination results determined by the first determination unit 142 and the second determination unit 144 includes, for example, reporting even if the first determination unit 142 determines that the monitoring direction of driver D is appropriate and the second determination unit 144 determines that driver D is not performing any behavior other than driving vehicle M, or reporting even if the first determination unit 142 determines that the monitoring direction of driver D is appropriate and determines that although driver D is performing any behavior other than driving, the impact on surrounding monitoring is less than the first threshold.

[0090] Figure 6 This is a diagram used to illustrate the change in the monitoring direction of driver D. In Figure 6 The example shows the driver D's monitoring direction moving from A3 to A4 within a specified time. The change in the HMI control unit 160 over the specified time ( Figure 6 A report is made when the change in angle (Δθ) is greater than or equal to the second threshold corresponding to the change, or when the velocity HV during the change of the monitoring direction is greater than or equal to the second threshold corresponding to the velocity. In this case, the monitoring directions A3 and A4 can be within or outside the monitored area AR1. Therefore, even if the monitoring direction is appropriate, but the movement Δθ or the velocity HV during the change is large, leading to a prediction that the focus cannot be on surrounding monitoring, a report can be made reliably with priority given to safety considerations. Figure 6 The movement Δθ and velocity HV shown in the monitoring direction indicate the movement and velocity in the horizontal direction (on the XY plane), but in the implementation, it can also be the movement and velocity in three dimensions (XYZ axes).

[0091] The second threshold can be changed based on the condition of vehicle M (e.g., lane change, left / right turn, collision avoidance). For example, when lane changes, left / right turns, collision avoidance, etc., are being performed through driving control by the driving control unit 180 or manual driving by the driver D, it is necessary to monitor not only the direction of travel of vehicle M but also the surrounding area in the direction of travel of vehicle M. Therefore, the amount of change in the direction monitored by driver D and the speed of the change are greater than usual. Therefore, by increasing the second threshold compared to normal conditions when the vehicle is driving in such situations, redundant reports related to side-view and surrounding area monitoring during lane changes, left / right turns, and collision avoidance can be suppressed.

[0092] If the first determination unit 142 determines that the driver D's monitoring direction is inappropriate, and the second determination unit 144 determines that the driver D is engaging in activities other than driving the vehicle M, the HMI control unit 160 can output both the first and second report information, or output the report information of the one with the higher preset priority. If the first determination unit 142 determines that the driver D's monitoring direction is appropriate, and the second determination unit 144 determines that the driver D is not engaging in activities other than driving the vehicle M, the HMI control unit 160 terminates the reporting.

[0093] [Processing Flow]

[0094] The following describes the processing performed by the driving support device 100 according to the embodiment. The processing performed by the driving support device 100 will primarily focus on the reporting process based on the monitoring direction and behavior of the driver D.

[0095] Figure 7 This is a flowchart illustrating an example of the processing performed by the driving support device 100 in the embodiment. Figure 7 In the example, the monitoring direction recognition unit 124 identifies the monitoring direction of the driver D of vehicle M (step S100). Next, the state recognition unit 126 identifies the driver D's behavior other than driving (step S110). Next, the first determination unit 142 determines whether the driver D's monitoring direction is an appropriate monitoring direction (step S120). If the monitoring direction is determined to be appropriate, the second determination unit 144 determines whether the driver D is performing behavior other than driving (step S130).

[0096] If it is determined that driver D is engaging in behavior other than driving, the second determination unit 144 determines whether the impact on surrounding monitoring is less than a first threshold (step S140). If it is determined that the impact on surrounding monitoring is less than the first threshold, the HMI control unit 160 determines whether the amount of change in the monitoring direction or the speed of the change is greater than or equal to a second threshold (step S150). If it is determined that it is not greater than or equal to the second threshold (but less than the second threshold), the HMI control unit 160 suppresses the reporting to driver D (step S160).

[0097] If, in step S120, it is determined that the monitoring direction is inappropriate; in step S140, it is determined that the impact on surrounding monitoring is not less than the first threshold (but greater than the first threshold); or in step S150, it is determined that the change in monitoring direction or speed is greater than the second threshold, the HMI control unit 160 generates corresponding report information (e.g., first report information, second report information) and outputs the report information to the HMI 30, thereby reporting to the driver D (step S170). The process of this flowchart ends here. If, in step S130, it is determined that the driver D is not performing any actions other than driving, the process of this flowchart ends here.

[0098] [Variation Example]

[0099] In this implementation, the HMI control unit 160 may also generate information related to driver D's behavior other than driving, as identified by the state recognition unit 126, and output this information to the HMI 30. The HMI control unit 160 may also generate information to ask driver D whether the identified behavior other than driving is correct, output this information to the HMI 30, and update the driver state data 194 based on the result of the question (driver D's response). This allows for more accurate state recognition for each driver D.

[0100] The HMI control unit 160 can also suppress the aforementioned report when the driver D's monitoring direction is within the monitored area AR1, and then release the suppression and report when the driver D's monitoring direction moves outside the monitored area AR1. This allows the report control to be immediately activated (restored).

[0101] In this embodiment, the driving control unit 180 may continue driving control even when the first determination unit 142 determines that the driver D's monitoring direction is appropriate, and the second determination unit 144 determines that the driver D is performing an action other than driving the vehicle M, but the non-driving action has an impact on the surrounding monitoring that is less than a first threshold. Therefore, driving control can continue even with non-driving actions that have a low impact on the surrounding monitoring, thus suppressing changes in control and achieving stable driving control.

[0102] If, during the execution of driving control, the driving control unit 180 determines, by the first determination unit 142, that the driver D's monitoring direction is inappropriate (or if the inappropriateness has persisted for a predetermined time or more since the determination), it terminates driving control. The driving control unit 180 may also terminate driving control if, during the execution of driving control, the second determination unit 144 determines that the driver D is engaging in an activity other than driving the vehicle M, and that this activity has an impact on surrounding monitoring that exceeds a first threshold (or if the aforementioned state has persisted for a predetermined time or more). In this case, the HMI control unit 160 may also cause the HMI 30 to output information indicating the termination of the driving control operation, along with either a first report or a second report, to the driver D.

[0103] According to the embodiments described above, the reporting control device includes: an identification unit 120 that identifies the monitoring direction and state of the driver of the vehicle M (an example of a moving body); a first determination unit 142 that determines whether the driver's monitoring direction is appropriate based on the identification result of the identification unit 120; a second determination unit 144 that determines whether the driver is engaging in any behavior other than driving the vehicle M based on the identification result; and a reporting control unit (HMI 30, HMI control unit 160) that controls the reporting to the driver D based on the determination result determined by the first determination unit 142 and the determination result determined by the second determination unit 144. The reporting control unit suppresses the reporting when the first determination unit 142 determines that the driver's monitoring direction is appropriate, and although the second determination unit 144 determines that the driver is engaging in any behavior other than driving the vehicle M, the non-driving behavior has an impact on the surrounding monitoring that is less than a first threshold. This allows for more appropriate reporting based on the driver's state.

[0104] For example, according to the implementation method, no report is made when a driver is performing an action other than driving (work) but the monitoring direction is appropriate and the action other than driving has little impact on the surrounding monitoring. This can suppress excessive reporting, and as a result, more appropriate reports can be made based on the driver's condition.

[0105] According to the implementation method, reporting is suppressed in cases of short-duration actions or actions that can be performed with one hand (e.g., removing or putting on jewelry, eating, etc.), thereby suppressing excessive reporting. According to the implementation method, reporting is made when it is determined that the monitoring direction is inappropriate, the amount of change in the monitoring direction, or the speed is above a second threshold, thereby enabling more appropriate reporting based on the driver's state.

[0106] The implementation methods described above can be performed as follows.

[0107] A reporting control device, wherein,

[0108] The report control device includes:

[0109] A storage medium that stores computer-readable instructions; and

[0110] The processor connected to the storage medium,

[0111] The processor performs the following processing by executing computer-readable instructions:

[0112] Identify the driver's monitoring direction and the driver's state;

[0113] The appropriateness of the driver's monitoring direction is determined based on the identification results.

[0114] Based on the identification result, it is determined whether the driver is engaging in any behavior other than driving the moving body;

[0115] Controlling reports to the driver based on the determination result; and

[0116] If it is determined that the driver's monitoring direction is appropriate, and although it is determined that the driver is engaging in behavior other than driving the vehicle, the non-driving behavior has an impact on the surrounding monitoring of less than a first threshold, the report is suppressed.

[0117] The above describes specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way. Various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A reporting control device, wherein, The reporting control device includes: The identification unit identifies the monitoring direction of the driver of the moving body and the state of the driver; The first determination unit determines whether the driver's monitoring direction is appropriate based on the recognition result of the recognition unit; The second determination unit determines, based on the identification result, whether the driver is engaging in any behavior other than driving the mobile body; as well as The reporting control unit controls the reporting to the driver based on the determination results made by the first determination unit and the determination results made by the second determination unit. The reporting control unit suppresses the report when the first determination unit determines that the driver's monitoring direction is appropriate, and although the second determination unit determines that the driver is performing an action other than driving the vehicle, the action other than driving has an impact on the surrounding monitoring of less than a first threshold.

2. The reporting control device according to claim 1, wherein, Actions whose impact on the surrounding surveillance is less than a first threshold include actions that are pre-planned to end within a specified time.

3. The reporting control device according to claim 1, wherein, Actions that have less than a first threshold impact on the surrounding surveillance include actions that the driver could perform with one hand.

4. The reporting control device according to claim 1, wherein, Actions that have less impact on the surrounding surveillance than a first threshold include the driver removing or putting on clothing items.

5. The reporting control device according to claim 1, wherein, Behaviors that have less than a first threshold impact on the surrounding surveillance include the driver's diet-related behaviors.

6. The reporting control device according to claim 1, wherein, If the first determination unit determines that the driver's monitoring direction is inappropriate, the reporting control unit will issue the report regardless of the determination result of the second determination unit.

7. The reporting control device according to claim 1, wherein, The identification unit identifies the amount of change or the speed at which the driver's monitoring direction changes within a specified time. When the change or speed exceeds a second threshold, the reporting control unit shall report regardless of the determination results made by the first determination unit and the second determination unit.

8. A reporting control method, wherein, The reporting control method causes the computer to perform the following processing: Identify the driver's monitoring direction and the driver's state; The appropriateness of the driver's monitoring direction is determined based on the identification results. Based on the identification result, it is determined whether the driver is engaging in any behavior other than driving the moving body; Based on the determination results, the reporting to the driver is controlled; as well as If it is determined that the driver's monitoring direction is appropriate, and although it is determined that the driver is engaging in behavior other than driving the vehicle, the non-driving behavior has an impact on the surrounding monitoring that is less than a first threshold, the report is suppressed.

9. A storage medium storing a program, wherein, The program causes the computer to perform the following processes: Identify the driver's monitoring direction and the driver's state; The appropriateness of the driver's monitoring direction is determined based on the identification results. Based on the identification result, it is determined whether the driver is engaging in any behavior other than driving the moving body; The reporting to the driver is controlled based on the results of the determination; as well as If it is determined that the driver's monitoring direction is appropriate, and although it is determined that the driver is engaging in behavior other than driving the vehicle, the non-driving behavior has an impact on the surrounding monitoring that is less than a first threshold, the report is suppressed.

Images