Information processing apparatus, information processing system, information processing method, and storage medium

Abstract

An information processing device, an information processing system, an information processing method, and a storage medium capable of detecting an abnormality of a driver are provided. The information processing device includes an acquisition unit that acquires a face image that is an image including a face of an occupant of a vehicle; a fellow passenger determination unit that determines whether a fellow passenger of the vehicle is present based on the face image; a driving ability derivation unit that derives driving evaluation information that is information related to a driving ability of a driver of the vehicle; an abnormality determination unit that, in a case where it is determined that the fellow passenger is present, determines whether the driver has an abnormality based on a comparison result between the derived driving evaluation information and reference driving evaluation information; and a notification control unit that causes a notification unit to perform notification in a case where it is determined that the driver has the abnormality.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and storage medium. Background Technology

[0002] In recent years, efforts have intensified to provide sustainable transportation systems that also consider vulnerable groups among transportation participants, particularly the elderly, people with disabilities, and children. Research and development, especially related to the development of mobility facilities for the elderly and people with disabilities, are particularly focused on further improving the safety and convenience of transportation. (Japanese Patent Application Publication No. 2020-71528, Japanese Patent Application Publication No. 2017-16568) Summary of the Invention

[0003] However, drivers whose cognitive abilities have declined due to factors such as aging may engage in dangerous driving behaviors, such as failing to see potential risks. To ensure traffic safety, it is important to monitor drivers' cognitive abilities (driving skills) daily and detect and notify them of any decline in cognitive ability at an early stage.

[0004] This invention was made in consideration of such circumstances, and one of its objectives is to provide an information processing device, information processing system, information processing method, and storage medium capable of detecting abnormalities in drivers, thereby contributing to the development of sustainable transportation systems.

[0005] [Methods used to solve problems]

[0006] The information processing device, information processing system, information processing method, and storage medium involved in this invention adopt the following structure.

[0007] (1): One aspect of the present invention relates to an information processing apparatus, wherein the information processing apparatus comprises: an acquisition unit that acquires a facial image, the facial image being an image containing the face of a vehicle occupant; a passenger determination unit that determines whether a passenger of the vehicle exists based on the facial image; a driving ability derivation unit that derives driving evaluation information, the driving evaluation information being information relating to the driving ability of the driver of the vehicle; an anomaly determination unit that, when it is determined that a passenger exists, determines whether the driver has experienced an anomaly based on a comparison result between the derived driving evaluation information and baseline driving evaluation information; and a notification control unit that, when it is determined that the driver has experienced an anomaly, causes the notification unit to issue a notification.

[0008] (2): In the above (1) scheme, the driving evaluation information is information representing the degree of danger of driving. When the abnormality determination unit determines that the passenger is present and the derived driving evaluation information shows that the degree of danger is higher compared with the benchmark driving evaluation information, it determines that the driver has an abnormality.

[0009] (3): In the above scheme (1) or (2), the driving evaluation information is information representing the degree of danger of driving. When the anomaly determination unit determines that there is no passenger and the derived driving evaluation information shows a higher degree of danger compared with the benchmark driving evaluation information, the anomaly determination unit determines that the driver has not caused any abnormality.

[0010] (4): In any of the above schemes (1) to (3), the benchmark driving evaluation information is derived from a single driving that serves as the benchmark.

[0011] (5): In any of the above schemes (1) to (4), the benchmark driving evaluation information is information set based on the driver's driving.

[0012] (6): In any of the above schemes (1) to (4), the benchmark driving evaluation information is derived from driving under the condition that the passenger is not present.

[0013] (7): In any of the above schemes (1) to (4), the benchmark driving evaluation information is derived from driving based on the determination that the passenger is present.

[0014] (8): In any of the above schemes (1) to (4), the benchmark driving evaluation information is derived from a specified number of driving sessions.

[0015] (9): In any of the above schemes (1) to (8), the acquisition unit acquires sound information inside the vehicle, and the information processing device further includes a driving instruction determination unit. When the passenger is present, the driving instruction determination unit determines whether the driver has received a driving instruction based on the sound information and the facial image. The driving instruction is an instruction related to the driving operation of the vehicle. When the abnormality determination unit determines that the driving instruction has been received, it does not determine whether the driver has experienced an abnormality.

[0016] (10): In the above-mentioned (9) solution, the information processing device further includes a storage control unit, which stores the first driving evaluation information, the second driving evaluation information and the third driving evaluation information in the storage unit respectively. The first driving evaluation information is the driving evaluation information when the passenger is not present, the second driving evaluation information is the driving evaluation information when the driving instruction is received, and the third driving evaluation information is the driving evaluation information when the driving instruction is not received.

[0017] (11): In the above (10) scheme, the information processing device further includes a comparison unit that compares the first driving evaluation information, the second driving evaluation information and the third driving evaluation information, and the notification control unit causes the notification unit to output information relating to the comparison result compared by the comparison unit.

[0018] (12): In any of the above schemes (1) to (10), the acquisition unit acquires the surrounding image of the vehicle, and the information processing device further comprises: a risk detection unit that detects risky objects in the surrounding image; and a miss determination unit that determines whether the driver has missed the risky object based on information relating to the location of the risky object and information relating to the facial image, and the driving ability derivation unit derives the driving evaluation information based on the determination result of the miss determination unit.

[0019] (13): In any of the above schemes (1) to (11), the driving ability derivation unit derives the driving evaluation information based on some or all of the information relating to the speed of the vehicle and the information relating to the direction of travel of the vehicle.

[0020] (14): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing apparatus of any of the above (1) to (13); a first terminal device for use by the driver and a notification unit not controlled by the notification control unit; and a second terminal device for use by a user different from the driver and a notification unit controlled by the notification control unit, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

[0021] (15): One aspect of the present invention relates to an information processing method, wherein the information processing method causes a computer to perform the following processing: acquiring a facial image, the facial image being an image containing the face of a vehicle occupant; determining, based on the facial image, whether a passenger in the vehicle exists; deriving driving evaluation information, the driving evaluation information being information relating to the driving ability of the driver of the vehicle; if it is determined that a passenger exists, determining, based on a comparison result between the derived driving evaluation information and benchmark driving evaluation information, whether the driver has acted abnormally; and if it is determined that the driver has acted abnormally, causing a notification unit to issue a notification.

[0022] (16): One aspect of the present invention relates to a storage medium storing a program, wherein the program causes a computer to perform the following processing: acquiring a facial image, the facial image being an image containing the face of a vehicle occupant; determining, based on the facial image, whether a passenger in the vehicle is present; deriving driving evaluation information, the driving evaluation information being information relating to the driving ability of the driver of the vehicle; if it is determined that a passenger is present, determining, based on a comparison result between the derived driving evaluation information and baseline driving evaluation information, whether the driver has acted abnormally; and if it is determined that the driver has acted abnormally, causing a notification unit to issue a notification.

[0023] [Invention Effects]

[0024] According to (1)-(16), the above-described scheme of the present invention can detect the reduction in the driver's cognitive ability. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating an example of the structure and usage environment of a driving monitoring device that includes the information processing apparatus according to the first embodiment.

[0026] Figure 2 This is a diagram illustrating an example of the driving monitoring device according to the first embodiment.

[0027] Figure 3 This is a diagram showing an example of an image obtained by the driving monitoring device according to the first embodiment.

[0028] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device according to the first embodiment.

[0029] Figure 5 This is a diagram illustrating an example of information detected by the detection unit according to the first embodiment.

[0030] Figure 6This diagram illustrates an example of an alarm triggered by the alarm unit according to the first embodiment.

[0031] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device according to the first embodiment.

[0032] Figure 8 This is a diagram illustrating an example of driving evaluation information involved in the first embodiment.

[0033] Figure 9 This is a diagram illustrating an example of a comparison between the baseline driving evaluation information and the driving evaluation information involved in the first embodiment.

[0034] Figure 10 This is a diagram illustrating another example of a comparison between the baseline driving evaluation information and the driving evaluation information involved in the first embodiment.

[0035] Figure 11 This is a flowchart illustrating an example of the processing flow performed by the exception determination unit according to the first embodiment.

[0036] Figure 12 This is a diagram illustrating an example of the structure of a driving monitoring device that includes the information processing apparatus according to the second embodiment.

[0037] Figure 13 This is a diagram illustrating an example of driving capability information involved in the second embodiment.

[0038] Figure 14 This diagram illustrates an example of a notification made by the first terminal device (notification unit) according to the second embodiment.

[0039] Figure 15 This diagram illustrates the exception notification processing in the information processing system according to the second embodiment. Detailed Implementation

[0040] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the rearward direction of the vehicle is defined as the negative X direction, the rightward direction of the vehicle's width relative to the positive X direction is defined as the positive Y direction, the leftward direction is defined as the negative Y direction, and the direction orthogonal to the X and Y directions and the height direction of the vehicle is defined as the positive Z direction.

[0041] <First Implementation>

[0042] [Overall Structure]

[0043] Figure 1This diagram illustrates an example of the structure and usage environment of a driver monitoring device 1, including the information processing device 2 according to the first embodiment. The information processing device 2 is, for example, a driver monitoring device 1 retrofitted into a vehicle (hereinafter referred to as "vehicle M") such as a dashcam. That is, the driver monitoring device 1 (information processing device 2) can be mounted and removed from the vehicle M via a mounting / unmounting unit (hereinafter referred to as "mounting / unmounting unit 18"). The information processing device 2 is, for example, a device not connected to the vehicle M's in-vehicle network. An in-vehicle network refers to an in-vehicle network that connects to the vehicle's control-related ECU (Electronic Control Unit), which uses communication standards such as CAN (Controller Area Network), and is connected to the vehicle M's drive, braking, steering, various vehicle sensors, driving-related operating components, and operation buttons. The vehicle M's network for HMI (Human Machine Interface) can also be excluded from the above-described in-vehicle network.

[0044] Figure 2 This diagram illustrates an example of a driver monitoring device 1. The driver monitoring device 1 is, for example, installed near the windshield of a vehicle M, and located near the front of the driver's seat. The driver monitoring device 1 stores images (moving images) captured by the camera unit 10, for example, in the storage unit 70 (described later). The driver monitoring device 1 may also be, for example, equipped with a display device such as an LCD (Liquid Crystal Display) and capable of displaying the images (moving images) stored in the storage unit 70 on the display device.

[0045] The driver monitoring device 1 includes, for example, a camera unit 10, a microphone 16, a loading and unloading unit 18, and an information processing unit 2. In the illustrated example, the camera unit 10 includes a front camera 12 and a rear camera 14.

[0046] The front camera 12 and the rear camera 14 are each, for example, digital cameras utilizing solid-state imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) captures images of the area in front of the vehicle M from the location where the driver monitoring device 1 is installed. The rear camera 14 repeatedly (periodically) captures images of the area behind the vehicle M and the interior of the vehicle M from the location where the driver monitoring device 1 is installed. The horizontal viewing angle captured by the front camera 12 and the rear camera 14 is, for example, 180° or more. That is, the driver monitoring device 1 (camera unit 10) captures a 360° range around the vehicle M from its installation location. The front camera 12 and the rear camera 14 may each be, for example, fisheye cameras including fisheye lenses. Alternatively, the front camera 12 and the rear camera 14 may each be wide-angle cameras or the like. In this embodiment, the use of the front camera 12 and the rear camera 14 is described, but other cameras may be used instead of these or other alternatives. That is, the camera unit 10 can be configured to capture images used in this embodiment (peripheral images and facial images described later).

[0047] Microphone 16 repeatedly (periodically) receives sound from inside the vehicle M. Microphone 16 can be located anywhere on the driver monitoring device 1. For example, microphone 16 can be located next to rear camera 14. One or more microphones 16 can be installed anywhere on the driver monitoring device 1. Microphone 16 is configured to receive sound (e.g., voice) from inside the vehicle. The sound received by microphone 16 is referred to as "sound information".

[0048] The loading / unloading part 18 is, for example, a component for mounting the driver monitoring device 1 onto the vehicle M. The loading / unloading part 18 can be, for example, any supporting component such as a suction cup, a seal, or a bracket. The loading / unloading part 18 may also include a power supply cable (wiring, etc.) for supplying power to the driver monitoring device 1.

[0049] Figure 3 This is a diagram illustrating an example of an image acquired by the driver monitoring device 1. That is, Figure 3 This diagram illustrates an example of an image captured by the camera unit 10. The front camera 12 captures images of the front of the vehicle M through the windshield, the driver's side side window, and the passenger side side window. The rear camera 14 captures images of the interior of the vehicle M (including the driver and passengers) and the rear of the vehicle M through the driver's side side window, the passenger side side window, and the rear window.

[0050] In the image captured by the front camera 12 (hereinafter referred to as the "front image"), the subject includes objects such as other vehicles, pedestrians, bicycles, fixed objects, road markings, etc., that can be seen through the front windshield and exist in front of the vehicle M, as well as objects that can be seen through the left and right (front) side windows of the vehicle M.

[0051] The image captured by the rear camera 14 (hereinafter referred to as the "rear image") shows the interior of the vehicle M, as well as objects visible through the rear window that are located behind the vehicle M, and objects visible through the left and right side windows that are located to the left and right (rear) of the vehicle M. Therefore, in the rear image captured by the rear camera 14, the driver of the vehicle M equipped with the driver monitoring device 1 is also reflected as the subject.

[0052] The front and rear images include images of the surrounding conditions of the vehicle M (hereinafter referred to as "surrounding images"). The rear image includes an image of the face of the occupant, including the driver of the vehicle M (hereinafter referred to as "facial image"). The driver monitoring device 1 can be installed at any location where the camera unit 10 can capture the surrounding images and the facial image.

[0053] Information processing device 2 (refer to) Figure 1 Based on the results obtained by referring to the surrounding images and facial images output by the camera unit 10, the alarm and notification are output via the first terminal device T1. Sometimes the driver monitoring device 1 and the first terminal device T1 are combined and referred to as "information processing system 100".

[0054] The first terminal device T1 is, for example, a mobile terminal device used by a driver driving a vehicle M equipped with a driver monitoring device 1, such as a smartphone or tablet. The first terminal device T1 executes, for example, an application for receiving alarms and notifications from the information processing device 2. The application causes a display device to show an image based on information sent by the information processing device 2 (e.g., alarm information and notification information described later), and causes a speaker to emit sound. The first terminal device T1 is also referred to as the alarm unit 82 and the notification unit 84. The first terminal device T1 is used, for example, in a state where it can be detachably mounted on the vehicle M. For example, a bracket for the first terminal device T1 with a mounting and detaching mechanism is provided on one or both of the first terminal device T1 and the vehicle M, and the first terminal device T1 is supported by the bracket.

[0055] Alternatively, the navigation, display, and speaker of vehicle M can be used instead of the first terminal device T1. That is, the navigation, display, and speaker of vehicle M can also output alarms and notifications based on the instructions of information processing device 2. That is, the navigation, display, and speaker of vehicle M can also be alarm unit 82 and notification unit 84. Alternatively, the display unit (alarm unit 82, notification unit 84) and speaker (alarm unit 82, notification unit 84) can be provided on the driver monitoring device 1 instead of the first terminal device T1.

[0056] like Figure 1 As shown, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, and a face orientation detection unit 36. The processing unit 40 includes a missed detection unit 41, a passenger detection unit 42, a driving instruction detection unit 43, a vehicle status recognition unit 44, a driving ability derivation unit 45, and an anomaly detection unit 46. The control unit 50 includes an alarm control unit 52 and a notification control unit 54.

[0057] The acquisition unit 20, detection unit 30, processing unit 40, and control unit 50 are equipped with hardware processors such as CPUs (Central Processing Units) and storage devices (storage devices with non-transitory storage media) storing programs (software). The processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented using hardware such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units) (including the circuitry). The functions of each component can also be realized through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs.

[0058] The program (software) can be pre-stored in the storage unit 70 (a storage device with a non-temporary storage medium) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it can be stored in a removable storage medium (a non-temporary storage medium) such as a memory card, and installed in the storage device by assembling the storage medium into the driving monitoring device 1. The program (software) can also be pre-downloaded from other computer devices via short-range communication or wide-area communication using an application program executed in the first terminal device T1, and then sent from the first terminal device T1, thereby being installed in the storage device.

[0059] The storage unit 70 stores, for example, driving capability information 72, programs (such as the programs described above), and various other information. The storage unit 70 can also be implemented using the various storage devices described above or EEPROM (Electrically Erasable Programmable Read Only Memory). The communication unit 60 is an interface for wireless or wired communication with information processing devices (external devices) such as the first terminal device T1.

[0060] The driver monitoring device 1 (information processing device 2) performs at least two functions: a missed detection alarm and an anomaly notification. The missed detection alarm is a process that detects a driver's missed detection of a risky object based on surrounding images and facial images, and triggers an alarm at the alarm unit 82. The anomaly notification is a process that detects an anomaly (reduction in cognitive ability) in the driver based on surrounding images and facial images, and triggers a notification at the notification unit 84.

[0061] A risky object refers to an object, for example, that vehicle M should avoid. A risky object is, for example, an object that would interfere with vehicle M when it is moving as it is, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose approach level becomes more than a certain level after a specified time, taking into account the object's position, direction of movement, and speed, as well as the position, method of movement, and speed of vehicle M. The object can also be a traffic participant such as a pedestrian, bicycle, or vehicle, or an object other than a traffic participant such as an object placed on the road or a fallen object.

[0062] [Handling of Missing Alarms]

[0063] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device 1 (information processing device 2).

[0064] The acquisition unit 20 acquires front and rear images from the imaging unit 10. Therefore, the acquisition unit 20 establishes a link between the peripheral images and the facial images to acquire these images. The imaging unit 10 repeatedly (periodically) captures the peripheral images and facial images, thus the acquisition unit 20 repeatedly (periodically) acquires these images. The acquisition unit 20 can also establish a link between the peripheral images and facial images and the time they were captured (the shooting time) to acquire these images.

[0065] The risk detection unit 32 detects risky objects in the surrounding image. "Detecting risky objects" refers to, for example, detecting the presence of a risky object and the position of that object relative to the vehicle M (driver monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M can be pre-stored in the storage unit 70. The risk detection unit 32 can also detect the position of the risky object relative to the vehicle M based on this reference information.

[0066] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, and also detects the direction of the risk object's presence (approach direction) relative to the vehicle M. "Direction of presence of the risk object" refers to, for example, the direction of presence of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, "direction of presence of the risk object" will sometimes be referred to as "risk presence direction".

[0067] The gaze detection unit 34 detects the driver's gaze in a facial image. Specifically, the gaze detection unit 34 detects the driver's gaze by analyzing the facial image using a prescribed image analysis method. "Gaze detection" refers to, for example, detecting the position of the driver's eyeballs and the direction of the gaze (the direction of the eyeballs).

[0068] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the facial image. Specifically, the face orientation detection unit 36 ​​detects the orientation of the driver's face by parsing the facial image using a prescribed image parsing method. The face orientation detection unit 36 ​​may also detect the position of the driver's face in addition to its orientation. Hereinafter, the information combining "face orientation" and "face position" is sometimes referred to as "facial information." The face orientation detection unit 36 ​​detects facial information in the facial image.

[0069] The detection unit 30 can also be configured to detect lanes, pedestrian crossings, etc., in addition to the aforementioned risk objects, lines of sight, and facial information. Figure 5 This is a diagram illustrating an example of information detected by the detection unit 30. In Figure 5In the example, the pedestrians, other vehicles, lanes, and crosswalks detected by the detection unit 30 are displayed based on the vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that may become risk objects. In the detection information, instead of the vehicle M, the direction of the line of sight, and the direction of the face, the traffic participants (objects) that may become risk objects detected by the detection unit 30 may be displayed based on the vehicle M, the direction of the vehicle M, and the direction of travel of the vehicle M.

[0070] Leak detection unit 41 (refer to) Figure 4 This method determines whether a driver has missed a hazard based on the location and / or direction of the hazard, line of sight, and facial information. "Risk location" and "direction of the hazard" are each examples of "information relating to the location of the hazard." Line of sight and facial information are examples of "information relating to facial images."

[0071] For example, reference information (not shown) indicating the relationship between the orientation of the driver's eyes (gaze) and / or the orientation of their face in the facial image and their relative orientation with respect to the direction of travel of the vehicle M can also be pre-stored in the storage unit 70. The omission determination unit 41 can also refer to this reference information to determine whether the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists. If the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists, the omission determination unit 41 can also determine that the driver visually identifies a risky object. The so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can be, for example, the case where there is a risky position on a straight line along the orientation of the gaze and / or the orientation of the face. Alternatively, the so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can also be the case where there is a risky position in a cone-shaped area with the position of the viewpoint (eyeball) and / or the face as the apex and having a central axis along the orientation of the gaze and / or the orientation of the face that expands laterally as it moves forward. If the direction of the driver's gaze and / or the direction of the face does not match the location of the risky object, the omission determination unit 41 may also determine that the driver has not visually identified the risky object.

[0072] The omission determination unit 41 can also determine that the driver has missed the risk object if the state of not visually recognizing the risk object continues for a predetermined time or longer. Alternatively, the omission determination unit 41 can determine that the driver has not missed the risk object if the driver visually recognizes the risk object, or if the state of not visually recognizing the risk object is resolved before the predetermined time has elapsed. However, the method by which the omission determination unit 41 determines omissions is not limited to the above and can be appropriately modified.

[0073] The omission detection unit 41 generates omission detection information based on the above detection results. The omission detection information includes, for example, the presence or absence of omissions and the direction of risk.

[0074] The alarm control unit 52 causes the first terminal device T1 (alarm unit 82) to output an alarm based on the leak detection information. For example, the alarm control unit 52 sends alarm information to the first terminal device T1 via the communication unit 60. The alarm information includes, for example, information indicating whether an alarm is needed and the direction of the risk. For example, if the leak detection information includes information indicating "a leak exists," the alarm control unit 52 generates alarm information including information indicating "an alarm is needed." For example, if the leak detection information includes information indicating "no leak exists," the alarm control unit 52 generates alarm information including information indicating "no alarm is needed." The first terminal device T1 outputs an alarm based on the alarm information.

[0075] Figure 6 This diagram illustrates an example of an alarm triggered by the first terminal device T1 (alarm unit 82). Figure 6 In the example, the first terminal device T1 displays identifiable entries indicating the direction of the risk (D1 and D2 in the diagram). Furthermore, the first terminal device T1 issues warnings by differentiating the display format of the risk direction when a risky object is missed (i.e., the warning message includes a message indicating "warning is needed") and when no risky object is missed (i.e., the warning message includes a message indicating "warning is not needed"). Specifically, when a risky object is missed, the risk direction is displayed in a more emphasized manner compared to when it is not missed. This emphasis could be based on color, flashing icons, or other visually illuminating elements to make the risk direction easier for the driver to recognize. The first terminal device T1 may also issue warnings using methods such as loudspeaker announcements or illuminated lights (A in the diagram), in addition to or instead of the above. The specific method of issuing the warning can be appropriately modified as long as it is identifiable by the driver.

[0076] The alarm issued by the first terminal device T1 (alarm unit 82) can also be issued during the time when the driver is driving the vehicle M (i.e., in real time). Examples of "the time when the driver is driving the vehicle M" include when the vehicle M is moving or when the vehicle M is temporarily stopped. For example, the alarm information sent by the alarm control unit 52 can also be issued at the aforementioned times. Alternatively, the alarm information sent by the alarm control unit 52 can be issued at any time, and the first terminal device T1 (alarm unit 82) controls the timing of the alarm to issue the alarm at the aforementioned times.

[0077] [Abnormal Notification Handling]

[0078] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device 1 (information processing device 2).

[0079] In addition to the aforementioned front and rear images, the acquisition unit 20 further acquires sound information from the microphone 16. Thus, the acquisition unit 20 establishes a link between the peripheral image, facial image, and sound information. The camera unit 10 repeatedly (periodically) captures peripheral images and facial images, and the microphone 16 similarly repeatedly (periodically) acquires sound information; therefore, the acquisition unit 20 repeatedly (periodically) acquires peripheral images, facial images, and sound information. The acquisition unit 20 can also establish a link between the peripheral images, facial images, and sound information and the time they were captured (the shooting time).

[0080] The passenger determination unit 42 determines whether a passenger of vehicle M exists based on a facial image. If the facial image contains a passenger of vehicle M, the passenger determination unit 42 determines that a passenger exists; if the facial image does not contain a passenger of vehicle M, it determines that a passenger does not exist. A passenger of vehicle M refers to an occupant other than the driver of vehicle M.

[0081] The driving instruction determination unit 43 determines whether the driver received driving instructions from a passenger based on voice information and facial image. Specifically, if the passenger determination unit 42 determines that a passenger is present, the driving instruction determination unit 43 determines whether the driver received driving instructions from the passenger based on voice information and facial image. Driving instructions refer to instructions relating to driving operations of the vehicle M. Driving instructions may be, for example, instructions indicating the presence of a dangerous object, or instructions specifying the speed and direction of travel of the vehicle M. Driving instructions simply require that the driver anticipates and performs steering operations or speed adjustments for the vehicle M upon receiving the instruction.

[0082] The driving instruction determination unit 43 can also detect the orientation of the passenger's face by analyzing facial images using a prescribed image analysis method. The driving instruction determination unit 43 can also detect the content of the passenger's voice by analyzing sound information using a prescribed sound analysis method. The driving instruction determination unit 43 can determine that a driving instruction has been received if the passenger's face is facing the driver and the content of the passenger's voice contains information related to driving instructions. The driving instruction determination unit 43 can also determine that a driving instruction has not been received if the passenger's face is not facing the driver or if the content of the passenger's voice does not contain information related to driving instructions.

[0083] The vehicle state recognition unit 44 identifies the vehicle state, which is the state of the vehicle M, based on surrounding images. Vehicle state refers to, for example, the amount of change in the vehicle M's speed or direction of travel. For example, reference information (not shown) indicating the relationship between the changes in the positions of objects in multiple surrounding images arranged sequentially in time series and the speed of the vehicle M, and reference information (not shown) indicating the relationship between the changes in the positions of objects in multiple surrounding images arranged sequentially in time series and the direction of travel (the steering state of the vehicle M), can also be pre-stored in the storage unit 70. The vehicle state recognition unit 44 can also identify the state of the vehicle M based on this reference information. By recognizing the vehicle state, the vehicle state recognition unit 44 can identify instances such as the driver suddenly turning the steering wheel, rapid acceleration, or sudden braking.

[0084] The information processing device 2 can also replace the vehicle state recognition unit 44, or, in addition to the vehicle state recognition unit 44, identify the changes in the speed and direction of travel of the vehicle M (vehicle state) based on information detected by the acceleration sensor, the speed information of the vehicle M, and the steering information. For example, the information processing device 2 can also acquire acceleration information detected by the acceleration sensor equipped on the driver monitoring device 1 through the acquisition unit 20, and the processing unit 40 can analyze the acquired acceleration information to identify the vehicle state. For example, the information processing device 2 can also acquire information related to the speed of the vehicle M and information related to steering from the vehicle M via the communication unit 60, and identify the vehicle state based on the acquired information.

[0085] Among the vehicle states identified by the information processing device 2 via the communication unit 60, there may be, for example, turning left or right without using the turn signal, speeding or slowing down relative to the speed limit, and high frequency of vehicle horn use. If the anomaly determination unit 46, described later, identifies these vehicle states, the assessment is that the danger level is high.

[0086] The driving ability export unit 45 exports driving evaluation information, which is information related to the driver's driving ability. Figure 8 This is a diagram illustrating an example of driving evaluation information. In Figure 8 In the chart, the vertical axis represents frequency, and the horizontal axis represents hazard level. Regarding the vertical axis, the upward direction is positive, and the higher the direction, the higher the frequency. Regarding the horizontal axis, the rightward direction is positive, and the further to the right, the higher the hazard level. Figure 8The system includes driving evaluation information L. Driving evaluation information L represents the correlation between the degree of danger and frequency of driving during a specified period. Regarding driving ability, for example, the more times the omission determination unit 41 determines that the driver has missed a risky object, or the greater the changes in speed or direction of travel of vehicle M detected by the vehicle state recognition unit 44, the higher the degree of danger is evaluated. Conversely, the fewer times the omission determination unit 41 determines that the driver has missed a risky object, or the smaller the changes in speed or direction of travel of vehicle M detected by the vehicle state recognition unit 44, the lower the degree of danger is evaluated. The degree of danger can be expressed numerically or using high, medium, and low levels.

[0087] While driving vehicle M, the driving ability derivation unit 45 periodically acquires hazard information indicating the degree of driving hazard based on the judgment result determined by the oversight judgment unit 41 and the vehicle's state. Based on the multiple hazard information acquired within a specified period, the driving ability derivation unit 45 counts the number of times each hazard occurs, calculates the frequency, and derives driving evaluation information L. That is, the frequency, for example, represents the number of times driving corresponding to each hazard level has been performed. The specified period is a pre-set period. The specified period can be, for example, a single driving session or multiple (specified number) driving sessions. A single driving session can, for example, represent driving from engine start-up to engine shutdown.

[0088] The anomaly determination unit 46 determines whether the driver has experienced an anomaly based on a comparison between the baseline driving evaluation information and the driving evaluation information. The baseline driving evaluation information is, for example, set based on past driving data performed by the driver of vehicle M. Alternatively, the baseline driving evaluation information may be set based on driving evaluation information derived from one or more past driving experiences that serve as a baseline. The baseline driving evaluation information may also be set based on the average driving ability of an able-bodied person.

[0089] Figure 9 This is a diagram illustrating an example of a comparison between baseline driving evaluation information and driving evaluation information. (And...) Figure 8 Similarly, in Figure 9 In the chart, the vertical axis represents frequency, and the horizontal axis represents hazard level. Figure 9The system includes baseline driving evaluation information L1 and driving evaluation information L2. The anomaly determination unit 46 calculates the mode difference D, which is the absolute value of the difference between the modes MO1 and MO2. The mode MO1 represents the hazard level corresponding to the mode of the baseline driving evaluation information L1, and the mode MO2 represents the hazard level corresponding to the mode of the driving evaluation information L2. For example, the anomaly determination unit 46 may determine that the driver has experienced an anomaly if the mode difference D is above a preset value (threshold). Alternatively, the anomaly determination unit 46 may determine that the driver has not experienced an anomaly if the mode difference D is below the preset value. The preset value can be preset or set based on the state of the driving path of the vehicle M detected by the detection unit 30, the surrounding environment, etc. For example, when the vehicle M is driving on a road with consecutive sharp curves, or in a foggy environment, the preset value can be set to a large value. This is because it is predicted that under such conditions, the change in the vehicle M's direction of travel will be greater, and the number of times a risky object is missed will increase.

[0090] Figure 10 These are other examples of comparisons between baseline driving evaluation information and driving evaluation information. (Compared to...) Figure 9 The explanation will focus on the differences between them. Figure 10 In this case, the mode difference D is less than a set value. The driving evaluation information L2 indicates that the driver is engaging in low-frequency but high-risk driving while driving, which is the source of the driving evaluation information L2. The anomaly determination unit 46 can also determine that the driver has experienced an anomaly if the driving evaluation information L2 contains a hazard level of V or higher. Alternatively, the anomaly determination unit 46 can determine that the driver has not experienced an anomaly if the driving evaluation information L2 does not contain a hazard level of V or higher. The hazard level V is a value set based on the baseline driving evaluation information L1. The hazard level V can be, for example, the highest hazard level contained in the baseline driving evaluation information L1. The hazard level V can also be, for example, a hazard level in the baseline driving evaluation information L1 where the hazard level is higher than the mode MO1 and the frequency is higher than a specified frequency.

[0091] However, the method by which the abnormality determination unit 46 determines whether the driver has experienced an abnormality is not limited to the above and can be appropriately modified.

[0092] The anomaly determination unit 46 generates anomaly determination information based on the determination result. The anomaly determination information includes, for example, whether the driver has experienced an anomaly, and driving evaluation information.

[0093] In this way, if the derived driving evaluation information indicates a higher level of danger compared to the baseline driving evaluation information, the anomaly determination unit 46 determines that the driver has committed an anomaly, thereby detecting a decrease in the driver's driving ability.

[0094] The anomaly determination unit 46 can also change the setting value based on the presence or absence of a passenger. The anomaly determination unit 46 can also decrease the setting value if it determines that a passenger is present. This is because the risk of driving with a passenger present tends to be lower than the risk of driving without a passenger, therefore, even if the mode difference D is small, the driver's cognitive ability is likely to be reduced. The anomaly determination unit 46 can also increase the setting value if it determines that a passenger is absent. This is because the risk of driving without a passenger tends to be higher than the risk of driving with a passenger, therefore, even if the mode difference D is large, the driver's cognitive ability is likely not reduced.

[0095] In this way, the setting value is set according to whether there is a passenger. Thus, even if the derived driving evaluation information indicates a higher level of danger compared with the benchmark driving evaluation information, the abnormality determination unit 46 determines that the driver has not experienced any abnormality when there is no passenger. This can prevent false detection of a decline in the driver's cognitive ability.

[0096] The anomaly determination unit 46 can also change the baseline driving evaluation information used for comparison based on whether or not a passenger is present. The baseline driving evaluation information used for comparison when a passenger is present is called the first baseline driving evaluation information, and the baseline driving evaluation information used for comparison when a passenger is absent is called the second baseline driving evaluation information. The first baseline driving evaluation information is set based on driving with a passenger present, and the second baseline driving evaluation information is set based on driving without a passenger. Like the baseline driving evaluation information described above, both the first and second baseline driving evaluation information can be set based on the driver's past driving experience or based on the average driving ability of able-bodied individuals.

[0097] Thus, when first and second benchmark evaluation information are set, if the passenger determination unit 46 determines that a passenger is present by the passenger determination unit 42, it determines whether the driver has experienced an abnormality based on the comparison result between the first benchmark driving evaluation information and the driving evaluation information. If the passenger determination unit 42 determines that no passenger is present, the abnormality determination unit 46 determines whether the driver has experienced an abnormality based on the comparison result between the second benchmark driving evaluation information and the driving evaluation information.

[0098] If the driving instruction determination unit 43 determines that a driving instruction has been received, the anomaly determination unit 46 may not need to determine whether the driver has experienced an anomaly. When a driving instruction has been received, the driving evaluation information derived by the driving ability derivation unit 45 may show a reduced risk level because the driver receives the driving instruction from a passenger. That is, by receiving the driving instruction from a passenger, the likelihood of missing a dangerous object, or sudden changes in the vehicle M's speed or direction of travel is high. Thus, in situations where the likelihood of a change in driving ability is high, not determining whether the driver has experienced an anomaly allows for proper detection of any decline in the driver's cognitive ability.

[0099] The notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information. For example, the notification control unit 54 sends the notification information to the first terminal device T1 via the communication unit 60. For example, the notification control unit 54 may also send the first notification information to the first terminal device T1 if the anomaly determination information includes information indicating that the driver is abnormal (i.e., if the anomaly determination unit 46 determines that the driver has an abnormality). The first notification information may include, for example, information indicating driving evaluation information. The notification control unit 54 may also not send the first notification information to the first terminal device T1 if the anomaly determination information includes information indicating that the driver is not abnormal (i.e., if the anomaly determination unit 46 determines that the driver is not abnormal).

[0100] The first terminal device T1 (notification unit 84) outputs a notification based on the first notification information. The notification may also be made by displaying a string, image, or the like obtained based on the first notification information on the display device of the first terminal device T1 (notification unit 84).

[0101] The notification issued by the first terminal device T1 (notification unit 84) may also include a suggestion to seek medical treatment at a medical institution (e.g., a hospital the driver frequently visits). In this case, the first terminal device T1 (notification unit 84) may also allow the driver to input information indicating whether the driver agrees to the medical treatment. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a virtual button or the like indicating agreement to the medical treatment. Such a button or the like functions as an agreement information acquisition unit, which acquires information indicating the driver's intention to agree to the medical treatment (hereinafter referred to as agreement information). Furthermore, if the driver operates the first terminal device T1 (notification unit 84) and the agreement information acquisition unit acquires the agreement to the medical treatment information, the first terminal device T1 (notification unit 84) may send a notification to a designated information processing terminal indicating that the driver has agreed to the medical treatment. The designated information processing terminal may, for example, be a terminal used by personnel belonging to a medical institution (e.g., the second terminal device T2 described later). Specifically, the specified information processing terminal may also be a terminal used by personnel belonging to a medical institution (e.g., doctors, nurses, etc.).

[0102] However, as long as the form is recognizable by the driver, the form of the notification issued by the first terminal device T1 (notification unit 84) is not particularly limited and can be appropriately changed.

[0103] The notification sent by the first terminal device T1 (notification unit 84) may occur at a time other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. For example, the first notification message sent by the notification control unit 54 may also occur at the aforementioned times. Alternatively, the first notification message sent by the notification control unit 54 may occur at any time, and the first terminal device T1 (notification unit 84) may control the timing of the notification to ensure that the notification occurs at the aforementioned times.

[0104] [Processing Flow]

[0105] Figure 11 This is a flowchart illustrating an example of the processing flow performed by the information processing device 2. Figure 11 The process shown in the flowchart can begin, for example, at a predetermined time after driving ends, or repeatedly at a predetermined cycle, or it can be started by the instruction of the driver, etc.

[0106] First, the passenger determination unit 42 determines whether a passenger exists (step S100). If a passenger is determined to exist (step S100: "Yes"), the driving instruction determination unit 43 determines whether the driver has received a driving instruction (step S102). If the driver has not received a driving instruction (step S102: "No"), the anomaly determination unit 46 compares the driving evaluation information with the first benchmark driving evaluation information (step S104). Next, the anomaly determination unit 46 determines whether the driver has experienced an anomaly based on the comparison result between the driving evaluation information and the first benchmark driving evaluation information (step S108). At this time, the anomaly determination unit 46 may also generate anomaly determination information. If the driver has experienced an anomaly (step S108: "Yes"), the notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information (step S110). After that, the information processing device 2 ends. Figure 11 The flowchart shown illustrates the processing.

[0107] On the other hand, if it is determined that there is no passenger (step S100: "No"), the anomaly determination unit 46 compares the driving evaluation information with the second benchmark driving evaluation information (step S106). Next, the anomaly determination unit 46 determines whether the driver has experienced an anomaly based on the comparison result between the driving evaluation information and the second benchmark driving evaluation information (step S108). At this time, the anomaly determination unit 46 may also generate anomaly determination information. If it is determined that the driver has experienced an anomaly (step S108: "Yes"), the notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information (step S110). After that, the information processing device 2 ends. Figure 11 The flowchart shown illustrates the processing.

[0108] If it is determined that the driver has received driving instructions (step S102: "Yes"), or if it is determined that the driver has not acted abnormally (step S108: "No"), the information processing device 2 ends. Figure 11 The flowchart shown illustrates the processing.

[0109] According to the first embodiment described above, the information processing device 2 includes: an acquisition unit 20 that acquires a facial image, which is an image including the face of a vehicle occupant; a passenger determination unit 42 that determines whether a passenger is present in the vehicle based on the facial image; a driving ability derivation unit 45 that derives driving evaluation information, which is information relating to the driver's driving ability; an anomaly determination unit 46 that, when it is determined that a passenger is present, determines whether the driver has experienced an anomaly based on a comparison between the derived driving evaluation information and baseline driving evaluation information; and a notification control unit 54 that, when it is determined that the driver has experienced an anomaly, causes the notification unit to issue a notification. This allows for the detection of a decline in the driver's cognitive abilities. For example, by providing early notification to elderly drivers about information related to cognitive decline, accidents can be prevented. It can extend driving lifespan and provide a safe and secure driving experience, enabling a richer life through the extension of healthy lifespan.

[0110] <Second Implementation>

[0111] Next, the second embodiment will be described. The basic structure is the same as that of the first embodiment. Therefore, the same reference numerals are used to label the same structures and their descriptions are omitted; only the differences will be described.

[0112] Figure 12 This diagram illustrates an example of the structure of a driver monitoring device 1A including the information processing apparatus 2A according to the second embodiment. The information processing apparatus 2A according to the second embodiment differs from the information processing apparatus 2 according to the first embodiment in that the processing unit 40 further includes a storage control unit 47 and a comparison unit 48 (see also [reference]). Figure 1 ).

[0113] The storage control unit 47 establishes a link between the exported date and time, driving evaluation information, whether there are passengers, and whether driving instructions are given, and stores these information in the storage unit 70. The exported date and time is the date and time at which the driving evaluation information was exported. The exported date and time can also be obtained, for example, from date and time information contained in surrounding images, facial images, or audio information. The aforementioned processing performed by the storage control unit 47 is performed, for example, whenever driving evaluation information is exported by the driving ability export unit 45. Hereinafter, the information stored in the storage unit 70 will be referred to as driving ability information 72.

[0114] Figure 13This diagram illustrates an example of driving ability information 72. The storage control unit 47 repeatedly performs the aforementioned storage process, thereby accumulating multiple groups (hereinafter referred to as export information groups J) in the driving ability information 72, including export date and time, driving evaluation information, whether there are passengers, and whether there are driving instructions. Specifically, the storage control unit 47 generates export information group J whenever driving evaluation information is exported from the driving ability export unit 45. Furthermore, the storage control unit 47 updates the driving ability information 72 by appending the generated export information group J to the driving ability information 72.

[0115] return Figure 12 The comparison unit 48 generates comparison information based on the driving ability information 72. Specifically, the comparison unit 48 compares first driving evaluation information (driving evaluation information when no passenger is present), second driving evaluation information (driving evaluation information when driving instructions are received), and third driving evaluation information (driving evaluation information when no driving instructions are received) to generate information relating to the comparison results, i.e., comparison information. The comparison information may, for example, be information obtained by comparing the mode of the hazard level obtained based on the first driving evaluation information, the mode of the hazard level obtained based on the second driving evaluation information, and the mode of the hazard level obtained based on the third driving evaluation information.

[0116] The comparison unit 48 may perform the above-described processing (comparison processing) whenever driving evaluation information is exported by the driving ability export unit 45, or it may perform the above-described processing based on instructions from the driver, etc. The comparison unit 48 may perform the above-described processing based on all exported information groups J contained in the driving ability information 72, or it may perform the above-described processing based on a portion of the exported information groups J contained in the driving ability information 72. A portion of the exported information groups J may, for example, be an exported information group J whose export date and time are included in the period specified by the driver, etc.

[0117] At this time, the notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the comparison information. For example, the notification control unit 54 may also send second notification information to the first terminal device T1 whenever the comparison unit 48 performs comparison processing. The second notification information may, for example, include information relating to the first driving evaluation information, the second driving evaluation information, and the third driving evaluation information.

[0118] The first terminal device T1 (notification unit 84) outputs a notification based on the second notification information. Figure 14 This diagram illustrates an example of a notification made by the first terminal device T1 (notification unit 84). Regarding... Figure 14The chart shown uses frequency on the vertical axis and driving hazard level on the horizontal axis. The chart includes first driving evaluation information H1, second driving evaluation information H2, and third driving evaluation information H3. First driving evaluation information H1, second driving evaluation information H2, and third driving evaluation information H3 represent the average of each of the multiple first driving evaluation information, multiple second driving evaluation information, and multiple third driving evaluation information included in driving ability information 72. The first terminal device T1 (notification unit 84) displays the comparison results of the first driving evaluation information H1, second driving evaluation information H2, and third driving evaluation information H3 as an image.

[0119] According to the second embodiment described above, a decline in the driver's cognitive ability can be detected. By displaying the comparison results as an image on the first terminal device T1, the driver can objectively assess their own driving ability. When a third party (instructor) such as a driving school seeks to improve the driver's driving ability, having the instructor assess the driver's driving ability numerically can help determine appropriate guidance.

[0120] <Third Implementation Method>

[0121] Next, the third embodiment will be described. The basic structure is the same as that of the first embodiment. Therefore, the same reference numerals will be used to label the same structures and their descriptions will be omitted. Only the differences will be described.

[0122] Figure 15 This diagram illustrates the exception notification processing in the information processing system 100B according to the third embodiment. Figure 15 As shown, the information processing system 100B according to the third embodiment includes a second terminal device T2 in addition to the driving monitoring device 1 and the first terminal device T1.

[0123] The second terminal device T2 is an information processing device used by a user different from the driver using the first terminal device T1 (e.g., an employee of a hospital or insurance company). The second terminal device T2 may also be, for example, a smartphone, tablet, personal computer, etc.

[0124] In the third embodiment, the driving monitoring device 1 (notification control unit 54) sends the first notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the third embodiment, the notification unit 84 controlled by the notification control unit 54 is not the first terminal device T1 but the second terminal device T2. In the second terminal device T2, for example, an application program for receiving notifications from the information processing device 2 is running. The application program causes the display device to display images, strings, etc., obtained based on the information (first notification information, etc.) sent by the information processing device 2, or causes the speaker to emit sound.

[0125] The second terminal device T2 uses information input by the user to notify the first terminal device T1. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0126] The input unit 92 and the transmission control unit 94 are equipped with hardware processors such as CPUs and storage devices (storage devices with non-transitory storage media) that store programs (software). The processor executes the programs to implement the functions of each component. Some or all of these components can be implemented using hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component can be implemented through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with information processing devices such as the first terminal device T1.

[0127] The input unit 92 accepts user operations and inputs them to the second terminal device T2. The input unit 92 can be configured using existing input devices such as keyboards, clicking devices (mouse, tablet, etc.), buttons, or touch panels. The input unit 92 can also be an interface for connecting an input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in the input device based on the user's operation to the second terminal device T2. The input unit 92 can be configured arbitrarily as long as it is capable of inputting instructions obtained based on user operations to the second terminal device T2.

[0128] For example, the user confirms the content of the notification sent by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2, and studies the content that should be notified to the driver. The content to be notified to the driver may include, for example, recommendations to seek medical attention at a medical institution for the purpose of detecting a possible decline in cognitive ability. The user inputs information, including the content of the studied notification, into the second terminal device T2 via the operation input unit 92. The transmission control unit 94 transmits the information input by the user to the first terminal device T1 via the communication unit 96. The first terminal device T1 then notifies the driver based on the transmitted information.

[0129] The information processing system 100B can also be configured to restrict the transmission of a first notification message from the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 by the driver operating the first terminal device T1 or the driving monitoring device 1. For example, the driver can operate the first terminal device T1 or the driving monitoring device 1 to generate restriction information indicating the presence or absence of restrictions on the transmission of the first notification message, and store the generated restriction information in the storage unit 70. Furthermore, the alarm control unit 52 can also decide whether to send the first notification message to the second terminal device T2 based on the restriction information stored in the storage unit 70. With this structure, the decision on whether to inform the user of the second terminal device T2 of any abnormal information involving the driver is made according to the driver's wishes. Therefore, the driver's privacy is easily protected.

[0130] According to the third embodiment described above, the information processing system 100B includes: an information processing device 2; a first terminal device T1, which is used by the driver and is a notification unit 84 that is not controlled by the notification control unit 54; and a second terminal device T2, which is used by a user different from the driver and is also controlled by the notification control unit 54. The second terminal device T2 causes the first terminal device T1 to make notifications based on information input by the user's operation. Thus, it is possible to provide the driver using the first terminal device T1 with notification content obtained from research conducted by the user using the second terminal device T2. For example, when a medical worker uses the second terminal device T2, abnormality determination information can be used for medical judgment to detect abnormalities in the driver.

[0131] <Variation Example>

[0132] The scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0133] Alternatively, information relating to the determination results of missed sightings by the missed sighting determination unit 41 may be collected in the storage unit 70, and the first terminal device T1 (alarm unit 82) may provide the driver with entries (hereinafter referred to as "review entries") generated based on the collected information for the driver to review their past driving. Review entries may include, for example, information (strings, images, etc.) that the driver can identify as potentially dangerous objects that they may have missed. Review entries may be generated by the driving monitoring device 1 (information processing device 2) or by the first terminal device T1 (alarm unit 82).

[0134] In the information processing system 100B according to the third embodiment, the driving monitoring device 1 (notification control unit 54) can also send first notification information to both the first terminal device T1 and the second terminal device T2. In other words, the notification unit 84 that controls the notifications from both the first terminal device T1 and the second terminal device T2 can also be controlled by the notification control unit 54.

[0135] In the first embodiment, the driving monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as different devices, but they can also be configured as an integrated device. For example, the driving monitoring device 1 may actually include all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, alarm unit 82, and notification unit 84. Alternatively, all these functions may actually be installed in the first terminal device T1 or the vehicle M. Similarly, in the second embodiment, all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and alarm unit 82 may actually be installed in the driving monitoring device 1, and all these functions may actually be installed in the first terminal device T1.

[0136] The driving monitoring device 1 can also be actually installed using multiple information processing devices. For example, the functions of the information processing device 2 can also be actually installed in the first terminal device T1 or the vehicle M (e.g., ECU). Alternatively, the driving monitoring device 1 can be actually installed using a device such as a cloud. For example, in the driving monitoring device 1, the detection unit 30, processing unit 40, control unit 50, and storage unit 70 can also be actually installed in different information processing devices. For example, the storage unit 70 can also be distributed among multiple information processing devices.

[0137] The implementation methods and variations described above can be represented as follows.

[0138] An information processing device comprising:

[0139] Storage medium, which stores computer-readable instructions; and

[0140] The processor, which is connected to the storage medium,

[0141] The processor performs the following processing by executing computer-readable instructions:

[0142] Obtain a facial image, which is an image containing the faces of the occupants of the vehicle;

[0143] The presence of a passenger in the vehicle is determined based on the facial image.

[0144] Export driving evaluation information, which is information concerning the driver's driving ability in the vehicle;

[0145] If it is determined that the passenger is present, a comparison between the derived driving evaluation information and the baseline driving evaluation information is used to determine whether the driver has acted abnormally; and

[0146] If it is determined that the driver has malfunctioned, the notification unit will issue a notification.

[0147] Without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements, and the above embodiments and variations can also be appropriately combined.

Claims

1. An information processing device, wherein, The information processing device includes: The acquisition unit acquires a facial image, which is an image containing the face of a vehicle occupant; The passenger determination unit determines whether there are any passengers in the vehicle based on the facial image. The driving ability derivation unit derives driving evaluation information, which is information relating to the driving ability of the driver of the vehicle. The anomaly determination unit determines whether the driver has committed an anomaly based on a comparison between the derived driving evaluation information and the baseline driving evaluation information when it determines that the passenger is present. as well as The notification control unit will issue a notification if it determines that the driver has experienced an abnormality.

2. The information processing apparatus according to claim 1, wherein, The driving evaluation information is information indicating the degree of danger of driving. If the anomaly determination unit determines that the driver has acted abnormally when it determines that the passenger is present and the derived driving evaluation information shows a higher degree of danger compared to the baseline driving evaluation information.

3. The information processing apparatus according to claim 1 or 2, wherein, The driving evaluation information is information indicating the degree of danger of driving. If the anomaly determination unit determines that there is no passenger and the derived driving evaluation information shows a higher degree of danger compared to the benchmark driving evaluation information, it determines that the driver has not experienced any abnormality.

4. The information processing apparatus according to claim 1 or 2, wherein, The benchmark driving evaluation information is derived from a single drive that serves as the benchmark.

5. The information processing apparatus according to claim 1 or 2, wherein, The benchmark driving evaluation information is information set based on the driver's driving.

6. The information processing apparatus according to claim 1 or 2, wherein, The baseline driving evaluation information is derived from driving under conditions where the passenger is not present.

7. The information processing apparatus according to claim 1 or 2, wherein, The baseline driving evaluation information is derived from driving in situations where the passenger is present.

8. The information processing apparatus according to claim 1 or 2, wherein, The benchmark driving evaluation information is derived from a specified number of driving sessions.

9. The information processing apparatus according to claim 1, wherein, The acquisition unit acquires audio information from inside the vehicle. The information processing device further includes a driving instruction determination unit, which, in the presence of the passenger, determines, based on the voice information and the facial image, whether the driver has received driving instructions related to the driving operation of the vehicle. If the anomaly determination unit determines that it has received the driving instruction, it will not determine whether the driver has committed an anomaly.

10. The information processing apparatus according to claim 9, wherein, The information processing device further includes a storage control unit, which stores first driving evaluation information, second driving evaluation information and third driving evaluation information in the storage unit respectively. The first driving evaluation information is the driving evaluation information when the passenger is not present, the second driving evaluation information is the driving evaluation information when the driving instruction is received, and the third driving evaluation information is the driving evaluation information when the driving instruction is not received.

11. The information processing apparatus according to claim 10, wherein, The information processing device further includes a comparison unit that compares the first driving evaluation information, the second driving evaluation information, and the third driving evaluation information. The notification control unit causes the notification unit to output information relating to the comparison results compared by the comparison unit.

12. The information processing apparatus according to claim 1, wherein, The acquisition unit acquires images of the surrounding area of ​​the vehicle. The information processing device also includes: The risk detection unit detects risky objects in the surrounding image; as well as The omission determination unit determines whether the driver has missed the risky object based on information relating to the location of the risky object and information relating to the facial image. The driving ability derivation unit derives the driving evaluation information based on the judgment result of the omission judgment unit.

13. The information processing apparatus according to claim 1, wherein, The driving ability derivation unit derives the driving evaluation information based on some or all of the information relating to the vehicle's speed and the vehicle's direction of travel.

14. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to claim 1 or 2; A first terminal device, which is used by the driver and is not the notification unit controlled by the notification control unit; as well as The second terminal device is for use by a user different from the driver, and the notification unit is controlled by the notification control unit. The second terminal device causes the first terminal device to make a notification based on information input by the user through the user's operation.

15. An information processing method, wherein, The information processing method causes the computer to perform the following processing: Obtain a facial image, which is an image containing the faces of the occupants of the vehicle; The presence of a passenger in the vehicle is determined based on the facial image. Export driving evaluation information, which is information concerning the driver's driving ability in the vehicle; If it is determined that the passenger is present, the driver is determined to have acted abnormally based on the comparison between the derived driving evaluation information and the baseline driving evaluation information. as well as If it is determined that the driver has malfunctioned, the notification unit will issue a notification.

16. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Obtain a facial image, which is an image containing the faces of the occupants of the vehicle; The presence of a passenger in the vehicle is determined based on the facial image. Export driving evaluation information, which is information concerning the driver's driving ability in the vehicle; If it is determined that the passenger is present, the driver is determined to have acted abnormally based on the comparison between the derived driving evaluation information and the baseline driving evaluation information. as well as If it is determined that the driver has malfunctioned, the notification unit will issue a notification.

Images