Information processing apparatus, information processing system, information processing method, and storage medium

Abstract

An information processing apparatus, an information processing system, an information processing method, and a storage medium capable of detecting an abnormality of a driver are provided. The information processing apparatus includes a risk detection unit that detects a risk object located in the periphery of a vehicle; an acquisition unit that acquires a facial image of a driver of the vehicle; a miss determination unit that determines whether or not the driver misses the risk object based on information related to the position of the risk object and information related to the facial image; an abnormality determination unit that determines whether or not the driver has an abnormality based on a change tendency of the presence or absence of the miss; and a notification control unit that causes a notification unit to perform notification in a case where it is determined that the driver has an abnormality.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and storage medium. Background Technology

[0002] In recent years, efforts have been intensifying to provide sustainable transportation systems that also take into account vulnerable groups among transportation participants, particularly the elderly, people with disabilities, and children. Research and development are particularly focused on further improving the safety and convenience of transportation through developments related to mobility facilities for the elderly and people with disabilities (see, for example, Patent Documents 1 and 2 below).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2020-71528

[0004] Patent Document 2: Japanese Patent Application Publication No. 2017-16568 Summary of the Invention

[0005] When a driver has conditions such as visual impairment, cognitive impairment, or accumulated fatigue, their ability to avoid dangerous objects is reduced. However, drivers are sometimes unaware of these conditions or the accumulation of fatigue. When drivers continue driving without realizing their illnesses, fatigue, and the resulting reduced abilities, it can compromise traffic safety and convenience.

[0006] The solutions involved in this invention were made in consideration of such circumstances, and one of their objectives is to provide an information processing device, information processing system, information processing method and storage medium capable of detecting abnormalities in drivers, thereby contributing to the development of sustainable transportation systems.

[0007] To address the aforementioned issues, the present invention employs the following solution.

[0008] (1): One aspect of the present invention relates to an information processing apparatus, wherein the information processing apparatus comprises: a risk detection unit that detects risky objects located around a vehicle; an acquisition unit that acquires a facial image of the driver of the vehicle; a miss determination unit that determines whether the driver has missed the risky object based on information relating to the location of the risky object and information relating to the facial image; an anomaly determination unit that determines whether the driver has committed an anomaly based on the changing trend of whether the miss has occurred; and a notification control unit that, if it is determined that the driver has committed an anomaly, causes the notification unit to make a notification.

[0009] (2): In the above (1) scheme, the anomaly determination unit calculates the frequency value of the missed check for each of the multiple first calculation periods included in the first determination period. If the frequency value of the first calculation period is above a first threshold determined based on the frequency value of the past missed check only in a part of the multiple first calculation periods, the driver is determined to have experienced a first anomaly. If the notification control unit determines that the driver has experienced a first anomaly, the notification unit will make a first notification.

[0010] (3): In the above scheme (1) or (2), the anomaly determination unit calculates the frequency value of the missed check for each of the multiple second calculation periods included in the second determination period. If the frequency value of all the multiple second calculation periods is above a second threshold determined based on the frequency value of past missed checks, the driver is determined to have a second anomaly. If the notification control unit determines that the driver has a second anomaly, the notification unit will make a second notification.

[0011] (4): In any of the above schemes (1) to (3), the anomaly determination unit calculates the frequency value of the missed information for each of the multiple third calculation periods included in the third determination period. If the change value of the frequency value in the multiple third calculation periods is above a third threshold, the driver is determined to have experienced a third anomaly. If the notification control unit determines that the driver has experienced a third anomaly, the notification unit will issue a third notification.

[0012] (5): In any of the above schemes (1) to (4), the abnormality determination unit may determine the period during which the driver is sleepy based on the facial image, and determine whether the driver has an abnormality based on the presence or absence of the omission during the period other than the determined sleepy period.

[0013] (6): In the above (5) scheme, the anomaly determination unit may determine the period of drowsiness based on the number of times the driver's eyelids open and close and / or the degree of eyelid opening detected from the facial image.

[0014] (7): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing apparatus of any of the above (1) to (6); a first terminal device for use by the driver and not controlled by the notification control unit; and a second terminal device for use by a user different from the driver and controlled by the notification control unit, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

[0015] (8): One aspect of the present invention relates to an information processing method, wherein the information processing method causes a computer to perform the following processing: detecting a risky object located around a vehicle; obtaining a facial image of the driver of the vehicle; determining whether the driver has overlooked the risky object based on information relating to the location of the risky object and information relating to the facial image; determining whether the driver has acted abnormally based on the changing trend of whether the driver has overlooked the risky object; and, if it is determined that the driver has acted abnormally, causing a notification unit to issue a notification.

[0016] (9): One aspect of the present invention relates to a storage medium, which is a non-temporary storage medium that stores a program that can be read by a computer, wherein the program is used to cause the computer to perform the following processing: detecting a risky object located around a vehicle; acquiring a facial image of the driver of the vehicle; determining whether the driver has overlooked the risky object based on information relating to the location of the risky object and information relating to the facial image; determining whether the driver has acted abnormally based on the changing trend of whether the driver has overlooked the risky object; and, if it is determined that the driver has acted abnormally, causing a notification unit to issue a notification.

[0017] According to the solution of the present invention, abnormalities of the driver can be detected. Attached Figure Description

[0018] Figure 1 This is a diagram illustrating an example of the structure and usage environment of a driving monitoring device that includes the information processing apparatus according to the first embodiment.

[0019] Figure 2 This is a diagram illustrating an example of a driver monitoring device.

[0020] Figure 3 This is an example of an image obtained by a driving monitoring device.

[0021] Figure 4 This diagram illustrates the process of a driver monitoring device triggering a missed alarm.

[0022] Figure 5This is a diagram illustrating an example of information detected by the testing department.

[0023] Figure 6 This is a diagram illustrating an example of an alarm issued by the alarm department.

[0024] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device.

[0025] Figure 8 This is a diagram illustrating an example of determining resume information.

[0026] Figure 9 This is a flowchart illustrating an example of the processing flow performed by the exception handling unit.

[0027] Figure 10 This diagram illustrates the exception notification processing in the information processing system according to the second embodiment. Detailed Implementation

[0028] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the rearward direction of the vehicle is defined as the negative X direction, the rightward direction of the vehicle's width relative to the positive X direction is defined as the positive Y direction, the leftward direction is defined as the negative Y direction, and the direction orthogonal to the X and Y directions and the height direction of the vehicle is defined as the positive Z direction.

[0029] <First Implementation>

[0030] [Overall Structure]

[0031] Figure 1 This diagram illustrates an example of the structure and usage environment of a driver monitoring device 1, including the information processing device 2 according to the first embodiment. The information processing device 2 is, for example, a driver monitoring device 1 retrofitted into a vehicle (hereinafter referred to as "vehicle M") such as a dashcam. That is, the driver monitoring device 1 (information processing device 2) can be mounted and removed from the vehicle M via a mounting / unmounting unit (hereinafter referred to as "mounting / unmounting unit 16"). The information processing device 2 is, for example, a device not connected to the vehicle M's in-vehicle network. An in-vehicle network refers to an in-vehicle network that connects to the vehicle's control-related ECU (Electronic Control Unit), which uses communication standards such as CAN (Controller Area Network), and is connected to the vehicle M's drive, braking, steering, various vehicle sensors, driving-related operating components, and operation buttons. The vehicle M's network for HMI (Human Machine Interface) can also be excluded from the above-described in-vehicle network.

[0032] Figure 2 This diagram illustrates an example of a driver monitoring device 1. The driver monitoring device 1 is, for example, installed near the windshield of a vehicle M, and located near the front of the driver's seat. The driver monitoring device 1 stores images (moving images) captured by the camera unit 10, for example, in the storage unit 70 (described later). The driver monitoring device 1 may also be, for example, equipped with a display device such as an LCD (Liquid Crystal Display) and capable of displaying the images (moving images) stored in the storage unit 70 on the display device.

[0033] The driving monitoring device 1 includes, for example, a camera unit 10, a loading and unloading unit 16, and an information processing unit 2. In the illustrated example, the camera unit 10 includes a front camera 12 and a rear camera 14.

[0034] The front camera 12 and the rear camera 14 are each, for example, digital cameras utilizing solid-state imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) captures images of the area in front of the vehicle M from the location where the driver monitoring device 1 is installed. The rear camera 14 repeatedly (periodically) captures images of the area behind the vehicle M and the interior of the vehicle M from the location where the driver monitoring device 1 is installed. The horizontal viewing angle captured by the front camera 12 and the rear camera 14 is, for example, 180° or more. That is, the driver monitoring device 1 (camera unit 10) captures a 360° range around the vehicle M from its installation location. The front camera 12 and the rear camera 14 may each be, for example, fisheye cameras including fisheye lenses. Alternatively, the front camera 12 and the rear camera 14 may each be wide-angle cameras or the like. In this embodiment, the use of the front camera 12 and the rear camera 14 is described, but other cameras may be used instead of these or other alternatives. That is, the camera unit 10 can be configured to capture images used in this embodiment (peripheral images and facial images described later).

[0035] The mounting / unmounting part 16 is, for example, a component for mounting the driver monitoring device 1 onto the vehicle M. The mounting / unmounting part 16 can be, for example, any supporting component such as a suction cup, a seal, or a bracket. The mounting / unmounting part 16 may also include a power supply cable (wiring, etc.) for supplying power to the driver monitoring device 1.

[0036] Figure 3 This is a diagram illustrating an example of an image acquired by the driver monitoring device 1. That is, Figure 3This diagram illustrates an example of an image captured by the camera unit 10. The front camera 12 captures images of the front of the vehicle M through the windshield, the driver's side side window, and the passenger side side window. The rear camera 14 captures images of the interior of the vehicle M (including the driver and passengers) and the rear of the vehicle M through the driver's side side window, the passenger side side window, and the rear window.

[0037] In the image captured by the front camera 12 (hereinafter referred to as the "front image"), the subject includes objects such as other vehicles, pedestrians, bicycles, fixed objects, road markings, etc., that can be seen through the front windshield and exist in front of the vehicle M, as well as objects that can be seen through the left and right (front) side windows of the vehicle M.

[0038] The image captured by the rear camera 14 (hereinafter referred to as the "rear image") shows the interior of the vehicle M, as well as objects visible through the rear window that are located behind the vehicle M, and objects visible through the left and right side windows that are located to the left and right (rear) of the vehicle M. Therefore, in the rear image captured by the rear camera 14, the driver of the vehicle M equipped with the driver monitoring device 1 is also reflected as the subject.

[0039] The front and rear images include images of the surrounding conditions of the vehicle M (hereinafter referred to as "surrounding images"). The rear image includes an image of the driver's face of the vehicle M (hereinafter referred to as "face image"). The driver monitoring device 1 can be installed at any location where the camera unit 10 can capture the surrounding images and the face image.

[0040] Information processing device 2 (refer to) Figure 1 Based on the results obtained by referring to the surrounding images and facial images output by the camera unit 10, the alarm and notification are output via the first terminal device T1. Sometimes the driver monitoring device 1 and the first terminal device T1 are combined and referred to as "information processing system 100".

[0041] The first terminal device T1 is, for example, a mobile terminal device used by a driver driving a vehicle M equipped with a driver monitoring device 1, such as a smartphone or tablet. The first terminal device T1 executes, for example, an application for receiving alarms and notifications from the information processing device 2. The application causes a display device to show an image based on information sent by the information processing device 2 (e.g., alarm information and notification information described later), and causes a speaker to emit sound. The first terminal device T1 is also referred to as the alarm unit 82 and the notification unit 84. The first terminal device T1 is used, for example, in a state where it can be detachably mounted on the vehicle M. For example, a bracket for the first terminal device T1 with a mounting and detaching mechanism is provided on one or both of the first terminal device T1 and the vehicle M, and the first terminal device T1 is supported by the bracket.

[0042] Alternatively, the navigation, display, and speaker of vehicle M can be used instead of the first terminal device T1. That is, the navigation, display, and speaker of vehicle M can also output alarms and notifications based on the instructions of information processing device 2. That is, the navigation, display, and speaker of vehicle M can also be alarm unit 82 and notification unit 84. Alternatively, the display unit (alarm unit 82, notification unit 84) and speaker (alarm unit 82, notification unit 84) can be provided on the driver monitoring device 1 instead of the first terminal device T1.

[0043] like Figure 1 As shown, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, and a face orientation detection unit 36. The processing unit 40 includes a miss detection unit 42, a storage control unit 44, and an anomaly detection unit 46. The control unit 50 includes an alarm control unit 52 and a notification control unit 54.

[0044] The acquisition unit 20, detection unit 30, processing unit 40, and control unit 50 are equipped with hardware processors such as CPUs (Central Processing Units) and storage devices (storage devices with non-transitory storage media) storing programs (software). The processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented using hardware such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units) (including the circuitry). The functions of each component can also be realized through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs.

[0045] The program (software) can be pre-stored in the storage unit 70 (a storage device with a non-temporary storage medium) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it can be stored in a removable storage medium (a non-temporary storage medium) such as a memory card, and installed in the storage device by assembling the storage medium into the driving monitoring device 1. The program (software) can also be pre-downloaded from other computer devices via short-range communication or wide-area communication using an application program executed in the first terminal device T1, and then sent from the first terminal device T1, thereby being installed in the storage device.

[0046] The storage unit 70 stores, for example, decision history information 72, programs (such as the programs described above), and various other information. The storage unit 70 can also be implemented using the various storage devices described above or EEPROM (Electrically Erasable Programmable Read Only Memory). The communication unit 60 is an interface for wireless or wired communication with information processing devices (external devices) such as the first terminal device T1.

[0047] The driver monitoring device 1 (information processing device 2) performs at least two functions: a missed detection alarm and an anomaly notification. The missed detection alarm is a process that detects a driver's missed detection of a risky object based on surrounding images and facial images, and triggers an alarm on the alarm unit 82. The anomaly notification is a process that detects driver anomalies (fatigue, narrowed field of vision, cognitive impairment) based on the changing trend of the presence or absence of a missed detection (i.e., what kind of trend exists in the changing presence or absence of a missed detection) and notifies the notification unit 84 of the anomaly.

[0048] A risky object refers to an object, for example, that vehicle M should avoid. A risky object is, for example, an object that would interfere with vehicle M when it is moving as it is, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose approach level becomes more than a certain level after a specified time, taking into account the object's position, direction of movement, and speed, as well as the position, method of movement, and speed of vehicle M. The object can also be a traffic participant such as a pedestrian, bicycle, or vehicle, or an object other than a traffic participant such as an object placed on the road or a fallen object.

[0049] [Handling of Missing Alarms]

[0050] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device 1 (information processing device 2).

[0051] The acquisition unit 20 acquires front and rear images from the imaging unit 10. Therefore, the acquisition unit 20 establishes a link between the peripheral images and the facial images to acquire these images. The imaging unit 10 repeatedly (periodically) captures the peripheral images and facial images, thus the acquisition unit 20 repeatedly (periodically) acquires these images. The acquisition unit 20 can also establish a link between the peripheral images and facial images and the time they were captured (the shooting time) to acquire these images.

[0052] The risk detection unit 32 detects risky objects in the surrounding image. "Detecting risky objects" refers to, for example, detecting the presence of a risky object and the position of that object relative to the vehicle M (driver monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M can be pre-stored in the storage unit 70. The risk detection unit 32 can also detect the position of the risky object relative to the vehicle M based on this reference information.

[0053] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, and also detects the direction of the risk object's presence (approach direction) relative to the vehicle M. "Direction of presence of the risk object" refers to, for example, the direction of presence of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, "direction of presence of the risk object" will sometimes be referred to as "risk presence direction".

[0054] The gaze detection unit 34 detects the driver's gaze in a facial image. Specifically, the gaze detection unit 34 detects the driver's gaze by analyzing the facial image using a prescribed image analysis method. "Gaze detection" refers to, for example, detecting the position of the driver's eyeballs and the direction of the gaze (the direction of the eyeballs).

[0055] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the facial image. Specifically, the face orientation detection unit 36 ​​detects the orientation of the driver's face by parsing the facial image using a prescribed image parsing method. The face orientation detection unit 36 ​​may also detect the position of the driver's face in addition to its orientation. Hereinafter, the information combining "face orientation" and "face position" is sometimes referred to as "facial information." The face orientation detection unit 36 ​​detects facial information in the facial image.

[0056] The detection unit 30 can also be configured to detect lanes, pedestrian crossings, etc., in addition to the aforementioned risk objects, lines of sight, and facial information. Figure 5 This is a diagram illustrating an example of information detected by the detection unit 30. In Figure 5 In the example, pedestrians, other vehicles, lanes, and crosswalks detected by the detection unit 30 are displayed based on vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that may become risk objects. In the information detected by the detection unit 30, traffic participants (objects) that may become risk objects detected by the detection unit 30 may also be displayed based on vehicle M, the direction of vehicle M, and / or the direction of travel of vehicle M, instead of vehicle M, the direction of the line of sight, and the direction of the face.

[0057] Leak detection unit 42 (refer to) Figure 4 This method determines whether a driver has missed a hazard based on the location and / or direction of the hazard, line of sight, and facial information. "Risk location" and "direction of the hazard" are each examples of "information relating to the location of the hazard." Line of sight and facial information are examples of "information relating to facial images."

[0058] For example, reference information (not shown) indicating the relationship between the orientation of the driver's eyes (gaze) and / or the orientation of their face in the facial image and their relative orientation with respect to the direction of travel of the vehicle M can also be pre-stored in the storage unit 70. The omission determination unit 42 can also refer to this reference information to determine whether the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists. If the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists, the omission determination unit 42 can also determine that the driver visually identifies a risky object. The so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can be, for example, the case where there is a risky position on a straight line along the orientation of the gaze and / or the orientation of the face. Alternatively, the so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can also be the case where there is a risky position in a cone-shaped area with the position of the viewpoint (eyeball) and / or the face as the apex and having a central axis along the orientation of the gaze and / or the orientation of the face that expands laterally as it moves forward. If the direction of the driver's gaze and / or the direction of the face does not match the location of the risky object, the omission determination unit 42 may also determine that the driver has not visually identified the risky object.

[0059] The omission determination unit 42 can also determine that the driver has missed the risk object if the state of not visually recognizing the risk object continues for a predetermined time or longer. Alternatively, the omission determination unit 42 can determine that the driver has not missed the risk object if the driver visually recognizes the risk object, or if the state of not visually recognizing the risk object is resolved before the predetermined time has elapsed. However, the method by which the omission determination unit 42 determines omissions is not limited to the above and can be appropriately modified.

[0060] The omission detection unit 42 generates omission detection information based on the above detection results. The omission detection information includes, for example, the presence or absence of omissions and the direction of risk.

[0061] The alarm control unit 52 causes the first terminal device T1 (alarm unit 82) to output an alarm based on the leak detection information. For example, the alarm control unit 52 sends alarm information to the first terminal device T1 via the communication unit 60. The alarm information includes, for example, information indicating whether an alarm is needed and the direction of the risk. For example, if the leak detection information includes information indicating "a leak exists," the alarm control unit 52 generates alarm information including information indicating "an alarm is needed." For example, if the leak detection information includes information indicating "no leak exists," the alarm control unit 52 generates alarm information including information indicating "no alarm is needed." The first terminal device T1 outputs an alarm based on the alarm information.

[0062] Figure 6This diagram illustrates an example of an alarm triggered by the first terminal device T1 (alarm unit 82). Figure 6 In the example, the first terminal device T1 displays identifiable entries indicating the direction of the risk (D1 and D2 in the diagram). Furthermore, the first terminal device T1 issues warnings by differentiating the display format of the risk direction when a risky object is missed (i.e., the warning message includes a message indicating "warning is needed") and when no risky object is missed (i.e., the warning message includes a message indicating "warning is not needed"). Specifically, when a risky object is missed, the risk direction is displayed in a more emphasized manner compared to when it is not missed. This emphasis could be based on color, flashing icons, or other visually illuminating elements to make the risk direction easier for the driver to recognize. The first terminal device T1 may also issue warnings using methods such as loudspeaker announcements or illuminated lights (A in the diagram), in addition to or instead of the above. The specific method of issuing the warning can be appropriately modified as long as it is identifiable by the driver.

[0063] The alarm issued by the first terminal device T1 (alarm unit 82) can also be issued during the time when the driver is driving the vehicle M (i.e., in real time). Examples of "the time when the driver is driving the vehicle M" include when the vehicle M is moving or when the vehicle M is temporarily stopped. For example, the alarm information sent by the alarm control unit 52 can also be issued at the aforementioned times. Alternatively, the alarm information sent by the alarm control unit 52 can be issued at any time, and the first terminal device T1 (alarm unit 82) controls the timing of the alarm to issue the alarm at the aforementioned times.

[0064] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device 1 (information processing device 2).

[0065] The storage control unit 44 establishes a link between the determination time, the presence or absence of a potential object to be missed, and the facial image, and stores this information in the storage unit 70. The determination time, for example, corresponds to the time when the surrounding image and / or facial image was captured, used in determining whether a missed object was present. The aforementioned processing (storage processing) performed by the storage control unit 44 is performed, for example, whenever the missed object determination unit 42 generates missed object determination information. Hereinafter, the information stored in the storage unit 70 will be referred to as determination history information 72.

[0066] Figure 8This diagram illustrates an example of the judgment history information 72. The storage control unit 44 repeatedly performs the aforementioned storage process, thereby accumulating multiple groups (hereinafter referred to as judgment information groups J) in the judgment history information 72, including the judgment time, whether there was a missed judgment, and facial image information. Specifically, the storage control unit 44 generates judgment information group J whenever the missed judgment unit 42 generates missed judgment information. Furthermore, the storage control unit 44 appends the generated judgment information group J to the judgment history information 72, thereby updating the judgment history information 72.

[0067] return Figure 7 The anomaly determination unit 46 determines whether an anomaly has occurred in the driver based on the determination history information 72. Specifically, the anomaly determination unit 46 determines whether a first anomaly, a second anomaly, and a third anomaly have occurred in the driver based on the changing trend of whether or not the driver has been overlooked, as contained in the determination history information 72.

[0068] The first abnormality is, for example, accumulated fatigue or potential accumulated fatigue. The second abnormality is, for example, having or potentially having visual field defects such as narrowing of the visual field. The third abnormality is, for example, having or potentially having cognitive impairment. The abnormality determination unit 46 generates abnormality determination information based on the determination result. The abnormality determination information includes, for example, information indicating the presence or absence of the first abnormality, information indicating the presence or absence of the second abnormality, and information indicating the presence or absence of the third abnormality.

[0069] Figure 9 This is a flowchart illustrating an example of the processing flow performed by the exception determination unit 46. Figure 9 The process shown in the flowchart can be started repeatedly at a predetermined cycle, or whenever the storage control unit 44 determines the update of the history information 72, or it can be started by the instruction of the driver or others.

[0070] First, the anomaly determination unit 46 reads the determination history information 72 from the storage unit 70 (step S202). Next, the anomaly determination unit 46 determines (estimates) the period of drowsiness based on the facial image contained in the read determination history information 72. The period of drowsiness is the period during which the driver is drowsy.

[0071] For example, the anomaly detection unit 46 can also detect the number of times the driver's eyelids open and close and / or the degree of eyelid opening from the facial image. Specifically, the anomaly detection unit 46 can also analyze the facial image using a prescribed image analysis method, thereby detecting the number of times the eyelids open and close and / or the degree of eyelid opening. Moreover, the anomaly detection unit 46 can also determine the period of drowsiness based on the detected number of times the eyelids open and close and / or the degree of eyelid opening. For example, the anomaly detection unit 46 can also determine that the driver is drowsy if the number of times the eyelids open and close during a prescribed period is above a prescribed threshold or if the degree of eyelid opening is below a prescribed threshold. Furthermore, the period during which the driver is determined to be drowsy can be defined as the drowsy period.

[0072] Next, the anomaly determination unit 46 generates exclusion history information (step S206) that excludes information corresponding to the determined sleepiness period (determination information group J) from the determination history information 72. The exclusion history information is the information in the determination history information 72 that becomes the object of the anomaly determination described below. That is, the anomaly determination unit 46 does not set all information in the determination history information 72 as the object of anomaly determination, but rather performs anomaly determination based on whether or not there are any omissions in periods other than sleepiness. By performing this preprocessing to exclude information corresponding to the sleepiness period from the determination history information 72, the accuracy of the anomaly determination performed by the anomaly determination unit 46 can be improved.

[0073] After performing step S206, the exception determination unit 46 executes steps S208-S214 (hereinafter referred to as the first process), steps S216-S222 (hereinafter referred to as the second process), and steps S224-S230 (hereinafter referred to as the third process) in parallel. The first process involves a first exception. The second process involves a second exception. The third process involves a third exception. However, the exception determination unit 46 may also perform two or three of the first, second, and third processes sequentially.

[0074] In step S208, the anomaly determination unit 46 calculates the frequency value of the missed information based on the exclusion history information for each of the multiple first calculation periods included in the first determination period.

[0075] Here, the "first determination period" is the period used to determine the presence or absence of the first anomaly. The first determination period is, for example, preset. The first determination period is, for example, one week (i.e., a 7-day period). However, the length of the first determination period can be appropriately changed.

[0076] The "first calculated period" is the period used to calculate the frequency value for detecting omissions in the determination of the presence or absence of the first anomaly. The first calculated period is shorter than the first determination period. The first calculated period is, for example, preset. Multiple first calculated periods can also be obtained by dividing the first determination period into a predetermined number of periods. For example, if the first determination period is one week (7 days), the first determination period can also include seven first calculated periods. In this case, each first calculated period is one day. However, the number of first calculated periods included in the first determination period and the length of each first calculated period can be appropriately changed.

[0077] The "frequency of missed sightings" represents how often a driver misses a risky object, with a higher frequency resulting in a larger value. The frequency value can also be defined as the number of times a driver misses a risky object. However, the definition of the frequency value is not limited to this and can be appropriately modified. For example, the frequency value can also be the "missed sighting rate." The missed sighting rate is, for example, defined as the number of times a driver misses a risky object divided by the total number of risky objects that occur.

[0078] Next, the anomaly determination unit 46 determines whether the frequency value is above the first threshold during only a portion of the multiple first calculation periods included in the first determination period (step S210).

[0079] For example, if the first determination period is 7 days and each first calculation period is 1 day, and the anomaly determination unit 46 determines that "the frequency value of the first calculation period is only above the first threshold" if the frequency value is above the first threshold for only 1 or 2 days within the 7-day period. That is, if the anomaly determination unit 46 detects a frequency value change trend (a change trend of missing values) such as the frequency value temporarily being above the first threshold for 1 day and then falling below the first threshold a few days later, it determines that "the frequency value of the first calculation period is only above the first threshold". The anomaly determination unit 46 may also determine that "the frequency value of the first calculation period is only above the first threshold" if the frequency value of the first calculation period is above the first threshold in less than a predetermined proportion of the multiple first calculation periods included in the first determination period. Conversely, if there are no days where the frequency value is above the first threshold, and the frequency value is above the first threshold every day within the 7-day period, the anomaly determination unit 46 does not determine that "the frequency value of the first calculation period is only above the first threshold".

[0080] The first threshold is determined based on past frequency values ​​of missed inspections. For example, the first threshold may also be determined based on the average frequency values ​​of missed inspections by drivers of vehicle M over a specified period in the past. For example, the first threshold may be a value larger than the average of past frequency values. Specifically, the first threshold may also be a value obtained by adding or multiplying the average of past frequency values ​​by a specified value. For example, the calculated frequency value may be stored (accumulated) in the storage unit 70 each time it is calculated in step S208. Furthermore, the anomaly determination unit 46 may also calculate the first threshold based on past frequency values ​​stored in the storage unit 70.

[0081] If the anomaly determination unit 46 determines that "the frequency value of a portion of the first calculation period is above the first threshold" (step S210: "Yes"), it determines that the driver has experienced a first anomaly (step S212). If the anomaly determination unit 46 does not determine that "the frequency value of a portion of the first calculation period is above the first threshold" (step S210: "No"), it determines that the driver has not experienced a first anomaly (step S214).

[0082] When a driver exhibits the first abnormality (accumulated fatigue or its likelihood), the frequency of missed glances increases compared to when the first abnormality is absent. However, fatigue can be alleviated through sleep, etc. Therefore, the increase in the frequency of missed glances is only temporary, and it is expected that the frequency will return to normal (i.e., to the same level as the previous frequency) after one or several days. The first treatment described above can detect the first abnormality by detecting such a temporary increase in the frequency of missed glances.

[0083] In step S216, the anomaly determination unit 46 calculates the frequency value of the missed information based on the excluded history information for each of the multiple second calculated periods included in the second determination period.

[0084] Here, the "second determination period" is the period used to determine the presence or absence of a second anomaly. The second determination period may be preset, for example. The second determination period may also be longer than the first determination period. For example, the second determination period may range from one month to several years. However, the length of the second determination period can be appropriately changed.

[0085] The "second calculation period" is the period used to calculate the frequency value for detecting omissions in the determination of the presence or absence of a second anomaly. The second calculation period is shorter than the second determination period. The second calculation period is, for example, preset. Multiple second calculation periods can also be periods obtained by dividing the second determination period into a predetermined number of periods. For example, if the second determination period is one month (31 days), the second determination period can also include thirty-one second calculation periods. In this case, each second calculation period is one day. However, the number of second calculation periods included in the second determination period and the length of each second calculation period can be appropriately changed.

[0086] Next, the anomaly determination unit 46 determines whether the frequency value is above the second threshold during all the second calculation periods included in the second determination period (step S218).

[0087] For example, if the second determination period is 31 days and each second calculation period is 1 day, and the anomaly determination unit 46 determines that "the frequency value is above the second threshold for all second calculation periods" if the frequency value is above the second threshold for each day during the 31-day period. That is, if the anomaly determination unit 46 determines that "the frequency value is above the second threshold for all second calculation periods" if the frequency value remains above the second threshold throughout the second calculation periods, then the anomaly determination unit 46 does not determine that "the frequency value is above the second threshold for all second calculation periods" if there are days with a frequency value less than the second threshold.

[0088] The second threshold is determined based on past frequency values ​​of missed inspections. For example, the second threshold may also be determined based on the average frequency values ​​of missed inspections by drivers of vehicle M over a specified period. For example, the second threshold may be a value larger than the average of past frequency values. Specifically, the second threshold may also be a value obtained by adding or multiplying the average of past frequency values ​​by a specified value. For example, the calculated frequency value may be stored (accumulated) in the storage unit 70 each time a frequency value is calculated in step S216. Furthermore, the anomaly determination unit 46 may also calculate the second threshold based on past frequency values ​​stored in the storage unit 70. The second threshold may be the same as the first threshold or a different value.

[0089] If the anomaly determination unit 46 determines that "the frequency value during the entire second calculation period is above the second threshold" (step S218: "Yes"), it determines that the driver has experienced a second anomaly (step S220). If the anomaly determination unit 46 does not determine that "the frequency value during the entire second calculation period is above the second threshold" (step S218: "No"), it determines that the driver has not experienced a second anomaly (step S222).

[0090] When a driver exhibits a second anomaly (visual field defects such as narrowing or the possibility thereof), the frequency of missed vision increases compared to the absence of this anomaly. Visual field defects, unlike fatigue, do not resolve, therefore the increase in the frequency of missed vision is expected to continue. The second treatment described above can detect the second anomaly by detecting the continuation of this increase in the frequency of missed vision.

[0091] In step S224, the anomaly determination unit 46 calculates the frequency value of the missed information based on the exclusion history information for each of the multiple third calculation periods included in the third determination period.

[0092] Here, the "third determination period" is the period used to determine the presence or absence of a third anomaly. The third determination period may be preset, for example. The third determination period may also be longer than the first determination period. The third determination period may be, for example, from one month to several years. However, the length of the third determination period can be appropriately varied. The third determination period may be the same as or different from the second determination period.

[0093] The "third calculation period" is the period used to calculate the frequency value for detecting omissions in the determination of the presence or absence of a third anomaly. The third calculation period is shorter than the third determination period. The third calculation period is, for example, preset. Multiple third calculation periods can also be obtained by dividing the third determination period into a predetermined number of periods. For example, if the third determination period is one month (31 days), the third determination period can also contain thirty-one third calculation periods. In this case, each third calculation period is one day. However, the number of third calculation periods included in the third determination period and the length of each third calculation period can be appropriately changed. The third calculation period can be the same as or different from the second calculation period.

[0094] Next, the anomaly determination unit 46 determines whether the change value of the frequency value in the multiple third calculation periods included in the third determination period is above the third threshold (step S226).

[0095] Here, "variance of frequency values" refers to the magnitude of the change in frequency values ​​over multiple third calculation periods, with a larger value indicating a greater change in frequency values. The variation value can also be defined, for example, as the difference between the maximum and minimum frequency values ​​over multiple third calculation periods. However, the definition of the variation value is not limited to this and can be appropriately modified. For example, the variation value can also be the variance, standard deviation, etc., of the frequency values ​​over multiple third calculation periods.

[0096] The third threshold can be predetermined, for example. Alternatively, the third threshold can be determined based on the variation of past frequency values. In this case, the third threshold can also be a value larger than the variation of past frequency values. Specifically, the third threshold can also be a value obtained by adding or multiplying the variation of past frequency values ​​by a predetermined value. For example, the calculated frequency value can be stored (accumulated) in the storage unit 70 each time a frequency value is calculated in step S224. Furthermore, the anomaly determination unit 46 can also calculate the variation of past frequency values ​​and the third threshold based on the past frequency values ​​stored in the storage unit 70. Alternatively, the calculated variation value can be stored (accumulated) in the storage unit 70 each time the anomaly determination unit 46 calculates a variation value. Furthermore, the anomaly determination unit 46 can also calculate the third threshold based on past variation values ​​stored in the storage unit 70. The third threshold can be the same as the first threshold and the second threshold, or it can be a different value.

[0097] If the anomaly determination unit 46 determines that "the change in frequency values ​​during multiple third calculation periods is greater than or equal to the third threshold" (step S226: "Yes"), it determines that the driver has experienced a third anomaly (step S228). If the anomaly determination unit 46 does not determine that "the change in frequency values ​​during multiple third calculation periods is greater than or equal to the third threshold" (step S226: "No"), it determines that the driver has not experienced a third anomaly (step S230).

[0098] The inventors of this application have conducted intensive research and found that, in cases where the driver exhibits a third abnormality (cognitive impairment or its possibility), there is a tendency for the difference in the frequency of missed glances to increase between days with high and low frequency of missed glances, compared to cases without the third abnormality. The third processing described above can detect the third abnormality by detecting this increased difference in the frequency of missed glances.

[0099] After the first, second, and third processing are completed, the anomaly determination unit 46 generates anomaly determination information based on the above determination results (step S232). That is, the anomaly determination unit 46 generates anomaly determination information based on the determination results relating to the presence or absence of the first anomaly, the presence or absence of the second anomaly, and the presence or absence of the third anomaly.

[0100] return Figure 7The notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information. For example, the notification control unit 54 sends the notification information to the first terminal device T1 (notification unit 84) via the communication unit 60. For example, the notification control unit 54 may also send the notification information to the first terminal device T1 if the anomaly determination information includes at least one of information indicating that the driver has a first anomaly, information indicating that the driver has a second anomaly, and information indicating that the driver has a third anomaly (i.e., if the anomaly determination unit 46 determines that the driver has experienced at least one of the first to third anomalies). The notification information may include, for example, information indicating the driver's anomaly (first anomaly, second anomaly, third anomaly). The notification control unit 54 may also not send the notification information to the first terminal device T1 if the anomaly determination information includes all of the information indicating that the driver does not have a first anomaly, information indicating that the driver does not have a second anomaly, and information indicating that the driver does not have a third anomaly (i.e., if the anomaly determination unit 46 determines that the driver has not experienced any of the first, second, and third anomalies).

[0101] The first terminal device T1 (notification unit 84) outputs a notification based on notification information. The notification may also be made by displaying a string, image, or other object obtained based on the notification information on a display device of the first terminal device T1 (notification unit 84). The first terminal device T1 (notification unit 84) may also issue a notification corresponding to the content of the driver's abnormality (first abnormality, second abnormality, third abnormality). For example, if the notification information includes information indicating a first abnormality, the first terminal device T1 (notification unit 84) may also issue a notification indicating that the driver has accumulated or may accumulate fatigue. This notification is an example of a "first notification." If the notification information includes information indicating a second abnormality, the first terminal device T1 (notification unit 84) may also issue a notification indicating that the driver has or may have visual field defects. This notification is an example of a "second notification." If the notification information includes information indicating a third abnormality, the first terminal device T1 (notification unit 84) may also issue a notification indicating that the driver has or may have cognitive impairment. This notification is an example of a "third notification."

[0102] The first notification may also include a reminder to improve lifestyle habits to avoid accumulating fatigue. Notifications issued by the first terminal device T1 (notification unit 84) (e.g., second and third notifications) may also include a recommendation to seek medical treatment at a medical institution (e.g., a hospital the driver frequently visits). In this case, the first terminal device T1 (notification unit 84) may also allow the driver to input information indicating whether or not they agree to the medical treatment. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a hypothetical button indicating agreement to the medical treatment. This button functions as an agreement information acquisition unit, which acquires information indicating the driver's willingness to accept the medical treatment (hereinafter referred to as agreement information). Furthermore, if the driver operates the first terminal device T1 (notification unit 84) and the agreement information acquisition unit acquires the agreement information, the first terminal device T1 (notification unit 84) may send a notification to a designated information processing terminal indicating that the driver has agreed to the medical treatment. The specified information processing terminal may also be a terminal used in a medical institution (e.g., the second terminal device T2 described later). Specifically, the specified information processing terminal may also be a terminal used by personnel belonging to the medical institution (e.g., physicians, nurses, etc.).

[0103] However, as long as the form is recognizable by the driver, the form of the notification issued by the first terminal device T1 (notification unit 84) is not particularly limited and can be appropriately changed.

[0104] The notification sent by the first terminal device T1 (notification unit 84) can also be sent at times other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. For example, the notification information sent by the notification control unit 54 can also be sent at the aforementioned times. Alternatively, the notification information sent by the notification control unit 54 can be sent at any time, and the first terminal device T1 (notification unit 84) controls the timing of the notification to ensure that the notification is sent at the aforementioned times.

[0105] According to the first embodiment described above, the information processing device 2 includes: a risk detection unit 32 that detects risky objects located around the vehicle M; an acquisition unit 20 that acquires a facial image of the driver of the vehicle M; a miss determination unit 42 that determines whether the driver has missed a risky object based on information relating to the location of the risky object and information relating to the facial image; an anomaly determination unit 46 that determines whether the driver has committed an anomaly based on the changing trend of whether or not a miss has occurred; and a notification control unit 54 that, when it is determined that the driver has committed an anomaly, causes the notification unit 84 to issue a notification. Thus, it is possible to detect driver anomalies.

[0106] <Second Implementation>

[0107] Next, the second embodiment will be described. The basic structure is the same as that of the first embodiment. Therefore, the same reference numerals are used to label the same structures and their descriptions are omitted; only the differences will be described.

[0108] Figure 10 This diagram illustrates the exception notification processing in the information processing system 100A according to the second embodiment. Figure 10 As shown, the information processing system 100A according to the second embodiment includes a second terminal device T2 in addition to the driving monitoring device 1 and the first terminal device T1.

[0109] The second terminal device T2 is an information processing device used by a user different from the driver using the first terminal device T1 (e.g., an employee of a hospital or insurance company). The second terminal device T2 may also be, for example, a smartphone, tablet, personal computer, etc.

[0110] In the second embodiment, the driving monitoring device 1 (notification control unit 54) sends the notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the second embodiment, the notification unit 84 controlled by the notification control unit 54 is the second terminal device T2, not the first terminal device T1. The second terminal device T2, for example, executes an application program for receiving notifications from the information processing device 2. The application program causes the display device to display images, strings, etc., obtained based on the information (notification information, etc.) sent by the information processing device 2, or causes the speaker to emit sound.

[0111] The second terminal device T2 uses information input by the user to notify the first terminal device T1. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0112] The input unit 92 and the transmission control unit 94 are equipped with hardware processors such as CPUs and storage devices (storage devices with non-transitory storage media) that store programs (software). The processor executes the programs to implement the functions of each component. Some or all of these components can be implemented using hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component can be implemented through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with information processing devices such as the first terminal device T1.

[0113] The input unit 92 accepts user operations and inputs them to the second terminal device T2. The input unit 92 can be configured using existing input devices such as keyboards, clicking devices (mouse, tablet, etc.), buttons, or touch panels. The input unit 92 can also be an interface for connecting an input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in the input device based on the user's operation to the second terminal device T2. The input unit 92 can be configured arbitrarily as long as it is capable of inputting instructions obtained based on user operations to the second terminal device T2.

[0114] For example, the user confirms the content of the notification sent by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2, and studies the content that should be notified to the driver. The content to be notified to the driver may include, for example, recommendations to seek medical attention at a medical institution for the purpose of detecting the possibility of cognitive impairment, visual field defects, or other diseases (abnormalities). The user inputs information, including the studied notification content, into the second terminal device T2 via the operation input unit 92. The transmission control unit 94 transmits the information input by the user to the first terminal device T1 via the communication unit 96. The first terminal device T1 then notifies the driver based on the transmitted information.

[0115] The information processing system 100A can also be configured to restrict the transmission of notification information from the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 by the driver operating the first terminal device T1 or the driving monitoring device 1. For example, the driver can operate the first terminal device T1 or the driving monitoring device 1 to generate restriction information indicating the presence or absence of such restriction on the transmission of notification information, and store the generated restriction information in the storage unit 70. Furthermore, the alarm control unit 52 can also decide whether to send notification information to the second terminal device T2 based on the restriction information stored in the storage unit 70. With this structure, the decision on whether to inform the user of the second terminal device T2 of any abnormal information involving the driver is made according to the driver's wishes. Therefore, the driver's privacy is easily protected.

[0116] According to the second embodiment described above, the information processing system 100A includes: an information processing device 2; a first terminal device T1, which is used by the driver and is a notification unit 84 that is not controlled by the notification control unit 54; and a second terminal device T2, which is used by a user different from the driver and is also controlled by the notification control unit 54 to notify the first terminal device T1, wherein the second terminal device T2 causes the first terminal device T1 to make a notification based on information input by the user's operation. Thus, it is possible to provide the driver using the first terminal device T1 with notification content obtained from research conducted by the user using the second terminal device T2.

[0117] <Variation Example>

[0118] The scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0119] For example, in the above embodiment, the risk detection unit 32 detects risky objects by image analysis of the surrounding image, but the structure of the risk detection unit 32 is not limited to this. For example, various sensors installed in the vehicle M, etc., and the ECU of the vehicle M that manages them may be connected to the network NW, and the risk detection unit 32 may detect risky objects based on the signals output from these sensors. Examples of sensors include radar devices, LIDAR (Light Detection and Range) devices, etc. The method by which the risk detection unit 32 detects risky objects is not limited to these examples and can be appropriately modified.

[0120] The driver monitoring device 1 (information processing device 2) may also perform other alarm processing (hereinafter referred to as auxiliary alarm processing) in addition to the aforementioned omission alarm processing. Auxiliary alarm processing may include, for example, collision alarm processing that detects the possibility of a collision between a risky object and vehicle M based on the position and relative speed of the risky object relative to vehicle M and triggers an alarm from alarm unit 82. In collision alarm processing, the relative speed of the risky object relative to vehicle M may be calculated, for example, by the processing unit 40 analyzing the time change of the risky position detected by the risk detection unit 32. Alternatively, when the processing unit 40 detects the possibility of a collision between a risky object and vehicle M, the alarm control unit 52 triggers an alarm from alarm unit 82. Auxiliary alarm processing may also be performed during the driver's driving of vehicle M (i.e., in real time), similar to omission alarm processing. In the configuration where auxiliary alarm processing is performed in the driver monitoring device 1 (information processing device 2), the alarm from alarm unit 82 based on the omission determination result of omission determination unit 42 may not be executed. Even in this case, the omission determination performed by the omission determination unit 42 is executed in order to perform the above-mentioned abnormal notification processing.

[0121] Alternatively, information relating to the results of the omission determination unit 42 in determining omissions may be collected in the storage unit 70 (e.g., determination history information 72), and the first terminal device T1 (alarm unit 82) may provide the driver with entries (hereinafter referred to as "review entries") generated based on the collected information for the driver to review their past driving. Review entries may include, for example, information (strings, images, etc.) that the driver can identify as potentially overlooked objects. Review entries may be generated by the driving monitoring device 1 (information processing device 2) or by the first terminal device T1 (alarm unit 82).

[0122] In the information processing system 100A according to the second embodiment, the driving monitoring device 1 (notification control unit 54) can also send notification information to both the first terminal device T1 and the second terminal device T2. In other words, the notification unit 84 that controls the notifications from both the first terminal device T1 and the second terminal device T2 can also be controlled by the notification control unit 54.

[0123] In the first embodiment, the driver monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as different devices, but they can also be configured as an integrated device. For example, the driver monitoring device 1 may actually include all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, alarm unit 82, and notification unit 84. Alternatively, all these functions may actually be included in the first terminal device T1 or the vehicle M. Similarly, in the second embodiment, all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and alarm unit 82 may actually be included in the driver monitoring device 1, and all these functions may also be included in the first terminal device T1. A camera other than the camera included in the driver monitoring device 1 may also perform some or all of the functions of the camera unit 10. For example, a driver monitoring camera that captures the driver may capture facial images.

[0124] The driving monitoring device 1 can also be actually installed using multiple information processing devices. For example, the functions of the information processing device 2 can also be actually installed in the first terminal device T1 or the vehicle M (e.g., ECU). Alternatively, the driving monitoring device 1 can be actually installed using a device such as a cloud. For example, in the driving monitoring device 1, the detection unit 30, processing unit 40, control unit 50, and storage unit 70 can also be actually installed in different information processing devices. For example, the storage unit 70 can also be distributed among multiple information processing devices.

[0125] The implementation methods and variations described above can be represented as follows.

[0126] An information processing device comprising:

[0127] Storage medium, which stores computer-readable instructions; and

[0128] The processor, which is connected to the storage medium,

[0129] The processor performs the following processing by executing computer-readable instructions:

[0130] Detect hazardous objects located around the vehicle;

[0131] Obtain the facial image of the driver of the vehicle;

[0132] The driver is determined to have missed seeing the risky object based on information relating to the location of the risky object and information relating to the facial image.

[0133] The driver's abnormality is determined based on the changing trend of whether or not the leak has occurred; and

[0134] If it is determined that the driver has malfunctioned, the notification unit will issue a notification.

[0135] Without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements, and the above embodiments and variations can also be appropriately combined.

Claims

1. An information processing device, wherein, The information processing device includes: The risk detection department detects risky objects located around the vehicle; The acquisition unit acquires a facial image of the driver of the vehicle; The omission determination unit determines whether the driver has missed the risky object based on information relating to the location of the risky object and information relating to the facial image; The anomaly determination unit determines whether the driver has experienced an anomaly based on the changing trend of whether or not the leak has been detected. as well as The notification control unit will issue a notification if it determines that the driver has experienced an abnormality.

2. The information processing apparatus according to claim 1, wherein, The anomaly determination unit calculates the frequency value of the missed checks for each of the plurality of first calculation periods included in the first determination period. If the frequency value in any of the first calculation periods, only a portion of the plurality of first calculation periods, is above a first threshold determined based on past frequency values ​​of missed checks, the unit determines that the driver has experienced a first anomaly. If the notification control unit determines that the driver has experienced a first abnormality, it will issue a first notification.

3. The information processing apparatus according to claim 1 or 2, wherein, The anomaly determination unit calculates the frequency value of the missed checks for each of the plurality of second calculation periods included in the second determination period. If the frequency value for all of the plurality of second calculation periods is above a second threshold determined based on past frequency values ​​of missed checks, the unit determines that the driver has experienced a second anomaly. If the notification control unit determines that the driver has experienced a second abnormality, it will issue a second notification.

4. The information processing apparatus according to claim 1 or 2, wherein, The anomaly determination unit calculates the frequency value of the missed detection for each of the plurality of third calculation periods included in the third determination period. If the change value of the frequency value in the plurality of third calculation periods is greater than or equal to a third threshold, it determines that the driver has experienced a third anomaly. If the notification control unit determines that the driver has experienced a third abnormality, it will issue a third notification.

5. The information processing apparatus according to claim 1 or 2, wherein, The anomaly determination unit determines the period during which the driver is drowsy based on the facial image, and determines whether the driver has experienced an anomaly based on the presence or absence of the missed information during periods other than the determined drowsy period.

6. The information processing apparatus according to claim 5, wherein, The anomaly determination unit determines the period of drowsiness based on the number of times the driver's eyelids open and close and / or the degree of eyelid opening detected from the facial image.

7. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to claim 1 or 2; A first terminal device, which is used by the driver and is not the notification unit controlled by the notification control unit; as well as The second terminal device is used by a user different from the driver, and the notification unit is controlled by the notification control unit. The second terminal device causes the first terminal device to make a notification based on information input by the user through the user's operation.

8. An information processing method, wherein, The information processing method causes the computer to perform the following processes: Detect hazardous objects located around the vehicle; Obtain the facial image of the driver of the vehicle; The driver is determined to have missed seeing the risky object based on information relating to the location of the risky object and information relating to the facial image. The driver is determined to have experienced an abnormality based on the changing trend of whether or not the leak has been detected. as well as If it is determined that the driver has malfunctioned, the notification unit will issue a notification.

9. A storage medium that stores a program and is a non-transitory storage medium readable by a computer, wherein, The program is used to cause the computer to perform the following processes: Detect hazardous objects located around the vehicle; Obtain the facial image of the driver of the vehicle; The driver is determined to have missed seeing the risky object based on information relating to the location of the risky object and information relating to the facial image. The driver is determined to have experienced an abnormality based on the changing trend of whether or not the leak has been detected. as well as If it is determined that the driver has malfunctioned, the notification unit will issue a notification.

Images