Information processing apparatus, information processing system, information processing method, and storage medium

Abstract

An information processing apparatus, an information processing system, an information processing method, and a storage medium capable of detecting an abnormality of a driver are provided. The information processing apparatus includes an acquisition unit that acquires, in a time series, face images that are images including a face of a driver of a vehicle, an extraction unit that extracts, in each of the face images, a feature amount of the face of the driver, and an abnormality determination unit that determines whether or not the driver has an abnormality based on a change tendency of the feature amount.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and storage medium. Background Technology

[0002] In recent years, efforts have intensified to provide sustainable transportation systems that also consider vulnerable groups among transportation participants, particularly the elderly, people with disabilities, and children. Research and development, especially related to the development of mobility facilities for the elderly and people with disabilities, are particularly focused on further improving the safety and convenience of transportation. (Japanese Patent Application Publication No. 2020-71528, Japanese Patent Application Publication No. 2017-16568) Summary of the Invention

[0003] However, in cases of cognitive impairment, there is a tendency for frequent and drastic mood swings, and even in dangerous situations, a tendency for emotions to remain largely unchanged (inattentiveness). These tendencies may also be observed in drivers with cognitive impairment. When drivers continue driving without realizing the onset of cognitive impairment, traffic safety may be compromised. Therefore, it is important to routinely monitor drivers' emotions (facial expressions) and detect and report any decline in cognitive ability at an early stage.

[0004] This invention was made in consideration of such circumstances, and one of its objectives is to provide an information processing device, information processing system, information processing method, and storage medium capable of detecting abnormalities in drivers, thereby contributing to the development of sustainable transportation systems.

[0005] [Methods used to solve problems]

[0006] The information processing device, information processing system, information processing method, and storage medium involved in this invention adopt the following structure.

[0007] (1): One aspect of the present invention relates to an information processing apparatus, wherein the information processing apparatus comprises: an acquisition unit that acquires a facial image in a time sequence, the facial image being an image including the face of a driver of a vehicle; an extraction unit that extracts feature quantities of the driver's face based on the facial image; and an anomaly determination unit that determines whether the driver has experienced an anomaly based on the changing trend of the feature quantities.

[0008] (2): In the above (1) scheme, the extraction unit extracts the feature amount of the driver's face from each of the facial images obtained in time sequence.

[0009] (3): In the above (2) scheme, the information processing device further includes an export unit that exports the change between a first feature quantity and a second feature quantity. The first feature quantity is extracted from the facial image obtained at a first time point, and the second feature quantity is extracted from the facial image obtained at a second time point that is closer to the first time point. The anomaly determination unit determines whether the driver has experienced an anomaly based on the change quantity.

[0010] (4): In the above (3) scheme, the derivation unit calculates the change amount for a group of two consecutive facial images in the facial images obtained in a time sequence, and the anomaly determination unit determines whether the driver has an anomaly based on the number of the number of changes that are above a threshold among the calculated multiple changes.

[0011] (5): In the above (4) scheme, the exporting unit calculates the change amount for each of the two consecutive sets of the facial images.

[0012] (6): In any of the above schemes (3) to (5), the acquisition unit also acquires the surrounding image of the vehicle, the information processing device further includes a risk detection unit that detects risky objects in the surrounding image, and the anomaly determination unit determines that the driver has an anomaly when no risky object is detected and the change amount corresponding to the time point of acquiring the surrounding image is above a threshold.

[0013] (7): In any of the above schemes (3) to (5), the acquisition unit also acquires the surrounding image of the vehicle, the information processing device further includes a risk detection unit that detects a risky object in the surrounding image, and the anomaly determination unit determines that the driver has an anomaly when it detects the risky object and the amount of change corresponding to the time point of acquiring the surrounding image is less than a threshold.

[0014] (8): In any of the above schemes (1) to (7), the information processing device further includes a notification control unit, which causes the notification unit to make a notification when the abnormality determination unit determines that the driver has an abnormality.

[0015] (9): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing device of any of the above (1) to (8); a first terminal device used by the driver; and a second terminal device used in a medical institution, wherein the information processing device, upon determining that the driver has an abnormality, causes the first terminal device to advise the driver to seek medical treatment at the medical institution, and the information processing device, based on information input by the driver to the first terminal device indicating acceptance of the medical treatment, causes the second terminal device to make a notification.

[0016] (10): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing device of any of the above (1) to (8); a first terminal device used by the driver; and a second terminal device used by a user different from the driver, the second terminal device being controlled and notified by the information processing device, the second terminal device causing the first terminal device to make a notification based on information input by the user's operation.

[0017] (11): One aspect of the present invention relates to an information processing method, wherein the information processing method causes a computer to perform the following processing: acquiring a facial image in a time sequence, the facial image being an image of the face of a driver of a vehicle; extracting feature quantities of the driver's face based on the facial image; and determining whether the driver has experienced an abnormality based on the changing trend of the feature quantities.

[0018] (12): One aspect of the present invention relates to a storage medium storing a program, wherein the program causes a computer to perform the following processing: acquiring a facial image in a time sequence, the facial image being an image including the face of a driver of a vehicle; extracting feature quantities of the driver's face based on the facial image; and determining whether the driver has experienced an abnormality based on the changing trend of the feature quantities.

[0019] [Invention Effects]

[0020] According to (1)-(12), the above-described scheme of the present invention can detect the reduction of the driver's cognitive ability. Attached Figure Description

[0021] Figure 1 This is a diagram illustrating an example of the structure and usage environment of a driving monitoring device that includes the information processing apparatus according to the first embodiment.

[0022] Figure 2 This is a diagram illustrating an example of the driving monitoring device according to the first embodiment.

[0023] Figure 3 This is a diagram showing an example of an image obtained by the driving monitoring device according to the first embodiment.

[0024] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device according to the first embodiment.

[0025] Figure 5 This is a diagram showing an example of information detected by the detection unit according to the first embodiment.

[0026] Figure 6 This diagram illustrates an example of an alarm triggered by the alarm unit according to the first embodiment.

[0027] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device according to the first embodiment.

[0028] Figure 8 This is a diagram illustrating an example of the change information involved in the first embodiment.

[0029] Figure 9 This is a diagram illustrating an example of the relationship between the amount of change and the frequency involved in the first embodiment.

[0030] Figure 10 This is a flowchart illustrating an example of the processing flow performed by the information processing apparatus according to the first embodiment.

[0031] Figure 11 This is a flowchart illustrating an example of the processing flow performed by the information processing apparatus according to the first embodiment.

[0032] Figure 12 This diagram illustrates the exception notification processing in the information processing system according to the second embodiment. Detailed Implementation

[0033] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the rearward direction of the vehicle is defined as the negative X direction, the rightward direction of the vehicle's width relative to the positive X direction is defined as the positive Y direction, the leftward direction is defined as the negative Y direction, and the direction orthogonal to the X and Y directions and the height direction of the vehicle is defined as the positive Z direction.

[0034] <First Implementation>

[0035] [Overall Structure]

[0036] Figure 1This diagram illustrates an example of the structure and usage environment of a driver monitoring device 1, including the information processing device 2 according to the first embodiment. The information processing device 2 is, for example, a driver monitoring device 1 retrofitted into a vehicle (hereinafter referred to as "vehicle M") such as a dashcam. That is, the driver monitoring device 1 (information processing device 2) can be mounted and removed from the vehicle M via a mounting / unmounting unit (hereinafter referred to as "mounting / unmounting unit 16"). The information processing device 2 is, for example, a device not connected to the vehicle M's in-vehicle network. An in-vehicle network refers to an in-vehicle network that connects to the vehicle's control-related ECU (Electronic Control Unit), which uses communication standards such as CAN (Controller Area Network), and is connected to the vehicle M's drive, braking, steering, various vehicle sensors, driving-related operating components, and operation buttons. The vehicle M's network for HMI (Human Machine Interface) can also be excluded from the above-described in-vehicle network.

[0037] Figure 2 This diagram illustrates an example of a driver monitoring device 1. The driver monitoring device 1 is, for example, installed near the windshield of a vehicle M, and located near the front of the driver's seat. The driver monitoring device 1 stores images (moving images) captured by the camera unit 10, for example, in the storage unit 70 (described later). The driver monitoring device 1 may also be, for example, equipped with a display device such as an LCD (Liquid Crystal Display) and capable of displaying the images (moving images) stored in the storage unit 70 on the display device.

[0038] The driving monitoring device 1 includes, for example, a camera unit 10, a loading and unloading unit 16, and an information processing unit 2. In the illustrated example, the camera unit 10 includes a front camera 12 and a rear camera 14.

[0039] The front camera 12 and the rear camera 14 are each, for example, digital cameras utilizing solid-state imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) captures images of the area in front of the vehicle M from the location where the driver monitoring device 1 is installed. The rear camera 14 repeatedly (periodically) captures images of the area behind the vehicle M and the interior of the vehicle M from the location where the driver monitoring device 1 is installed. The horizontal viewing angle captured by the front camera 12 and the rear camera 14 is, for example, 180° or more. That is, the driver monitoring device 1 (camera unit 10) captures a 360° range around the vehicle M from its installation location. The front camera 12 and the rear camera 14 may each be, for example, fisheye cameras including fisheye lenses. Alternatively, the front camera 12 and the rear camera 14 may each be wide-angle cameras or the like. In this embodiment, the use of the front camera 12 and the rear camera 14 is described, but other cameras may be used instead of these or other alternatives. That is, the camera unit 10 can be configured to capture images used in this embodiment (peripheral images and facial images described later).

[0040] The mounting / unmounting part 16 is, for example, a component for mounting the driver monitoring device 1 onto the vehicle M. The mounting / unmounting part 16 can be, for example, any supporting component such as a suction cup, a seal, or a bracket. The mounting / unmounting part 16 may also include a power supply cable (wiring, etc.) for supplying power to the driver monitoring device 1.

[0041] Figure 3 This is a diagram illustrating an example of an image acquired by the driver monitoring device 1. That is, Figure 3 This diagram illustrates an example of an image captured by the camera unit 10. The front camera 12 captures images of the front of the vehicle M through the windshield, the driver's side side window, and the passenger side side window. The rear camera 14 captures images of the interior of the vehicle M (including the driver and passengers) and the rear of the vehicle M through the driver's side side window, the passenger side side window, and the rear window.

[0042] In the image captured by the front camera 12 (hereinafter referred to as the "front image"), the subject includes objects such as other vehicles, pedestrians, bicycles, fixed objects, road markings, etc., that can be seen through the front windshield and exist in front of the vehicle M, as well as objects that can be seen through the left and right (front) side windows of the vehicle M.

[0043] The image captured by the rear camera 14 (hereinafter referred to as the "rear image") shows the interior of the vehicle M, as well as objects visible through the rear window that are located behind the vehicle M, and objects visible through the left and right side windows that are located to the left and right (rear) of the vehicle M. Therefore, in the rear image captured by the rear camera 14, the driver of the vehicle M equipped with the driver monitoring device 1 is also reflected as the subject.

[0044] The front and rear images include images of the surrounding conditions of the vehicle M (hereinafter referred to as "surrounding images"). The rear image includes an image of the driver's face of the vehicle M (hereinafter referred to as "face image"). The driver monitoring device 1 can be installed at any location where the camera unit 10 can capture the surrounding images and the face image.

[0045] Information processing device 2 (refer to) Figure 1 Based on the results obtained by referring to the surrounding images and facial images output by the camera unit 10, the alarm and notification are output via the first terminal device T1. Sometimes the driver monitoring device 1 and the first terminal device T1 are combined and referred to as "information processing system 100".

[0046] The first terminal device T1 is, for example, a mobile terminal device used by a driver driving a vehicle M equipped with a driver monitoring device 1, such as a smartphone or tablet. The first terminal device T1 executes, for example, an application for receiving alarms and notifications from the information processing device 2. The application causes a display device to show an image based on information sent by the information processing device 2 (e.g., alarm information and notification information described later), and causes a speaker to emit sound. The first terminal device T1 is also referred to as the alarm unit 82 and the notification unit 84. The first terminal device T1 is used, for example, in a state where it can be detachably mounted on the vehicle M. For example, a bracket for the first terminal device T1 with a mounting and detaching mechanism is provided on one or both of the first terminal device T1 and the vehicle M, and the first terminal device T1 is supported by the bracket.

[0047] Alternatively, the navigation, display, and speaker of vehicle M can be used instead of the first terminal device T1. That is, the navigation, display, and speaker of vehicle M can also output alarms and notifications based on the instructions of information processing device 2. That is, the navigation, display, and speaker of vehicle M can also be alarm unit 82 and notification unit 84. Alternatively, the display unit (alarm unit 82, notification unit 84) and speaker (alarm unit 82, notification unit 84) can be provided on the driver monitoring device 1 instead of the first terminal device T1.

[0048] like Figure 1As shown, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, and a face orientation detection unit 36. The processing unit 40 includes a miss detection unit 41, an extraction unit 42, an export unit 43, a storage control unit 44, and an anomaly detection unit 45. The control unit 50 includes an alarm control unit 52 and a notification control unit 54.

[0049] The acquisition unit 20, detection unit 30, processing unit 40, and control unit 50 are equipped with hardware processors such as CPUs (Central Processing Units) and storage devices (storage devices with non-transitory storage media) storing programs (software). The processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented using hardware such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units) (including the circuitry). The functions of each component can also be realized through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs.

[0050] The program (software) can be pre-stored in the storage unit 70 (a storage device with a non-temporary storage medium) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it can be stored in a removable storage medium (a non-temporary storage medium) such as a memory card, and installed in the storage device by assembling the storage medium into the driving monitoring device 1. The program (software) can also be pre-downloaded from other computer devices via short-range communication or wide-area communication using an application program executed in the first terminal device T1, and then sent from the first terminal device T1, thereby being installed in the storage device.

[0051] The storage unit 70 stores, for example, change information 72, programs (such as the programs described above), and various other information. The storage unit 70 can also be implemented using the various storage devices described above or EEPROM (Electrically Erasable Programmable Read Only Memory). The communication unit 60 is an interface for wireless or wired communication with information processing devices (external devices) such as the first terminal device T1.

[0052] The driver monitoring device 1 (information processing device 2) performs at least two functions: a missed detection alarm and an anomaly notification. The missed detection alarm is a process that detects a driver's missed detection of a risky object based on surrounding images and facial images, and triggers an alarm at the alarm unit 82. The anomaly notification is a process that detects an anomaly (reduction in cognitive ability) in the driver based on surrounding images and facial images, and triggers a notification at the notification unit 84.

[0053] A risky object refers to an object, for example, that vehicle M should avoid. A risky object is, for example, an object that would interfere with vehicle M when it is moving as it is, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose approach level becomes more than a certain level after a specified time, taking into account the object's position, direction of movement, and speed, as well as the position, method of movement, and speed of vehicle M. The object can also be a traffic participant such as a pedestrian, bicycle, or vehicle, or an object other than a traffic participant such as an object placed on the road or a fallen object.

[0054] [Handling of Missing Alarms]

[0055] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device 1 (information processing device 2).

[0056] The acquisition unit 20 acquires front and rear images from the imaging unit 10. Therefore, the acquisition unit 20 establishes a link between the peripheral images and the facial images to acquire these images. The imaging unit 10 repeatedly (periodically) captures the peripheral images and facial images, thus the acquisition unit 20 repeatedly (periodically) acquires these images. The acquisition unit 20 can also establish a link between the peripheral images and facial images and the time they were captured (the shooting time) to acquire these images.

[0057] The risk detection unit 32 detects risky objects in the surrounding image. "Detecting risky objects" refers to, for example, detecting the presence of a risky object and the position of that object relative to the vehicle M (driver monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M can be pre-stored in the storage unit 70. The risk detection unit 32 can also detect the position of the risky object relative to the vehicle M based on this reference information.

[0058] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, and also detects the direction of the risk object's presence (approach direction) relative to the vehicle M. "Direction of presence of the risk object" refers to, for example, the direction of presence of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, "direction of presence of the risk object" will sometimes be referred to as "risk presence direction".

[0059] The gaze detection unit 34 detects the driver's gaze in a facial image. Specifically, the gaze detection unit 34 detects the driver's gaze by analyzing the facial image using a prescribed image analysis method. "Gaze detection" refers to, for example, detecting the position of the driver's eyeballs and the direction of the gaze (the direction of the eyeballs).

[0060] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the facial image. Specifically, the face orientation detection unit 36 ​​detects the orientation of the driver's face by parsing the facial image using a prescribed image parsing method. The face orientation detection unit 36 ​​may also detect the position of the driver's face in addition to its orientation. Hereinafter, the information combining "face orientation" and "face position" is sometimes referred to as "facial information." The face orientation detection unit 36 ​​detects facial information in the facial image.

[0061] The detection unit 30 can also be configured to detect lanes, pedestrian crossings, etc., in addition to the aforementioned risk objects, lines of sight, and facial information. Figure 5 This is a diagram illustrating an example of information detected by the detection unit 30. In Figure 5 In the example, the pedestrians, other vehicles, lanes, and crosswalks detected by the detection unit 30 are shown based on the vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that may become risk objects. In the detection information, instead of the vehicle M, the direction of the line of sight, and the direction of the face, the traffic participants (objects) that may become risk objects detected by the detection unit 30 may be shown based on the vehicle M, the direction of the vehicle M, and / or the direction of travel of the vehicle M.

[0062] Leak detection unit 41 (refer to) Figure 4 This method determines whether a driver has missed a hazard based on the location and / or direction of the hazard, line of sight, and facial information. "Risk location" and "direction of the hazard" are each examples of "information relating to the location of the hazard." Line of sight and facial information are examples of "information relating to facial images."

[0063] For example, reference information (not shown) indicating the relationship between the orientation of the driver's eyes (gaze) and / or the orientation of their face in the facial image and their relative orientation with respect to the direction of travel of the vehicle M can also be pre-stored in the storage unit 70. The omission determination unit 41 can also refer to this reference information to determine whether the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists. If the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists, the omission determination unit 41 can also determine that the driver visually identifies a risky object. The so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can be, for example, the case where there is a risky position on a straight line along the orientation of the gaze and / or the orientation of the face. Alternatively, the so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can also be the case where there is a risky position in a cone-shaped area with the position of the viewpoint (eyeball) and / or the face as the apex and having a central axis along the orientation of the gaze and / or the orientation of the face that expands laterally as it moves forward. If the direction of the driver's gaze and / or the direction of the face does not match the location of the risky object, the omission determination unit 41 may also determine that the driver has not visually identified the risky object.

[0064] The omission determination unit 41 can also determine that the driver has missed the risk object if the state of not visually recognizing the risk object continues for a predetermined time or longer. Alternatively, the omission determination unit 41 can determine that the driver has not missed the risk object if the driver visually recognizes the risk object, or if the state of not visually recognizing the risk object is resolved before the predetermined time has elapsed. However, the method by which the omission determination unit 41 determines omissions is not limited to the above and can be appropriately modified.

[0065] The omission detection unit 41 generates omission detection information based on the above detection results. The omission detection information includes, for example, the presence or absence of omissions and the direction of risk.

[0066] The alarm control unit 52 causes the first terminal device T1 (alarm unit 82) to output an alarm based on the leak detection information. For example, the alarm control unit 52 sends alarm information to the first terminal device T1 via the communication unit 60. The alarm information includes, for example, information indicating whether an alarm is needed and the direction of the risk. For example, if the leak detection information includes information indicating "a leak exists," the alarm control unit 52 generates alarm information including information indicating "an alarm is needed." For example, if the leak detection information includes information indicating "no leak exists," the alarm control unit 52 generates alarm information including information indicating "no alarm is needed." The first terminal device T1 outputs an alarm based on the alarm information.

[0067] Figure 6This diagram illustrates an example of an alarm triggered by the first terminal device T1 (alarm unit 82). Figure 6 In the example, the first terminal device T1 displays identifiable entries indicating the direction of the risk (D1 and D2 in the diagram). Furthermore, the first terminal device T1 issues warnings by differentiating the display format of the risk direction when a risky object is missed (i.e., the warning message includes a message indicating "warning is needed") and when no risky object is missed (i.e., the warning message includes a message indicating "warning is not needed"). Specifically, when a risky object is missed, the risk direction is displayed in a more emphasized manner compared to when it is not missed. This emphasis could be based on color, flashing icons, or other visually illuminating elements to make the risk direction easier for the driver to recognize. The first terminal device T1 may also issue warnings using methods such as loudspeaker announcements or illuminated lights (A in the diagram), in addition to or instead of the above. The specific method of issuing the warning can be appropriately modified as long as it is identifiable by the driver.

[0068] The alarm issued by the first terminal device T1 (alarm unit 82) can also be issued during the time when the driver is driving the vehicle M (i.e., in real time). Examples of "the time when the driver is driving the vehicle M" include when the vehicle M is moving or when the vehicle M is temporarily stopped. For example, the alarm information sent by the alarm control unit 52 can also be issued at the aforementioned times. Alternatively, the alarm information sent by the alarm control unit 52 can be issued at any time, and the first terminal device T1 (alarm unit 82) controls the timing of the alarm to issue the alarm at the aforementioned times.

[0069] [Abnormal Notification Handling]

[0070] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device 1 (information processing device 2).

[0071] The acquisition unit 20 acquires front and rear images from the camera unit 10 in the same manner as the aforementioned missed detection alarm processing. Thus, the acquisition unit 20 obtains the surrounding images by establishing a link between the surrounding images and the facial images. Since the camera unit 10 captures the surrounding images and facial images in a time sequence, the acquisition unit 20 acquires the surrounding images and facial images in a time sequence. The acquisition unit 20 can also obtain the surrounding images and facial images by establishing a link between the surrounding images and the time they were captured (the capture time).

[0072] The extraction unit 42 extracts facial features of the driver from facial images acquired in a time series. The extraction unit 42 can also extract facial features of the driver from multiple facial images acquired in a time series, respectively. For example, the extraction unit 42 can also extract facial features of the driver by analyzing the facial images using a prescribed image analysis method. The features extracted by the extraction unit 42 can also vary based on the driver's facial expression.

[0073] The export unit 43 exports the tendency of change in the feature quantity. Specifically, the export unit 43 exports the change between a first feature quantity and a second feature quantity, the first feature quantity being extracted from a facial image acquired at a first time point, and the second feature quantity being extracted from a facial image acquired at a second time point closer to the first time point. The change between the first feature quantity and the second feature quantity is, for example, the absolute value of the difference between the first feature quantity and the second feature quantity. The change quantity can be obtained simply based on the result of comparing the first feature quantity and the second feature quantity. The first time point and the second time point can also be two temporally consecutive time points.

[0074] The risk detection unit 32 detects risky objects in the surrounding image in the same way as the aforementioned missed alarm processing. In other words, the risk detection unit 32 determines whether there are risky objects.

[0075] The storage control unit 44 stores the export time, change amount, and presence or absence of risky objects in the storage unit 70 by establishing a relationship between these information and the storage unit 70. The export time is the time when the change amount is exported. The export time can be obtained, for example, from the shooting time information obtained by linking it to the facial image. The storage control unit 44 stores, for example, the presence or absence of risky objects in the surrounding image obtained at the same time as the export time in the storage unit 70 by linking the export time and the change amount. The above-described processing performed by the storage control unit 44 is performed, for example, whenever the change amount is exported by the export unit 43. Hereinafter, the information stored in the storage unit 70 will be referred to as change amount information 72.

[0076] Figure 8 This is a diagram illustrating an example of change information 72. The storage control unit 44 repeatedly performs the aforementioned processing, thereby accumulating multiple groups (hereinafter referred to as export information groups J) in the change information 72, including the export time, change amount, and whether there are any risky objects. That is, the storage control unit 44 generates export information groups J whenever the export unit 43 exports a change. Furthermore, the storage control unit 44 updates the change information 72 by appending the generated export information groups J to the change information 72.

[0077] The anomaly determination unit 45 determines whether the driver has experienced an anomaly based on the change information 72.

[0078] First, the anomaly determination unit 45 determines whether the change derived from the feature quantities extracted at a first time point and a second time point that are sequentially consecutive in time is above a preset first threshold. The first threshold can be preset or set based on the changing trends of multiple feature quantities extracted from multiple facial images of the driver in the past. For example, the first threshold can also be set based on the change derived when it is determined that the driver has not experienced any abnormality (cognitive ability has not declined).

[0079] Next, the anomaly determination unit 45 determines whether the driver has experienced an anomaly based on the number of times the change in a specified period reaches or exceeds a first threshold. The specified period can be, for example, a single driving session or a fixed time period. A single driving session refers, for example, driving from engine start-up to shutdown. A fixed time period refers, for example, a fixed period such as 10 minutes or 1 hour. The anomaly determination unit 45 determines that the driver has experienced an anomaly if the number of times the change in a specified period reaches or exceeds the first threshold is a set number. The anomaly determination unit 45 determines that the driver has not experienced an anomaly if the number of times the change in a specified period reaches or exceeds the first threshold is less than the set number. The set number can be set based on, for example, the length of the specified period or the number of past changes reaching or exceeding the first threshold. For example, the set number can also be set based on the number of times the change reaches or exceeds the first threshold when it is determined that the driver has not experienced an anomaly (cognitive ability has not declined).

[0080] Figure 9 This is a graph illustrating the relationship between change and frequency. Regarding... Figure 9 In the chart, the vertical axis represents frequency, and the horizontal axis represents the magnitude of change. The upward direction of the vertical axis is positive, with higher values ​​indicating higher frequencies. The rightward direction of the horizontal axis is positive, with greater values ​​indicating greater magnitudes of change. When comparing the driver's facial expressions at the first and second time points, greater changes in facial expressions correspond to greater magnitudes of change, and smaller changes in facial expressions correspond to smaller magnitudes of change. The magnitude of change can be represented numerically or by levels such as high, medium, and low.

[0081] exist Figure 9 This includes the change index L. The change index L represents the correlation between the magnitude and frequency of the change during driving within a specified period. The anomaly determination unit 45 counts the occurrences of each change based on multiple derived information groups J stored during the specified period, thereby calculating the frequency and deriving the change index L. That is, the frequency represents the number of times each change is derived.

[0082] The anomaly determination unit 45 determines whether a portion or all of the change quantity index L exists in the anomaly determination region AR. The anomaly determination region AR, in the graph, is the area where the change quantity is above a first threshold V and the frequency is above a set number of times N. That is, if a portion or all of the change quantity index L exists in the anomaly determination region AR, the number of times the change quantity exceeds the first threshold V is above the set number of times N. If none of the change quantity index L exists in the anomaly determination region AR, the number of times the change quantity exceeds the first threshold V is less than the set number of times N. In other words, if a portion or all of the change quantity index L exists in the anomaly determination region AR, the anomaly determination unit 45 determines that the driver has experienced an anomaly. If none of the change quantity index L exists in the anomaly determination region AR, the anomaly determination unit 45 determines that the driver has not experienced an anomaly.

[0083] The aforementioned processing (first determination processing) performed by the anomaly determination unit 45 can also be performed at times other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. When the first determination processing is performed at times other than when the driver is driving the vehicle M, the first determination processing is performed at predetermined intervals in the multiple derived information groups J stored since the processing date of the last first determination processing. The first determination processing performed by the anomaly determination unit 45 can also be performed every predetermined interval while the driver is driving the vehicle M. When the first determination processing is performed every predetermined interval, the derived information groups J contained in the change amount information 72 can also be deleted after the first determination processing is completed. In this case, the anomaly determination unit 45 performs the first determination processing using all the derived information groups J contained in the change amount information 72.

[0084] In this way, by determining whether a driver has experienced an abnormality based on the number of times the change in a certain value exceeds a first threshold, it is possible to detect a decline in the driver's cognitive abilities. A change exceeding the first threshold refers to significant changes in facial expression. A high frequency of changes exceeding the first threshold predicts a high number of dramatic emotional fluctuations, i.e., drastic emotional swings. Patients with cognitive impairment tend to exhibit drastic emotional swings. In other words, by judging the tendency of changes in characteristic values, a decline in the driver's cognitive abilities can be detected.

[0085] The anomaly determination unit 45 can also determine whether the driver has experienced an anomaly based on the presence and change of a risky object. If the anomaly determination unit 45 determines that no risky object exists based on the derived information group J and the change is greater than or equal to the second threshold, the driver has experienced an anomaly. If the anomaly determination unit 45 determines that no risky object exists based on the derived information group J and the change is less than the second threshold, the driver has not experienced an anomaly. If the anomaly determination unit 45 determines that a risky object exists based on the derived information group J and the change is less than the third threshold, the driver has experienced an anomaly. If the anomaly determination unit 45 determines that a risky object exists based on the derived information group J and the change is greater than or equal to the third threshold, the driver has not experienced an anomaly. The second and third thresholds can be preset or set based on the changing trends of multiple feature quantities extracted from the driver's facial image in the past. The second threshold is, for example, a value larger than the first threshold. The third threshold is, for example, a value smaller than the first threshold. The second and third thresholds can be the same value or different values. The second and / or third thresholds can be the same value as the first threshold or different values.

[0086] The aforementioned processing (second determination processing) performed by the anomaly determination unit 45 can also be performed at times other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. In the case of times other than when the driver is driving the vehicle M, the anomaly determination unit 45 performs second determination processing on all the multiple exported information groups J stored since the last time the second determination processing was performed. The second determination processing performed by the anomaly determination unit 45 can also be performed whenever a change in exported information is expressed. In the case of performing second determination processing whenever a change in exported information is expressed, the anomaly determination unit 45 performs the second determination by referring to the latest exported information group J contained in the change information 72.

[0087] Patients with cognitive impairment tend to have disorientation disorders. Disorientation disorders refer to a reduced ability to recognize one's own situation. For example, significant changes in a driver's facial expressions when no dangerous object is detected (low risk), or minimal changes in facial expressions when a dangerous object is detected (high risk), may indicate a decline in the driver's cognitive ability, i.e., cognitive impairment. Therefore, determining whether a driver has experienced an abnormality based on the presence or absence of a dangerous object and the magnitude of the change can appropriately detect a decline in the driver's cognitive ability.

[0088] However, the method by which the abnormality determination unit 45 determines whether the driver has experienced an abnormality is not limited to the above, and can be appropriately modified.

[0089] The anomaly determination unit 45 generates anomaly determination information based on the determination result. This anomaly determination information may include, for example, whether the driver has experienced an anomaly, and the change rate indicator. When the first determination process and the second determination process are executed at different times, the anomaly determination unit 45 may also generate anomaly determination information based on their respective determination results. In this case, the anomaly determination information may also include whether the driver has experienced an anomaly, the change rate indicator, and the type of determination process. The anomaly determination information may also be stored in the storage unit 70.

[0090] The notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information. For example, the notification control unit 54 sends the notification information to the first terminal device T1 via the communication unit 60. For example, the notification control unit 54 may also send the first notification information to the first terminal device T1 if the anomaly determination information includes information indicating that the driver is abnormal (i.e., if the anomaly determination unit 45 determines that the driver has an abnormality). The first notification information may include, for example, information indicating driving evaluation information. The notification control unit 54 may also not send the first notification information to the first terminal device T1 if the anomaly determination information includes information indicating that the driver is not abnormal (i.e., if the anomaly determination unit 45 determines that the driver is not abnormal).

[0091] The first terminal device T1 (notification unit 84) outputs a notification based on the first notification information. The notification may also be made by displaying a string, image, or the like obtained based on the first notification information on the display device of the first terminal device T1 (notification unit 84).

[0092] The notification issued by the first terminal device T1 (notification unit 84) may also include a suggestion to seek medical treatment at a medical institution (e.g., a hospital the driver frequently visits). In this case, the first terminal device T1 (notification unit 84) may also allow the driver to input information indicating whether the driver agrees to the medical treatment. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a virtual button or the like indicating agreement to the medical treatment. Such a button or the like functions as an agreement information acquisition unit, which acquires information indicating the driver's intention to agree to the medical treatment (hereinafter referred to as agreement information). Furthermore, if the driver operates the first terminal device T1 (notification unit 84) and the agreement information acquisition unit acquires the agreement information, the notification control unit 54 or the first terminal device T1 (notification unit 84) may cause a designated information processing terminal to issue a notification indicating that the driver has agreed to the medical treatment. The designated information processing terminal may, for example, be a terminal used in a medical institution (e.g., the second terminal device T2 described later). Specifically, the specified information processing terminal may also be a terminal used by personnel belonging to a medical institution (e.g., doctors, nurses, etc.).

[0093] However, as long as the form is recognizable by the driver, the form of the notification issued by the first terminal device T1 (notification unit 84) is not particularly limited and can be appropriately changed.

[0094] The notification sent by the first terminal device T1 (notification unit 84) may occur at a time other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. For example, the first notification message sent by the notification control unit 54 may also occur at the aforementioned times. Alternatively, the first notification message sent by the notification control unit 54 may occur at any time, and the first terminal device T1 (notification unit 84) may control the timing of the notification to ensure that the notification occurs at the aforementioned times.

[0095] [Processing Flow]

[0096] Figure 10 This is a flowchart illustrating an example of the processing flow performed by the information processing device 2. Figure 10 The process shown in the flowchart begins, for example, while vehicle M is in motion.

[0097] First, the acquisition unit 20 acquires a facial image and surrounding images (step S100). Next, the extraction unit 42 extracts facial features of the driver based on the facial image (step S102). Next, the export unit 43 exports the change in the feature values ​​(step S104). Specifically, the export unit 43 exports the change between a first feature value and a second feature value, the first feature value being extracted from the facial image acquired at a first time point, and the second feature value being extracted from the facial image acquired at a second time point closer to the first time point. Next, the risk detection unit 32 detects a risky object (step S106). In other words, the risk detection unit 32 determines whether a risky object exists. Next, the storage control unit 44 generates change information 72 (step S108). The change information 72 includes the export time, the change value, and whether a risky object exists. After this, the information processing device 2 terminates. Figure 10 The flowchart shown illustrates the processing.

[0098] The order of the above processes can also be changed. For example, the process in step S106 can be performed before the process in step S100. (The following will be discussed further.) Figure 11 The flowchart shown is the same.

[0099] Figure 11 This is a flowchart illustrating an example of the processing flow performed by the information processing device 2. Figure 11 The process shown in the flowchart can begin at a predetermined time, such as after driving ends, or repeatedly at a predetermined cycle, or it can be started by the instruction of the driver, etc.

[0100] First, the anomaly determination unit 45 refers to the change amount information 72 to determine whether the number of times the change amount reaches or exceeds the first threshold during the specified period is greater than or equal to a set number (step S200). If the number of times the change amount reaches or exceeds the first threshold during the specified period is less than the set number (step S200: "No"), the anomaly determination unit 45 determines whether there is a stored export information group J where the change amount reaches or exceeds the second threshold during the specified period and it is determined that there is no risk object (step S202). If there is no stored export information group J where the change amount reaches or exceeds the second threshold and it is determined that there is no risk object (step S202: "No"), the anomaly determination unit 45 determines whether there is a stored export information group J where the change amount is less than the third threshold and it is determined that there is a risk object (step S204). If there is no stored export information group J where the change amount is less than the third threshold and it is determined that there is a risk object (step S204: "No"), the anomaly determination unit 45 determines that the driver has not experienced any anomalies (step S206).

[0101] On the other hand, if the number of times the change amount reaches or exceeds the first threshold during the specified period is set (step S200: "Yes"), if there is a set of export information group J with a change amount exceeding the second threshold and it is determined that there is no risk object (step S202: "Yes"), or if there is a set of export information group J with a change amount less than the third threshold and it is determined that there is a risk object (step S204: "Yes"), the abnormality determination unit 45 determines that the driver has an abnormality (step S208).

[0102] After the processing in step S206 or step S208 is completed, the anomaly determination unit 45 generates anomaly determination information (step S210). Next, the notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information (step S212). Afterwards, the information processing device 2 terminates. Figure 11 The flowchart shown illustrates the processing.

[0103] The processes in steps S200 to S210 and the process in step S212 described above can also be performed at different times. For example, the processes in steps S200 to S210 can be performed at predetermined intervals, and the generated anomaly determination information can be stored in the storage unit 70. Then, at a time specified by the driver, the anomaly determination unit 45 can also perform the process in step S212 with reference to the anomaly determination information.

[0104] According to the first embodiment described above, the information processing device 2 includes: an acquisition unit that acquires facial images in a time sequence, the facial images including the face of the driver of the vehicle; an extraction unit that extracts feature quantities of the driver's face from each facial image; and an anomaly determination unit that determines whether an anomaly has occurred in the driver based on the changing trend of the feature quantities. This enables the detection of a decline in the driver's cognitive abilities. For example, early notification to elderly drivers of information related to a decline in cognitive abilities can prevent accidents from happening. It can extend driving lifespan and provide a safe and secure driving experience, enabling a richer life through the extension of healthy lifespan.

[0105] <Second Implementation>

[0106] Next, the second embodiment will be described. The basic structure is the same as that of the first embodiment. Therefore, the same reference numerals are used to label the same structures and their descriptions are omitted; only the differences will be described.

[0107] Figure 12 This diagram illustrates the exception notification processing in the information processing system 100A according to the second embodiment. Figure 12As shown, the information processing system 100A according to the second embodiment includes a second terminal device T2 in addition to the driving monitoring device 1 and the first terminal device T1.

[0108] The second terminal device T2 is an information processing device used by a user different from the driver using the first terminal device T1 (e.g., an employee of a hospital or insurance company). The second terminal device T2 may also be, for example, a smartphone, tablet, personal computer, etc.

[0109] In the second embodiment, the driving monitoring device 1 (notification control unit 54) sends the first notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the second embodiment, the notification unit 84 controlled by the notification control unit 54 is the second terminal device T2 instead of the first terminal device T1. In the second terminal device T2, for example, an application program for receiving notifications from the information processing device 2 is running. The application program causes the display device to display images, strings, etc., obtained based on the information (first notification information, etc.) sent by the information processing device 2, or causes the speaker to emit sound.

[0110] The second terminal device T2 uses information input by the user to notify the first terminal device T1. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0111] The input unit 92 and the transmission control unit 94 are equipped with hardware processors such as CPUs and storage devices (storage devices with non-transitory storage media) that store programs (software). The processor executes the programs to implement the functions of each component. Some or all of these components can be implemented using hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component can be implemented through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with information processing devices such as the first terminal device T1.

[0112] The input unit 92 accepts user operations and inputs them to the second terminal device T2. The input unit 92 can be configured using existing input devices such as keyboards, clicking devices (mouse, tablet, etc.), buttons, or touch panels. The input unit 92 can also be an interface for connecting an input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in the input device based on the user's operation to the second terminal device T2. The input unit 92 can be configured arbitrarily as long as it is capable of inputting instructions obtained based on user operations to the second terminal device T2.

[0113] For example, the user confirms the content of the notification sent by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2, and studies the content that should be notified to the driver. The content to be notified to the driver may include, for example, recommendations to seek medical attention at a medical institution for the purpose of detecting a possible decline in cognitive ability. The user inputs information, including the content of the studied notification, into the second terminal device T2 via the operation input unit 92. The transmission control unit 94 transmits the information input by the user to the first terminal device T1 via the communication unit 96. The first terminal device T1 then notifies the driver based on the transmitted information.

[0114] The information processing system 100A can also be configured to restrict the transmission of a first notification message from the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 by the driver operating the first terminal device T1 or the driving monitoring device 1. For example, the driver can operate the first terminal device T1 or the driving monitoring device 1 to generate restriction information indicating the presence or absence of restrictions on the transmission of the first notification message, and store the generated restriction information in the storage unit 70. Furthermore, the alarm control unit 52 can also decide whether to send the first notification message to the second terminal device T2 based on the restriction information stored in the storage unit 70. With this structure, the decision on whether to inform the user of the second terminal device T2 of any abnormal information involving the driver is made according to the driver's wishes. Therefore, the driver's privacy is easily protected.

[0115] According to the second embodiment described above, the information processing system 100A includes: an information processing device 2; a first terminal device T1, which is used by the driver and is a notification unit 84 that is not controlled by the notification control unit 54; and a second terminal device T2, which is used by a user different from the driver and is also controlled by the notification control unit 54. The second terminal device T2 causes the first terminal device T1 to make notifications based on information input by the user's operation. Thus, it is possible to provide the driver using the first terminal device T1 with notification content obtained from research conducted by the user using the second terminal device T2. For example, when a medical worker uses the second terminal device T2, abnormality determination information can be used for medical judgment to detect abnormalities in the driver.

[0116] <Variation Example>

[0117] The scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0118] For example, the anomaly determination unit 45 may set the change index L derived when the driver is determined not to have an anomaly (no decline in cognitive ability) as the baseline change index L1, and compare the baseline change index L1 with the change index L to determine whether the driver has an anomaly. The anomaly determination unit 45 may also determine that the driver has an anomaly if the absolute value of the difference between the change corresponding to the mode of the baseline change index L1 and the change corresponding to the mode of the change index L is above a threshold. Alternatively, the anomaly determination unit 45 may determine that the driver has not an anomaly if the absolute value of the difference between the change corresponding to the mode of the baseline change index L1 and the change corresponding to the mode of the change index L is less than a threshold. The baseline change index L1 may also be set based on the average driving ability of able-bodied individuals.

[0119] For example, the driver monitoring device 1 (information processing device 2) may perform other alarm processing (hereinafter referred to as auxiliary alarm processing) in addition to the aforementioned omission alarm processing. Auxiliary alarm processing may, for example, be a collision alarm processing that detects the possibility of a collision between a risky object and vehicle M based on the position and relative speed of the risky object relative to vehicle M and triggers an alarm from alarm unit 82. In collision alarm processing, the relative speed of the risky object relative to vehicle M may be calculated, for example, by the processing unit 40 analyzing the time change of the risky position detected by the risk detection unit 32. If the processing unit 40 detects the possibility of a collision between the risky object and vehicle M, the alarm control unit 52 may also trigger alarm unit 82. Auxiliary alarm processing may also be performed during the driver's driving of vehicle M (i.e., in real time), similar to omission alarm processing. In the configuration where auxiliary alarm processing is performed in the driver monitoring device 1 (information processing device 2), the alarm from alarm unit 82 based on the omission determination result of omission determination unit 41 may not be executed. Even in this case, the omission determination performed by the omission determination unit 41 is also performed in order to carry out the above-mentioned abnormal notification processing.

[0120] Alternatively, information relating to the determination results of missed sightings by the missed sighting determination unit 41 may be collected in the storage unit 70, and the first terminal device T1 (alarm unit 82) may provide the driver with entries (hereinafter referred to as "review entries") generated based on the collected information for the driver to review their past driving. Review entries may include, for example, information (strings, images, etc.) that the driver can identify as potentially dangerous objects that they may have missed. Review entries may be generated by the driving monitoring device 1 (information processing device 2) or by the first terminal device T1 (alarm unit 82).

[0121] In the information processing system 100A according to the second embodiment, the driving monitoring device 1 (notification control unit 54) can also send first notification information to both the first terminal device T1 and the second terminal device T2. In other words, the notification unit 84 that controls the notifications from both the first terminal device T1 and the second terminal device T2 can also be controlled by the notification control unit 54.

[0122] In the first embodiment, the driving monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as different devices, but they can also be configured as an integrated device. For example, the driving monitoring device 1 may actually include all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, alarm unit 82, and notification unit 84. Alternatively, all these functions may actually be installed in the first terminal device T1 or the vehicle M. Similarly, in the second embodiment, all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and alarm unit 82 may actually be installed in the driving monitoring device 1, and all these functions may actually be installed in the first terminal device T1.

[0123] The driving monitoring device 1 can also be actually installed using multiple information processing devices. For example, the functions of the information processing device 2 can also be actually installed in the first terminal device T1 or the vehicle M (e.g., ECU). Alternatively, the driving monitoring device 1 can be actually installed using a device such as a cloud. For example, in the driving monitoring device 1, the detection unit 30, processing unit 40, control unit 50, and storage unit 70 can also be actually installed in different information processing devices. For example, the storage unit 70 can also be distributed among multiple information processing devices.

[0124] The implementation methods and variations described above can be represented as follows.

[0125] An information processing device comprising:

[0126] Storage medium, which stores computer-readable instructions; and

[0127] The processor, which is connected to the storage medium,

[0128] The processor performs the following processing by executing computer-readable instructions:

[0129] Facial images are acquired in a time series, including images of the driver's face in the vehicle.

[0130] Extract facial features of the driver from each facial image; and

[0131] The driver's behavior is determined based on the trend of changes in characteristic quantities.

[0132] Without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements, and the above embodiments and variations can also be appropriately combined.

Claims

1. An information processing device, wherein, The information processing device includes: The acquisition unit acquires facial images in a time sequence, the facial images including the face of the vehicle's driver; An extraction unit extracts feature quantities of the driver's face based on the facial image; as well as The anomaly determination unit determines whether the driver has experienced an anomaly based on the changing trend of the characteristic quantity.

2. The information processing apparatus according to claim 1, wherein, The extraction unit extracts the feature quantity of the driver's face from each of the facial images acquired in a time sequence.

3. The information processing apparatus according to claim 2, wherein, The information processing device further includes an export unit that exports the change between a first feature quantity and a second feature quantity. The first feature quantity is extracted from the facial image acquired at a first time point, and the second feature quantity is extracted from the facial image acquired at a second time point earlier than the first time point. The anomaly determination unit determines whether the driver has experienced an anomaly based on the change amount.

4. The information processing apparatus according to claim 3, wherein, The derivation unit calculates the change amount for a group of two temporally consecutive facial images obtained in a time-series manner. The anomaly determination unit determines whether the driver has experienced an anomaly based on the number of the calculated multiple changes that are above a threshold.

5. The information processing apparatus according to claim 4, wherein, The output unit calculates the change amount for each of the two consecutive sets of facial images.

6. The information processing apparatus according to any one of claims 3 to 5, wherein, The acquisition unit also acquires images of the vehicle's surroundings. The information processing device also includes a risk detection unit that detects risky objects in the surrounding images. If the anomaly determination unit does not detect the risky object and the change amount corresponding to the time point when the surrounding image is acquired is above a threshold, it determines that the driver has an anomaly.

7. The information processing apparatus according to any one of claims 3 to 5, wherein, The acquisition unit also acquires images of the vehicle's surroundings. The information processing device also includes a risk detection unit that detects risky objects in the surrounding images. The anomaly determination unit determines that the driver has an anomaly when it detects the risky object and the change in the amount of change corresponding to the time point when the surrounding image was obtained is less than a threshold.

8. The information processing apparatus according to any one of claims 1 to 5, wherein, The information processing device also includes a notification control unit, which causes the notification unit to issue a notification when the anomaly determination unit determines that the driver has experienced an anomaly.

9. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to any one of claims 1 to 5; The first terminal device used by the driver; as well as Second terminal devices used in medical institutions If the information processing device determines that the driver has experienced an abnormality, it will instruct the first terminal device to advise the driver to seek medical attention at a medical institution. The information processing device causes the second terminal device to make a notification based on information input by the driver to the first terminal device indicating acceptance of the medical examination.

10. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to any one of claims 1 to 5; The first terminal device used by the driver; and A second terminal device used by a user different from the driver. The second terminal device is controlled and notified by the information processing device. The second terminal device causes the first terminal device to issue a notification based on information input by the user through the user's operation.

11. An information processing method, wherein, The information processing method causes the computer to perform the following processing: Facial images are acquired in a time series, including images of the driver's face in the vehicle. Based on the facial image, extract the feature quantities of the driver's face; as well as The driver is determined to have experienced an abnormality based on the changing trend of the aforementioned characteristic quantity.

12. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Facial images are acquired in a time series, including images of the driver's face in the vehicle. Based on the facial image, extract the feature quantities of the driver's face; as well as The driver is determined to have experienced an abnormality based on the changing trend of the aforementioned characteristic quantity.

Images