Information processing apparatus, information processing system, information processing method, and storage medium

Abstract

An information processing apparatus, an information processing system, an information processing method, and a storage medium capable of detecting a cognitive impairment of a driver. The information processing apparatus includes an acquisition unit that acquires a surrounding image of a vehicle and a face image of a driver of the vehicle; a risk detection unit that detects a risk object in the surrounding image; a visual recognition determination unit that determines whether or not the driver visually recognizes the risk object based on information regarding a position of the risk object and information regarding the face image; an abnormality determination unit that determines whether or not the driver has a first abnormality based on whether or not an evasive risk action that reduces a risk degree is detected after it is determined that the driver visually recognizes the risk object, the risk degree indicating a degree of risk of the vehicle colliding with the risk object; and a notification control unit that causes a notification unit to perform a first notification in a case where it is determined that the driver has the first abnormality.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and storage medium. Background Technology

[0002] In recent years, efforts have intensified to provide sustainable transportation systems that also take into account vulnerable groups among transportation participants, particularly the elderly, people with disabilities, and children. Research and development, particularly focused on improving the safety and convenience of transportation through developments related to mobility facilities for the elderly and people with disabilities, are particularly dedicated to achieving this (see, for example, Patent Document 1 below).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-20745 Summary of the Invention

[0004] When a driver suffers from cognitive impairment, their ability to avoid dangerous objects is reduced. However, drivers are sometimes unaware of their cognitive impairment. When drivers continue driving without being aware of their cognitive impairment and the resulting impairment, it can impair traffic safety and convenience.

[0005] The solutions involved in this invention were made in consideration of such circumstances, and one of their objectives is to provide an information processing device, information processing system, information processing method and storage medium capable of detecting cognitive impairment in drivers, thereby contributing to the development of sustainable transportation systems.

[0006] To address the aforementioned issues, the present invention employs the following solution.

[0007] (1): One aspect of the present invention relates to an information processing apparatus, wherein the information processing apparatus comprises: an acquisition unit that acquires a surrounding image of a vehicle and a facial image of the driver of the vehicle; a risk detection unit that detects a risky object in the surrounding image; a visual recognition determination unit that determines whether the driver has visually recognized the risky object based on information relating to the location of the risky object and information relating to the facial image; an anomaly determination unit that determines whether the driver has experienced a first anomaly based on whether a risk avoidance action that reduces the risk level is detected after determining that the driver has visually recognized the risky object, the risk level indicating the degree of risk of the vehicle colliding with the risky object; and a notification control unit that, in the case that the driver has experienced the first anomaly, causes the notification unit to make a first notification.

[0008] (2): In the above (1) scheme, the information processing device may also include an alarm control unit, which will sound an alarm when the alarm control unit determines that the driver has visually identified the risky object but has not detected the risk avoidance action.

[0009] (3): In the above (2) scheme, it is also possible that after determining that the driver has visually identified the risky object, even if the risk avoidance action is not detected, when the driver's face orientation and / or line of sight orientation is detected based on the facial image and the risk avoidance action is detected, the alarm control unit will not cause the alarm unit to issue the alarm.

[0010] (4): In any of the above (1) to (3), if the orientation of the driver's face and / or the orientation of the driver's gaze changes by more than a predetermined threshold based on the facial image, the anomaly determination unit determines that the driver has experienced a second anomaly, and the notification control unit causes the notification unit to issue a second notification when it determines that the driver has experienced the second anomaly.

[0011] (5): In any of the above (1) to (4), the anomaly determination unit may determine that the driver has committed the first anomaly if the anomaly determination unit does not detect the event of the driver taking the risk avoidance action after the driver visually identifies the risk object more than a specified number of times within a specified period.

[0012] (6): In any of the above schemes (1) to (5), the notification control unit may cause the notification unit to make the first notification at a time other than when the driver is driving the vehicle.

[0013] (7): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing device of any of the above (1) to (6); a first terminal device for use by the driver and being the notification unit; and a second terminal device for use in a medical institution, wherein the first notification includes a recommendation to receive treatment at the medical institution, and the first terminal device causes the second terminal device to make the notification based on information input by the driver indicating an acceptance of the treatment.

[0014] (8): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing apparatus of any one of the above (1) to (6); a first terminal device for use by the driver and not controlled by the notification control unit of the first notification; and a second terminal device for use by a user different from the driver and controlled by the notification control unit of the first notification, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

[0015] (9): One aspect of the present invention relates to an information processing method, wherein the information processing method causes a computer to perform the following processing: acquiring a surrounding image of a vehicle and a facial image of the driver of the vehicle; detecting a risky object in the surrounding image; determining whether the driver has visually recognized the risky object based on information relating to the location of the risky object and information relating to the facial image; determining whether the driver has experienced a first anomaly based on whether a risk avoidance action to reduce the risk level is detected after determining that the driver has visually recognized the risky object, the risk level representing the degree of risk of the vehicle colliding with the risky object; and, if the driver has experienced the first anomaly, causing a notification unit to issue a first notification.

[0016] (10): One aspect of the present invention relates to a storage medium that is a non-transitory storage medium that stores a program that can be read by a computer, wherein the program is configured to cause the computer to perform the following processing: acquiring a surrounding image of a vehicle and a facial image of the driver of the vehicle; detecting a risky object in the surrounding image; determining whether the driver has visually recognized the risky object based on information relating to the location of the risky object and information relating to the facial image; determining whether the driver has experienced a first anomaly based on whether a risk avoidance action to reduce the risk level is detected after determining that the driver has visually recognized the risky object, the risk level representing the degree of risk of the vehicle colliding with the risky object; and, in the case that the driver has experienced the first anomaly, causing a notification unit to issue a first notification.

[0017] According to the solution of the present invention, it is possible to detect cognitive impairment in drivers. Attached Figure Description

[0018] Figure 1 This is a diagram illustrating an example of the structure and usage environment of a driving monitoring device that includes the information processing apparatus according to the first embodiment.

[0019] Figure 2 This is a diagram illustrating an example of a driver monitoring device.

[0020] Figure 3 This is an example of an image obtained by a driving monitoring device.

[0021] Figure 4 This diagram illustrates the process of a driver monitoring device triggering a missed alarm.

[0022] Figure 5 This is a diagram illustrating an example of information detected by the testing department.

[0023] Figure 6 This diagram illustrates an example of a missed alarm issued by the alarm department.

[0024] Figure 7 This diagram illustrates the special alarm processing performed by the driver monitoring device.

[0025] Figure 8 This is a flowchart illustrating an example of a process that determines whether a decision-making department needs to perform a process.

[0026] Figure 9 This diagram illustrates the abnormal notification processing performed by the driving monitoring device.

[0027] Figure 10 This is an example diagram representing resume information.

[0028] Figure 11 This is a flowchart illustrating an example of the processing flow performed by the exception handling unit.

[0029] Figure 12 This diagram illustrates the exception notification processing in the information processing system according to the second embodiment. Detailed Implementation

[0030] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the rearward direction of the vehicle is defined as the negative X direction, the rightward direction of the vehicle's width relative to the positive X direction is defined as the positive Y direction, the leftward direction is defined as the negative Y direction, and the direction orthogonal to the X and Y directions and the height direction of the vehicle is defined as the positive Z direction.

[0031] <First Implementation>

[0032] [Overall Structure]

[0033] Figure 1This diagram illustrates an example of the structure and usage environment of a driver monitoring device 1, including the information processing device 2 according to the first embodiment. The information processing device 2 is, for example, a driver monitoring device 1 retrofitted into a vehicle (hereinafter referred to as "vehicle M") such as a dashcam. That is, the driver monitoring device 1 (information processing device 2) can be mounted and removed from the vehicle M via a mounting / unmounting unit (hereinafter referred to as "mounting / unmounting unit 16"). The information processing device 2 is, for example, a device not connected to the vehicle M's in-vehicle network. An in-vehicle network refers to an in-vehicle network that connects to the vehicle's control-related ECU (Electronic Control Unit), which uses communication standards such as CAN (Controller Area Network), and is connected to the vehicle M's drive, braking, steering, various vehicle sensors, driving-related operating components, and operation buttons. The vehicle M's network for HMI (Human Machine Interface) can also be excluded from the above-described in-vehicle network.

[0034] Figure 2 This diagram illustrates an example of a driver monitoring device 1. The driver monitoring device 1 is, for example, installed near the windshield of a vehicle M, and located near the front of the driver's seat. The driver monitoring device 1 stores images (moving images) captured by the camera unit 10, for example, in the storage unit 70 (described later). The driver monitoring device 1 may also be, for example, equipped with a display device such as an LCD (Liquid Crystal Display) and capable of displaying the images (moving images) stored in the storage unit 70 on the display device.

[0035] The driving monitoring device 1 includes, for example, a camera unit 10, a loading and unloading unit 16, and an information processing unit 2. In the illustrated example, the camera unit 10 includes a front camera 12 and a rear camera 14.

[0036] The front camera 12 and the rear camera 14 are each, for example, digital cameras utilizing solid-state imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) captures images of the area in front of the vehicle M from the location where the driver monitoring device 1 is installed. The rear camera 14 repeatedly (periodically) captures images of the area behind the vehicle M and the interior of the vehicle M from the location where the driver monitoring device 1 is installed. The horizontal viewing angle captured by the front camera 12 and the rear camera 14 is, for example, 180° or more. That is, the driver monitoring device 1 (camera unit 10) captures a 360° range around the vehicle M from its installation location. The front camera 12 and the rear camera 14 may each be, for example, fisheye cameras including fisheye lenses. Alternatively, the front camera 12 and the rear camera 14 may each be wide-angle cameras or the like. In this embodiment, the use of the front camera 12 and the rear camera 14 is described, but other cameras may be used instead of these or other alternatives. That is, the camera unit 10 can be configured to capture images used in this embodiment (peripheral images and facial images described later).

[0037] The mounting / unmounting part 16 is, for example, a component for mounting the driver monitoring device 1 onto the vehicle M. The mounting / unmounting part 16 can be, for example, any supporting component such as a suction cup, a seal, or a bracket. The mounting / unmounting part 16 may also include a power supply cable (wiring, etc.) for supplying power to the driver monitoring device 1.

[0038] Figure 3 This is a diagram illustrating an example of an image acquired by the driver monitoring device 1. That is, Figure 3 This diagram illustrates an example of an image captured by the camera unit 10. The front camera 12 captures images of the front of the vehicle M through the windshield, the driver's side side window, and the passenger side side window. The rear camera 14 captures images of the interior of the vehicle M (including the driver and passengers) and the rear of the vehicle M through the driver's side side window, the passenger side side window, and the rear window.

[0039] In the image captured by the front camera 12 (hereinafter referred to as the "front image"), the subject includes objects such as other vehicles, pedestrians, bicycles, fixed objects, road markings, etc., that can be seen through the front windshield and exist in front of the vehicle M, as well as objects that can be seen through the left and right (front) side windows of the vehicle M.

[0040] The image captured by the rear camera 14 (hereinafter referred to as the "rear image") shows the interior of the vehicle M, as well as objects visible through the rear window that are located behind the vehicle M, and objects visible through the left and right side windows that are located to the left and right (rear) of the vehicle M. Therefore, in the rear image captured by the rear camera 14, the driver of the vehicle M equipped with the driver monitoring device 1 is also reflected as the subject.

[0041] The front and rear images include images of the surrounding conditions of the vehicle M (hereinafter referred to as "surrounding images"). The rear image includes an image of the driver's face of the vehicle M (hereinafter referred to as "face image"). The driver monitoring device 1 can be installed at any location where the camera unit 10 can capture the surrounding images and the face image.

[0042] Information processing device 2 (refer to) Figure 1 Based on the results obtained by referring to the surrounding images and facial images output by the camera unit 10, an alarm and notification are output via the first terminal device T1. The notification may include a suggestion to seek medical treatment at a medical institution (details will be described later). The first terminal device T1 sends a notification to the second terminal device T2 based on information input by the driver of vehicle M indicating acceptance of medical treatment. Sometimes, the driver monitoring device 1, the first terminal device T1, and the second terminal device T2 are combined and referred to as "information processing system 100".

[0043] The driving monitoring device 1, the first terminal device T1, and the second terminal device T2 are connected communicatively via a network NW. The network NW can be a wireless communication network or a wired communication network. For example, the network NW can be constructed using the Internet, a LAN (Local Area Network), Bluetooth (registered trademark), etc. The network NW can also be constructed by combining multiple networks.

[0044] The first terminal device T1 is, for example, a mobile terminal device used by a driver driving a vehicle M equipped with a driver monitoring device 1, such as a smartphone or tablet. The first terminal device T1 executes, for example, an application for receiving alarms and notifications from the information processing device 2. The application causes a display device to show an image based on information sent by the information processing device 2 (e.g., alarm information and notification information described later), and causes a speaker to emit sound. The first terminal device T1 is also referred to as the alarm unit 82 and the notification unit 84. The first terminal device T1 is used, for example, in a state where it can be detachably mounted on the vehicle M. For example, a bracket for the first terminal device T1 with a mounting and detaching mechanism is provided on one or both of the first terminal device T1 and the vehicle M, and the first terminal device T1 is supported by the bracket.

[0045] Alternatively, the navigation, display, and speaker of vehicle M can be used instead of the first terminal device T1. That is, the navigation, display, and speaker of vehicle M can also output alarms and notifications based on the instructions of information processing device 2. That is, the navigation, display, and speaker of vehicle M can also be alarm unit 82 and notification unit 84. Alternatively, the display unit (alarm unit 82, notification unit 84) and speaker (alarm unit 82, notification unit 84) can be provided on the driver monitoring device 1 instead of the first terminal device T1.

[0046] The second terminal device T2 is a terminal device used in a medical institution. The second terminal device T2 can also be a terminal used by personnel belonging to the medical institution (e.g., doctors, nurses, etc.). The second terminal device T2 is an information processing device used by a user different from the driver using the first terminal device T1. The second terminal device T2 can also be, for example, a smartphone, tablet, personal computer, etc. The second terminal device T2 may run an application for receiving notifications sent by the first terminal device T1. The application causes a display device to display an image based on information sent by the first terminal device T1 and causes a speaker to emit sound.

[0047] like Figure 1 As shown, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, and a face orientation detection unit 36. The processing unit 40 includes a visual recognition determination unit 41, a miss determination unit 42, an evaluation unit 43, a need determination unit 44, a storage control unit 45, and an anomaly determination unit 46. The control unit 50 includes an alarm control unit 52 and a notification control unit 54.

[0048] The acquisition unit 20, detection unit 30, processing unit 40, and control unit 50 are equipped with hardware processors such as CPUs (Central Processing Units) and storage devices (storage devices with non-transitory storage media) storing programs (software). The processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented using hardware such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units) (including the circuitry). The functions of each component can also be realized through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs.

[0049] The program (software) can be pre-stored in the storage unit 70 (a storage device with a non-temporary storage medium) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it can be stored in a removable storage medium (a non-temporary storage medium) such as a memory card, and installed in the storage device by assembling the storage medium into the driving monitoring device 1. The program (software) can also be pre-downloaded from other computer devices via short-range communication or wide-area communication using an application program that runs in the first terminal device T1 and / or the second terminal device T2, and then sent from the first terminal device T1 and / or the second terminal device T2, thereby being installed in the storage device.

[0050] The storage unit 70 stores, for example, resume information 72, programs (such as the programs described above), and various other information. The storage unit 70 can also be implemented using the various storage devices described above or EEPROM (Electrically Erasable Programmable Read Only Memory). The communication unit 60 is an interface for wireless or wired communication with information processing devices (external devices) such as the first terminal device T1 and the second terminal device T2 via a network NW.

[0051] The driver monitoring device 1 (information processing device 2) performs at least three functions: a missed detection alarm, a special alarm, and an anomaly notification. The missed detection alarm is triggered by detecting a driver's failure to observe a dangerous object based on surrounding and facial images, causing the alarm unit 82 to sound an alarm (hereinafter referred to as a missed detection alarm). The special alarm is triggered when the driver visually identifies a dangerous object based on surrounding and facial images but fails to take evasive action, causing the alarm unit 82 to sound an alarm (hereinafter referred to as a special alarm). The anomaly notification is triggered when the driver visually identifies a dangerous object and, based on whether the driver has taken evasive action, detects an anomaly in the driver (e.g., cognitive impairment, visual field defects, etc.) and causes the notification unit 84 to issue a notification.

[0052] A risky object refers to an object, for example, that vehicle M should avoid. A risky object is, for example, an object that would interfere with vehicle M when it is moving as it is, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose approach level becomes more than a certain level after a specified time, taking into account the object's position, direction of movement, and speed, as well as the position, method of movement, and speed of vehicle M. The object can also be a traffic participant such as a pedestrian, bicycle, or vehicle, or an object other than a traffic participant such as an object placed on the road or a fallen object.

[0053] Risk avoidance actions refer to actions that reduce the degree of risk. Risk degree refers to an indicator of the extent to which vehicle M risks colliding with a hazardous object. Examples of risk avoidance actions include braking and steering (handling) towards the opposite side of the hazardous object.

[0054] [Handling of Missing Alarms]

[0055] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device 1 (information processing device 2).

[0056] The acquisition unit 20 acquires front and rear images from the imaging unit 10. Therefore, the acquisition unit 20 establishes a link between the peripheral images and the facial images to acquire these images. The imaging unit 10 repeatedly (periodically) captures the peripheral images and facial images, thus the acquisition unit 20 repeatedly (periodically) acquires these images. The acquisition unit 20 can also establish a link between the peripheral images and facial images and the time they were captured (the shooting time) to acquire these images.

[0057] The risk detection unit 32 detects risky objects in the surrounding image. "Detecting risky objects" refers to, for example, detecting the presence of a risky object and the position of that object relative to the vehicle M (driver monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M can be pre-stored in the storage unit 70. The risk detection unit 32 can also detect the position of the risky object relative to the vehicle M based on this reference information.

[0058] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, and also detects the direction of the risk object's presence (approach direction) relative to the vehicle M. "Direction of presence of the risk object" refers to, for example, the direction of presence of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, "direction of presence of the risk object" will sometimes be referred to as "risk presence direction".

[0059] The gaze detection unit 34 detects the driver's gaze in a facial image. Specifically, the gaze detection unit 34 detects the driver's gaze by analyzing the facial image using a prescribed image analysis method. "Gaze detection" refers to, for example, detecting the position of the driver's eyeballs and the direction of the gaze (the direction of the eyeballs).

[0060] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the facial image. Specifically, the face orientation detection unit 36 ​​detects the orientation of the driver's face by parsing the facial image using a prescribed image parsing method. The face orientation detection unit 36 ​​may also detect the position of the driver's face in addition to its orientation. Hereinafter, the information combining "face orientation" and "face position" is sometimes referred to as "facial information." The face orientation detection unit 36 ​​detects facial information in the facial image.

[0061] The detection unit 30 can also be configured to detect lanes, pedestrian crossings, etc., in addition to the aforementioned risk objects, lines of sight, and facial information. Figure 5 This is a diagram illustrating an example of information detected by the detection unit 30. In Figure 5 In the example, pedestrians, other vehicles, lanes, and crosswalks detected by the detection unit 30 are displayed based on vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that may become risk objects. In the information detected by the detection unit 30, traffic participants (objects) that may become risk objects detected by the detection unit 30 may also be displayed based on vehicle M, the direction of vehicle M, and / or the direction of travel of vehicle M, instead of vehicle M, the direction of the line of sight, and the direction of the face.

[0062] Visual recognition and determination unit 41 (refer to) Figure 4 The system determines whether the driver has visually identified a risky object based on the risk's location and / or direction of presence, as well as line of sight and / or facial information. "Risk location" and "direction of presence" are each examples of "information relating to the location of a risky object." "Line of sight" and "facial information" are each examples of "information relating to facial images."

[0063] For example, reference information (not shown) indicating the relationship between the orientation of the driver's eyes (gaze) and / or the orientation of their face in the facial image and their relative orientation with respect to the direction of travel of the vehicle M can also be pre-stored in the storage unit 70. The visual recognition determination unit 41 can also refer to this reference information to determine whether the orientation of the gaze and / or the orientation of the face coincides with the direction of the risk. If the orientation of the gaze and / or the orientation of the face coincides with the direction of the risk, the visual recognition determination unit 41 can also determine that the driver is visually recognizing a risky object. The so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction of the risk" can, for example, be the case where there is a risky position on a straight line along the orientation of the gaze and / or the orientation of the face. Alternatively, the so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction of the risk" can also be the case where there is a risky position in a cone-shaped region with the position of the viewpoint (eyeball) and / or the face as the apex and having a central axis along the orientation of the gaze and / or the orientation of the face that expands laterally as it moves forward. If the direction of the driver's gaze and / or the direction of the face does not match the location of the hazard object, the visual recognition determination unit 41 may also determine that the driver has not visually recognized the hazard object.

[0064] The visual recognition determination unit 41 generates visual recognition determination information based on the above determination result. The visual recognition determination information includes, for example, the presence or absence of visual recognition and the direction of risk. However, the method for determining visual recognition performed by the visual recognition determination unit 41 is not limited to the above and can be appropriately modified.

[0065] The omission detection unit 42 determines whether the driver has missed any potentially dangerous objects based on visual recognition information.

[0066] For example, the omission determination unit 42 may determine that the driver has missed the risk object if the state of not visually recognizing the risk object continues for a predetermined time or longer. The omission determination unit 42 may also determine that the driver has not missed the risk object if the driver visually recognizes the risk object, or if the state of not visually recognizing the risk object is resolved before the predetermined time has elapsed. However, the method by which the omission determination unit 42 determines omissions is not limited to the above and can be appropriately modified.

[0067] The omission detection unit 42 generates omission detection information based on the above detection results. The omission detection information includes, for example, the presence or absence of omissions and the direction of risk.

[0068] The alarm control unit 52 causes the first terminal device T1 (alarm unit 82) to output a missed detection alarm based on the missed detection determination information. For example, the alarm control unit 52 sends the missed detection alarm information to the first terminal device T1 via the communication unit 60. The missed detection alarm information includes, for example, information indicating whether a missed detection alarm is needed and the direction of risk existence. For example, if the missed detection determination information includes information indicating "a missed detection exists," the alarm control unit 52 generates missed detection alarm information including information indicating "a missed detection alarm is needed." For example, if the missed detection determination information includes information indicating "no missed detection exists," the alarm control unit 52 generates missed detection alarm information including information indicating "a missed detection alarm is not needed." The first terminal device T1 outputs a missed detection alarm based on the missed detection alarm information.

[0069] Figure 6 This diagram illustrates an example of a leak detection alarm triggered by the first terminal device T1 (alarm unit 82). Figure 6 In the example, the first terminal device T1 displays identifiable entries indicating the direction of the risk (D1 and D2 in the diagram). Furthermore, the first terminal device T1 provides oversight warnings by differentiating the display format of the risk direction when a risky object is missed (i.e., the oversight warning information includes a message indicating "oversight warning needed") and when no risky object is missed (i.e., the oversight warning information includes a message indicating "oversight warning not needed"). Specifically, when a risky object is missed, the risk direction is displayed in a more emphasized manner compared to when no oversight occurs. This emphasis could be based on color, flashing icons, or other visual cues that make the risk direction easier for the driver to visually identify. The first terminal device T1 may also provide oversight warnings through speaker announcements, light illumination, or other means, in addition to or instead of the above (A in the diagram). The specific method of providing the oversight warning can be appropriately modified as long as it is identifiable by the driver.

[0070] The oversight alarm issued by the first terminal device T1 (alarm unit 82) can also be issued during the moment when the driver is driving the vehicle M (i.e., in real time). Examples of the moment when the driver is driving the vehicle M include when the vehicle M is moving or when the vehicle M is temporarily stopped. For example, the oversight alarm information sent by the alarm control unit 52 can also be issued at the aforementioned moment. Alternatively, the oversight alarm information sent by the alarm control unit 52 can be issued at any moment, and the first terminal device T1 (alarm unit 82) controls the timing of the oversight alarm to issue the oversight alarm at the aforementioned moment.

[0071] [Special Alarm Handling]

[0072] Figure 7 This diagram illustrates the special alarm processing performed by the driving monitoring device 1 (information processing device 2).

[0073] The evaluation unit 43 evaluates the degree of risk based on the risk location. For example, the evaluation unit 43 may calculate the relative velocity of the risk object relative to the vehicle M (hereinafter referred to as the risk relative velocity) based on the time change of the risk location detected by the risk detection unit 32. Furthermore, the evaluation unit 43 may evaluate the degree of risk based on the risk location (the relative position of the risk object relative to the vehicle M) and the risk relative velocity. For example, the evaluation unit 43 may calculate the probability that the risk object will collide with the vehicle M within a specified time based on the risk location and the risk relative velocity. Moreover, the higher the calculated probability, the higher the risk level evaluated by the evaluation unit 43; conversely, the lower the calculated probability, the lower the risk level evaluated by the evaluation unit 43.

[0074] Whether a special alarm is needed is determined by the determination unit 44, for example, based on the degree of risk, whether the driver has visually recognized the risky object, and line of sight and / or facial information. Regarding the presence or absence of visual recognition, this is obtained, for example, through the processing performed by the visual recognition determination unit 41 in the aforementioned missed detection alarm processing (see...). Figure 4 Similarly, gaze and facial information are obtained, for example, through processing performed by the gaze detection unit 34 and the face orientation detection unit 36 ​​in the omission alarm processing (see reference). Figure 4 The decision unit 44 generates information on whether a special alarm is needed, including a decision result on whether a special alarm is needed.

[0075] Figure 8 This is a flowchart illustrating an example of the process executed by the decision-making unit 44 regarding whether a decision is needed. Figure 8 The process shown, for example, begins whenever there is a visual identification of a risky object by the driver (i.e. whenever there is a visual identification determination made by the visual identification determination unit 41).

[0076] First, the visual recognition determination unit 41 determines whether a risk avoidance action by the driver has been detected within a predetermined time (hereinafter referred to as t1) from the moment the driver visually recognizes the risk object (step S102). t1 is, for example, a predetermined value. t1 can also be determined based on the time typically required for a driver with normal cognitive abilities (e.g., without cognitive impairment or visual field defects) from visual recognition of the risk object to the moment a risk avoidance action is taken. The need determination unit 44 can also detect the risk avoidance action based, for example, on the level of risk evaluated by the evaluation unit 43. The need determination unit 44 can also detect the risk avoidance action if, for example, there is a decrease in the level of risk evaluated by the evaluation unit 43. If a risk avoidance action is detected within t1 (step S102: Yes), the need determination unit 44 generates a need for a special alarm message as "no special alarm needed" (step S104). After that, the need determination unit 44 ends. Figure 8 The flowchart shown illustrates the processing.

[0077] If no risk avoidance action is detected within t1 (step S102: No), the determination unit 44 determines whether a change in the driver's facial orientation and / or line of sight, and subsequent risk avoidance action, has been detected within a predetermined time (hereinafter referred to as t2) from the moment the driver visually identifies the risky object (step S106). t2 is set to a value longer than t1. t2 is, for example, a predetermined value. If a change in facial orientation and / or line of sight, and subsequent risk avoidance action, is detected within t2 (step S106: Yes), the determination unit 44 generates a special alarm message indicating "no special alarm required" (step S104). Afterward, the determination unit 44 concludes the process. Figure 8 The flowchart shown illustrates the processing.

[0078] If no change in facial orientation and / or gaze direction is detected within t2, or if no subsequent risk avoidance action is detected even if a change in facial orientation and / or gaze direction is detected (step S106: No), the determination unit 44 generates a special alarm message indicating whether a special alarm is needed (step S108). Afterwards, the determination unit 44 concludes the process. Figure 8 The flowchart shown illustrates the processing.

[0079] Based on the above processing, if no evasive action is detected after the driver visually identifies a hazardous object, a special alert is deemed necessary. In cases where the driver suffers from cognitive impairment, the reduced cognitive ability caused by the impairment may lead to situations where "no evasive action is taken despite visually identifying a hazardous object." By detecting such events and informing the driver with a special alert, accidents caused by the failure to take evasive action (such as a collision between the hazardous object and vehicle M) can be prevented.

[0080] On the other hand, when a driver has visual field defects (e.g., narrowed field of vision), the driver cannot perceive a risky object when it is located in the area of ​​the visual field defect (hereinafter referred to as the defect area). That is, even if the direction of the driver's gaze and facial orientation aligns with the direction of the risk, making it appear as if the driver can visually identify the risky object, the driver may still not perceive it. In such cases, when the driver's facial orientation and / or gaze orientation changes, a phenomenon may occur where "the risky object moves away from the defect area, and the driver becomes able to perceive the risky object that was previously unrecognizable." In view of this phenomenon, in the above processing, after determining that the driver has visually identified the risky object, even if no evasive action by the driver is detected, if a change in the driver's facial orientation and / or gaze orientation is detected based on the facial image, and an evasive action is detected, it is determined that no special alarm is needed. Therefore, it is possible to more accurately determine whether a special alarm is needed.

[0081] return Figure 7 The alarm control unit 52, based on the direction of the risk and whether special alarm information is needed, causes the first terminal device T1 (alarm unit 82) to output a special alarm. A special alarm is an example of an "alarm." The direction of the risk is obtained, for example, through the processing performed by the risk detection unit 32 in the aforementioned leak detection alarm processing (see reference). Figure 4 For example, the alarm control unit 52 sends special alarm information to the first terminal device T1 via the communication unit 60. The special alarm information may include, for example, whether special alarm information is needed and the direction of the risk.

[0082] The first terminal device T1 outputs a special alarm based on special alarm information. The special alarm can also be triggered by displaying strings, images, etc., on the display device of the first terminal device T1 (alarm unit 82) based on the special alarm information. For example, similar to the aforementioned oversight alarm, if the special alarm information includes a statement indicating whether a special alarm is needed, and the entry identifies the direction of the risk, the direction of the risk can be displayed in a more emphasized form, thereby triggering the special alarm. For example, the direction of the risk can be displayed in the special alarm in a form that is even more emphasized than the oversight alarm described above. The first terminal device T1 can also, in addition to or instead of the above, trigger the special alarm through sound from a speaker, illumination of a light, etc. In this case, the special alarm can also be triggered in a form that is even more emphasized than the oversight alarm. For example, the special alarm can be triggered by sounding at a louder volume, flashing light at a higher frequency, or illuminating light with a stronger intensity compared to the oversight alarm. The specific method of the special alarm can be appropriately modified as long as it can be recognized by the driver.

[0083] The special alarm issued by the first terminal device T1 (alarm unit 82) may also be issued during the driver's operation of the vehicle M (i.e., in real time). For example, the transmission of special alarm information by the alarm control unit 52 may also be issued at the aforementioned time. Alternatively, the transmission of special alarm information by the alarm control unit 52 may be issued at any time, and the first terminal device T1 (alarm unit 82) may control the timing of the special alarm to ensure that the special alarm is issued at the aforementioned time.

[0084] [Abnormal Notification Handling]

[0085] Figure 9 This diagram illustrates the abnormal notification processing performed by the driving monitoring device 1 (information processing device 2).

[0086] The storage control unit 45 establishes a link between the time, risk level, whether the driver visually identifies a risky object, and gaze and / or facial information, and stores this information in the storage unit 70. The time, for example, corresponds to the time of capturing surrounding images and / or facial images used in detecting the risk location, determining the presence or absence of visual identification, and detecting gaze and facial information. The risk level is obtained, for example, through processing performed by the evaluation unit 43 in the aforementioned special alarm processing (see reference). Figure 7 The presence or absence of visual recognition is obtained, for example, through the processing performed by the visual recognition determination unit 41 in the aforementioned leak alarm processing (see reference). Figure 4 Similarly, gaze and facial information are obtained, for example, through processing performed by the gaze detection unit 34 and the face orientation detection unit 36 ​​in the omission alarm processing (see reference). Figure 4 ).

[0087] The aforementioned processing (storage processing) performed by the storage control unit 45 is performed, for example, whenever a risk level assessment, a determination of the presence or absence of visual recognition, or a detection of gaze and / or facial information is performed. Hereinafter, the information stored in the storage unit 70 will be referred to as resume information 72.

[0088] Figure 10 This diagram illustrates an example of the resume information 72. The storage control unit 45 repeatedly performs the aforementioned storage process, thereby accumulating multiple groups (hereinafter referred to as information groups G) of information including time, risk level, presence or absence of visual recognition, and gaze and / or facial information in the resume information 72. Specifically, the storage control unit 45 generates information group G whenever it performs a risk level assessment, determines the presence or absence of visual recognition, or detects gaze and / or facial information. Furthermore, the storage control unit 45 updates the resume information 72 by appending the generated information group G to the resume information 72.

[0089] return Figure 9 The anomaly determination unit 46 determines whether the driver has experienced an anomaly based on the driver history information 72. Specifically, the anomaly determination unit 46 determines whether the driver has experienced a first anomaly and whether the driver has experienced a second anomaly. The first anomaly is, for example, having or potentially having cognitive impairment. The second anomaly is, for example, having or potentially having visual field defects such as narrowing of the field of vision. The anomaly determination unit 46 generates anomaly determination information based on the determination results. The anomaly determination information includes, for example, information indicating the presence or absence of the first anomaly and information indicating the presence or absence of the second anomaly.

[0090] Figure 11 This is a flowchart illustrating an example of the processing flow performed by the exception determination unit 46. Figure 11 The flowchart shown can be processed repeatedly at a predetermined cycle, or whenever the storage control unit 45 updates the history information 72, or it can be started by the instruction of the driver or others.

[0091] First, the exception determination unit 46 reads the history information 72 from the storage unit 70 (step S202). After performing the processing in step S202, the exception determination unit 46 executes the processing in steps S204 to S210 and steps S212 to S218 in parallel. However, the exception determination unit 46 may also perform the processing in steps S204 to S210 and steps S212 to S218 sequentially. The processing in steps S204 to S210 involves handling the first exception. The processing in steps S212 to S218 involves handling the second exception.

[0092] In step S204, the anomaly determination unit 46 counts the number of times the first event occurred during the first period based on the read history information 72. The first period is a predetermined period. For example, the first period could be the period from a predetermined point in the past to the present.

[0093] The first event is, for example, an event that satisfies both conditions (1-1) and (1-2).

[0094] (1-1): "After determining that the driver has visually identified a risky object, no risk avoidance action was detected within t3."

[0095] (1-2): "After determining that the driver has visually identified a risky object, no change in facial orientation and / or line of sight orientation is detected within t4" or "After determining that the driver has visually identified a risky object, no risk avoidance action is detected within t4 even after a change in facial orientation and / or line of sight orientation is detected".

[0096] Here, t3 is a specified time. t3 is, for example, a predetermined value. t3 can also be determined based on the time typically required for a driver with normal cognitive abilities (e.g., without cognitive impairment or visual field defects) to visually identify a hazardous object and take evasive action. t3 can also be, for example, the same value as t1 described above. t4 is a specified time. t4 is set to a value longer than t3. t4 is, for example, a predetermined value. t4 can also be, for example, the same value as t2 described above.

[0097] The anomaly determination unit 46 can detect events that satisfy the above conditions (1-1) by referring to the time information, risk level information, and visual recognition presence / absence information contained in the history information 72. The anomaly determination unit 46 can also detect events that satisfy the above conditions (1-2) by referring to the time information, risk level information, visual recognition presence / absence information, and facial orientation and / or gaze direction information contained in the history information 72. The anomaly determination unit 46 can also, similarly to the need-to-determine unit 44, consider a decrease in risk level as the presence of risk avoidance action, thereby detecting risk avoidance action.

[0098] Next, the anomaly determination unit 46 determines whether the number of times the first event occurred during the first period is greater than or equal to a predetermined number (step S206). The predetermined number can be, for example, one time or multiple times. If the number of times the first event occurred is greater than or equal to the predetermined number (step S206: Yes), the anomaly determination unit 46 determines that the driver has a first anomaly (step S208). If the number of times the first event occurred is less than or equal to the predetermined number (step S206: No), the anomaly determination unit 46 determines that the driver does not have a first anomaly (step S210).

[0099] Based on the processing of steps S204 to S210 above, it is determined whether the driver has experienced the first anomaly based on whether an evasive action is detected after the driver has visually identified a risky object.

[0100] By setting the specified number of occurrences to multiple times, the accuracy of the first anomaly determination can be improved. That is, even if an event (the first event) in which a driver visually recognizes a dangerous object but fails to take evasive action occurs only once, the driver may not necessarily have cognitive impairment. For example, there is a possibility that the driver occasionally failed to take evasive action due to inattentive driving. By adopting a structure that determines the first anomaly only after the first event has occurred multiple times, misjudgments caused by the accidental occurrence of such first events can be suppressed, thus improving the accuracy of the determination.

[0101] By setting conditions (1-2), after determining that the driver has visually identified a risky object, even if no evasive action is detected, if a change in the driver's facial orientation and / or line of sight is detected based on the facial image and an evasive action is detected, it is determined that no first anomaly has occurred. As described above, when a driver has visual field defects, they may be temporarily unable to recognize the risky object based on the positional relationship between the defect and the risky object, but it is possible that they can recognize the risky object based on changes in facial orientation and / or line of sight. In such cases, a structure that does not determine it as the first anomaly is adopted, thereby improving the accuracy of the determination of the first anomaly. It should be noted that, as described later, the anomaly determination unit 46 determines that the driver has a second anomaly (i.e., an anomaly involving visual field defects) in such cases.

[0102] In step S212, the anomaly determination unit 46 counts the number of times the second event occurred during the second period based on the read history information 72. The second period is a predetermined period. For example, the second period could be the period from a certain point in the past to the present. The second period could be the same as the first period described above, or it could be a different period from the first period described above.

[0103] The second event is, for example, an event that satisfies at least one of the following conditions (2-1) and (2-2).

[0104] (2-1): "After determining that the driver has visually identified a risky object, a change in the direction of the face and / or the direction of the gaze is detected within t4, and subsequent risk avoidance actions are taken."

[0105] (2-2): "Above a specified threshold for changes in the direction of the driver's face and / or gaze"

[0106] The anomaly detection unit 46 can detect events that meet the above conditions (2-1) by referring to the time information, risk level information, and visual recognition presence / absence information contained in the resume information 72. The anomaly detection unit 46 can also detect events that meet the above conditions (2-2) by referring to information on facial orientation and / or gaze direction. The phrase "above a predetermined threshold for changes in facial orientation and / or gaze direction" refers, for example, to changes in facial orientation and / or gaze direction by a predetermined angle or more within a predetermined time period.

[0107] Next, the anomaly determination unit 46 determines whether the number of times the second event occurred during the second period is greater than or equal to a predetermined number (step S214). The predetermined number can be, for example, once or multiple times. The predetermined number in step S214 can be the same as or different from the predetermined number in step S206 described above. If the number of times the second event occurred is greater than or equal to the predetermined number (step S214: Yes), the anomaly determination unit 46 determines that the driver has a second anomaly (step S216). If the number of times the second event occurred is less than or equal to the predetermined number (step S214: No), the anomaly determination unit 46 determines that the driver does not have a second anomaly (step S218).

[0108] Based on the processing in steps S212 to S218 described above, if a change in the driver's facial orientation and / or gaze direction is detected based on the facial image, and subsequently, a risk avoidance action is detected, the driver is determined to have a second anomaly. Furthermore, if the change in the driver's facial orientation and / or gaze direction detected based on the facial image exceeds a predetermined threshold, the driver is also determined to have a second anomaly. Sometimes, when the driver is aware of visual field defects, they may actively change the orientation of their face and / or gaze while driving the vehicle M. By setting condition (2-2), visual field defects can also be detected based on such driver actions.

[0109] After the processing in steps S204 to S210 and steps S212 to S218 is completed, the anomaly determination unit 46 generates anomaly determination information based on the above determination results. That is, the anomaly determination unit 46 generates anomaly determination information based on the determination results involving whether or not a first anomaly exists and the determination results involving whether or not a second anomaly exists.

[0110] return Figure 9 The notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information. For example, the notification control unit 54 sends the notification information to the first terminal device T1 (notification unit 84) via the communication unit 60. For example, the notification control unit 54 may also send the notification information to the first terminal device T1 if the anomaly determination information includes at least one of information indicating that the driver has a first anomaly and information indicating that the driver has a second anomaly (i.e., if the anomaly determination unit 46 determines that the driver has experienced at least one of the first anomaly and the second anomaly). The notification information may include, for example, information indicating the driver's anomaly (first anomaly, second anomaly). The notification control unit 54 may also not send the notification information to the first terminal device T1 if the anomaly determination information includes both information indicating that the driver does not have a first anomaly and information indicating that the driver does not have a second anomaly (i.e., if the anomaly determination unit 46 determines that the driver has not experienced either the first or the second anomaly).

[0111] The first terminal device T1 (notification unit 84) outputs a notification based on notification information. The notification may, for example, be made by displaying a string, image, or other object obtained based on the notification information on a display device of the first terminal device T1 (notification unit 84). The first terminal device T1 (notification unit 84) may also issue a notification corresponding to the content of the driver's abnormality (first abnormality, second abnormality). For example, if the notification information includes information indicating a first abnormality, the first terminal device T1 (notification unit 84) may issue a notification indicating that the driver has or may have cognitive impairment. This notification is an example of a "first notification." If the notification information includes information indicating a second abnormality, the first terminal device T1 (notification unit 84) may issue a notification indicating that the driver has or may have visual field defects. This notification is an example of a "second notification."

[0112] The notifications (first notification and second notification) issued by the first terminal device T1 (notification unit 84) may include a suggestion to receive treatment at a medical institution (e.g., a hospital the driver frequently visits). In this case, the first terminal device T1 (notification unit 84) may also allow the driver to input information indicating whether the driver agrees to receive treatment at the medical institution. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a hypothetical button or the like indicating agreement to receive treatment. Such a button or the like functions as an agreement information acquisition unit, which acquires information indicating the driver's intention to receive treatment at the medical institution (hereinafter referred to as agreement information). Furthermore, if the driver operates the first terminal device T1 (notification unit 84) and the agreement information acquisition unit acquires the agreement information, the first terminal device T1 (notification unit 84) may also cause the second terminal device T2 to issue a notification indicating that the driver has agreed to receive treatment.

[0113] However, as long as the form is recognizable by the driver, the form of the notification issued by the first terminal device T1 (notification unit 84) is not particularly limited and can be appropriately changed.

[0114] The notification sent by the first terminal device T1 (notification unit 84) can also be sent at times other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. For example, the notification information sent by the notification control unit 54 can also be sent at the aforementioned times. Alternatively, the notification information sent by the notification control unit 54 can be sent at any time, and the first terminal device T1 (notification unit 84) controls the timing of the notification to ensure that the notification is sent at the aforementioned times.

[0115] According to the first embodiment described above, the information processing device 2 includes: an acquisition unit 20 that acquires a surrounding image of the vehicle M and a facial image of the driver of the vehicle M; a risk detection unit 32 that detects a risky object in the surrounding image; a visual recognition determination unit 41 that determines whether the driver has visually recognized the risky object based on information relating to the location of the risky object and information relating to the facial image; an anomaly determination unit 46 that determines whether the driver has experienced a first anomaly based on whether a risk avoidance action that reduces the risk level is detected after determining that the driver has visually recognized the risky object, where the risk level represents the degree of risk of the vehicle M colliding with the risky object; and a notification control unit 54 that, when it is determined that the driver has experienced a first anomaly, causes the notification unit 84 to issue a notification. Thus, it is possible to detect cognitive impairment in the driver.

[0116] <Second Implementation>

[0117] Next, the second embodiment will be described, in which the basic structure is the same as that in the first embodiment. Therefore, the same reference numerals will be used to label the same structures and their descriptions will be omitted; only the differences will be described.

[0118] Figure 12 This diagram illustrates the exception notification processing in the information processing system 100A according to the second embodiment. The second terminal device T2 in the second embodiment can also be a terminal device used in a medical institution, similar to that in the first embodiment. Alternatively, the second terminal device T2 in the second embodiment can also be an information processing device used by employees of an insurance company, etc.

[0119] In the second embodiment, the driving monitoring device 1 (notification control unit 54) sends the notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the second embodiment, the notification unit 84, which controls the notifications (first notification, second notification) by the notification control unit 54, is the second terminal device T2 instead of the first terminal device T1. In the second terminal device T2, for example, an application program for receiving notifications from the information processing device 2 is executed. The application program causes the display device to display images, strings, etc., obtained based on the information (notification information, etc.) sent by the information processing device 2, or causes the speaker to emit sound.

[0120] The second terminal device T2 uses information input by the user to notify the first terminal device T1. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0121] The input unit 92 and the transmission control unit 94 are equipped with hardware processors such as CPUs and storage devices (storage devices with non-transitory storage media) that store programs (software). The processor executes the programs to implement the functions of each component. Some or all of these components can be implemented using hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component can be implemented through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with information processing devices such as the information processing device 2 and the first terminal device T1 via a network NW.

[0122] The input unit 92 accepts user operations and inputs them to the second terminal device T2. The input unit 92 can be configured using existing input devices such as keyboards, clicking devices (mouse, tablet, etc.), buttons, or touch panels. The input unit 92 can also be an interface for connecting an input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in the input device based on the user's operation to the second terminal device T2. The input unit 92 can be configured arbitrarily as long as it is capable of inputting instructions obtained based on user operations to the second terminal device T2.

[0123] For example, the user confirms the content of the notification sent by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2, and studies the content that should be notified to the driver. The content to be notified to the driver may include the possibility of discovering diseases (abnormalities) such as cognitive impairment or visual field defects, and advice to seek medical treatment at a medical institution for such diseases. The user inputs information, including the studied notification content, into the second terminal device T2 via the operation input unit 92. The transmission control unit 94 transmits the information input by the user to the first terminal device T1 via the communication unit 96. The first terminal device T1 then notifies the driver based on the transmitted information.

[0124] The information processing system 100A can also be configured to restrict the transmission of notification information from the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 by the driver operating the first terminal device T1 or the driving monitoring device 1. For example, the driver can operate the first terminal device T1 or the driving monitoring device 1 to generate restriction information indicating the presence or absence of such restriction on the transmission of notification information, and store the generated restriction information in the storage unit 70. Furthermore, the alarm control unit 52 can also decide whether to send notification information to the second terminal device T2 based on the restriction information stored in the storage unit 70. With this structure, the decision on whether to inform the user of the second terminal device T2 of any abnormal information involving the driver is made according to the driver's wishes. Therefore, the driver's privacy is easily protected.

[0125] According to the second embodiment described above, the information processing system 100A includes: an information processing device 2; a first terminal device T1, which is used by the driver and is a notification unit 84 that is not controlled by the notification control unit 54; and a second terminal device T2, which is used by a user different from the driver and is also controlled by the notification control unit 54 to notify the first terminal device T1, wherein the second terminal device T2 causes the first terminal device T1 to make a notification based on information input by the user's operation. Thus, it is possible to provide the driver using the first terminal device T1 with notification content obtained from research conducted by the user using the second terminal device T2.

[0126] <Variation Example>

[0127] The scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0128] For example, in the above embodiment, it is not necessary for the determination unit 44 and the anomaly determination unit 46 to detect risk avoidance actions based on the reduction in the risk level evaluated by the evaluation unit 43, but the method for detecting risk avoidance actions is not limited to this. For example, an acceleration sensor may be installed in the driver monitoring device 1 and the first terminal device T1, and risk avoidance actions such as braking and steering operations may be detected based on the detected acceleration information. Alternatively, various sensors equipped in the vehicle M and the ECU of the vehicle M that manages them may be connected to the network NW, and risk avoidance actions may be detected based on the signals output from these sensors. The method for detecting risk avoidance actions is not limited to these examples and can be appropriately modified.

[0129] In the above embodiment, the anomaly determination unit 46 establishes a history information 72 based on time, risk level, presence or absence of visual recognition, and gaze and / or facial information to determine the presence or absence of a first anomaly and a second anomaly. However, the method for determining anomalies is not limited to this. For example, it could also indicate special alarm processing (see [reference]). Figure 8 The results of various judgments in the process (e.g., the results of judgments in steps S102 and S104) are stored (accumulated) in the storage unit 70 each time a judgment is made. Furthermore, the exception determination unit 46 can also determine an exception based on the information stored (accumulated) in the storage unit 70. The method for determining the presence or absence of an exception is not limited to these examples and can be appropriately modified.

[0130] The driver monitoring device 1 (information processing device 2) may also perform other alarm processing (hereinafter referred to as auxiliary alarm processing) in addition to the aforementioned omission alarm processing. Auxiliary alarm processing may include, for example, collision alarm processing. Collision alarm processing may also be a process that detects the probability of a collision between a risky object and vehicle M (e.g., the aforementioned risk level) based on the position and relative speed of the risky object relative to vehicle M, and triggers an alarm by the alarm unit 82. In collision alarm processing, the relative speed of the risky object relative to vehicle M may be calculated, for example, by the processing unit 40 analyzing the time change of the risk position detected by the risk detection unit 32. Alternatively, when the processing unit 40 detects the probability of a collision between a risky object and vehicle M (e.g., when the risk level exceeds a predetermined threshold), the alarm control unit 52 triggers an alarm by the alarm unit 82. Auxiliary alarm processing may also be performed during the driver's driving of vehicle M (i.e., in real time), similar to omission alarm processing. In the configuration where the driver monitoring device 1 (information processing device 2) performs auxiliary alarm processing, an alarm based on the omission determination result of the omission determination unit 42 (omission alarm) may not be performed. Even in this case, the determination of the presence or absence of visual recognition by the visual recognition determination unit 41 is performed in order to carry out the special alarm processing and abnormal notification processing described above.

[0131] Alternatively, information relating to the judgment result of the missed detection by the missed detection unit 42 may be collected in the storage unit 70, and the first terminal device T1 (alarm unit 82) may provide the driver with entries (hereinafter referred to as "review entries") generated based on the collected information for the driver to review their past driving. Review entries may include, for example, information (strings, images, etc.) that the driver can identify as potentially dangerous objects that they may have missed. Review entries may be generated by the driving monitoring device 1 (information processing device 2) or by the first terminal device T1 (alarm unit 82).

[0132] In the information processing systems 100 and 100A described in the above embodiments, the driving monitoring device 1 (notification control unit 54) can also send notification information to both the first terminal device T1 and the second terminal device T2. In other words, the notification unit 84 that controls the notifications (first notification, second notification) between the first terminal device T1 and the second terminal device T2 can also be controlled by the notification control unit 54.

[0133] In the first embodiment, the driver monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as different devices, but they can also be configured as an integrated device. For example, the driver monitoring device 1 may actually include all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, alarm unit 82, and notification unit 84. Alternatively, all these functions may actually be included in the first terminal device T1 or the vehicle M. Similarly, in the second embodiment, all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and alarm unit 82 may actually be included in the driver monitoring device 1, and all these functions may also be included in the first terminal device T1. A camera other than the one included in the driver monitoring device 1 may also perform some or all of the functions of the camera unit 10. For example, a driver monitoring camera that captures the driver may capture facial images.

[0134] The driving monitoring device 1 can also be actually installed using multiple information processing devices. For example, the functions of the information processing device 2 can also be actually installed in the first terminal device T1 or the vehicle M (e.g., ECU). Alternatively, the driving monitoring device 1 can be actually installed using a device such as a cloud. For example, in the driving monitoring device 1, the detection unit 30, processing unit 40, control unit 50, and storage unit 70 can also be actually installed in different information processing devices. For example, the storage unit 70 can also be distributed among multiple information processing devices.

[0135] The implementation methods and variations described above can be represented as follows.

[0136] An information processing device comprising:

[0137] Storage medium, which stores computer-readable instructions; and

[0138] The processor, which is connected to the storage medium,

[0139] The processor performs the following processing by executing computer-readable instructions:

[0140] Obtain images of the vehicle's surroundings and the driver's face.

[0141] Detect risky objects in the surrounding image;

[0142] The driver is determined to have visually identified the risky object based on information relating to the location of the risky object and information relating to the facial image.

[0143] Based on whether any risk-avoidance actions to reduce the risk level are detected after the driver has visually identified the hazardous object, it is determined whether the driver has experienced a first anomaly. This risk level represents the degree of risk of a collision between the vehicle and the hazardous object.

[0144] If it is determined that the driver has experienced the first abnormality, the notification unit will issue a first notification.

[0145] Without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements, and the above embodiments and variations can also be appropriately combined.

Claims

1. An information processing device, wherein, The information processing device includes: The acquisition unit acquires images of the vehicle's surroundings and the driver's face. The risk detection unit detects risky objects in the surrounding image; The visual recognition determination unit determines whether the driver has visually recognized the risky object based on information relating to the location of the risky object and information relating to the facial image. The anomaly determination unit determines whether the driver has experienced a first anomaly based on whether a risk avoidance action that reduces the risk level is detected after the driver has visually identified the risk object. The risk level indicates the degree of risk of the vehicle colliding with the risk object. as well as The notification control unit, upon determining that the driver has experienced the first abnormality, will issue a first notification.

2. The information processing apparatus according to claim 1, wherein, The information processing device also includes an alarm control unit, which, if it is determined that the driver has visually identified the risky object but has not detected any risk avoidance action, causes the alarm control unit to sound an alarm.

3. The information processing apparatus according to claim 2, wherein, After determining that the driver has visually identified the risky object, even if the risk avoidance action is not detected, if the risk avoidance action is subsequently detected based on a change in the driver's facial orientation and / or line of sight in the facial image, the alarm control unit will not trigger the alarm.

4. The information processing apparatus according to any one of claims 1 to 3, wherein, If the facial image detects a change in the orientation of the driver's face and / or the direction of their gaze exceeding a predetermined threshold, the anomaly determination unit determines that the driver has experienced a second anomaly. If the notification control unit determines that the driver has experienced the second abnormality, it will issue a second notification.

5. The information processing apparatus according to any one of claims 1 to 3, wherein, If, within a specified period, no risk avoidance action by the driver is detected after the driver has visually identified the risky object, and such event occurs more than a specified number of times, the anomaly determination unit determines that the driver has committed the first anomaly.

6. The information processing apparatus according to any one of claims 1 to 3, wherein, The notification control unit causes the notification unit to send the first notification at a time other than when the driver is driving the vehicle.

7. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to any one of claims 1 to 3; A first terminal device, which is used by the driver and is the notification unit; and The second terminal device, used in medical institutions The first notification includes a recommendation to seek treatment at the medical facility mentioned. The first terminal device causes the second terminal device to make a notification based on information input by the driver indicating consent to the medical examination.

8. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to any one of claims 1 to 3; A first terminal device, which is used by the driver and is not the notification unit that controls the first notification by the notification control unit; as well as The second terminal device is for use by a user different from the driver, and is the notification unit that controls the first notification from the notification control unit. The second terminal device causes the first terminal device to make a notification based on information input by the user through the user's operation.

9. An information processing method, wherein, The information processing method causes the computer to perform the following processes: Obtain images of the vehicle's surroundings and the driver's face. Detect risky objects in the surrounding image; The driver is determined to have visually identified the risky object based on information relating to the location of the risky object and information relating to the facial image. Whether the driver has experienced a first abnormality is determined based on whether any risk avoidance action to reduce the risk level is detected after the driver has visually identified the risk object. This risk level indicates the degree of risk of the vehicle colliding with the risk object. as well as If it is determined that the driver has experienced the first abnormality, the notification unit will issue a first notification.

10. A storage medium that stores a non-transitory storage medium capable of being read by a computer, wherein, The program is used to cause the computer to perform the following processes: Obtain images of the vehicle's surroundings and the driver's face. Detect risky objects in the surrounding image; The driver is determined to have visually identified the risky object based on information relating to the location of the risky object and information relating to the facial image. Whether the driver has experienced a first abnormality is determined based on whether any risk avoidance action to reduce the risk level is detected after the driver has visually identified the risk object. This risk level indicates the degree of risk of the vehicle colliding with the risk object. as well as If it is determined that the driver has experienced the first abnormality, the notification unit will issue a first notification.

Images