Information processing apparatus, information processing system, information processing method, and storage medium

Abstract

The present application provides an information processing device, an information processing system, an information processing method, and a storage medium capable of detecting a narrow field of view of a driver. The information processing device includes: a risk detection unit that detects a risk object located in the periphery of a vehicle; an alarm control unit that causes an alarm display unit provided in the vehicle to display an alarm corresponding to the position of the risk object; an acquisition unit that acquires a facial image of a driver of the vehicle; an identification determination unit that determines whether the driver can recognize the alarm by the alarm display unit based on the position at which the alarm is displayed and information related to the facial image; a storage control unit that causes history information in which the position at which the alarm is displayed, whether the driver can recognize the alarm, and a line of sight detected from the facial image are associated to be stored in a storage unit; and an evaluation unit that evaluates the visual recognition characteristics of the driver based on the history information.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and storage medium. Background Technology

[0002] In recent years, efforts have been intensifying to provide sustainable transportation systems that also take into account vulnerable groups among transportation participants, particularly the elderly, people with disabilities, and children. Research and development are particularly focused on further improving the safety and convenience of transportation through developments related to mobility facilities for the elderly and people with disabilities (see, for example, Patent Document 1 below).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2021-144505 Summary of the Invention

[0004] When a driver suffers from a condition called narrowed field of vision (visual impairment), their ability to avoid dangerous objects is reduced. However, drivers are sometimes unaware of their narrowed field of vision. When drivers continue driving without realizing the narrowed field of vision and the resulting impairment, it can impair traffic safety and convenience.

[0005] The solutions involved in this invention were made in consideration of such circumstances, and one of their objectives is to provide an information processing device, information processing system, information processing method, and storage medium capable of detecting a driver's narrow field of vision, thereby contributing to the development of sustainable transportation systems.

[0006] To address the aforementioned issues, the present invention employs the following solution.

[0007] (1): One aspect of the present invention relates to an information processing apparatus, wherein the information processing apparatus comprises: a risk detection unit that detects a risky object located around a vehicle; an alarm control unit that causes an alarm display unit provided on the vehicle to display an alarm corresponding to the location of the risky object; an acquisition unit that acquires a facial image of the driver of the vehicle; a recognition determination unit that determines whether the driver can recognize the alarm issued by the alarm display unit based on the location where the alarm is displayed and information relating to the facial image; a storage control unit that stores in a storage unit historical information relating the location where the alarm is displayed, whether the driver can recognize the alarm, and a gaze detected from the facial image; and an evaluation unit that evaluates the driver's visual recognition characteristics based on the historical information.

[0008] (2): In the above (1) scheme, if the evaluated visual recognition characteristics are abnormal, the alarm control unit may cause the alarm display unit to display the location change of the alarm based on the information input by the driver's operation.

[0009] (3): In the above (2) scheme, the information processing device may also include: a passenger detection unit that detects passengers riding in the vehicle; and a reporting control unit that causes the reporting unit to report when the location where the alarm is displayed is changed and the passenger is detected.

[0010] (4): In any of the above schemes (1) to (3), the information processing device may also include a notification control unit that causes the notification unit to issue a notification when the evaluated visual recognition characteristics are abnormal.

[0011] (5): In the above (4) scheme, the notification control unit may also make the notification unit perform the notification at a time other than when the driver is driving the vehicle.

[0012] (6): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing device of the above-described (4) or (5) aspect; a first terminal device for use by the driver and being the notification unit; and a second terminal device used in a medical institution, wherein the notification includes a recommendation to receive treatment at the medical institution, and the first terminal device causes the second terminal device to make the notification based on information input by the driver indicating an acceptance of the treatment.

[0013] (7): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing device of the above-described (4) or (5); a first terminal device for use by the driver and not controlled by the notification control unit; and a second terminal device for use by a user different from the driver and controlled by the notification control unit, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

[0014] (8): One aspect of the present invention relates to an information processing method, wherein the information processing method causes a computer to perform the following processing: detecting a risky object located around a vehicle; causing an alarm display unit provided on the vehicle to display an alarm corresponding to the location of the risky object; acquiring a facial image of the driver of the vehicle; determining, based on the location where the alarm is displayed and information relating to the facial image, whether the driver can recognize the alarm issued by the alarm display unit; storing in a storage unit a history information linking the location where the alarm is displayed, whether the driver can recognize the alarm, and a gaze detected from the facial image; and evaluating the driver's visual recognition characteristics based on the history information.

[0015] (9): One aspect of the present invention relates to a storage medium that is a non-transitory storage medium capable of being read by a computer and storing a program, wherein the program is used to cause the computer to perform the following processing: detecting a risky object located around a vehicle; causing an alarm display unit provided on the vehicle to display an alarm corresponding to the location of the risky object; acquiring a facial image of the driver of the vehicle; determining, based on the location where the alarm is displayed and information relating to the facial image, whether the driver can recognize the alarm issued by the alarm display unit; storing in a storage unit historical information that establishes a connection between the location where the alarm is displayed, whether the driver can recognize the alarm, and a gaze detected from the facial image; and evaluating the driver's visual recognition characteristics based on the historical information.

[0016] According to the solution of the present invention, it is possible to detect a driver's narrow field of vision. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating an example of the structure and usage environment of a driving monitoring device that includes the information processing apparatus according to the first embodiment.

[0018] Figure 2 This is a diagram illustrating an example of a driver monitoring device.

[0019] Figure 3 This is an example of an image obtained by a driving monitoring device.

[0020] Figure 4 This diagram illustrates the risk warning processing performed by the driving monitoring device.

[0021] Figure 5 This is a diagram illustrating an example of information detected by the testing department.

[0022] Figure 6 This is an example diagram showing an alarm displayed on the alarm display unit.

[0023] Figure 7 This diagram illustrates the identification and judgment processing performed by the driver monitoring device.

[0024] Figure 8 This is an example diagram representing resume information.

[0025] Figure 9 This diagram illustrates the abnormal notification processing performed by the driving monitoring device.

[0026] Figure 10 This diagram illustrates an example of a change in the location of an alarm displayed by the alarm display unit.

[0027] Figure 11 This diagram illustrates the passenger report processing performed by the driver monitoring device.

[0028] Figure 12 This diagram illustrates the exception notification processing in the information processing system according to the second embodiment.

[0029] Figure 13 This is an example diagram showing an alarm displayed on the alarm display unit.

[0030] Figure 14 This is an example diagram showing an alarm displayed on the alarm display unit. Detailed Implementation

[0031] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the rearward direction of the vehicle is defined as the negative X direction, the rightward direction of the vehicle's width relative to the positive X direction is defined as the positive Y direction, the leftward direction is defined as the negative Y direction, and the direction orthogonal to the X and Y directions and the height direction of the vehicle is defined as the positive Z direction.

[0032] <First Implementation>

[0033] [Overall Structure]

[0034] Figure 1This diagram illustrates an example of the structure and usage environment of a driver monitoring device 1, including the information processing device 2 according to the first embodiment. The information processing device 2 is, for example, a driver monitoring device 1 retrofitted into a vehicle (hereinafter referred to as "vehicle M") such as a dashcam. That is, the driver monitoring device 1 (information processing device 2) can be mounted and removed from the vehicle M via a mounting / unmounting unit (hereinafter referred to as "mounting / unmounting unit 16"). The information processing device 2 is, for example, a device not connected to the vehicle M's in-vehicle network. An in-vehicle network refers to an in-vehicle network that connects to the vehicle's control-related ECU (Electronic Control Unit), which uses communication standards such as CAN (Controller Area Network), and is connected to the vehicle M's drive, braking, steering, various vehicle sensors, driving-related operating components, and operation buttons. The vehicle M's network for HMI (Human Machine Interface) can also be excluded from the above-described in-vehicle network.

[0035] Figure 2 This diagram illustrates an example of a driver monitoring device 1. The driver monitoring device 1 is, for example, installed near the windshield of a vehicle M, and located in front of the driver. The driver monitoring device 1 stores images (moving images) captured by the camera unit 10, for example, in the storage unit 70 (described later). The driver monitoring device 1 may also be, for example, equipped with a display device such as an LCD (Liquid Crystal Display) and capable of displaying the images (moving images) stored in the storage unit 70 on the display device.

[0036] The driving monitoring device 1 includes, for example, a camera unit 10, a loading and unloading unit 16, a reporting unit 18, and an information processing unit 2. In the illustrated example, the camera unit 10 includes a front camera 12 and a rear camera 14.

[0037] The front camera 12 and the rear camera 14 are each, for example, digital cameras utilizing solid-state imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) captures images of the area in front of the vehicle M from the location where the driver monitoring device 1 is installed. The rear camera 14 repeatedly (periodically) captures images of the area behind the vehicle M and the interior of the vehicle M from the location where the driver monitoring device 1 is installed. The horizontal viewing angle captured by the front camera 12 and the rear camera 14 is, for example, 180° or more. That is, the driver monitoring device 1 (camera unit 10) captures a 360° range around the vehicle M from its installation location. The front camera 12 and the rear camera 14 may each be, for example, fisheye cameras including fisheye lenses. Alternatively, the front camera 12 and the rear camera 14 may each be wide-angle cameras or the like. In this embodiment, the use of the front camera 12 and the rear camera 14 is described, but other cameras may be used instead of these or other alternatives. That is, the camera unit 10 can be configured to capture images used in this embodiment (surrounding images, facial images, and in-vehicle images described later).

[0038] The mounting / unmounting section 16 is, for example, a component for mounting the driver monitoring device 1 onto the vehicle M. The mounting / unmounting section 16 can be any component such as a suction cup, seal, bracket, or other support member. The mounting / unmounting section 16 may also include a power supply cable (wiring, etc.) for supplying power to the driver monitoring device 1. Regarding the reporting section 18, see the following description.

[0039] Figure 3 This is a diagram illustrating an example of an image acquired by the driver monitoring device 1. That is, Figure 3 This diagram illustrates an example of an image captured by the camera unit 10. The front camera 12 captures images of the front of the vehicle M through the windshield, the driver's side side window, and the passenger side side window. The rear camera 14 captures images of the interior of the vehicle M (including the driver and passengers) and the rear of the vehicle M through the driver's side side window, the passenger side side window, and the rear window.

[0040] In the image captured by the front camera 12 (hereinafter referred to as the "front image"), the subject includes objects such as other vehicles, pedestrians, bicycles, fixed objects, road markings, etc., that can be seen through the front windshield and exist in front of the vehicle M, as well as objects that can be seen through the left and right (front) side windows of the vehicle M.

[0041] The image captured by the rear camera 14 (hereinafter referred to as the "rear image") shows the interior of the vehicle M, as well as objects visible through the rear window that are located behind the vehicle M, and objects visible through the left and right side windows that are located to the left and right (rear) of the vehicle M. Therefore, in the rear image captured by the rear camera 14, the driver of the vehicle M equipped with the driver monitoring device 1 is also reflected as the subject.

[0042] The front and rear images include images of the surrounding conditions of vehicle M (hereinafter referred to as "surrounding images"). The rear image includes an image of the driver's face of vehicle M (hereinafter referred to as "face image"). The rear image includes an image of the interior (cabin interior) of vehicle M (hereinafter referred to as "cabin interior image"). The driver monitoring device 1 can be installed at any location where the camera unit 10 can capture the surrounding images, face images, and cabin interior images.

[0043] Information processing device 2 (refer to) Figure 1 Based on the results of referencing the surrounding images output by the camera unit 10, the alarm display unit 82 (referencing) displays the results. Figure 2 The information processing device 2 outputs an alarm based on the result of referring to the facial image. The notification can include a recommendation to a medical institution for treatment (details will be described later). The first terminal device T1 sends a notification to the second terminal device T2 based on information input by the driver of vehicle M indicating acceptance of treatment. Sometimes, the driver monitoring device 1, the first terminal device T1, the second terminal device T2, and the alarm display unit 82 are combined and referred to as "information processing system 100".

[0044] The driver monitoring device 1, the first terminal device T1, the second terminal device T2, and the alarm display unit 82 are connected in a communicable manner via a network NW. The network NW can be a wireless communication network or a wired communication network. The network NW can also be constructed using, for example, the Internet, or using LAN (Local Area Network), Bluetooth (registered trademark), etc. The network NW can also be constructed by combining multiple networks.

[0045] The first terminal device T1 is, for example, a smartphone, tablet, or other portable terminal device used by a driver to drive a vehicle M equipped with a driver monitoring device 1. The first terminal device T1 executes, for example, an application program for receiving notifications from the information processing device 2. The application program causes a display device to show an image based on information sent by the information processing device 2 (e.g., notification information described later), and causes a speaker to emit sound. The first terminal device T1 is also referred to as a notification unit 84.

[0046] Alternatively, the navigation, display, and speaker of vehicle M can be used instead of the first terminal device T1. That is, the navigation, display, and speaker of vehicle M can also output notifications based on the instructions of information processing device 2. In other words, the navigation, display, and speaker of vehicle M can also be a notification unit 84. Alternatively, a display unit (notification unit 84), a speaker (notification unit 84), etc., can be provided on the driver monitoring device 1 instead of the first terminal device T1.

[0047] The second terminal device T2 is a terminal device used in a medical institution. The second terminal device T2 can also be a terminal used by personnel belonging to the medical institution (e.g., doctors, nurses, etc.). The second terminal device T2 is an information processing device for use by a user different from the driver using the first terminal device T1. The second terminal device T2 can also be, for example, a smartphone, tablet, personal computer, etc. In the second terminal device T2, for example, an application program for receiving notifications from the first terminal device T1 is executed. The application program causes a display device to display an image based on information sent by the first terminal device T1 and causes a speaker to emit sound.

[0048] The alarm display unit (alarm display device) 82 displays the alarm based on the control performed by the information processing unit 2. That is, the alarm display unit 82 outputs visually recognizable alarms. For example... Figure 2 As shown, the warning display unit 82 may, for example, be a bar-shaped light installed on the vehicle M. The bar-shaped light, serving as the warning display unit 82, may be arranged along the lower edge of the windshield. The two ends of the bar-shaped light may, for example, be located at the lower corners of the windshield. The warning display unit 82 has a communication interface (not shown) for wireless or wired communication with information processing devices (external devices) such as the driver monitoring device 1, the first terminal device T1, and the second terminal device T2 via a network NW (described later).

[0049] like Figure 1 As shown, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, a face orientation detection unit 36, and a passenger detection unit 38. The processing unit 40 includes an identification determination unit 42, a storage control unit 44, an evaluation unit 46, and an anomaly determination unit 48. The control unit 50 includes an alarm control unit 52, a notification control unit 54, and a reporting control unit 56.

[0050] The acquisition unit 20, detection unit 30, processing unit 40, and control unit 50 are equipped with hardware processors such as CPUs (Central Processing Units) and storage devices (storage devices with non-transitory storage media) storing programs (software). The processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented using hardware such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units) (including the circuitry). The functions of each component can also be realized through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs.

[0051] The program (software) can be pre-stored in a storage device (not shown) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory (a storage device with a non-temporary storage medium), or it can be stored in a removable storage medium (a non-temporary storage medium) such as a memory card, and installed in the storage device by being mounted on the driving monitoring device 1 via the storage medium. Alternatively, the program (software) can be pre-downloaded from other computer devices via short-range or wide-area communication using an application program executed in the first terminal device T1 and / or the second terminal device T2, and then sent from the first terminal device T1 and / or the second terminal device T2, thereby installing in the storage device.

[0052] The storage unit 70 stores, for example, setting information 72, history information 74, programs (such as the programs described above), and various other information. The storage unit 70 can also be implemented using the various storage devices described above or EEPROM (Electrically Erasable Programmable Read Only Memory). The communication unit 60 is an interface for wireless or wired communication with information processing devices (external devices) such as the first terminal device T1 and the second terminal device T2, and the alarm display unit 82 via a network NW.

[0053] The driver monitoring device 1 (information processing device 2) performs at least risk warning processing, identification and determination processing, anomaly notification processing, and passenger report processing. Risk warning processing detects risky objects located around the vehicle M based on surrounding images and displays an alarm on the alarm display unit 82. Identification and determination processing determines whether the driver can recognize the alarm issued by the alarm display unit 82. Anomaly notification processing detects visual impairments in the driver based on whether the driver can recognize the alarm and notifies the notification unit 84. Anomaly notification processing also includes changing the position of the alarm displayed on the alarm display unit 82 based on the driver's instructions (alarm position change processing). Passenger report processing involves the reporting unit 18 reporting when the alarm display position changes and a passenger is detected.

[0054] A risky object refers to an object that vehicle M should avoid, for example. A risky object is, for example, an object that would interfere with vehicle M when it is moving as it is, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose approach level becomes more than a certain level after a specified time, taking into account the object's position, direction of movement, and speed, as well as the position, method of movement, and speed of vehicle M. A risky object can also be an object approaching vehicle M from the front. The object can be a pedestrian, bicycle, vehicle, or other traffic participants, or an object placed on the road, a fallen object, or other object other than a traffic participant.

[0055] [Risk Alert Handling (Setting Information 72 to "Normal" Case)]

[0056] Figure 4 This diagram illustrates the risk warning processing performed by the driving monitoring device 1 (information processing device 2).

[0057] The acquisition unit 20 acquires peripheral images from the camera unit 10. The acquisition unit 20 repeatedly (periodically) captures peripheral images, thus acquiring peripheral images repeatedly (periodically). The acquisition unit 20 can also acquire peripheral images by establishing a relationship between the peripheral images and the time when the peripheral images were captured (capture time).

[0058] The risk detection unit 32 detects risky objects in the surrounding image. "Detecting risky objects" refers to, for example, detecting the presence of a risky object and the position of that object relative to the vehicle M (driver monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M can be pre-stored in the storage unit 70. The risk detection unit 32 can also detect the position of the risky object relative to the vehicle M based on this reference information.

[0059] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, and also detects the direction of the risk object's presence (approach direction) relative to the vehicle M. "Direction of presence of the risk object" refers to, for example, the direction of presence of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, "direction of presence of the risk object" will sometimes be referred to as "risk presence direction".

[0060] Figure 5 This is a diagram illustrating an example of information detected by the detection unit 30. In Figure 5 In the example, pedestrians and other vehicles detected by the detection unit 30 are displayed based on the vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that may become risk objects. In the information detected by the detection unit 30, instead of the vehicle M, the direction of the line of sight, and the direction of the face, traffic participants (objects) that may become risk objects detected by the detection unit 30 may be displayed based on the vehicle M, the direction of the vehicle M, and / or the direction of travel of the vehicle M. Figure 5 In the example, the orientation of the face detected in the anomaly notification processing described later is also shown. The detection unit 30 can also be configured to detect lanes, pedestrian crossings, etc., in... Figure 5 The example also shows the lanes and pedestrian crossings detected by the detection unit 30.

[0061] return Figure 4 The alarm control unit 52 causes the alarm display unit 82 to display an alarm based on the risk location detected by the risk detection unit 32 and the setting information 72 stored in the storage unit 70. For example, the alarm control unit 52 sends alarm information to the alarm display unit 82 via the communication unit 60. The alarm information includes, for example, information indicating the content of the alarm that the alarm display unit 82 should display. The alarm display unit 82 displays the alarm based on the alarm information.

[0062] Figure 6 This diagram illustrates an example of an alarm displayed by the alarm display unit 82. In the event that a dangerous object is approaching the vehicle M from the left (front left), the alarm control unit 52, for example, causes the left-side area of ​​the alarm display unit 82 to illuminate. Figure 6 (Lower left). In the event that a dangerous object is approaching vehicle M from the right (front right), the alarm control unit 52, for example, causes the right-side area of ​​the alarm display unit 82 to illuminate (…). Figure 6 (Lower right). The alarm display unit 82 provides an alarm to the driver by emitting light in this way. Hereinafter, the area in the alarm display unit 82 that emits light for the purpose of providing an alarm will be referred to as the luminous area AR.

[0063] Here, the alarm information mentioned above includes alarm location information indicating the location where the alarm display unit 82 should display the alarm. Figure 6 In the example, the alarm location information can also be information specifying the luminous area AR. The alarm control unit 52 can, for example, determine the displayable range RG and display length L based on the risk location and setting information 72, and determine the luminous area AR based on the determined displayable range RG and display length L. The following describes an example of the method for determining the displayable range RG and display length L, and the luminous area AR based on them.

[0064] The displayable range RG is the range within which the alarm display unit 82 can display the alarm (i.e., the luminous area AR can be configured). The displayable range RG is determined, for example, based on setting information 72. Setting information 72 is, for example, information that can take a "normal" or "special" value. Setting information 72 is preset to "normal". When setting information 72 is "normal", the alarm control unit 52 determines the displayable range RG in such a way that the left end RG1 of the displayable range RG coincides with the left end 82l of the alarm display unit 82, and the right end RGr of the displayable range RG coincides with the right end 82r of the alarm display unit 82. The case where setting information 72 is "special" will be explained together with the abnormal notification processing described later.

[0065] In the case where a hazardous object is approaching vehicle M from the left ( Figure 6 The alarm control unit 52 determines the luminous area AR such that the left end of the luminous area AR aligns with the left end RG1 of the displayable range RG, and the right end of the luminous area AR aligns with the position P1. Here, the position P1 is a position to the right of the left end RG1 of the displayable range RG, away from the display length L. The display length L is determined, for example, based on the risk position. For example, the closer the risky object is to the vehicle M, the longer the display length L is determined, and the further away the risky object is from the vehicle M, the shorter the display length L is determined. Thus, the luminous area AR is determined such that the left end RG1 of the displayable range RG is set as the base (left end), and the position of the front end (right end) varies according to the risk position. The driver can identify the degree to which a risky object approaching from the left is approaching the vehicle M based on the display length L of the luminous area AR.

[0066] In the case where a hazardous object is approaching vehicle M from the right ( Figure 6The alarm control unit 52 determines the luminous area AR such that the right end of the luminous area AR aligns with the right end RGr of the displayable range RG, and the left end of the luminous area AR is located at position Pr. Here, position Pr is a position that moves to the left of the display length L from the right end RGr of the displayable range RG. The display length L can also be determined in the same way as described above. Thus, the luminous area AR is determined such that the right end RGr of the displayable range RG is set as the base (right end), and the position of the front end (left end) varies according to the risk position. The driver can identify the degree to which a dangerous object approaching the vehicle M from the right based on the display length L of the luminous area AR.

[0067] As described above, the alarm control unit 52 causes the alarm display unit 82 to display an alarm corresponding to the location of the hazardous object and the setting information 72. The alarm information may also include information such as the luminous intensity, luminous color, and flashing speed of the designated luminous area AR. The alarm control unit 52 may also determine the luminous intensity, luminous color, and flashing speed of the luminous area AR based on the location of the hazardous object.

[0068] The alarm displayed by the alarm display unit 82 can also be performed, for example, during the time when the driver is driving the vehicle M (i.e., in real time). Examples of "the time when the driver is driving the vehicle M" include when the vehicle M is moving, when the vehicle M is temporarily stopped, etc. For example, the alarm information sent by the alarm control unit 52 can also be performed at the aforementioned times. Alternatively, the alarm information sent by the alarm control unit 52 can be performed at any time, and the alarm display unit 82 can control the timing of the alarm display so that the alarm display occurs at the aforementioned times.

[0069] [Identification and Judgment Processing]

[0070] Figure 7 This diagram illustrates the identification and judgment processing performed by the driving monitoring device 1 (information processing device 2).

[0071] The acquisition unit 20 acquires facial images from the camera unit 10. The acquisition unit 20 repeatedly (periodically) captures facial images, thus repeatedly (periodically) acquiring facial images. The acquisition unit 20 can also acquire facial images by linking them to the time they were captured (the shooting time).

[0072] The gaze detection unit 34 detects the driver's gaze in a facial image. Specifically, the gaze detection unit 34 analyzes the facial image using a prescribed image analysis method, thereby detecting the driver's gaze. "Gaze detection" refers, for example, to detecting the position of the driver's eyeballs and the direction of the gaze (the direction of the eyeballs).

[0073] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the face image (see also...). Figure 5 Specifically, the face orientation detection unit 36 ​​analyzes the facial image using a prescribed image analysis method, thereby detecting the orientation of the driver's face. The face orientation detection unit 36 ​​can also detect the position of the driver's face in addition to its orientation. Hereinafter, the information combining "face orientation" and "face position" will sometimes be referred to as "facial information." The face orientation detection unit 36 ​​detects facial information in the facial image.

[0074] The recognition and determination unit 42 determines whether the driver can recognize the alarm based on the location of the alarm displayed on the alarm display unit 82, as well as the driver's gaze and / or facial information. "Gaze" and "facial information" are examples of "information involving facial images". The recognition and determination unit 42 may also obtain the location of the alarm based, for example, on the alarm information (alarm location information) output by the alarm control unit 52 in the above-described risk alarm processing.

[0075] For example, reference information (not shown) indicating the relationship between the orientation of the driver's eyes (gaze) and / or the orientation of their face in the facial image and their relative orientation with respect to the vehicle M in a predetermined direction (e.g., forward direction) can also be pre-stored in the storage unit 70. The recognition determination unit 42 can also refer to this reference information to determine whether the orientation of the gaze and / or the orientation of the face matches the location of the alarm. If the orientation of the gaze and / or the orientation of the face matches the location of the alarm, the recognition determination unit 42 can also determine that the driver has visually recognized the alarm. The so-called "the orientation of the gaze and / or the orientation of the face matches the direction of the risk" can, for example, be the case where there is a risk location on a straight line along the orientation of the gaze and / or the orientation of the face. Alternatively, the so-called "the orientation of the gaze and / or the orientation of the face matches the direction of the risk" can also be the case where there is a risk location in a cone-shaped region with the position of the viewpoint (eyeball) and / or the face as the apex, having a central axis along the orientation of the gaze and / or the orientation of the face, and widening laterally as it moves forward. If the direction of the driver's gaze and / or the direction of the face does not match the location of the alarm, the identification determination unit 42 may also determine that the driver has not visually recognized the alarm.

[0076] The identification determination unit 42 can also determine that the driver has not recognized the alarm if the state of the alarm not being visually recognized by the driver continues for a predetermined time or longer. The identification determination unit 42 can also determine that the driver has recognized the alarm if the driver visually recognizes the alarm, or if the state of the alarm not being visually recognized by the driver is cleared before the predetermined time has elapsed (i.e., the driver visually recognized the alarm even if it was not initially visually recognized before the predetermined time had elapsed). However, the method of identification determined by the identification determination unit 42 is not limited to the above and can be appropriately modified.

[0077] The identification determination unit 42 generates identification determination information based on the above determination result. The identification determination information includes, for example, the presence or absence of identification, the location of the alarm, and the line of sight. The determination performed by the identification determination unit 42 (generation of identification determination information) can also be performed whenever the alarm control unit 52 generates alarm information. In other words, the determination performed by the identification determination unit 42 (generation of identification determination information) can also be performed whenever the alarm display unit 82 displays an alarm.

[0078] The storage control unit 44 establishes a link between the time and the identification determination information (presence or absence of identification, location of the alarm, and line of sight) and stores it in the storage unit 70. The time corresponds, for example, to the time when the surrounding image used in line-of-sight detection was captured, or the time when the alarm was displayed on the alarm display unit 82 (display time). The aforementioned processing (storage processing) performed by the storage control unit 44 is performed, for example, whenever identification determination information is generated by the identification determination unit 42. Hereinafter, the information stored in the storage unit 70 will be referred to as history information 74.

[0079] Figure 8 This diagram illustrates an example of the history information 74. The storage control unit 44 repeatedly performs the aforementioned storage process, thereby accumulating multiple groups (hereinafter referred to as information groups G) in the history information 74, including information such as time, alarm location, presence or absence of identification, and line-of-sight. Specifically, the storage control unit 44 generates information group G whenever identification determination information is generated by the identification determination unit 42. Furthermore, the storage control unit 44 appends the generated information group G to the history information 74, thereby updating the history information 74. The history information 74 is used, for example, in the exception notification processing described below.

[0080] [Abnormal Notification Handling]

[0081] Figure 9 This diagram illustrates the abnormal notification processing performed by the driving monitoring device 1 (information processing device 2).

[0082] The evaluation unit 46 reads the history information 74 from the storage unit 70 and evaluates the driver's visual recognition characteristics based on the read history information 74. The reading of the history information 74 and the evaluation of visual recognition characteristics performed by the evaluation unit 46 can begin repeatedly at a predetermined period, or whenever the history information 74 is updated by the storage control unit 44, or can be initiated by instruction from the driver or others. The evaluation unit 46 generates evaluation information that assesses the driver's visual recognition characteristics.

[0083] Evaluation information can also be, for example, information calculated by dividing frequency information, which relates to the frequency at which the driver can recognize the alarm, into each direction within the driver's field of vision (hereinafter referred to as the direction within the field of vision). In other words, evaluation information can also be mapping information that establishes a relationship between frequency information and the direction within the driver's field of vision.

[0084] For example, the evaluation unit 46 can also calculate the alarm vector and the line-of-sight vector for each information group G. The alarm vector is a vector that starts from the position of the eyeball and ends at the position of the alarm. The line-of-sight vector is a vector that starts from the position of the eyeball and points towards the line of sight (eyeball) and has a predetermined magnitude. Furthermore, the evaluation unit 46 can determine which visual field direction the position of the alarm in information group G corresponds to by calculating the angle between the alarm vector and the line-of-sight vector. By determining the visual field direction based on the multiple information groups G contained in the history information 74, and by statistically processing the presence or absence of recognition for each visual field direction, frequency information can be calculated for each visual field direction. The frequency information can be, for example, the recognition rate. The recognition rate can also be defined as the value obtained by dividing the number of times the driver recognizes an alarm displayed in that visual field direction by the total number of times an alarm is displayed in that visual field direction.

[0085] The anomaly determination unit 48 determines whether there is an anomaly in the driver's visual recognition characteristics based on the evaluation information. For example, the anomaly determination unit 48 determines whether the driver has a narrow field of vision (visual field defect).

[0086] For example, the anomaly determination unit 48 can also compare the frequency information evaluated as described above with a predetermined threshold for each direction within the field of view. Furthermore, the anomaly determination unit 48 can also determine whether there are any directions within the field of view where alarm non-recognition (alarm oversight) occurs more frequently than the predetermined threshold. The anomaly determination unit 48 can also determine that the driver's visual recognition characteristics are abnormal (narrow field of view) if there are any directions within the field of view where alarm non-recognition occurs more frequently than the predetermined threshold. The anomaly determination unit 48 can also determine that the driver's visual recognition characteristics are not abnormal (narrow field of view) if there are no directions within the field of view where alarm non-recognition occurs more frequently than the predetermined threshold.

[0087] The aforementioned threshold can also be set to different values ​​for each direction within the visual field. Specifically, the threshold can be set such that the recognition rate decreases as one moves from the center of the visual field outwards. This is because even healthy individuals without visual field defects generally tend to experience a decline in vision and attention as they move away from the center of their visual field. The threshold can also be a value predetermined based on the visual recognition characteristics of a typical healthy individual. Alternatively, the threshold can be a value obtained by the anomaly determination unit 48 from past evaluations of the driver's visual recognition characteristics. In this case, the evaluation unit 46 can also link the evaluation information to time and store it in the storage unit 70, and the anomaly determination unit 48 can read the evaluation information stored in the storage unit 70 to perform the aforementioned anomaly determination.

[0088] The anomaly determination unit 48 generates anomaly determination information based on the above determination results. The anomaly determination information includes information indicating whether the driver's visual recognition characteristics are abnormal (narrow field of vision). The anomaly determination information may also include information indicating the direction within the field of vision where alarm non-recognition occurs more frequently than a specified threshold (hereinafter referred to as the direction of cognitive abnormality).

[0089] The notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information. For example, the notification control unit 54 sends the notification information to the first terminal device T1 (notification unit 84) via the communication unit 60. For example, the notification control unit 54 may also send the notification information to the first terminal device T1 if the anomaly determination information includes information indicating the presence of an anomaly in the driver's visual recognition characteristics (narrow field of vision) (i.e., if an anomaly exists in the evaluated visual recognition characteristics). It may also include information indicating the direction of the cognitive anomaly in the notification information if the anomaly determination information includes information indicating the absence of an anomaly in the driver's visual recognition characteristics (narrow field of vision) (i.e., if no anomaly exists in the evaluated visual recognition characteristics). The notification control unit 54 may also not send the notification information to the first terminal device T1 if the anomaly determination information includes information indicating the absence of an anomaly in the driver's visual recognition characteristics (narrow field of vision) (i.e., if no anomaly exists in the evaluated visual recognition characteristics).

[0090] The first terminal device T1 (notification unit 84) outputs a notification based on notification information. The notification may also be made by displaying a string, image, or other object obtained based on the notification information on a display device of the first terminal device T1 (notification unit 84). The first terminal device T1 (notification unit 84) may also issue a notification indicating that the driver has experienced or may experience visual impairment. When information indicating the direction of the cognitive abnormality is included in the notification information, the first terminal device T1 (notification unit 84) may also output an item (e.g., a string, image, etc.) that the driver or others can recognize as the direction of the cognitive abnormality.

[0091] The notification issued by the first terminal device T1 (notification unit 84) may also include a suggestion to seek medical treatment at a medical institution (e.g., a hospital the driver frequently visits). In this case, the first terminal device T1 (notification unit 84) may also allow the driver to input information indicating whether the driver agrees to the medical treatment. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a virtual button or the like indicating agreement to the medical treatment. Such a button or the like functions as an agreement information acquisition unit, which acquires information indicating the driver's intention to agree to the medical treatment (hereinafter referred to as agreement information). Furthermore, if the driver operates the first terminal device T1 (notification unit 84) and the agreement information acquisition unit acquires the agreement information, the first terminal device T1 (notification unit 84) may also cause the second terminal device T2 to issue a notification indicating that the driver has agreed to the medical treatment.

[0092] However, as long as the form is recognizable by the driver, the form of the notification issued by the first terminal device T1 (notification unit 84) is not particularly limited and can be appropriately changed.

[0093] The notification sent by the first terminal device T1 (notification unit 84) can also be sent at times other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, after the driver has finished driving the vehicle M, or when the driver has left the vehicle M and is at home. For example, the notification information sent by the notification control unit 54 can also be sent at the aforementioned times. Alternatively, the notification information sent by the notification control unit 54 can be sent at any time, and the first terminal device T1 (notification unit 84) controls the timing of the notification to ensure that the notification is sent at the aforementioned times.

[0094] Alternatively, if the driver operates the first terminal device T1 (notification unit 84) and the information acquisition unit obtains the permission information, the first terminal device T1 may update the setting information 72 stored in the storage unit 70 (alarm position change processing). Specifically, the first terminal device T1 may also send a message to the information processing device 2 via the network NW indicating its intention to change the setting information 72 to "special". Moreover, the information processing device 2 that obtains this message may also change (update) the setting information 72 from "normal" to "special". Alternatively, the first terminal device T1 (notification unit 84) may display a hypothetical button other than the permission information acquisition unit, and update the setting information 72 when the driver operates that button.

[0095] [Risk Alert Handling (Setting Information 72 as "Special")]

[0096] As described above, in risk alarm processing, the alarm control unit 52 causes the alarm display unit 82 to display an alarm based on the setting information 72 and the risk location detected by the risk detection unit 32 (see reference). Figure 4 The setting information 72 is changed to "Special" as described above, thereby changing the location of the alarm displayed through risk alarm processing from the case where the setting information 72 is "Normal". The following details this change in alarm location. The processing performed by the acquisition unit 20 and the risk detection unit 32... Figure 4 The same applies to the other two cases, so the explanation is omitted.

[0097] Figure 10 This diagram illustrates an example of a change in the location of an alarm displayed by the alarm display unit 82. Figure 10 The area shown in the accompanying diagram, along with the reference NR, is the normal field of vision. The normal field of vision NR refers to the area within driver D's field of vision where driver D can normally perceive the alarm (i.e., not in the direction of the cognitive abnormality). For example... Figure 10 As shown, when setting information 72 is "normal" and the display range RG is set to the entire alarm display unit 82, the entire or part of the illuminated area AR (alarm) may be located outside the normal field of vision NR. In this case, the driver D cannot correctly identify the display length L of the illuminated area AR (alarm).

[0098] Therefore, the alarm control unit 52 adjusts the displayable range RG differently depending on whether the setting information 72 is "special" or "normal". Specifically, when the setting information 72 is "special", the alarm control unit 52 reduces the displayable range RG to a range closer to the driver's seat (D) compared to when the setting information 72 is "normal". Figure 10 In the example, the left end RG1 of the visible range RG is located to the right (i.e., on the driver's side), thus reducing the visible range RG. This change in the visible range RG also changes the position of the illuminated area AR (i.e., the position where the alarm is displayed).

[0099] For example, when the setting information 72 is set to "special", the alarm control unit 52 can determine that the visible range RG is within the normal field of vision NR. The alarm control unit 52 can also estimate the normal field of vision NR based on evaluation information generated by the evaluation unit 46. Furthermore, the alarm control unit 52 can determine that the visible range RG is within the estimated normal field of vision NR. Therefore, the luminous area AR (alarm) is also within the normal field of vision NR. In this way, by narrowing the visible range RG to the driver's side (for example, making the visible range RG within the normal field of vision NR), the alarm can be properly recognized by the driver D who experiences a narrowed field of vision.

[0100] [Passenger Report Handling]

[0101] When the driver changes the setting information 72 to "special", the position of the alarm displayed on the alarm display unit 82 changes as described above. However, passengers viewing the alarm displayed in the changed position may feel uncomfortable. Therefore, the driver monitoring device 1 (information processing device 2) performs passenger reporting processing to inform passengers of this intention when the alarm position changes.

[0102] Figure 11 This diagram illustrates the passenger report processing performed by the driver monitoring device 1 (information processing device 2).

[0103] The acquisition unit 20 acquires images of the interior of the vehicle from the camera unit 10. The acquisition unit 20 repeatedly (periodically) captures images of the interior of the vehicle, thus the acquisition unit 20 repeatedly (periodically) acquires images of the interior of the vehicle.

[0104] The passenger detection unit 38 detects passengers in the in-vehicle images. A passenger refers to a person who is riding in the vehicle M and is not the driver. Specifically, the passenger detection unit 38 analyzes the in-vehicle images using a prescribed image analysis method to detect passengers. "Detecting passengers" means, for example, detecting the presence of passengers.

[0105] The report control unit 56 triggers the report unit 18 to report based on the presence or absence of a passenger detected by the passenger detection unit 38 and the setting information 72 stored in the storage unit 70. Specifically, the report control unit 56 triggers the report unit 18 to report when the setting information 72 is "special" (i.e., the location of the alarm display has been changed) and a passenger is detected by the passenger detection unit 38. The report control unit 56 may also prevent the report unit 18 from reporting once until a new passenger is detected by the passenger detection unit 38.

[0106] The reporting unit 18 makes a report under the control of the reporting control unit 56. The reporting unit 18 may be, for example, a speaker, a display device, etc. For example, the reporting unit 18 makes a report based on strings, images, sounds, etc., under the control of the reporting control unit 56. The report made by the reporting unit 18 may also include content such as "informing that the location of the alarm display has been changed".

[0107] According to the first embodiment described above, the information processing device includes: a risk detection unit 32 that detects risky objects located around the vehicle M; an alarm control unit 52 that causes an alarm display unit 82 installed on the vehicle M to display an alarm corresponding to the location of the risky object; an acquisition unit 20 that acquires a facial image of the driver of the vehicle M; a recognition determination unit 42 that determines whether the driver can recognize the alarm issued by the alarm control unit 52 based on the location where the alarm is displayed and information related to the facial image; a storage control unit 44 that stores historical information 74, which establishes a connection between the location where the alarm is displayed, whether the driver can recognize the alarm, and the gaze detected from the facial image, in a storage unit 70; and an evaluation unit 46 that evaluates the driver's visual recognition characteristics based on the historical information 74. Thus, it is possible to detect narrowing of the driver's field of vision.

[0108] <Second Implementation>

[0109] Next, the second embodiment will be described. The basic structure is the same as that of the first embodiment. Therefore, the same reference numerals are used to label the same structures and their descriptions are omitted; only the differences will be described.

[0110] Figure 12 This diagram illustrates the exception notification processing in the information processing system 100A according to the second embodiment. The second terminal device T2 in the second embodiment can also be a terminal device used in a medical institution, similar to that in the first embodiment. Alternatively, the second terminal device T2 in the second embodiment can also be an information processing device used by insurance company employees, etc. The second terminal device T2 can be any information processing device used by a user different from the driver.

[0111] In the second embodiment, the driving monitoring device 1 (notification control unit 54) sends the notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the second embodiment, the notification unit 84 controlled by the notification control unit 54 is the second terminal device T2, not the first terminal device T1. The second terminal device T2, for example, executes an application program for receiving notifications from the information processing device 2. The application program causes the display device to display images, strings, etc., obtained based on the information (notification information, etc.) sent by the information processing device 2, or causes the speaker to emit sound.

[0112] The second terminal device T2 uses information input by the user to notify the first terminal device T1. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0113] The input unit 92 and the transmission control unit 94 are equipped with hardware processors such as CPUs and storage devices (storage devices with non-transitory storage media) storing programs (software), and the processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented by hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component can be realized through the cooperation of software and hardware. Some or all of these components can also be implemented by dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with information processing devices such as the information processing device 2 and the first terminal device T1 via a network NW.

[0114] The input unit 92 accepts user operations and inputs them to the second terminal device T2. The input unit 92 can be configured using existing input devices such as keyboards, clicking devices (mouse, tablet, etc.), buttons, or touch panels. The input unit 92 can also be an interface for connecting an input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in the input device based on the user's operation to the second terminal device T2. The input unit 92 can be configured arbitrarily as long as it is capable of inputting instructions obtained based on user operations to the second terminal device T2.

[0115] For example, the user confirms the content of the notification sent by the driving monitoring device 1 (notification control unit 54) to the second terminal device T2, and studies the content that should be notified to the driver. The content to be notified to the driver may include, for example, a recommendation to a medical institution for treatment of a patient with a narrow field of vision. The user inputs information, including the studied notification content, into the second terminal device T2 via the operation input unit 92. The transmission control unit 94 transmits the information input by the user to the first terminal device T1 via the communication unit 96. The first terminal device T1 then notifies the driver based on the transmitted information.

[0116] The information processing system 100A can also be configured to restrict the transmission of notification information from the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 by the driver operating the first terminal device T1 or the driving monitoring device 1. For example, the driver can operate the first terminal device T1 or the driving monitoring device 1 to generate restriction information indicating the presence or absence of such restriction on the transmission of notification information, and store the generated restriction information in the storage unit 70. Furthermore, the alarm control unit 52 can also decide whether to send notification information to the second terminal device T2 based on the restriction information stored in the storage unit 70. With this structure, the decision on whether to inform the user of the second terminal device T2 of any abnormal information involving the driver is made according to the driver's wishes. Therefore, the driver's privacy is easily protected.

[0117] According to the second embodiment described above, the information processing system 100A includes: an information processing device 2; a first terminal device T1, which is used by the driver and is a notification unit 84 that is not controlled by the notification control unit 54; and a second terminal device T2, which is used by a user different from the driver and is also controlled by the notification control unit 54 to notify the first terminal device T1, wherein the second terminal device T2 causes the first terminal device T1 to make a notification based on information input by the user's operation. Thus, it is possible to provide the driver using the first terminal device T1 with notification content obtained from research conducted by the user using the second terminal device T2.

[0118] <Third Implementation Method>

[0119] Next, the third embodiment will be described. The basic structure is the same as that of the first embodiment. Therefore, the same reference numerals are used to label the same structures and their descriptions are omitted; only the differences are described.

[0120] Figure 13 This is a diagram showing an example of an alarm displayed by the alarm display unit 82. Figure 13 The example shown corresponds to the case where setting information 72 is "normal". Figure 6 In the example shown, regarding the luminous area AR, the left end RG1 (i.e., the left end of the alarm display unit 82) or the right end RGr (i.e., the right end of the alarm display unit 82) of the displayable range RG is set as the base end, and the position of the front end (position P1, Pr) varies depending on the risk location. Figure 13 The example shown is an example involving the third embodiment, where the shape of the light-emitting region AR is similar to... Figure 6 The examples shown are different from those involved in the first embodiment. Therefore, in... Figure 13In the example shown, the end of the luminous area AR does not necessarily coincide with the left end RG1 (i.e., the left end of the alarm display unit 82) or the right end RGr (i.e., the right end of the alarm display unit 82) of the displayable range RG. This will be explained in detail below.

[0121] exist Figure 13 In the upper section, the pedestrian, acting as the hazardous object RK, is positioned sufficiently far from the vehicle M, and the warning display unit 82 does not illuminate. Figure 13 In the middle section, when the hazardous object RK approaches relative to the vehicle M, a portion of the alarm display unit 82 illuminates. Here, the alarm control unit 52 causes the position of the illuminated area AR to change according to the position of the hazardous object RK (the hazardous object RK relative to the vehicle M). Thus, in Figure 13 In the next section, as the risky object RK moves to the right, the position of the luminous area AR also moves to the right. The display length L of the luminous area AR can, for example, be constant and independent of its position.

[0122] Detailed illustrations are omitted, but when setting information 72 to "special," similar to the first embodiment, the alarm control unit 52 narrows the visible range RG to a range closer to the driver's seat. Specifically, the alarm control unit 52 may also determine the visible range RG to be within the normal field of vision NR (see also...). Figure 10 Furthermore, the alarm control unit 52 can also change the position of the luminous area AR within this reduced displayable range RG. Thus, the same effect as in the first embodiment can be achieved. That is, it enables drivers experiencing narrowed vision to correctly recognize the alarm.

[0123] <Variation Example>

[0124] The scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0125] For example, in the embodiments described above, if the risky object meets the prescribed conditions, the alarm control unit 52 may cause the alarm display unit 82 to display an alarm with a more emphasized form compared to the case where the risky object does not meet the prescribed conditions. This "more emphasized form of alarm" is also called a warning. The prescribed conditions may, for example, be a situation where the risky object is rapidly approaching the vehicle M. The prescribed conditions may also be based on the speed of the risky object (e.g., its relative speed relative to the vehicle M) and its relative position relative to the vehicle M. The prescribed conditions may also be combinations of conditions such as the risky object's relative speed relative to the vehicle M being above a threshold, the risky object's relative distance relative to the vehicle M being less than a threshold, or similar conditions.

[0126] Figure 14 This diagram shows an example of an alarm displayed by the alarm display unit 82. Figure 14 In the previous section, the pedestrian, identified as the hazard object RK, crossed to the right, causing the alarm display unit 82 to stop illuminating. Afterwards, in Figure 14 In the next section, a person riding a bicycle (a new hazardous object RK) suddenly appears from the side of the vehicle M. As a result, the alarm control unit 52 causes the alarm display unit 82 to illuminate the entire alarm display unit 82 (displayable area RG) and changes the color of the illumination. Illuminating the entire alarm display unit 82 (displayable area RG) and changing the color of the illumination is an example of a "more emphasized alarm." For example, the alarm display unit 82 may illuminate yellow when the specified conditions are not met, and red when the specified conditions are met. The alarm control unit 52 may also, regardless of whether the setting information 72 is "normal" or "special," display a more emphasized alarm as described above when the hazardous object meets the specified conditions.

[0127] The alarm display unit 82 is not limited to a bar-shaped light; it can also be a display device installed in the vehicle M. As long as it is a device installed in a position forward of the driver's seat and capable of displaying the alarm, the type of alarm display unit 82 can be appropriately changed. In such an alarm display unit 82, the same effect as in the above-described embodiment can be obtained by changing the position of the alarm display by the alarm control unit 52. The alarm display unit 82 can also be a device installed in the vehicle M retrofitted, or it can be pre-assembled in the vehicle M.

[0128] In the above embodiment, the risk detection unit 32 detects risky objects by image analysis of surrounding images, but the structure of the risk detection unit 32 is not limited to this. For example, various sensors installed in the vehicle M, etc., and the ECU of the vehicle M that manages them may be connected to the network NW, and the risk detection unit 32 may detect risky objects based on signals output from these sensors. Examples of sensors include radar devices, LIDAR (Light Detection and Ranging), etc. The method by which the risk detection unit 32 detects risky objects is not limited to these examples and can be appropriately modified.

[0129] In the above embodiment, the passenger detection unit 38 detects passengers by analyzing images of the interior of the vehicle, but the structure of the passenger detection unit 38 is not limited to this. For example, various sensors installed in the vehicle M, etc., and the ECU of the vehicle M that manages them may be connected to the network NW, and the passenger detection unit 38 may detect passengers based on signals output from these sensors. Examples of sensors include thermal sensors and weight sensors installed in each seat of the vehicle M. The method by which the passenger detection unit 38 detects passengers is not limited to these examples and can be appropriately modified.

[0130] The driver monitoring device 1 (information processing device 2) may also perform other alarm processing (hereinafter referred to as auxiliary alarm processing) in addition to the risk alarm processing described above (or in lieu of the risk alarm processing described above), and perform an alarm corresponding to the risk location in the other alarm processing. Auxiliary alarm processing may, for example, be a missed detection alarm processing. Missed detection alarm processing, for example, is a process that detects the driver's missed detection of a risky object based on the risk location and information involving facial images (e.g., gaze, facial information) and displays an alarm on the alarm display unit 82. Auxiliary alarm processing may also be performed in real time while the driver is driving the vehicle M, similar to risk alarm processing. When auxiliary alarm processing is performed, the aforementioned recognition determination processing may also include a process of determining whether the driver can recognize the alarm displayed in the auxiliary alarm processing. The aforementioned alarm location change processing may also include a process of changing the location of the alarm displayed in the auxiliary alarm processing.

[0131] In the information processing systems 100 and 100A described in the above embodiments, the driver monitoring device 1 (notification control unit 54) can also send notification information to both the first terminal device T1 and the second terminal device T2. In other words, both the first terminal device T1 and the second terminal device T2 can be notification units 84 controlled by the notification control unit 54. If the alarm display unit 82 is equipped with a display device, it can also function as a notification unit 84.

[0132] In the first embodiment, the driver monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as separate devices, but they can also be configured as a single unit. For example, the driver monitoring device 1 may actually have all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and notification unit 84. Alternatively, all these functions may actually be installed in the first terminal device T1 or the vehicle M. Similarly, in the second embodiment, all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, and storage unit 70 may actually be installed in the driver monitoring device 1, and all these functions may actually be installed in the first terminal device T1. A camera other than the camera included in the driver monitoring device 1 may also perform some or all of the functions of the camera unit 10. For example, a driver monitoring camera that captures the driver may capture facial images.

[0133] The driving monitoring device 1 can also be installed using multiple information processing devices. For example, the functions of the information processing device 2 can also be installed in the first terminal device T1 or the vehicle M (e.g., ECU). Alternatively, the driving monitoring device 1 can be installed using a cloud or similar device. For example, in the driving monitoring device 1, the detection unit 30, processing unit 40, control unit 50, and storage unit 70 can also be installed in different information processing devices. For example, the storage unit 70 can also be distributed among multiple information processing devices.

[0134] The implementation methods and variations described above can be represented as follows.

[0135] An information processing device includes:

[0136] Storage medium, which stores computer-readable instructions; and

[0137] The processor, which is connected to the storage medium,

[0138] The processor performs the following processing by executing computer-readable instructions:

[0139] Detect hazardous objects located around the vehicle;

[0140] The alarm display unit installed on the vehicle displays an alarm corresponding to the location of the dangerous object;

[0141] Obtain the facial image of the driver of the vehicle;

[0142] Based on the location where the alarm is displayed and information involving the facial image, it is determined whether the driver can recognize the alarm issued by the alarm display unit;

[0143] The system stores in its storage unit the location where the alarm will be displayed, whether the driver can recognize the alarm, and the history information linked to the gaze detected from the facial image; and

[0144] The driver's visual recognition characteristics are evaluated based on the resume information.

[0145] Without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements, and the above embodiments and variations can also be appropriately combined.

Claims

1. An information processing device, wherein, The information processing device includes: The risk detection department detects risky objects located around the vehicle; An alarm control unit causes an alarm display unit installed in the vehicle to display an alarm corresponding to the location of the dangerous object; The acquisition unit acquires a facial image of the driver of the vehicle; The identification and determination unit determines, based on the location where the alarm is displayed and information relating to the facial image, whether the driver can recognize the alarm issued by the alarm display unit; The storage control unit stores the location where the alarm is displayed, whether the driver can recognize the alarm, and the history information that is linked to the gaze detected from the facial image in the storage unit. as well as The evaluation department evaluates the driver's visual recognition characteristics based on the resume information.

2. The information processing apparatus according to claim 1, wherein, If the evaluated visual recognition characteristics are abnormal, the alarm control unit, based on information input by the driver, causes the alarm display unit to display the location change of the alarm.

3. The information processing apparatus according to claim 2, wherein, The information processing device also includes: Passenger detection unit, which detects passengers riding in the vehicle; and The reporting control unit will issue a report if the location where the alarm is displayed has been changed and the passenger has been detected.

4. The information processing apparatus according to any one of claims 1 to 3, wherein, The information processing device further includes a notification control unit, which causes the notification unit to issue a notification when the evaluated visual recognition characteristics are abnormal.

5. The information processing apparatus according to claim 4, wherein, The notification control unit causes the notification unit to send the notification at times other than when the driver is driving the vehicle.

6. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to claim 4; A first terminal device, used by the driver, and serving as the notification unit; and The second terminal device, which is used in medical institutions, The notification included a recommendation to seek treatment at that medical facility. The first terminal device sends a notification to the second terminal device based on information input by the driver indicating consent to the medical examination.

7. An information processing system, wherein, The information processing system has the following features: The information processing apparatus according to claim 4; A first terminal device, which is used by the driver and is not the notification unit that controls the notification by the notification control unit; as well as The second terminal device is for use by a user different from the driver, and the notification unit that controls the notification is controlled by the notification control unit. The second terminal device causes the first terminal device to issue a notification based on information input by the user through the user's operation.

8. An information processing method, wherein, The information processing method causes the computer to perform the following processes: Detect hazardous objects located around the vehicle; The alarm display unit installed on the vehicle displays an alarm corresponding to the location of the dangerous object; Obtain the facial image of the driver of the vehicle; Based on the location where the alarm is displayed and information involving the facial image, it is determined whether the driver can recognize the alarm issued by the alarm display unit; The system stores the historical information that links the location where the alarm will be displayed, whether the driver can recognize the alarm, and the gaze detected from the facial image in the storage unit. as well as The driver's visual recognition characteristics are evaluated based on the resume information.

9. A storage medium that stores a program and is a non-transitory storage medium readable by a computer, wherein, The program is used to cause the computer to perform the following processes: Detect hazardous objects located around the vehicle; The alarm display unit installed on the vehicle displays an alarm corresponding to the location of the dangerous object; Obtain the facial image of the driver of the vehicle; Based on the location where the alarm is displayed and information involving the facial image, it is determined whether the driver can recognize the alarm issued by the alarm display unit; The system stores the historical information that links the location where the alarm will be displayed, whether the driver can recognize the alarm, and the gaze detected from the facial image in the storage unit. as well as The driver's visual recognition characteristics are evaluated based on the resume information.

Images