Information processing apparatus, information processing system, information processing method, and storage medium

Abstract

An information processing apparatus, an information processing system, an information processing method, and a storage medium capable of detecting information on a narrow field of view of a driver. The information processing apparatus includes an acquisition unit that acquires a surrounding image of a vehicle and a facial image of a driver of the vehicle; a risk detection unit that detects a risk object in the surrounding image; a miss determination unit that determines whether or not the driver misses the risk object based on information on a position of the risk object and information on the facial image; a generation unit that generates field-of-view mapping information based on the information on the position of the risk object, the presence or absence of the miss, and a line of sight detected from the facial image, the field-of-view mapping information including a first region including a center of a field of view of the driver and a second region located outside the first region; and an abnormality determination unit that determines that the driver has an abnormality when the miss occurs by a threshold value or more than a predetermined reference value in the second region of the field-of-view mapping information.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing system, information processing method, and storage medium. Background Technology

[0002] In recent years, efforts have been intensifying to provide sustainable transportation systems that also take into account vulnerable groups among transportation participants, particularly the elderly, people with disabilities, and children. Research and development are particularly focused on further improving the safety and convenience of transportation through developments related to mobility facilities for the elderly and people with disabilities (see, for example, Patent Documents 1 and 2 below).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2020-71528

[0004] Patent Document 2: Japanese Patent Application Publication No. 2017-16568 Summary of the Invention

[0005] When a driver suffers from a condition called narrowed field of vision (visual impairment), their ability to avoid dangerous objects is reduced. However, drivers are sometimes unaware of their narrowed field of vision. When drivers continue driving without realizing the narrowed field of vision and the reduced ability caused by it, it can impair traffic safety and convenience.

[0006] The solutions involved in this invention were made in consideration of such circumstances, and one of their objectives is to provide an information processing device, information processing system, information processing method, and storage medium capable of detecting a driver's narrow field of vision, thereby contributing to the development of sustainable transportation systems.

[0007] To address the aforementioned issues, the present invention employs the following solution.

[0008] (1): One aspect of the present invention relates to an information processing apparatus, wherein the information processing apparatus comprises: an acquisition unit that acquires a peripheral image of a vehicle and a facial image of the driver of the vehicle; a risk detection unit that detects a risky object in the peripheral image; a miss determination unit that determines whether the driver has missed the risky object based on information relating to the location of the risky object and information relating to the facial image; a generation unit that generates vision mapping information that establishes a relationship between information relating to the frequency of the miss and the direction within the driver's field of vision based on information relating to the location of the risky object, the presence or absence of the miss, and a gaze detected from the facial image, wherein the vision mapping information includes a first region including the center of the driver's field of vision and a second region located outside the first region; and an anomaly determination unit that determines that the driver has an anomaly if the miss exceeds a predetermined threshold compared to a predetermined reference value in the second region of the vision mapping information.

[0009] (2): In the above (1) scheme, the information processing device may also include a gaze detection unit for detecting the gaze from the facial image.

[0010] (3): In the above scheme (1) or (2), the information processing device may also include a notification control unit, which causes the notification unit to make a notification when it is determined that the driver has an abnormality.

[0011] (4): In the above (3) scheme, the notification control unit may also make the notification at a time other than when the driver is driving the vehicle.

[0012] (5): In any of the above (1) to (4), the information processing device may also include an alarm control unit that causes the alarm unit to sound an alarm based on information relating to the location of the risky object.

[0013] (6): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: an information processing device, which is the information processing device of any of the above (1) to (5), and further comprises a notification control unit that causes the notification unit to make a notification when it is determined that the driver has an abnormality; a first terminal device, which is used by the driver and is not controlled by the notification control unit; and a second terminal device, which is used by a user different from the driver and is controlled by the notification control unit, wherein the second terminal device causes the first terminal device to make a notification based on information input by the user's operation.

[0014] (7): One aspect of the present invention relates to an information processing method, wherein the information processing method causes a computer to perform the following processing: acquiring a peripheral image of a vehicle and a facial image of the driver of the vehicle; detecting a risky object in the peripheral image; determining, based on information relating to the location of the risky object and information relating to the facial image, whether the driver has missed seeing the risky object; generating, based on information relating to the location of the risky object, whether the driver has missed seeing the object, and a gaze detected from the facial image, a vision mapping information that links information relating to the frequency of the missed seeing to the direction within the driver's field of vision, and the vision mapping information including a first region including the center of the driver's field of vision and a second region located outside the first region; and determining that the driver has acted abnormally if the missed seeing exceeds a predetermined threshold compared to a predetermined reference value in the second region of the vision mapping information.

[0015] (8): One aspect of the present invention relates to a storage medium that is a non-transitory storage medium that stores a program that can be read by a computer, wherein the program is configured to cause the computer to perform the following processing: acquiring a peripheral image of a vehicle and a facial image of the driver of the vehicle; detecting a risky object in the peripheral image; determining, based on information relating to the location of the risky object and information relating to the facial image, whether the driver has missed the risky object; generating, based on information relating to the location of the risky object, whether the driver has missed the object, and a gaze detected from the facial image, visual field mapping information that establishes a correlation between information relating to the frequency of the missed object and the direction within the driver's field of vision, wherein the visual field mapping information includes a first region including the center of the driver's field of vision and a second region located outside the first region; and determining that the driver has acted abnormally if the missed object exceeds a predetermined threshold compared to a predetermined reference value in the second region of the visual field mapping information.

[0016] According to the solution of the present invention, it is possible to detect a driver's narrow field of vision. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating an example of the structure and usage environment of a driving monitoring device that includes the information processing apparatus according to the first embodiment.

[0018] Figure 2 This is a diagram illustrating an example of a driver monitoring device.

[0019] Figure 3 This is an example of an image obtained by a driving monitoring device.

[0020] Figure 4 This diagram illustrates the process of a driver monitoring device triggering a missed alarm.

[0021] Figure 5 This is a diagram illustrating an example of information detected by the testing department.

[0022] Figure 6 This is a diagram illustrating an example of an alarm issued by the alarm department.

[0023] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device.

[0024] Figure 8 This is a diagram illustrating an example of determining resume information.

[0025] Figure 9 This is a diagram representing an example of view mapping information.

[0026] Figure 10 This is a flowchart illustrating an example of the processing flow performed by the exception handling unit.

[0027] Figure 11 This is a diagram representing an example of view mapping information.

[0028] Figure 12 This diagram illustrates the exception notification processing in the information processing system according to the second embodiment. Detailed Implementation

[0029] Hereinafter, embodiments of the information processing apparatus, information processing system, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings. In the following description, the forward direction of the vehicle is defined as the positive X direction, the rearward direction of the vehicle is defined as the negative X direction, the rightward direction of the vehicle's width relative to the positive X direction is defined as the positive Y direction, the leftward direction is defined as the negative Y direction, and the direction orthogonal to the X and Y directions and the height direction of the vehicle is defined as the positive Z direction.

[0030] <First Implementation>

[0031] [Overall Structure]

[0032] Figure 1This diagram illustrates an example of the structure and usage environment of a driver monitoring device 1, including the information processing device 2 according to the first embodiment. The information processing device 2 is, for example, a driver monitoring device 1 retrofitted into a vehicle (hereinafter referred to as "vehicle M") such as a dashcam. That is, the driver monitoring device 1 (information processing device 2) can be mounted and removed from the vehicle M via a mounting / unmounting unit (hereinafter referred to as "mounting / unmounting unit 16"). The information processing device 2 is, for example, a device not connected to the vehicle M's in-vehicle network. An in-vehicle network refers to an in-vehicle network that connects to the vehicle's control-related ECU (Electronic Control Unit), which uses communication standards such as CAN (Controller Area Network), and is connected to the vehicle M's drive, braking, steering, various vehicle sensors, driving-related operating components, and operation buttons. The vehicle M's network for HMI (Human Machine Interface) can also be excluded from the above-described in-vehicle network.

[0033] Figure 2 This diagram illustrates an example of a driver monitoring device 1. The driver monitoring device 1 is, for example, installed near the windshield of a vehicle M, and located near the front of the driver's seat. The driver monitoring device 1 stores images (moving images) captured by the camera unit 10, for example, in the storage unit 70 (described later). The driver monitoring device 1 may also be, for example, equipped with a display device such as an LCD (Liquid Crystal Display) and capable of displaying the images (moving images) stored in the storage unit 70 on the display device.

[0034] The driving monitoring device 1 includes, for example, a camera unit 10, a loading and unloading unit 16, and an information processing unit 2. In the illustrated example, the camera unit 10 includes a front camera 12 and a rear camera 14.

[0035] The front camera 12 and the rear camera 14 are each, for example, digital cameras utilizing solid-state imaging elements such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The front camera 12 repeatedly (periodically) captures images of the area in front of the vehicle M from the location where the driver monitoring device 1 is installed. The rear camera 14 repeatedly (periodically) captures images of the area behind the vehicle M and the interior of the vehicle M from the location where the driver monitoring device 1 is installed. The horizontal viewing angle captured by the front camera 12 and the rear camera 14 is, for example, 180° or more. That is, the driver monitoring device 1 (camera unit 10) captures a 360° range around the vehicle M from its installation location. The front camera 12 and the rear camera 14 may each be, for example, fisheye cameras including fisheye lenses. Alternatively, the front camera 12 and the rear camera 14 may each be wide-angle cameras or the like. In this embodiment, the use of the front camera 12 and the rear camera 14 is described, but other cameras may be used instead of these or other alternatives. That is, the camera unit 10 can be configured to capture images used in this embodiment (peripheral images and facial images described later).

[0036] The mounting / unmounting part 16 is, for example, a component for mounting the driver monitoring device 1 onto the vehicle M. The mounting / unmounting part 16 can be, for example, any supporting component such as a suction cup, a seal, or a bracket. The mounting / unmounting part 16 may also include a power supply cable (wiring, etc.) for supplying power to the driver monitoring device 1.

[0037] Figure 3 This is a diagram illustrating an example of an image acquired by the driver monitoring device 1. That is, Figure 3 This diagram illustrates an example of an image captured by the camera unit 10. The front camera 12 captures images of the front of the vehicle M through the windshield, the driver's side side window, and the passenger side side window. The rear camera 14 captures images of the interior of the vehicle M (including the driver and passengers) and the rear of the vehicle M through the driver's side side window, the passenger side side window, and the rear window.

[0038] In the image captured by the front camera 12 (hereinafter referred to as the "front image"), the subject includes objects such as other vehicles, pedestrians, bicycles, fixed objects, road markings, etc., that can be seen through the front windshield and exist in front of the vehicle M, as well as objects that can be seen through the left and right (front) side windows of the vehicle M.

[0039] The image captured by the rear camera 14 (hereinafter referred to as the "rear image") shows the interior of the vehicle M, as well as objects visible through the rear window that are located behind the vehicle M, and objects visible through the left and right side windows that are located to the left and right (rear) of the vehicle M. Therefore, in the rear image captured by the rear camera 14, the driver of the vehicle M equipped with the driver monitoring device 1 is also reflected as the subject.

[0040] The front and rear images include images of the surrounding conditions of the vehicle M (hereinafter referred to as "surrounding images"). The rear image includes an image of the driver's face of the vehicle M (hereinafter referred to as "face image"). The driver monitoring device 1 can be installed at any location where the camera unit 10 can capture the surrounding images and the face image.

[0041] Information processing device 2 (refer to) Figure 1 Based on the results obtained by referring to the surrounding images and facial images output by the camera unit 10, the alarm and notification are output via the first terminal device T1. Sometimes the driver monitoring device 1 and the first terminal device T1 are combined and referred to as "information processing system 100".

[0042] The first terminal device T1 is, for example, a mobile terminal device used by a driver driving a vehicle M equipped with a driver monitoring device 1, such as a smartphone or tablet. The first terminal device T1 executes, for example, an application for receiving alarms and notifications from the information processing device 2. The application causes a display device to show an image based on information sent by the information processing device 2 (e.g., alarm information and notification information described later), and causes a speaker to emit sound. The first terminal device T1 is also referred to as the alarm unit 82 and the notification unit 84. The first terminal device T1 is used, for example, in a state where it can be detachably mounted on the vehicle M. For example, a bracket for the first terminal device T1 with a mounting and detaching mechanism is provided on one or both of the first terminal device T1 and the vehicle M, and the first terminal device T1 is supported by the bracket.

[0043] Alternatively, the navigation, display, and speaker of vehicle M can be used instead of the first terminal device T1. That is, the navigation, display, and speaker of vehicle M can also output alarms and notifications based on the instructions of information processing device 2. That is, the navigation, display, and speaker of vehicle M can also be alarm unit 82 and notification unit 84. Alternatively, the display unit (alarm unit 82, notification unit 84) and speaker (alarm unit 82, notification unit 84) can be provided on the driver monitoring device 1 instead of the first terminal device T1.

[0044] like Figure 1As shown, the information processing device 2 includes, for example, an acquisition unit 20, a detection unit 30, a processing unit 40, a control unit 50, a communication unit 60, and a storage unit 70. The detection unit 30 includes a risk detection unit 32, a gaze detection unit 34, and a face orientation detection unit 36. The processing unit 40 includes a miss detection unit 42, a storage control unit 44, a generation unit 46, and an anomaly detection unit 48. The control unit 50 includes an alarm control unit 52 and a notification control unit 54.

[0045] The acquisition unit 20, detection unit 30, processing unit 40, and control unit 50 are equipped with hardware processors such as CPUs (Central Processing Units) and storage devices (storage devices with non-transitory storage media) storing programs (software). The processor executes the programs to realize the functions of each component. Some or all of these components can also be implemented using hardware such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units) (including the circuitry). The functions of each component can also be realized through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs.

[0046] The program (software) can be pre-stored in the storage unit 70 (a storage device with a non-temporary storage medium) of the information processing device 2, such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory, or it can be stored in a removable storage medium (a non-temporary storage medium) such as a memory card, and installed in the storage device by assembling the storage medium into the driving monitoring device 1. The program (software) can also be pre-downloaded from other computer devices via short-range communication or wide-area communication using an application program executed in the first terminal device T1, and then sent from the first terminal device T1, thereby being installed in the storage device.

[0047] The storage unit 70 stores, for example, decision history information 72, view mapping information 74, programs (such as the programs described above), and various other information. The storage unit 70 can also be implemented using the various storage devices described above or EEPROM (Electrically Erasable Programmable Read Only Memory). The communication unit 60 is an interface for wireless or wired communication with information processing devices (external devices) such as the first terminal device T1.

[0048] The driver monitoring device 1 (information processing device 2) performs at least two functions: a missed view alarm and an anomaly notification. The missed view alarm is a process that detects a driver's missed view of a risky object based on surrounding images and facial images, and triggers an alarm on the alarm unit 82. The anomaly notification is a process that detects an anomaly (narrow field of vision) in the driver based on surrounding images and facial images, and triggers a notification on the anomaly on the notification unit 84.

[0049] A risky object refers to an object, for example, that vehicle M should avoid. A risky object is, for example, an object that would interfere with vehicle M when it is moving as it is, or an object that has the potential to interfere. An object that has the potential to interfere is, for example, an object whose approach level becomes more than a certain level after a specified time, taking into account the object's position, direction of movement, and speed, as well as the position, method of movement, and speed of vehicle M. The object can also be a traffic participant such as a pedestrian, bicycle, or vehicle, or an object other than a traffic participant such as an object placed on the road or a fallen object.

[0050] [Handling of Missing Alarms]

[0051] Figure 4 This diagram illustrates the omission alarm processing performed by the driving monitoring device 1 (information processing device 2).

[0052] The acquisition unit 20 acquires front and rear images from the imaging unit 10. Therefore, the acquisition unit 20 establishes a link between the peripheral images and the facial images to acquire these images. The imaging unit 10 repeatedly (periodically) captures the peripheral images and facial images, thus the acquisition unit 20 repeatedly (periodically) acquires these images. The acquisition unit 20 can also establish a link between the peripheral images and facial images and the time they were captured (the shooting time) to acquire these images.

[0053] The risk detection unit 32 detects risky objects in the surrounding image. "Detecting risky objects" refers to, for example, detecting the presence of a risky object and the position of that object relative to the vehicle M (driver monitoring device 1) (hereinafter referred to as the risk position). For example, reference information (not shown) indicating the relationship between the risk position in the surrounding image and its relative position to the vehicle M can be pre-stored in the storage unit 70. The risk detection unit 32 can also detect the position of the risky object relative to the vehicle M based on this reference information.

[0054] The risk detection unit 32 detects the relative position of the risk object with respect to the vehicle M, and also detects the direction of the risk object's presence (approach direction) relative to the vehicle M. "Direction of presence of the risk object" refers to, for example, the direction of presence of a risk object approaching the vehicle M (driving monitoring device 1) relative to the vehicle M (driving monitoring device 1). Hereinafter, "direction of presence of the risk object" will sometimes be referred to as "risk presence direction".

[0055] The gaze detection unit 34 detects the driver's gaze in a facial image. Specifically, the gaze detection unit 34 detects the driver's gaze by analyzing the facial image using a prescribed image analysis method. "Gaze detection" refers to, for example, detecting the position of the driver's eyeballs and the direction of the gaze (the direction of the eyeballs).

[0056] The face orientation detection unit 36 ​​detects at least the orientation of the driver's face in the facial image. Specifically, the face orientation detection unit 36 ​​detects the orientation of the driver's face by parsing the facial image using a prescribed image parsing method. The face orientation detection unit 36 ​​may also detect the position of the driver's face in addition to its orientation. Hereinafter, the information combining "face orientation" and "face position" is sometimes referred to as "facial information." The face orientation detection unit 36 ​​detects facial information in the facial image.

[0057] The detection unit 30 can also be configured to detect lanes, pedestrian crossings, etc., in addition to the aforementioned risk objects, lines of sight, and facial information. Figure 5 This is a diagram illustrating an example of information detected by the detection unit 30. In Figure 5 In the example, pedestrians, other vehicles, lanes, and crosswalks detected by the detection unit 30 are displayed based on vehicle M, the direction of the line of sight, and the direction of the face. Pedestrians and other vehicles are examples of traffic participants (objects) that may become risk objects. In the information detected by the detection unit 30, traffic participants (objects) that may become risk objects detected by the detection unit 30 may also be displayed based on vehicle M, the direction of vehicle M, and / or the direction of travel of vehicle M, instead of vehicle M, the direction of the line of sight, and the direction of the face.

[0058] Leak detection unit 42 (refer to) Figure 4 This method determines whether a driver has missed a hazard based on the location and / or direction of the hazard, line of sight, and facial information. "Risk location" and "direction of the hazard" are each examples of "information relating to the location of the hazard." Line of sight and facial information are examples of "information relating to facial images."

[0059] For example, reference information (not shown) indicating the relationship between the orientation of the driver's eyes (gaze) and / or the orientation of their face in the facial image and their relative orientation with respect to the direction of travel of the vehicle M can also be pre-stored in the storage unit 70. The omission determination unit 42 can also refer to this reference information to determine whether the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists. If the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists, the omission determination unit 42 can also determine that the driver visually identifies a risky object. The so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can be, for example, the case where there is a risky position on a straight line along the orientation of the gaze and / or the orientation of the face. Alternatively, the so-called "the orientation of the gaze and / or the orientation of the face coincides with the direction where the risk exists" can also be the case where there is a risky position in a cone-shaped area with the position of the viewpoint (eyeball) and / or the face as the apex and having a central axis along the orientation of the gaze and / or the orientation of the face that expands laterally as it moves forward. If the direction of the driver's gaze and / or the direction of the face does not match the location of the risky object, the omission determination unit 42 may also determine that the driver has not visually identified the risky object.

[0060] The omission determination unit 42 can also determine that the driver has missed the risk object if the state of not visually recognizing the risk object continues for a predetermined time or longer. Alternatively, the omission determination unit 42 can determine that the driver has not missed the risk object if the driver visually recognizes the risk object, or if the state of not visually recognizing the risk object is resolved before the predetermined time has elapsed. However, the method by which the omission determination unit 42 determines omissions is not limited to the above and can be appropriately modified.

[0061] The omission detection unit 42 generates omission detection information based on the above detection results. The omission detection information includes, for example, the presence or absence of omissions and the direction of risk.

[0062] The alarm control unit 52 causes the first terminal device T1 (alarm unit 82) to output an alarm based on the leak detection information. For example, the alarm control unit 52 sends alarm information to the first terminal device T1 via the communication unit 60. The alarm information includes, for example, information indicating whether an alarm is needed and the direction of the risk. For example, if the leak detection information includes information indicating "a leak exists," the alarm control unit 52 generates alarm information including information indicating "an alarm is needed." For example, if the leak detection information includes information indicating "no leak exists," the alarm control unit 52 generates alarm information including information indicating "no alarm is needed." The first terminal device T1 outputs an alarm based on the alarm information.

[0063] Figure 6This diagram illustrates an example of an alarm triggered by the first terminal device T1 (alarm unit 82). Figure 6 In the example, the first terminal device T1 displays identifiable entries indicating the direction of the risk (D1 and D2 in the diagram). Furthermore, the first terminal device T1 issues warnings by differentiating the display format of the risk direction when a risky object is missed (i.e., the warning message includes a message indicating "warning is needed") and when no risky object is missed (i.e., the warning message includes a message indicating "warning is not needed"). Specifically, when a risky object is missed, the risk direction is displayed in a more emphasized manner compared to when it is not missed. This emphasis could be based on color, flashing icons, or other visually illuminating elements to make the risk direction easier for the driver to recognize. The first terminal device T1 may also issue warnings using methods such as loudspeaker announcements or illuminated lights (A in the diagram), in addition to or instead of the above. The specific method of issuing the warning can be appropriately modified as long as it is identifiable by the driver.

[0064] The alarm issued by the first terminal device T1 (alarm unit 82) can also be issued during the time when the driver is driving the vehicle M (i.e., in real time). Examples of "the time when the driver is driving the vehicle M" include when the vehicle M is moving or when the vehicle M is temporarily stopped. For example, the alarm information sent by the alarm control unit 52 can also be issued at the aforementioned times. Alternatively, the alarm information sent by the alarm control unit 52 can be issued at any time, and the first terminal device T1 (alarm unit 82) controls the timing of the alarm to issue the alarm at the aforementioned times.

[0065] [Abnormal Notification Handling]

[0066] Figure 7 This diagram illustrates the abnormal notification processing performed by the driving monitoring device 1 (information processing device 2). Abnormal notification processing is also known as narrow field of vision notification processing.

[0067] The storage control unit 44 stores the determination time, the aforementioned risk location, whether there is a missed risk object, and the connection established with the line of sight in the storage unit 70. The determination time corresponds, for example, to the time when the surrounding images and / or facial images used in the detection of risk locations, the determination of whether a missed object has been seen, and the detection of the line of sight are captured. Risk location is an example of "information relating to the location of a risk object". The aforementioned processing (storage processing) performed by the storage control unit 44 is performed, for example, whenever the missed object determination unit 42 generates missed object determination information. Hereinafter, the information stored in the storage unit 70 will be referred to as determination history information 72.

[0068] Figure 8This diagram illustrates an example of the judgment history information 72. The storage control unit 44 repeatedly performs the aforementioned storage process, thereby accumulating multiple groups (hereinafter referred to as judgment information groups J) in the judgment history information 72, including information such as the judgment time, risk location, whether there was a missed inspection, and line of sight. Specifically, the storage control unit 44 generates judgment information group J whenever the missed inspection judgment unit 42 generates missed inspection judgment information. Furthermore, the storage control unit 44 appends the generated judgment information group J to the judgment history information 72, thereby updating the judgment history information 72.

[0069] return Figure 7 The generation unit 46 generates vision mapping information 74 based on the judgment history information 72.

[0070] First, let's illustrate an example of the content of the field-of-view mapping information 74. Figure 9 This is a diagram illustrating an example of field-of-view mapping information 74. Field-of-view mapping information 74 is information that links the frequency of occurrence of objects at risk of being missed by the driver with directions within the driver's field of vision.

[0071] The field-of-view mapping information 74 includes, for example, multiple regions AR arranged in a two-dimensional (e.g., a two-dimensional lattice) pattern. Each region AR corresponds to a direction within the driver's field of vision (hereinafter referred to as a field-of-view direction). For example, among the multiple regions AR, there is a central region ARa corresponding to the center (front) of the driver's field of vision, and multiple non-central regions ARb that are regions AR other than the central region ARa. The relative position of each non-central region ARb in the field-of-view mapping information 74 with respect to the central region ARa corresponds to the relative position of each field-of-view direction within the driver's field of vision with respect to the center of the field of vision. The multiple regions AR may also be configured such that the area encompassing them all includes the entire portion of the windshield reflected in the driver's field of vision while driving with their gaze directed toward the direction of travel of the vehicle M. In other words, the outermost region AR (non-central region ARb) among the multiple regions AR may also correspond to the outer periphery of the windshield or a portion beyond the outer periphery in the driver's field of vision.

[0072] The field-of-view mapping information 74 includes a first region AR1 and a second region AR2. The first region AR1 is the region that includes the center of the driver's field of vision (i.e., the central region ARa). The second region AR2 is a ring-shaped region located outside the first region AR1. In the illustrated example, the first region AR1 coincides with the central region ARa, and the second region AR2 coincides with the region that combines all the regions AR (non-central regions ARb) excluding the central region ARa. That is, the boundary between the first region AR1 and the second region AR2 coincides with the periphery of the central region ARa. However, the boundary between the first region AR1 and the second region AR2 is not limited to this and can be appropriately changed. For example, the boundary can also be set in such a way that the first region AR1 contains multiple regions AR. Information indicating the boundary between the first region AR1 and the second region AR2 (i.e., information indicating which region of the first region AR1 and the second region AR2 each region AR belongs to) can also be pre-stored in the storage unit 70.

[0073] The numerical values ​​(0.8, 0.9, 1.0, etc.) recorded in each AR area represent the driver's awareness rate of risky objects occurring in that AR area (within the field of vision). The awareness rate is defined, for example, as "1 - (missed detection rate)". The missed detection rate is defined as the number of times a risky object occurring in that AR area is missed, divided by the total number of times a risky object occurs in that AR area. The awareness rate is an example of "information on the frequency of missed detections of risky objects". Hereinafter, "information on the frequency of missed detections of risky objects" is sometimes referred to as "missed detection frequency information".

[0074] Here, Figure 9 The visual field mapping information 74 shown is information for healthy individuals (i.e., those without cognitive impairment, visual field defects, or other similar conditions). Generally speaking, such as Figure 9 As shown, in the visual field mapping information 74 of healthy individuals, the recognition rate is highest in the central region ARa, and decreases symmetrically as one moves away from the central region ARa. This is because, generally speaking, visual acuity and attention decline further away from the center of the visual field.

[0075] Next, an example of a method for generating the field-of-view mapping information 74 will be described. The generation unit 46 generates the field-of-view mapping information 74 as described above based on the determination history information 72. For example, the generation unit 46 determines, based on the risk location and line of sight, which direction the risky object is located within the field of view (in other words, which region AR it corresponds to in the field-of-view mapping information 74) for each determination information group J contained in the field-of-view mapping information 74.

[0076] For example, the generation unit 46 can also calculate the risk vector and the line-of-sight vector. The risk vector is a vector that starts from the position of the eyeball and ends at the risk location. The line-of-sight vector is a vector with a predetermined magnitude that starts from the position of the eyeball and extends along the direction of the line of sight (eyeball). Furthermore, the generation unit 46 can also calculate the angle between the risk vector and the line-of-sight vector, thereby determining the direction within the field of vision of the aforementioned risky object.

[0077] The generation unit 46 can also perform prescribed statistical processing based on the determined direction within the field of view and whether there are any missed views included in the judgment information group J, thereby calculating the recognition rate (missed view frequency information) in each AR region. The recognition rate is calculated for all AR regions, thus completing the generation of the field of view mapping information 74. For example, the generation unit 46 can also calculate the recognition rate (generation of field of view mapping information 74) based on the judgment information group J belonging to a prescribed calculation period (e.g., the most recent year) in the judgment history information 72.

[0078] The generation unit 46 establishes a link between the generated field-view mapping information 74 and information indicating the calculation period, and stores it in the storage unit 70. The generation of the field-view mapping information 74 by the generation unit 46 can also be performed repeatedly at a predetermined period. In this case, the period for generating the field-view mapping information 74 by the generation unit 46 can also be set based on the calculation period. Alternatively, the generation of the field-view mapping information 74 can be performed via instructions from the driver or other personnel. The generation unit 46 repeatedly generates the field-view mapping information 74, thereby storing (accumulating) multiple field-view mapping information 74s with different calculation periods in the storage unit 70.

[0079] The content and generation method of the field-of-view mapping information 74 are not limited to the examples described above and can be appropriately modified. For example, the number of AR regions included in the field-of-view mapping information 74, and the width of the field of view corresponding to each AR region, can be appropriately modified. The omission rate or the number of times the driver missed a risky object (or did not miss it) can also be used as omission frequency information. As long as the omission frequency information is linked to the direction within the driver's field of view, the content of the field-of-view mapping information 74 can be appropriately modified. The generation method of the field-of-view mapping information 74 can be appropriately modified based on its content.

[0080] return Figure 7 The anomaly determination unit 48 determines whether the driver has experienced an anomaly (i.e., narrowed field of vision) based on the field of vision mapping information 74. The anomaly determination unit 48 generates anomaly determination information based on the determination result. The anomaly determination information includes information indicating the presence or absence of the aforementioned driver anomaly.

[0081] Figure 10 This is a flowchart illustrating an example of the processing flow performed by the exception determination unit 48. Figure 10The process shown in the flowchart can, for example, start repeatedly at a predetermined cycle, or start whenever the generation unit 46 generates the view mapping information 74, or start upon instruction from the driver or others.

[0082] First, the anomaly determination unit 48 reads the field of view mapping information 74 from the storage unit 70 (step S102). Specifically, the anomaly determination unit 48 reads the field of view mapping information 74 from the storage unit 70, which is the object for determining whether an anomaly (narrow field of view) for the driver is present or absent. For example, the anomaly determination unit 48 may also read the latest information stored in the field of view mapping information 74 in the storage unit 70.

[0083] Next, the anomaly determination unit 48 determines whether multiple incidents were missed in the second region AR2 of the read field mapping information 74 (step S104). Figure 11 This is a diagram representing an example of field-of-view mapping information 74, and also a diagram representing an example of multiple missed shots in the second region AR2. In Figure 11 In the example shown, with, for example Figure 9 Compared to the visual field mapping information 74 of an able-bodied person as shown, the cognitive rate is reduced in AR regions belonging to at least a portion of the second AR region 2. On the other hand, in other AR regions, compared to, for example... Figure 9 Compared to the visual field mapping information 74 of an able-bodied person, no significant decrease in cognitive rate was found. Details of the judgment process in step S104 (hereinafter referred to as the visual field narrowing judgment process) will be described later.

[0084] return Figure 10 If it is determined that multiple instances of missed detections have occurred in the second region AR2 (step S104: "Yes"), the anomaly determination unit 48 recognizes an anomaly and generates anomaly determination information (step S106). The anomaly determination unit 48 may also include information indicating the specific region with the multiple missed detections (hereinafter referred to as the missing region) in the generated anomaly determination information. Afterwards, the anomaly determination unit 48 terminates. Figure 10 The flowchart shown illustrates the processing.

[0085] If no multiple instances of vision failure are detected in the second area AR2 (step S104: "No"), the anomaly determination unit 48 assumes no anomaly exists and generates anomaly determination information (step S108). That is, the anomaly determination unit 48 generates anomaly determination information that includes information indicating the driver does not have any anomalies. Afterwards, the anomaly determination unit 48 terminates. Figure 10 The flowchart shown illustrates the processing.

[0086] return Figure 7The notification control unit 54 causes the first terminal device T1 (notification unit 84) to output a notification based on the anomaly determination information. For example, the notification control unit 54 sends the notification information to the first terminal device T1 via the communication unit 60. For example, the notification control unit 54 may also send the notification information to the first terminal device T1 if the anomaly determination information includes information indicating that the driver is abnormal (i.e., if the anomaly determination unit 48 determines that the driver is abnormal). If information indicating a damaged area is included in the anomaly determination information, the notification information may also include information indicating the damaged area. If the anomaly determination information includes information indicating that the driver is not abnormal (i.e., if the anomaly determination unit 48 determines that the driver is not abnormal), the notification control unit 54 may choose not to send the notification information to the first terminal device T1.

[0087] The first terminal device T1 (notification unit 84) outputs a notification based on notification information. The notification may also be delivered by displaying a string, image, or other object obtained based on the notification information on a display device of the first terminal device T1 (notification unit 84). The first terminal device T1 (notification unit 84) may also notify the driver that a visual field defect has occurred or that a visual field defect may occur. When information indicating the defect area is included in the notification information, the first terminal device T1 (notification unit 84) may also output an item (e.g., a string, image, etc.) that the driver or others can recognize the defect area.

[0088] The notification issued by the first terminal device T1 (notification unit 84) may also include a suggestion to seek medical treatment at a medical institution (e.g., a hospital the driver frequently visits). In this case, the first terminal device T1 (notification unit 84) may also allow the driver to input information indicating whether the driver agrees to the medical treatment. Specifically, the display device of the first terminal device T1 (notification unit 84) may display a virtual button or the like indicating agreement to the medical treatment. Such a button or the like functions as an agreement information acquisition unit, which acquires information indicating the driver's intention to agree to the medical treatment (hereinafter referred to as agreement information). Furthermore, if the driver operates the first terminal device T1 (notification unit 84) and the agreement information acquisition unit acquires the agreement information, the first terminal device T1 (notification unit 84) may also send a notification to a designated information processing terminal indicating that the driver has agreed to the medical treatment. The designated information processing terminal may be, for example, a terminal used in a medical institution (e.g., the second terminal device T2 described later). Specifically, the specified information processing terminal may also be a terminal used by personnel belonging to a medical institution (e.g., doctors, nurses, etc.).

[0089] However, as long as the form is recognizable by the driver, the form of the notification issued by the first terminal device T1 (notification unit 84) is not particularly limited and can be appropriately changed.

[0090] The notification sent by the first terminal device T1 (notification unit 84) can also be sent at times other than when the driver is driving the vehicle M. Examples of "times other than when the driver is driving the vehicle M" include when the vehicle M is parked, when the driver has finished driving the vehicle M, and when the driver has left the vehicle M and is at home. For example, the notification information sent by the notification control unit 54 can also be sent at the aforementioned times. Alternatively, the notification information sent by the notification control unit 54 can be sent at any time, and the first terminal device T1 (notification unit 84) controls the timing of the notification to ensure that the notification is sent at the aforementioned times.

[0091] [Details of the assessment and treatment of narrow field of vision]

[0092] The following is an example illustrating the detailed procedures for determining and handling narrowed visual fields. However, the procedures for determining and handling narrowed visual fields are not limited to the example below and can be modified as appropriate.

[0093] The anomaly determination unit 48 reads, for example, the visual field mapping information 74 of an able-bodied person pre-stored in the storage unit 70 as the reference mapping information. Alternatively, the anomaly determination unit 48 may also read the past visual field mapping information 74 generated by the generation unit 46 as the reference mapping information. Then, the anomaly determination unit 48 compares the frequency of missed vision information (e.g., cognitive rate) with respect to the reference mapping information and the visual field mapping information 74 (which is the object of determining the presence or absence of an anomaly in the driver) for each region AR. In other words, the frequency of missed vision information contained in the reference mapping information is set as a reference value, and the difference between the frequency of missed vision information contained in the visual field mapping information 74 and the reference value is calculated for each region AR. Then, the anomaly determination unit 48 determines the region AR (hereinafter referred to as the frequently missed vision region ARc) where the difference from the reference value is above a predetermined threshold. The predetermined threshold can be a value greater than 0 or 0.

[0094] Furthermore, if the identified frequent viewing error area ARc is an area AR contained within the second area AR2, the anomaly determination unit 48 may determine that "frequent viewing errors occur in the second area AR2". In other words, if there is a frequent viewing error area ARc contained within the second area AR2, the anomaly determination unit 48 may determine that "frequent viewing errors occur in the second area AR2". That is, if a viewing error occurs in the second area AR2 of the field of view mapping information 74 that exceeds the aforementioned threshold compared to the aforementioned benchmark value, the anomaly determination unit 48 may determine that "frequent viewing errors occur in the area AR". The field of view mapping information 74 may also determine the area where the frequent viewing error area ARc is concentrated as the aforementioned defect area. If the identified frequent viewing error area ARc is an area AR contained within the first area AR1 (if there is a frequent viewing error area ARc contained within the first area AR1), the anomaly determination unit 48 may not determine that "frequent viewing errors occur in the second area AR2", and may generate anomaly determination information as if no anomaly exists. As described above, in this embodiment, the content of the generated anomaly determination information is determined based on which of the first region AR1 and the second region AR2 the frequently missed vision region ARc belongs to, and the notification information and its content are determined accordingly. That is, the anomaly determination unit 48 compares the missed vision frequency information (e.g., cognitive rate) with a reference value (e.g., the cognitive rate of an able-bodied person) for each of the regions AR included in the visual field mapping information 74, and determines whether the driver is normal or abnormal based on the result of the comparison.

[0095] According to the first embodiment described above, the information processing device 2 includes: an acquisition unit 20 that acquires a peripheral image of a vehicle M and a facial image of the driver of the vehicle M; a risk detection unit 32 that detects risky objects in the peripheral image; a miss determination unit 42 that determines whether the driver has missed a risky object based on information about the location of the risky object and information about the facial image; a generation unit 46 that generates vision mapping information 74 based on information about the location of the risky object, whether a miss has occurred, and the gaze detected from the facial image, which establishes information about the frequency of misses with the direction within the driver's field of vision, and the vision mapping information 74 includes a first region AR1 including the center of the driver's field of vision and a second region AR2 located outside the first region AR1; and an anomaly determination unit 48 that determines that the driver has an anomaly if a miss in the second region AR2 of the vision mapping information 74 exceeds a predetermined threshold compared to a predetermined reference value. Thus, it is possible to detect narrowing of the driver's field of vision.

[0096] <Second Implementation>

[0097] Next, the second embodiment will be described, in which the basic structure is the same as that in the first embodiment. Therefore, the same reference numerals will be used to label the same structures and their descriptions will be omitted; only the differences will be described.

[0098] Figure 12 This diagram illustrates the exception notification processing in the information processing system 100A according to the second embodiment. Figure 12 As shown, the information processing system 100A according to the second embodiment includes a second terminal device T2 in addition to the driving monitoring device 1 and the first terminal device T1.

[0099] The second terminal device T2 is an information processing device used by a user different from the driver using the first terminal device T1 (e.g., an employee of a hospital or insurance company). The second terminal device T2 may also be, for example, a smartphone, tablet, personal computer, etc.

[0100] In the second embodiment, the driving monitoring device 1 (notification control unit 54) sends the notification information to the second terminal device T2 instead of the first terminal device T1. That is, in the second embodiment, the notification unit 84 controlled by the notification control unit 54 is the second terminal device T2, not the first terminal device T1. The second terminal device T2, for example, executes an application program for receiving notifications from the information processing device 2. The application program causes the display device to display images, strings, etc., obtained based on the information (notification information, etc.) sent by the information processing device 2, or causes the speaker to emit sound.

[0101] The second terminal device T2 uses information input by the user to notify the first terminal device T1. The second terminal device T2 includes, for example, an input unit 92, a transmission control unit 94, and a communication unit 96.

[0102] The input unit 92 and the transmission control unit 94 are equipped with hardware processors such as CPUs and storage devices (storage devices with non-transitory storage media) that store programs (software). The processor executes the programs to implement the functions of each component. Some or all of these components can be implemented using hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or the functions of each component can be implemented through the cooperation of software and hardware. Some or all of these components can also be implemented using dedicated LSIs. The communication unit 96 is an interface for wireless or wired communication with information processing devices such as the first terminal device T1.

[0103] The input unit 92 accepts user operations and inputs them to the second terminal device T2. The input unit 92 can be configured using existing input devices such as keyboards, clicking devices (mouse, tablet, etc.), buttons, or touch panels. The input unit 92 can also be an interface for connecting an input device to the second terminal device T2. In this case, the input unit 92 inputs the input signal generated in the input device based on the user's operation to the second terminal device T2. The input unit 92 can be configured arbitrarily as long as it is capable of inputting instructions obtained based on user operations to the second terminal device T2.

[0104] For example, the user confirms the content of the notification sent by the driver monitoring device 1 (notification control unit 54) to the second terminal device T2, and studies the content that should be notified to the driver. The content to be notified to the driver may include, for example, recommendations to seek medical attention at a medical institution for the purpose of detecting visual field defects (abnormalities). The user inputs information, including the studied notification content, into the second terminal device T2 via the operation input unit 92. The transmission control unit 94 transmits the information input by the user to the first terminal device T1 via the communication unit 96. The first terminal device T1 then notifies the driver based on the transmitted information.

[0105] The information processing system 100A can also be configured to restrict the transmission of notification information from the driving monitoring device 1 (notification control unit 54) to the second terminal device T2 by the driver operating the first terminal device T1 or the driving monitoring device 1. For example, the driver can operate the first terminal device T1 or the driving monitoring device 1 to generate restriction information indicating the presence or absence of such restriction on the transmission of notification information, and store the generated restriction information in the storage unit 70. Furthermore, the alarm control unit 52 can also decide whether to send notification information to the second terminal device T2 based on the restriction information stored in the storage unit 70. With this structure, the decision on whether to inform the user of the second terminal device T2 of any abnormal information involving the driver is made according to the driver's wishes. Therefore, the driver's privacy is easily protected.

[0106] According to the embodiments described above, the information processing system 100A includes: an information processing device 2; a first terminal device T1, which is used by the driver and is a notification unit 84 that is not controlled by the notification control unit 54; and a second terminal device T2, which is used by a user different from the driver and is also controlled by the notification control unit 54 to notify the first terminal device T1, wherein the second terminal device T2 causes the first terminal device T1 to make notifications based on information input by the user through operation. Thus, it is possible to provide the driver using the first terminal device T1 with notification content obtained from research by the user using the second terminal device T2.

[0107] <Variation Example>

[0108] The scope of the present invention is not limited to the described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0109] For example, the driver monitoring device 1 (information processing device 2) may perform other alarm processing (hereinafter referred to as auxiliary alarm processing) in addition to the aforementioned omission alarm processing. Auxiliary alarm processing may, for example, be a collision alarm processing that detects the possibility of a collision between a risky object and vehicle M based on the position and relative speed of the risky object relative to vehicle M and triggers an alarm on alarm unit 82. In collision alarm processing, the relative speed of the risky object relative to vehicle M may be calculated, for example, by the processing unit 40 analyzing the time change of the risky position detected by the risk detection unit 32. "Relative speed of the risky object relative to vehicle M" is an example of "information relating to the position of the risky object." When the processing unit 40 detects the possibility of a collision between the risky object and vehicle M, the alarm control unit 52 may also trigger an alarm on alarm unit 82. Auxiliary alarm processing may also be performed (i.e., in real-time) during the driver's driving of vehicle M, similar to omission alarm processing. In the configuration where auxiliary alarm processing is performed in driver monitoring device 1 (information processing device 2), the alarm on alarm unit 82 may not be triggered based on the omission determination result of omission determination unit 42. Even in this case, the omission determination performed by the omission determination unit 42 is executed in order to perform the above-mentioned abnormal notification processing.

[0110] Alternatively, information relating to the judgment results of the missed detection determined by the missed detection determination unit 42 may be collected in the storage unit 70 (e.g., judgment history information 72), and the first terminal device T1 (alarm unit 82) may provide the driver with entries (hereinafter referred to as "review entries") generated based on the collected information for the driver to review their past driving. Review entries may include, for example, information (strings, images, etc.) that the driver can identify as potentially overlooked objects. Review entries may be generated by the driving monitoring device 1 (information processing device 2) or by the first terminal device T1 (alarm unit 82).

[0111] In the information processing system 100A according to the second embodiment, the driving monitoring device 1 (notification control unit 54) can also send notification information to both the first terminal device T1 and the second terminal device T2. In other words, the notification unit 84 that controls the notifications from both the first terminal device T1 and the second terminal device T2 can also be controlled by the notification control unit 54.

[0112] In the first embodiment, the driver monitoring device 1 (information processing device 2) and the first terminal device T1 are configured as different devices, but they can also be configured as an integrated device. For example, the driver monitoring device 1 may actually include all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, alarm unit 82, and notification unit 84. Alternatively, all these functions may actually be included in the first terminal device T1 or the vehicle M. Similarly, in the second embodiment, all the functions of the camera unit 10, acquisition unit 20, detection unit 30, processing unit 40, control unit 50, storage unit 70, and alarm unit 82 may actually be included in the driver monitoring device 1, and all these functions may also be included in the first terminal device T1. A camera other than the one included in the driver monitoring device 1 may also perform some or all of the functions of the camera unit 10. For example, a driver monitoring camera that captures the driver may capture facial images.

[0113] The driving monitoring device 1 can also be actually installed using multiple information processing devices. For example, the functions of the information processing device 2 can also be actually installed in the first terminal device T1 or the vehicle M (e.g., ECU). Alternatively, the driving monitoring device 1 can be actually installed using a device such as a cloud. For example, in the driving monitoring device 1, the detection unit 30, processing unit 40, control unit 50, and storage unit 70 can also be actually installed in different information processing devices. For example, the storage unit 70 can also be distributed among multiple information processing devices.

[0114] The implementation methods and variations described above can be represented as follows.

[0115] An information processing device comprising:

[0116] Storage medium, which stores computer-readable instructions; and

[0117] The processor, which is connected to the storage medium,

[0118] The processor performs the following processing by executing computer-readable instructions:

[0119] Obtain images of the vehicle's surroundings and the driver's face.

[0120] Detect risky objects in the surrounding image;

[0121] The driver is determined to have missed seeing the risky object based on information relating to the location of the risky object and information relating to the facial image.

[0122] Based on information regarding the location of the risky object, the presence or absence of missed sightings, and gazes detected from the facial image, a vision mapping information is generated that links information regarding the frequency of missed sightings to directions within the driver's field of vision. This vision mapping information includes a first region encompassing the center of the driver's field of vision and a second region located outside the first region.

[0123] If the missed view exceeds a predetermined threshold in the second region of the vision mapping information compared to a predetermined benchmark value, it is determined that the driver has malfunctioned.

[0124] Without departing from the spirit of the present invention, the constituent elements in the above embodiments can be appropriately replaced with well-known constituent elements, and the above embodiments and variations can also be appropriately combined.

Claims

1. An information processing device, wherein, The information processing device includes: The acquisition unit acquires images of the vehicle's surroundings and the driver's face. The risk detection unit detects risky objects in the surrounding image; The omission determination unit determines whether the driver has missed the risky object based on information relating to the location of the risky object and information relating to the facial image; The generation unit generates vision mapping information that links information about the frequency of the missed sighting with the direction within the driver's field of vision, based on information about the location of the risky object, the presence or absence of the missed sighting, and the gaze detected from the facial image. The vision mapping information includes a first region including the center of the driver's field of vision and a second region located outside the first region. as well as The anomaly determination unit determines that the driver has an anomaly if the missed view exceeds a predetermined threshold compared to a predetermined reference value in the second region of the vision mapping information.

2. The information processing apparatus according to claim 1, wherein, The information processing device further includes a gaze detection unit for detecting the gaze from the facial image.

3. The information processing apparatus according to claim 1 or 2, wherein, The information processing device also includes a notification control unit, which causes the notification unit to issue a notification when it determines that the driver has experienced an abnormality.

4. The information processing apparatus according to claim 3, wherein, The notification control unit causes the notification to be made at a time other than when the driver is driving the vehicle.

5. The information processing apparatus according to claim 1 or 2, wherein, The information processing device also includes an alarm control unit that triggers an alarm based on information relating to the location of the object at risk.

6. An information processing system, wherein, The information processing system has the following features: An information processing device, which is the information processing device according to claim 1 or 2, and further includes a notification control unit that causes the notification unit to make a notification when it is determined that the driver has an abnormality. A first terminal device, which is used by the driver and is not the notification unit controlled by the notification control unit; as well as The second terminal device is for use by a user different from the driver, and the notification unit is controlled by the notification control unit. The second terminal device causes the first terminal device to make a notification based on information input by the user.

7. An information processing method, wherein, The information processing method causes the computer to perform the following processes: Obtain images of the vehicle's surroundings and the driver's face. Detect risky objects in the surrounding image; The driver is determined to have missed seeing the risky object based on information relating to the location of the risky object and information relating to the facial image. Based on information about the location of the risky object, the presence or absence of the missed sight, and the gaze detected from the facial image, a vision mapping information is generated that links information about the frequency of the missed sight with the direction within the driver's field of vision, and the vision mapping information includes a first region including the center of the driver's field of vision and a second region located outside the first region. as well as If the missed view exceeds a predetermined threshold in the second region of the vision mapping information compared to a predetermined benchmark value, it is determined that the driver has malfunctioned.

8. A storage medium that stores a non-transitory storage medium capable of being read by a computer, wherein, The program is used to cause the computer to perform the following processes: Obtain images of the vehicle's surroundings and the driver's face. Detect risky objects in the surrounding image; The driver is determined to have missed seeing the risky object based on information relating to the location of the risky object and information relating to the facial image. Based on information about the location of the risky object, the presence or absence of the missed sight, and the gaze detected from the facial image, a vision mapping information is generated that links information about the frequency of the missed sight with the direction within the driver's field of vision, and the vision mapping information includes a first region including the center of the driver's field of vision and a second region located outside the first region. as well as If the missed view exceeds a predetermined threshold in the second region of the vision mapping information compared to a predetermined benchmark value, it is determined that the driver has malfunctioned.

Images