Information processing device, mobile device, method, and program

Abstract

The invention relates to an information processing apparatus, a mobile apparatus, a method, and a program. The present invention implements a configuration in which driver information and environment information are input, and in which a safety index value representing whether a driver in autonomous driving is in a state in which the driver can perform safe manual driving or a time at which the driver can return to manual driving is estimated. The present invention includes: a driver information acquisition unit that acquires driver information of a mobile device such as an automobile; an environment information acquisition unit that acquires environment information of the mobile device; and a safety determination unit to which driver information and environment information are input, and which learns and calculates a safety index value indicating whether or not a driver in the mobile device during autonomous driving is in a state in which the driver can perform safe manual driving. The safety determination unit also estimates a time at which it is possible to return to manual driving, which is a time before a driver in the mobile device in autonomous driving is able to start safe manual driving.

Description

[0001] This application is a divisional application of the PCT application entering the Chinese national phase, with an international filing date of January 29, 2019, national application number 201980010757.X, and invention name “Information processing device, mobile device, method and program”. Technical Field

[0002] The present disclosure relates to an information processing device, a mobile device, a method, and a program. More particularly, the present disclosure relates to an information processing device, a mobile device, a method, and a program for performing control to switch between automatic driving and manual driving. Background Art

[0003] In recent years, technologies related to autonomous driving have been actively developed.

[0004] The autonomous driving technology is a technology that enables automatic driving on roads by using various sensors such as position detection devices provided in vehicles (cars), and it is predicted that the autonomous driving technology will spread rapidly in the future.

[0005] However, under the current circumstances, autonomous driving is in the development stage, and it is believed that it will take time to achieve 100% autonomous driving, and it is predicted that for a period of time, driving will be performed while appropriately switching autonomous driving and manual driving performed by the driver.

[0006] For example, it is predicted that it will be necessary to travel in automatic driving mode on a straight road with a sufficient road width such as an expressway, and in a case where the vehicle leaves the expressway and stops at a desired location in a parking lot or on a mountain road with a narrow road width, etc., switch to manual driving mode and travel in accordance with the operation performed by the driver.

[0007] For example, while the vehicle is performing automatic driving, the driver does not need to direct his or her gaze in the forward direction, which is the direction in which the vehicle is traveling, and has the freedom to act such as dozing off, watching TV, reading, or sitting back facing and talking to someone sitting in the back seat.

[0008] In a vehicle that switches between automatic driving and manual driving, if it is necessary to switch from the automatic driving mode to the manual driving mode, the driver must start manual driving.

[0009] However, if the driver falls asleep during automated driving, for example, their alertness deteriorates. In other words, their level of consciousness has deteriorated. As described above, switching to manual driving mode while their alertness is reduced makes normal manual driving impossible, and in the worst case, an accident could occur.

[0010] In order to ensure driving safety, manual driving needs to be started when the driver is in a state of high alertness, in other words, when the driver is very conscious.

[0011] To do this, the driver's sobriety needs to be checked.

[0012] Patent Document 1 (Japanese Patent No. 4980988) is an example of a prior art technology that discloses a technique for checking a driver's alertness. This document discloses a configuration in which the deterioration state of the driver's attention is estimated by using facial information or biometric information such as the driver's line of sight or pupil diameter.

[0013] However, the prior art including Patent Document 1 does not disclose a configuration for controlling the notification timing required to safely return from autonomous driving to manual driving.

[0014] In the event that a switch from autonomous driving mode to manual driving mode is required, a notification is required to enable the driver to start manual driving.

[0015] If the notification timing is too late, manual driving is started in a state where the driver's alertness is low, in other words, in a state where the consciousness level has deteriorated, and there is a problem that the probability of an accident occurring increases.

[0016] Citation List

[0017] Patent Literature

[0018] Patent Document 1: Japanese Patent No. 4980988 Summary of the Invention

[0019] Problems to be solved by the present invention

[0020] The present disclosure has been made in view of, for example, the above-described program, and an object of the present disclosure is to provide an information processing device, a mobile device, a method, and a program that enable safe execution of switching from automatic driving to manual driving.

[0021] Solution to the problem

[0022] The first aspect of the present disclosure is:

[0023] An information processing device includes: a safety determination unit that receives driver information and environmental information as input and calculates a safety index value indicating whether a driver in a vehicle is in a state capable of performing safe manual driving during automatic driving.

[0024] Furthermore, a second aspect of the present disclosure is:

[0025] A mobile device, comprising:

[0026] a driver information acquiring unit configured to acquire driver information of the mobile device;

[0027] an environmental information acquiring unit configured to acquire environmental information of the mobile device; and

[0028] A safety determination unit receives driver information and environmental information as input and calculates a safety index value indicating whether the driver in the mobile device is in a state capable of performing safe manual driving during automatic driving.

[0029] Furthermore, a third aspect of the present disclosure is:

[0030] An information processing method performed by an information processing device, the information processing method comprising:

[0031] The safety determination unit receives driver information and environment information as input and performs processing for calculating a safety index value indicating whether a driver in the vehicle is in a state capable of performing safe manual driving during automatic driving.

[0032] Furthermore, a fourth aspect of the present disclosure is:

[0033] An information processing method executed by a mobile device, the information processing method comprising:

[0034] A driver information acquiring step in which a driver information acquiring unit acquires driver information of the mobile device;

[0035] An environmental information acquisition step in which an environmental information acquisition unit acquires environmental information of the mobile device; and

[0036] A safety determination step in which a safety determination unit receives driver information and environment information as input and calculates a safety index value indicating whether the driver in the mobile device is in a state capable of performing safe manual driving during automatic driving.

[0037] In addition, a fifth aspect of the present disclosure is:

[0038] A program causing an information processing device to execute information processing, the information processing comprising:

[0039] The safety determination unit receives driver information and environment information as input and calculates a safety index value indicating whether a driver in the vehicle is in a state capable of performing safe manual driving during automatic driving.

[0040] Note that the program according to the present disclosure is, for example, a program that can be provided to an information processing apparatus or a computer system that can execute various program codes by a storage medium or a communication medium that provides the program in a computer-readable form. By providing such a program in a computer-readable form, processing according to the program is executed on the information processing apparatus or the computer system.

[0041] Other objects, features and advantages of the present disclosure will be elucidated by a more detailed description of one example based on the later-described examples or drawings of the present disclosure. Note that the system described here is a configuration including a logical set of a plurality of apparatuses, and does not always include apparatuses with respective configurations in the same housing.

[0042] Effects of the Invention

[0043] By employing the configuration in one example of the present disclosure, a configuration is realized in which driver information and environmental information are input, and a safety index value indicating whether a driver who is performing automatic driving is in a state in which safe manual driving can be performed or a manual driving resumption available time is estimated.

[0044] Specifically, for example, the configuration includes a driver information acquisition unit that acquires driver information of a mobile apparatus such as a car, an environmental information acquisition unit that acquires environmental information of the mobile apparatus, and a safety determination unit that receives driver information and environmental information as input, learns and calculates a safety index value indicating whether a driver in the mobile apparatus during automatic driving is in a state in which safe manual driving can be performed. The safety determination unit also estimates a manual driving resumption available time including a time until the driver in the mobile apparatus during automatic driving can start safe manual driving.

[0045] By employing such a configuration, a configuration is realized in which driver information and environmental information are input, and a safety index value indicating whether a driver who is performing automatic driving is in a state in which safe manual driving can be performed or a manual driving resumption available time is estimated.

[0046] Note that the effects described here are illustrative only and are not limiting, and additional effects can be presented. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a diagram that explains one configuration example of a mobile apparatus according to the present disclosure.

[0048] Figure 2 is a diagram that explains one example of data displayed on a display unit of a mobile apparatus according to the present disclosure.

[0049] Figure 3 is a diagram that explains processing performed by a mobile apparatus according to the present disclosure.

[0050] Figure 4 is a diagram illustrating a configuration example of a mobile device according to the present disclosure.

[0051] Figure 5 is a diagram illustrating a configuration example of a mobile device according to the present disclosure.

[0052] Figure 6 is a diagram illustrating a sensor configuration example of a mobile device according to the present disclosure.

[0053] Figure 7 is a diagram illustrating one example of a mode switching sequence from an automatic driving mode to a manual driving mode performed by a mobile device according to the present disclosure.

[0054] Figure 8 1 is a diagram showing a flowchart illustrating an example of an operation sequence of automatic driving.

[0055] Figure 9 This is a diagram showing an example of a travel route in which an automatic driving permission / prohibition section determined by a driver setting a destination is constantly set or generated.

[0056] Figure 10 This is a diagram explaining information processing for displaying travel sections on a travel route.

[0057] Figure 11 FIG. 1 is a diagram showing an example of the finally performed travel section display.

[0058] Figure 12 1 is a diagram showing an example of a change in the travel section display over time (an example of scrolling).

[0059] Figure 13 1 is a diagram showing an example of a display of travel sections on a travel route displayed on a screen of a tablet terminal device (hereinafter simply referred to as a “tablet”).

[0060] Figure 14 : is a diagram showing an example of a state in which a driver performs a secondary task by actually using a tablet computer.

[0061] Figure 15 This diagram shows a state where a cautious travel section Sd is newly generated in the second section and a warning is given to the driver by flashing the display.

[0062] Figure 16 FIG. 1 is a diagram showing a state where a small window pops up on the screen of a tablet computer.

[0063] Figure 17 It is a diagram illustrating a specific configuration example and processing of a safety determination unit.

[0064] Figure 18 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process.

[0065] Figure 19 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process.

[0066] Figure 20 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process.

[0067] Figure 21 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process.

[0068] Figure 22 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process.

[0069] Figure 23 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process.

[0070] Figure 24 FIG. 1 is a diagram showing a flowchart of a sequence example of an explanation learning process. DETAILED DESCRIPTION

[0071] Details of the information processing apparatus, mobile apparatus, method, and program according to the present disclosure are described below with reference to the drawings. Note that the description is provided according to the following items.

[0072] 1. Overview of configuration and processing of mobile apparatus and information processing apparatus

[0073] 2. Specific configuration and processing example of mobile apparatus

[0074] 3. Mode switching sequence from autonomous driving mode to manual driving mode

[0075] 4. Operation sequence example of autonomous driving

[0076] 5. Safety determination processing and manual driving recovery available time estimation processing performed by mobile apparatus

[0077] 6. Sequence of processing performed by mobile apparatus and information processing apparatus according to the present disclosure

[0078] 7. Configuration example of information processing apparatus

[0079] 8. Overview of configuration according to the present disclosure

[0080] [1. Overview of Configuration and Processing of Mobile Device and Information Processing Device]

[0081] First, refer to Figure 1 and subsequent figures describe an overview of the configuration and processing of a mobile device and an information processing device.

[0082] The mobile device according to the present disclosure is, for example, a car capable of traveling while switching between automatic driving and manual driving.

[0083] In such a car, when it is necessary to switch from the automatic driving mode to the manual driving mode, the driver must start manual driving.

[0084] However, during execution of autonomous driving, various types of processing (secondary tasks) are performed by the driver.

[0085] For example, simply releasing the hands from the steering wheel includes situations like when the driver looks ahead of the car while driving, when the driver reads a book, or when the driver falls asleep.

[0086] The driver's alertness (level of consciousness) changes due to the differences between these treatments.

[0087] For example, if the driver falls asleep, their alertness deteriorates. In other words, their level of consciousness has deteriorated. With this level of alertness deteriorating, as mentioned above, normal manual driving is impossible. If a switch to manual driving mode is made in this state, in the worst-case scenario, an accident could occur.

[0088] In order to ensure driving safety, manual driving needs to be started when the driver is in a state of high alertness, in other words, when the driver is very conscious.

[0089] For this reason, it is necessary to change the notification timing of issuing a request to switch from the automatic driving to the manual driving according to the driver's alertness during execution of the automatic driving.

[0090] For example, in a case where the driver is facing forward and is looking at the road during execution of autonomous driving, the driver has a high level of alertness, in other words, the driver can start manual driving at any time.

[0091] In this case, it is sufficient to issue a notification to switch to manual driving at a moment before the time when manual driving is required, because the driver can immediately start safe manual driving.

[0092] However, in the case where the driver is dozing off during execution of autonomous driving, the driver has an extremely low level of alertness.

[0093] In this case, if a notification to switch to manual driving is issued before the time when manual driving is required, the driver must begin manual driving while unconscious. This increases the possibility of an accident. Therefore, as described above, when the driver's alertness is low, a notification to switch to manual driving should be issued at an earlier stage.

[0094] The mobile device or the information processing device that can be installed in the mobile device according to the present disclosure controls the timing of notification of switching to manual driving according to the driver's alertness, for example.

[0095] Furthermore, in one example of the present disclosure, in addition to driver information such as sobriety, the optimal timing of notification for switching to manual driving is calculated by using information such as the driving time of a mobile device, driver operation information, environmental information of a road, or information related to past accidents.

[0096] For example, in one example of the present disclosure, a learning process is performed to calculate the level of alertness by using each of the above-mentioned information (i.e., information such as the driving time of the mobile device, driver operation information, environmental information of the road, or information related to accidents that occurred in the past), and the optimal notification timing is calculated as a result of the learning process.

[0097] Note that the learning process may be performed in the car serving as the mobile device, or may be performed in a server capable of communicating with the car serving as the mobile device.

[0098] In addition, in one example of the present disclosure, in the above-mentioned notification timing calculation processing, information related to the individual driver (i.e., driving history, accident history, driving operation information after the driver's manual driving starts) and information related to the type of car, etc. are taken into consideration to perform processing.

[0099] In addition, the number of occupants or information related to loaded cargo is also used.

[0100] Note that in the case of using many types of information, the many types of information may be used in a state where the many types of information have been classified into information for advanced learning and information for online learning according to data characteristics.

[0101] Reference Figure 1 , and the following drawings, describe configurations and processes of a mobile device and an information processing device that can be installed in the mobile device according to the present disclosure.

[0102] Figure 1 is a diagram showing one configuration example of a car 10 as one example of a mobile device according to the present disclosure.

[0103] The information processing device according to the present disclosure is installed Figure 1 On the car 10 shown.

[0104] Figure 1 The illustrated automobile 10 is a vehicle that can be driven in two driving modes: a manual driving mode and an automatic driving mode.

[0105] In the manual driving mode, driving is performed based on an operation performed by the driver 20 , that is, an operation performed by a steering wheel or an operation performed by an accelerator, a brake, or the like.

[0106] In contrast, in the automatic driving mode, the driver 20 does not need to perform an operation, and driving is performed based on sensor information from, for example, a position sensor, other peripheral information detection sensors, and the like.

[0107] The position sensor is, for example, a GPS receiver, etc., and the peripheral information detection sensor is, for example, a camera, an ultrasonic sensor, a radar, light detection and ranging or laser imaging detection and ranging (LiDAR), sonar, etc.

[0108] Notice, Figure 1 1 is a diagram illustrating an overview of the present disclosure, and schematically shows main components. The detailed configuration will be described later.

[0109] like Figure 1 As shown, the automobile 10 includes a data processing unit 11 , a driver information acquisition unit 12 , an environment information acquisition unit 13 , a communication unit 14 , and a notification unit 15 .

[0110] The driver information acquisition unit 12 acquires, for example, information for determining the driver's alertness, driver operation information, etc. Specifically, the driver information acquisition unit 12 includes, for example, a camera that captures a facial image of the driver, an operation information acquisition unit for each operation unit (such as a steering wheel, accelerator, or brake), etc.

[0111] The environment information acquisition unit 13 acquires driving environment information of the car 10. Examples include image information of the front, rear, left, and right sides of the car, position information from GPS, information about surrounding obstacles from light detection and ranging or laser imaging detection and ranging (LiDAR), sonar, etc.

[0112] The data processing unit 11 receives the driver information acquired by the driver information acquisition unit 12 or the environmental information acquired by the environmental information acquisition unit 13 as input, and calculates a safety index value indicating whether the driver in the vehicle is able to perform safe manual driving during automatic driving, whether the driver is performing safe driving during manual driving, etc.

[0113] Furthermore, for example, in the case where switching from the automatic driving mode to the manual driving mode is required, processing for issuing a notification via the notification unit 15 is performed in such a manner that switching to the manual driving mode is performed.

[0114] It is assumed that the timing of this notification process is, for example, an optimal timing calculated by inputting to the driver information acquisition unit 12 and the environment information acquisition unit 13 .

[0115] In other words, it is assumed that a timing at which the driver 20 can start safe manual driving is used.

[0116] Specifically, if the driver's alertness is high, notification is issued immediately before the start of manual driving, for example, five seconds before. If the driver's alertness is low, processing is performed with ample time, for example, 20 seconds before the start of manual driving. The specific calculation of optimal notification timing will be described later.

[0117] The notification unit 15 includes a display unit, a sound output unit, or a vibrator of a steering wheel or a seat that issues the notification.

[0118] exist Figure 2 An example of a warning display performed on the display unit configuring the notification unit 15 is shown in FIG.

[0119] like Figure 2 As shown, various displays described below are performed on the display unit 30.

[0120] Driving mode information = "Autonomous Driving",

[0121] Warning display = "Please switch to manual driving."

[0122] In the area where the driving mode information is displayed, “Automatic Driving” is displayed during execution of the automatic driving mode, and “Manual Driving” is displayed during execution of the manual driving mode.

[0123] The area where the warning display information is displayed is a display area where the following display is performed while the automatic driving is executed in the automatic driving mode.

[0124] "Please switch to manual driving"

[0125] In addition, if Figure 1 As shown, the car 10 has a configuration capable of communicating with a server 30 via a communication unit 14 .

[0126] For example, a part of the process for calculating the appropriate time to output a notification, which is performed by the data processing unit 11 , and specifically, a learning process may be performed in the server 30 .

[0127] A specific example of this case is described later.

[0128] Figure 3 is a diagram illustrating a specific example of a process performed by a mobile device or an information processing device according to the present disclosure.

[0129] Figure 3 : is a diagram showing an example of setting appropriate timing for notification of a request to switch to manual driving during automatic driving execution in the automatic driving mode, and shows the following two examples of notification processing.

[0130] (a) Notification Processing When the Driver Has a High Level of Alertness During Execution of Automatic Driving

[0131] (b) Notification Processing in the Case Where the Driver Has Low Alertness During Execution of Automatic Driving

[0132] Example (a) is an example in which the driver is facing forward and looking at the road during the execution of autonomous driving. In this case, the driver has a high level of alertness, in other words, the driver can start manual driving at any time.

[0133] In this case, even if a notification to switch to manual driving is issued just before the time when manual driving is required, the driver can immediately start safe manual driving.

[0134] In example (b), for example, in a case where the driver is dozing off during execution of the autonomous driving, the driving has an extremely low alertness.

[0135] In this case, if a notification to switch to manual driving is issued just before the time when manual driving is required, the driver will start manual driving while unconscious, and there is a high possibility of an accident. Therefore, when the driver's alertness is low, as described above, it is necessary to issue a notification to switch to manual driving at an earlier stage.

[0136] [2. Specific Configuration and Processing Example of Mobile Device]

[0137] Next, refer to Figure 4 and subsequent drawings describe specific configuration and processing examples of the mobile device 10 according to the present disclosure.

[0138] Figure 4 1 shows a configuration example of the mobile device 100. Note that, hereinafter, in the case of distinguishing a vehicle provided with the mobile device 100 from other vehicles, the vehicle provided with the mobile device 100 is referred to as a host vehicle or a host vehicle.

[0139] The mobile device 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, an in-vehicle device 104, an output controller 105, an output unit 106, a drive system controller 107, a drive system 108, a main body system controller 109, a main body system 110, a storage device 111 and an automatic driving controller 112.

[0140] The input unit 101, the data acquisition unit 102, the communication unit 103, the output controller 105, the drive system controller 107, the main body system controller 109, the storage device 111, and the automatic driving controller 112 are connected to each other via a communication network 121. The communication network 121 includes, for example, an in-vehicle communication network that conforms to any standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), or FlexRay (registered trademark), a bus, etc. Note that in some cases, the various units of the mobile device 100 are directly connected without the communication network 121.

[0141] In addition, below, when each unit of the mobile device 100 performs communication via the communication network 121, the description of the communication network 121 is omitted. For example, when the input unit 101 and the automatic driving controller 112 perform communication via the communication network 121, the input unit 101 and the automatic driving controller 112 perform communication simply.

[0142] The input unit 101 includes devices used by passengers to input various types of data, instructions, and the like. For example, the input unit 101 includes operating devices such as a touch panel, buttons, microphone, switch, or joystick, as well as operating devices that can perform input operations by methods other than manual operation using voice or gestures. Furthermore, for example, the input unit 101 may be a remote control device using infrared or other radio waves, or an externally connected device such as a mobile device or a wearable device that corresponds to the operation of the mobile device 100. The input unit 101 generates input signals based on the data, instructions, and the like input by the passenger, and provides these input signals to the various units of the mobile device 100.

[0143] The data acquisition unit 102 includes various sensors and the like that acquire data used in processing performed by the mobile device 100 , and supplies the acquired data to the respective units of the mobile device 100 .

[0144] For example, the data acquisition unit 102 includes various sensors for detecting the state of the vehicle. Specifically, for example, the data acquisition unit 102 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting the amount of operation performed on the accelerator pedal, the amount of operation performed on the brake pedal, the steering angle of the steering wheel, the engine speed, the motor rotation speed, the wheel rotation speed, and the like.

[0145] In addition, for example, the data acquisition unit 102 includes various sensors that detect information related to the exterior of the vehicle. Specifically, for example, the data acquisition unit 102 includes an imaging device such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras. In addition, for example, the data acquisition unit 102 includes environmental sensors that detect weather, meteorological phenomena, etc., and peripheral information detection sensors that detect objects around the vehicle. Environmental sensors include, for example, raindrop sensors, fog sensors, sunlight sensors, snow sensors, etc. The peripheral information detection sensors include, for example, ultrasonic sensors, radars, light detection and ranging or laser imaging detection and ranging (LiDAR), sonar, etc.

[0146] For example, Figure 5 The following illustrates an example of installing various sensors that detect information related to the exterior of the vehicle. For example, imaging devices 7910, 7912, 7914, 7916, and 7918 are installed in at least one of the front nose, side mirrors, rear bumper, rear door, or upper portion of the windshield in the vehicle cabin of the vehicle 7900.

[0147] Imaging device 7910 included in the front nose and imaging device 7918 included in the upper portion of the windshield in the vehicle cabin primarily capture images of the front side of vehicle 7900. Imaging devices 7912 and 7914 included in the side mirrors primarily capture images of the sides of vehicle 7900. Imaging device 7916 included in the rear bumper or rear door primarily captures images of the rear side of vehicle 7900. Imaging device 7918 included in the upper portion of the windshield in the vehicle cabin is primarily used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, and the like. Furthermore, in future autonomous driving, when the vehicle turns right or left, its use can be expanded to detect pedestrians crossing the road after a right or left turn within a wider range, and further expanded to include objects approaching a crosswalk.

[0148] Notice, Figure 5 An example of the image capture range of each of imaging devices 7910, 7912, 7914, and 7916 is shown. Imaging range a represents the imaging range of imaging device 7910, located in the front nose; imaging ranges b and c represent the imaging ranges of imaging devices 7912 and 7914, respectively, located in the side mirrors; and imaging range d represents the imaging range of imaging device 7916, located in the rear bumper or rear door. For example, by superimposing image data captured by imaging devices 7910, 7912, 7914, and 7916, it is possible to obtain a bird's-eye view of vehicle 7900 from above, a full-circle stereoscopic display image of the vehicle's periphery surrounded by curved surfaces, and the like.

[0149] Sensors 7920, 7922, 7924, 7926, 7928, and 7930 located on the front, rear, sides, corners, and upper portion of the windshield in the interior of vehicle 7900 may be, for example, ultrasonic sensors or radars. Sensors 7920, 7926, and 7930 located on the front nose, rear bumper, rear doors, and upper portion of the windshield in the interior of vehicle 7900 may be, for example, LiDARs. These sensors 7920 to 7930 are primarily used to detect preceding vehicles, pedestrians, obstacles, and the like. These detection results can be further used to improve the display of three-dimensional objects in the aforementioned bird's-eye view display or all-around stereoscopic display.

[0150] return Figure 4 , the description of each component continues. The data acquisition unit 102 includes various sensors for detecting the current position of the vehicle. Specifically, for example, the data acquisition unit 102 includes a global navigation satellite system (GNSS) receiver that receives GNSS signals from GNSS satellites.

[0151] Furthermore, for example, the data acquisition unit 102 includes various sensors that detect in-vehicle information. Specifically, for example, the data acquisition unit 102 includes an imaging device for imaging the driver, a biometric sensor for detecting the driver's biological information, and a microphone for collecting sounds within the vehicle cabin. Biometric sensors are placed on, for example, seat surfaces, steering wheels, and other surfaces to detect the seated state of a passenger or the biological information of the driver steering the wheel. Biometric signals can include a variety of observable data, such as heart rate, pulse rate, blood flow, respiration, mind-body correlation, visual stimulation, brain waves, perspiration, head and eye posture and behavior, gaze, blinking, saccades, microsaccades, fixation, drifting, staring, or pupillary reaction of the iris. This observable biological activity information reflecting observable driving conditions is used by the safety determination unit 155, described later, to calculate the recovery notification timing, which is a characteristic inherent to the corresponding driver's recovery delay case, based on the recovery delay time characteristics, which are summarized as observable evaluation values ​​estimated through observation and associated with a log of the evaluation values.

[0152] Figure 6 Examples of various sensors included in data acquisition unit 102 and used to acquire information related to the driver in the vehicle are shown. For example, data acquisition unit 102 includes a time-of-flight camera, a stereo camera, a seat strain gauge, and the like as sensors for detecting the driver's position or posture. Furthermore, data acquisition unit 102 includes a facial (head) recognition device, a driver eye tracker, a driver head tracker, and the like as sensors for acquiring observable information on the driver's biological activity.

[0153] Furthermore, data acquisition unit 102 includes a vital sign detector as a detector for acquiring observable information on the driver's biological activity. Furthermore, data acquisition unit 102 includes a driver identification unit. Note that other authentication methods include knowledge authentication using passwords, personal identification numbers, etc., and biometric authentication using facial features, fingerprints, irises, voiceprints, etc.

[0154] The communication unit 103 performs communication with the in-vehicle device 104 and various off-vehicle devices, servers, base stations, and the like, and transmits data provided from various units of the mobile device 100 or provides received data to various units of the mobile device 100. Note that the communication protocols supported by the communication unit 103 are not particularly limited, and the communication unit 103 can also support multiple types of communication protocols.

[0155] For example, the communication unit 103 performs wireless communication with the in-vehicle device 104 by using wireless LAN, Bluetooth (registered trademark), near field communication (NFC), wireless USB (WUSB), etc. In addition, for example, the communication unit 103 performs wired communication with the in-vehicle device 104 via a connection terminal (and, if necessary, a cable) not shown by using a universal serial bus (USB), a high-definition multimedia interface (HDMI) (registered trademark), a mobile high-definition link (MHL), etc.

[0156] In addition, for example, the communication unit 103 communicates with equipment (e.g., an application server or a control server) existing on an external network (e.g., the Internet, a cloud network, or a company's private network) via a base station or an access point. In addition, for example, the communication unit 103 communicates with terminals (e.g., terminals of pedestrians or stores, or machine type communication (MTC) terminals) existing near the vehicle using peer-to-peer (P2P) technology.

[0157] Furthermore, the communication unit 103 performs V2X communications, such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, or vehicle-to-pedestrian communication. Furthermore, the communication unit 103 includes, for example, a beacon receiver that receives radio waves or electromagnetic waves transmitted from wireless stations installed on roads, etc., and acquires information related to the current location, traffic congestion, traffic regulations, required time, etc. Note that the communication unit can be used to pair with a leading vehicle traveling in a section, acquiring information acquired by the data acquisition unit installed in the leading vehicle as information during forward travel and using this information as supplementary data for the data acquisition unit 102 of the vehicle itself. This provides a means of further ensuring the safety of subsequent routes, particularly in tandem travel with a leading vehicle.

[0158] The in-vehicle device 104 includes, for example, a mobile device (tablet, smartphone, etc.) or a wearable device held by a passenger, an information device carried or installed to the vehicle, a navigation device that searches for a route to an arbitrary destination, etc. Note that if it is considered that the occupant is not always fixed to a seating fixed position due to the popularization of automated driving, the use can be expanded in the future to equipment that can be used in a detachable manner from a video reproduction device, game equipment, or other vehicle-mounted equipment. In the present example, the description is provided by using an example in which information related to a place where the intervention of the driver is required is presented only to the corresponding driver. However, the information can be further provided to a following vehicle in a platoon, etc., and in addition, by constantly providing the information to an operation management center of a passenger shuttle bus or a long-distance distribution commercial vehicle, the use of an appropriate combination with remote travel support can be realized.

[0159] The output controller 105 controls the output of each type of information to the passenger of the vehicle or outside the vehicle. For example, the output controller 105 controls the output of visual information (e.g., image data) and auditory information (e.g., sound data) from the output unit 106 by generating an output signal including at least one of visual information or auditory information and providing the output signal to the output unit 106. Specifically, for example, the output controller 105 combines image data captured by imaging devices different from each other in the data acquisition unit 102, generates an overview image, a panoramic image, etc., and provides an output signal including the generated image to the output unit 106. Further, for example, the output controller 105 generates sound data including a warning sound, a warning message, etc., with respect to a danger such as a collision, contact, or entry into a dangerous area, and provides an output signal including the generated sound data to the output unit 106.

[0160] The output unit 106 includes a device capable of outputting visual information or auditory information to the passenger of the vehicle or outside the vehicle. For example, the output unit 106 includes a display device, an instrument panel, an audio speaker, earphones, a wearable device such as a glasses-type display worn by a passenger, a projector, a lamp, etc. For example, the display device included in the output unit 106 can be a device including a normal display, or can be a device that displays auditory information in the field of view of the driver, such as a head-up display, a transmissive display, or a device having an augmented reality (AR) display function.

[0161] The drive system controller 107 controls the drive system 108 by generating various control signals and providing the various control signals to the drive system 108. Further, the drive system controller 107 provides a control signal to each unit other than the drive system 108 as needed, and issues a notification of the control state of the drive system 108, etc.

[0162] The drive system 108 includes various devices related to the drive system of the host vehicle. For example, the drive system 108 includes a drive power generation device (such as an internal combustion engine or a drive motor) that generates drive power, a drive power transmission mechanism that transmits the drive power to the wheels, a steering mechanism that adjusts the steering angle, a brake device that generates braking force, an anti-lock brake system (ABS), an electronic stability control (ESC), an electric power steering device, and the like.

[0163] The main body system controller 109 controls the main body system 110 by generating various control signals and supplying the various control signals to the main body system 110. In addition, the main body system controller 109 supplies control signals to various units other than the main body system 110 as needed, and issues a notification of the control state of the main body system 110 or the like.

[0164] The main body system 110 includes various devices of the main body system that are installed in the vehicle body. For example, the main body system 110 includes a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioner, various lamps (for example, a headlamp, a tail lamp, a brake lamp, a turn signal, a fog lamp, and the like), and the like.

[0165] The storage device 111 includes, for example, a read only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, and the like. The storage device 111 stores various programs, data, and the like used by the various units of the mobile device 100. For example, the storage device 111 stores map data such as a three-dimensional high-precision map (for example, a dynamic map), a global map that has lower precision and covers a wider area than the high-precision map, or a local map that includes information about the surroundings of the host vehicle.

[0166] The automatic driving controller 112 performs control related to automatic driving such as autonomous travel or driving support. Specifically, for example, the automatic driving controller 112 performs cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS) including collision avoidance or impact mitigation of the host vehicle, follow-up travel based on the distance between vehicles, vehicle speed maintenance travel, collision warning of the host vehicle, lane departure warning of the host vehicle, and the like. In addition, for example, the automatic driving controller 112 performs cooperative control aimed at autonomous driving or the like for operations performed independently of the driver. The automatic driving controller 112 includes a detection unit 131, a self-position estimation unit 132, a situation analyzer 133, a planning unit 134, and an action controller 135.

[0167] The detection unit 131 detects various types of information required for controlling automatic driving. The detection unit 131 includes an outside-vehicle information detection unit 141, an in-vehicle information detection unit 142, and a vehicle state detection unit 143.

[0168] The outside information detecting unit 141 performs processing for detecting information related to the outside of the host vehicle on the basis of data or signals from the respective units of the mobile device 100. For example, the outside information detecting unit 141 performs processing for detecting objects around the host vehicle, recognition processing and tracking processing, and processing for detecting distances and relative speeds between objects. Examples of objects to be detected include vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road signs, and the like.

[0169] Further, for example, the outside information detecting unit 141 performs processing for detecting the surrounding environment of the host vehicle. Examples of the surrounding environment to be detected include weather, air temperature, humidity, brightness, road surface state, and the like. The outside information detecting unit 141 provides data representing the results of the detection processing to the self-position estimating unit 132, the map analyzer 151, the traffic rule recognizer 152 and the situation recognizer 153 of the situation analyzer 133, the emergency avoidance unit 171 of the action controller 135, and the like.

[0170] For the information acquired by the outside information detecting unit 141, if the travel section as a section in which automatic driving travel can be performed is a section in which a local dynamic map that is constantly updated is provided from infrastructure, the information can be mainly provided from the infrastructure. Alternatively, travel can be performed in a state in which the information is constantly updated in advance from a preceding vehicle or a group of vehicles that travel in the corresponding section before intrusion into the section. Further, for example, in a case in which the local dynamic map is not constantly updated to the latest version by infrastructure, or in other cases, in particular, in a case of platooning or the like, in order to acquire road information just before a safer intrusion section, road environment information acquired from a lead vehicle that has intruded into a section can be further complementarily used. In many cases, it is determined whether automatic driving can be performed in a section in accordance with the presence or absence of such pre-information provided from infrastructure. The automatic driving availability information on a route provided from infrastructure is roughly equivalent to providing an invisible track as so-called "information". Note that the outside information detecting unit 141 is shown under the assumption that the outside information detecting unit 141 is installed in the host vehicle for convenience. However, by using information acquired by a preceding vehicle as "information", pre-predictability during travel can be further improved.

[0171] The in-vehicle information detection unit 142 performs processing for detecting in-vehicle information based on data or signals from the respective units of the mobile device 100. For example, the in-vehicle information detection unit 142 performs processing for authenticating and identifying the driver, processing for detecting the state of the driver, processing for detecting the passenger, processing for detecting the in-vehicle environment, and the like. Examples of the state of the driver to be detected include the physical condition, the degree of wakefulness, the degree of concentration, the degree of fatigue, the direction of the line of sight, detailed eye behavior, and the like.

[0172] Further, it is expected that in the future, automatic driving will be used in a state in which the driver is completely separated from the task of driving. The driver will temporarily doze off or will start another task, and the system will need to grasp the degree to which the wakefulness of the consciousness required to resume driving and the recovery have progressed. In other words, the driver monitoring system conventionally proposed mainly includes detection means for monitoring the deterioration of consciousness such as drowsiness. However, in the future, the driver will not intervene in driving at all. Therefore, the system will not include means for directly observing the degree of intervention by the driver in driving based on the stability of the steering by the steering device or the like, and will need to observe the transition of the recovery of the consciousness required for driving from a state in which the accurate consciousness state of the driver is unknown, grasp the accurate internal wakefulness state of the driver, and promote the intervention transfer from automatic driving to manual driving.

[0173] Therefore, the in-vehicle information detection unit 142 mainly has important two-level roles. The first role is passive monitoring of the state of the driver during automatic driving, and the second role is the cognition, perception, and determination of the driver's surroundings, and further the detection and determination of the operability of the steering device to the level at which manual driving can be performed after a recovery request is issued from the system and before the cautious driving section is reached. As control, a failure self-diagnosis can be further performed on the entire vehicle. In the case where the automatic driving function of the vehicle is deteriorated due to a failure of a part of the automatic driving function, the driver can similarly be prompted to resume manual driving as early as possible. Here, passive monitoring means detection means that do not request a responsive reaction of the consciousness from the driver, and does not exclude detection means that emit physical radio waves, light, or the like from the device and detect a response signal. In other words, it involves monitoring the state of the driver who is not conscious during a nap or the like, and it is assumed that the category that is not a cognitive responsive reaction of the driver is a passive method. An active responsive device that analyzes and evaluates a reflected or diffused signal due to the application of radio waves, infrared rays, or the like is not excluded. In contrast, means for requesting a consciousness response that requests a responsive reaction from the driver is active.

[0174] Examples of the in-vehicle environment to be detected include air temperature, humidity, brightness, odor, and the like. The in-vehicle information detection unit 142 provides data indicating the result of the detection process to the situation recognizer 153 of the situation analyzer 133 and the action controller 135. Note that, after the discovery system issues a drive resumption instruction to the driver, the driver cannot implement manual driving until the correct cutoff time, and in the case where it is determined that even if deceleration control is executed in a state where self-driving is maintained and a time delay is executed, the takeover is not timely executed, an instruction is issued to the emergency situation avoidance unit 171 or the like of the system, and a deceleration / retreat / stop process is started to cause the vehicle to retreat. In other words, even in the case where it is not timely made as an initial state, it is possible to obtain the arrival time to the takeover limit by causing the vehicle to start deceleration as early as possible.

[0175] The vehicle state detection unit 143 executes a process for detecting the state of the host vehicle on the basis of data or signals from the respective units of the mobile device 100. Examples of the state of the host vehicle to be detected include speed, acceleration, steering angle, presence or absence and content of abnormality, state of driving operation, position and inclination of an electric seat, state of a door lock, state of another in-vehicle device, and the like. The vehicle state detection unit 143 provides data indicating the result of the detection process to the situation recognizer 153 of the situation analyzer 133, the emergency situation avoidance unit 171 of the action controller 135, and the like.

[0176] The own position estimation unit 132 executes a process for estimating the position, posture, and the like of the host vehicle on the basis of data or signals from the respective units of the mobile device 100, such as the outside-vehicle information detection unit 141, the situation recognizer 153 of the situation analyzer 133, and the like. Furthermore, the own position estimation unit 132 generates a local map for estimating the own position (hereinafter referred to as an own position estimation map) as necessary.

[0177] For example, assume that the own position estimation map is a high-precision map using a technique such as simultaneous localization and mapping (SLAM). The own position estimation unit 132 provides data indicating the result of the estimation process to the map analyzer 151, the traffic rule recognizer 152, and the situation recognizer 153 of the situation analyzer 133, and the like. Furthermore, the own position estimation unit 132 stores the own position estimation map in the storage device 111.

[0178] The situation analyzer 133 executes a process for analyzing the situation of the host vehicle and the surroundings. The situation analyzer 133 includes the map analyzer 151, the traffic rule recognizer 152, the situation recognizer 153, the situation prediction unit 154, and the safety determination unit 155.

[0179] The map analyzer 151 performs processing for analyzing various maps stored in the storage device 111 as needed by using data or signals from various units of the mobile device 100 such as the self-position estimation unit 132, the outside information detection unit 141, and the like, and constructs a map including information required for automatic driving processing. The map analyzer 151 supplies the constructed map to the traffic rule recognizer 152, the situation recognizer 153, the situation prediction unit 154, the route planning unit 161, the action planning unit 162, the motion planning unit 163, and the like of the planning unit 134.

[0180] The traffic rule recognizer 152 performs processing for recognizing traffic rules around the host vehicle based on data or signals from various units of the mobile device 100 such as the self-position estimation unit 132, the outside information detection unit 141, the map analyzer 151, and the like. In this recognition processing, for example, the position and state of a traffic light around the host vehicle, the content of traffic regulations around the host vehicle, a traffic lane on which the host vehicle can travel, and the like are recognized. The traffic rule recognizer 152 supplies data indicating the result of the recognition processing to the situation prediction unit 154 and the like.

[0181] The situation recognizer 153 performs processing for recognizing situations related to the host vehicle based on data or signals from various units of the mobile device 100 such as the self-position estimation unit 132, the outside information detection unit 141, the inside information detection unit 142, the vehicle state detection unit 143, the map analyzer 151, and the like. For example, the situation recognizer 153 performs processing for recognizing the situation of the host vehicle, the situation around the host vehicle, the situation of the driver of the host vehicle, and the like. Further, the situation recognizer 153 generates a local map (hereinafter referred to as a situation recognition map) for recognizing the situation around the host vehicle as needed. For example, it is assumed that the situation recognition map is an occupancy grid map.

[0182] Examples of the situation of the host vehicle to be recognized include the position, posture, movement (e.g., speed, acceleration, moving direction, etc.) of the host vehicle, and the load amount of the cargo or the movement of the center of gravity of the vehicle body due to the loading of the cargo, which determines the motion characteristics of the host vehicle, the tire air pressure, the situation inherent to the vehicle and further inherent to the cargo loaded, such as the movement of the braking distance according to the brake pad wear situation, the maximum deceleration brake allowed to prevent the movement of the cargo at the time of load braking, the centrifugal relaxation limit speed in the curve travel according to the liquid load, and the recovery start timing required for the control variation of the vehicle itself and the load even under the completely same road environment such as the road surface wear coefficient, the curve of the bar, or the slope, and the like. Therefore, it is necessary to collect, learn, and reflect these various conditions in the optimum timing for the control. This is not something obtained by simply observing and monitoring the presence or absence, content, etc. of the abnormality of the host vehicle when determining the control timing based on the vehicle type or the load. In the transportation business or the like, the parameter for determining the increase in the grace time of the recovery desired to ensure the prescribed level of safety according to the characteristics inherent to the load is set to a fixed value in advance, but a method for uniformly determining all the notification timing determination conditions in the self-accumulation learning is not necessarily adopted.

[0183] Examples of the situation of the host vehicle to be recognized include the position, posture, movement (e.g., speed, acceleration, moving direction, etc.) of the host vehicle, and the load amount of the cargo or the movement of the center of gravity of the vehicle body due to the loading of the cargo, which determines the motion characteristics of the host vehicle, the tire air pressure, the situation inherent to the vehicle and further inherent to the cargo loaded, such as the movement of the braking distance according to the brake pad wear situation, the maximum deceleration brake allowed to prevent the movement of the cargo at the time of load braking, the centrifugal relaxation limit speed in the curve travel according to the liquid load, and the recovery start timing required for the control variation of the vehicle itself and the load even under the completely same road environment such as the road surface wear coefficient, the curve of the bar, or the slope, and the like. Therefore, it is necessary to collect, learn, and reflect these various conditions in the optimum timing for the control. This is not something obtained by simply observing and monitoring the presence or absence, content, etc. of the abnormality of the host vehicle when determining the control timing based on the vehicle type or the load. In the transportation business or the like, the parameter for determining the increase in the grace time of the recovery desired to ensure the prescribed level of safety according to the characteristics inherent to the load is set to a fixed value in advance, but a method for uniformly determining all the notification timing determination conditions in the self-accumulation learning is not necessarily adopted.

[0184] The situation recognizer 153 provides data indicating the result of the recognition processing (including the situation recognition map as necessary) to the self-position estimation unit 132, the situation prediction unit 154, and the like. Further, the situation recognizer 153 stores the situation recognition map in the storage device 111.

[0185] The situation prediction unit 154 performs processing for predicting the situation related to the host vehicle based on data or signals from the respective units of the mobile device 100, such as the map analyzer 151, the traffic rule recognizer 152, the situation recognizer 153, and the like. For example, the situation prediction unit 154 performs processing for predicting the situation of the host vehicle, the situation of the surroundings of the host vehicle, the situation of the driver, and the like.

[0186] Examples of the situation of the host vehicle to be predicted include behavior of the host vehicle, occurrence of an abnormality, distance that can be traveled, and the like. Examples of the situation around the host vehicle to be predicted include behavior of a moving object around the host vehicle, state change of a traffic light, change of an environment such as weather, and the like. Examples of the situation of the driver to be predicted include behavior of the driver, physical condition, and the like.

[0187] The situation prediction unit 154 provides data representing a result of the prediction processing to the route planning unit 161, the action planning unit 162, and the movement planning unit 163, and the like of the plan unit 134, together with data from the traffic rule recognizer 152 and the situation recognizer 153.

[0188] The safety determination unit 155 learns an optimum timing of recovery in accordance with a recovery action pattern of the driver, vehicle characteristics, and the like, and provides the learned information to the situation recognizer 153 and the like. Thereby, for example, the optimum timing required for the driver to normally recover from automatic driving to manual driving at a predetermined rate or more can be presented to the driver.

[0189] The route planning unit 161 plans a route to a destination on the basis of data or signals from the respective units (for example, the map analyzer 151, the situation prediction unit 154, and the like) of the mobile device 100. For example, the route planning unit 161 sets a route from a current position to a specified destination on the basis of a global map. Further, for example, the route planning unit 161 appropriately changes the route on the basis of a traffic jam, an accident, a traffic restriction, a work situation, and the like, a physical condition of the driver, and the like. The route planning unit 161 provides data representing the planned route to the action planning unit 162 and the like.

[0190] The action planning unit 162 plans an action of the host vehicle to safely travel on a route planned by the route planning unit 161 within a planned time on the basis of data or signals from the respective units (for example, the map analyzer 151, the situation prediction unit 154, and the like) of the mobile device 100. For example, the action planning unit 162 plans departure, stop, travel direction (for example, forward travel, backward travel, left turn, right turn, direction change, and the like), travel lane, travel speed, passing, and the like. The action planning unit 162 provides data representing the planned action of the host vehicle to the movement planning unit 163 and the like.

[0191] The movement planning unit 163 plans a movement of the host vehicle for realizing an action planned by the action planning unit 162 on the basis of data or signals from the respective units (for example, the map analyzer 151, the situation prediction unit 154, and the like) of the mobile device 100. For example, the movement planning unit 163 plans acceleration, deceleration, travel trajectory, and the like. The movement planning unit 163 provides data representing the planned movement of the host vehicle to the acceleration / deceleration controller 172 and the direction controller 173 of the movement controller 135 and the like.

[0192] The action controller 135 controls local actions. The action controller 135 includes an emergency avoidance unit 171, an acceleration / deceleration controller 172, and a direction controller 173.

[0193] The emergency avoidance unit 171 performs processing for detecting an emergency such as a collision, a contact, an entry into a dangerous area, an abnormality of the driver, and an abnormality of the vehicle, on the basis of the detection results of the outside information detection unit 141, the inside information detection unit 142, and the vehicle state detection unit 143. The emergency avoidance unit 171 plans an action of the host vehicle for avoiding the emergency, such as a sudden stop or a sharp turn, in a case where it is detected that an emergency occurs. The emergency avoidance unit 171 provides data indicating the planned action of the host vehicle to the acceleration / deceleration control unit 172, the direction control unit 173, and the like.

[0194] The acceleration / deceleration controller 172 controls acceleration / deceleration to achieve the action of the host vehicle planned by the action planning unit 163 or the emergency avoidance unit 171. For example, the acceleration / deceleration controller 172 calculates a control target value for the driving force generator or the brake device to achieve the planned acceleration / deceleration or sudden stop, and provides a control command indicating the calculated control target value to the drive system controller 107. Note that there are mainly two cases where an emergency occurs. Specifically, there are a case where an unexpected accident occurs due to an unexpected cause and it is not possible to promptly perform emergency recovery on a road that was initially determined to be safe during automatic driving on a travel route during automatic driving acquired from a local dynamic map or the like by infrastructure, and a case where the driver is difficult to accurately recover from automatic driving to manual driving.

[0195] The direction controller 173 controls the direction for achieving the action of the host vehicle planned by the action planning unit 163 or the emergency avoidance unit 171. For example, the direction controller 173 calculates a control target value for the steering mechanism to achieve the travel trajectory or the sharp turn planned by the action planning unit 163 or the emergency avoidance unit 171, and provides a control command indicating the calculated control target value to the drive system controller 107.

[0196] [3. Sequence of mode switching from the automatic driving mode to the manual driving mode]

[0197] Next, the takeover sequence from the automatic driving mode to the manual driving mode is described.

[0198] Figure 7 One example of the sequence of mode switching from the automatic driving mode to the manual driving mode performed by the automatic driving controller 112 is schematically shown.

[0199] In step S1, the driver is in a state of complete disengagement from driving. For example, in this state, the driver can perform a secondary task such as a nap, watching a video, focusing on a game, or a task using a visual tool such as a tablet or a smartphone. It is conceivable that the task using a visual tool such as a tablet or a smartphone is performed, for example, in a state where the driver seat has been displaced or in a seat different from the driver seat.

[0200] According to these states of the driver, when the driver approaches an interval on the route where resumption to manual driving is required, the time required for the driver to resume is expected to change significantly depending on the content of the current task. If the notification is issued just before the event arrives, the time required for the resume is insufficient, or in a case where the notification is issued too early to ensure a margin before the event arrives, the time until the timing actually required for the resume is too long. If the case where the notification is not issued at the accurate timing thus occurs repeatedly, the timing reliability of the notification timing of the system decreases, the driver lowers the awareness of the notification, as a result, the driver neglects to take the accurate measure, and this will lead to a high risk of takeover failure, and hinders the reliable performance of the secondary task. Therefore, in order for the driver to start the accurate measure for driving resumption in response to the notification, the system needs to optimize the notification timing.

[0201] As described above with reference to Figure 2 Step S2 indicates the timing of the manual driving resumption request notification. The notification of the driving resumption is issued to the driver by using a dynamic method such as vibration or in a visual or auditory manner. For example, the automated driving controller 112 monitors the stable state of the driver, masters the timing of the notification, and issues the notification at the appropriate timing. In other words, it is expected that the state of the driver performing the secondary task is constantly and passively monitored during the preceding stage passive monitoring period, the system can calculate the optimal timing of the notification, constantly and continuously perform passive monitoring during the step S1 period, and match the resumption timing and the resumption notification to the resumption characteristics inherent to the driver.

[0202] In other words, it is expected to learn the optimal resumption timing according to the resumption action pattern of the driver, the characteristics of the vehicle, and the like, and present the optimal timing to the driver, which is statistically obtained as required for the driver to normally resume from the automated driving to the manual driving at a predetermined certain rate or more. In this case, in a case where the driver does not respond to the notification during a certain period of time, a warning is issued using an alarm sound or the like.

[0203] In step S3, it is checked whether the driver has performed the seating resumption. In step S4, the state of the internal wakefulness of the driver is checked by analyzing the behavior of the face or the eyeball such as a saccade. In step S5, the stability of the actual turning situation of the driver is monitored. Then, in step S6, the takeover from the automated driving to the manual driving is completed.

[0204] [4. Example of operation sequence of autonomous driving]

[0205] Next, one example of an operation sequence of autonomous driving will be described with reference to the flowchart illustrated in Figure 8

[0206] Figure 8 The flowchart illustrated in FIG. 1 is a flowchart for explaining an operation sequence of implementing autonomous driving of the mobile device 100.

[0207] First, in step S11, driver authentication is performed. This driver authentication is performed by using either a knowledge authentication using a password, a personal identification number, or the like, or a biometric authentication using a face, a fingerprint, an iris of a pupil, a voiceprint, or the like, or by using both the knowledge authentication and the biometric authentication. By performing the driver authentication, as described above, even in a case where a plurality of drivers drive the same vehicle, information for determining a notification timing associated with each driver can be accumulated.

[0208] Next, in step S12, the driver operates the input unit 101 to set a destination. In this case, the driver performs an input operation based on a display on the dashboard

[0209] Note that the present example describes an example in a case where a route is set on the vehicle. However, remote advance reservation setting or the like can be performed in advance by using a smartphone before entering the vehicle or by using a personal computer before leaving home. Further, a system of the car can perform a pre-planned setting according to a schedule that the driver has assumed according to a schedule, can update and acquire LDM information of a road environment (that is, so-called local dynamic map (LDM) information that constantly updates travel map information of a road on which the vehicle travels with high precision), and the like, and can further perform an actual travel advice display or the like in a manner of a gatekeeper at the time of entering the vehicle or before entering the vehicle.

[0210] Next, in step S13, a travel section display on a travel route is started. This travel section display is performed, for example, on the dashboard, or is performed, for example, in a tablet or the like in which the driver performs a secondary task so that the travel section and a work window are arranged side by side. Thereby, the driver who performs a task in the work window can easily recognize a driver intervention necessary section and an autonomous driving available section on the travel route on a travel time axis from a current location.

[0211] ​In this travel section display, a forward schedule and information related to the approach to each location are presented. In this travel section display, a driver intervention necessary section and an automatic driving available section on a travel route are displayed on a reaching prediction time axis from a current location. Then, the driver intervention necessary section includes a manual driving section, a takeover section from automatic driving to manual driving, and a cautious driving section from automatic driving. Details of this travel section display are described later.

[0212] Next, in step S14, acquisition of LDM update information is started. According to the acquisition of LDM update information, the content of the travel section display can be changed to the latest state. Next, in step S15, travel is started. Next, in step S16, display of the travel section display is updated based on the position information of the host vehicle and the acquired LDM update information. Thereby, the travel section display is scroll-displayed according to travel in a manner that each section approaches the host vehicle.

[0213] The means of presenting information related to the forward travel environment relatively close with travel, the timing required for takeover of the corresponding host vehicle, and the like is not limited to the scroll means, and as other methods, a means in which the upcoming time interval is intuitively and clearly known, and the timing at which recovery to driving should be started can be known with reduced misrecognition is preferably adopted. For example, a time presentation method imitating an hourglass or a means for presenting the takeover remaining time directly to the user in the form of a watch-type equipment worn can be intervened.

[0214] Next, in step S17, the state of the driver is monitored. Next, in step S18, event change corresponding processing is executed. This event change corresponding processing includes mode switching processing for handling a case in which a switching point between automatic driving mode and manual driving mode already exists on a travel route or a cautious driving section is approaching, event occurrence processing for handling a case in which a mode switching point or a cautious driving section newly occurs on a travel route, and the like. Thereafter, the processing of steps S16 to S18 is appropriately repeated.

[0215] "Details of the travel section display"

[0216] Figure 9One example of a travel route determined by a driver setting a destination is shown. In this travel route, there are an automatic driving available section Sa, a manual driving section Sb, a takeover section Sc from automatic driving to manual driving, and a cautious driving section Sd from automatic driving to cautious driving. Here, the takeover section Sc always comes before the manual driving section Sb and requires the driver to be in a position to recover to manual driving. In addition, the cautious driving section Sd is a section in which driving can be performed at a reduced speed while maintaining automatic driving in a state in which the driver in the position to recover to manual driving performs cautious monitoring.

[0217] In the example shown, the automatic driving available section Sa is shown in green, the manual driving section Sb is shown in red, and the takeover section Sc and the cautious driving section Sd are shown in yellow. Note that the respective colors are shown by using different patterns for convenience.

[0218] In the travel section display on a display device such as a center information display or a tablet, the above-described sections of the travel route are displayed on an arrival prediction time axis from the current location. The automatic driving controller 112 performs information processing for the travel section display of the travel route based on the travel route information and the traffic information.

[0219] Figure 10 (a) shows each section on the travel route on a moving distance axis from the current location at a fixed scale. Figure 10 (b) shows the speed v(t) of the average road traffic flow at each location. Figure 10 (c) shows each section expressed on a moving distance axis converted from the speed v(t) by using a time axis. Thereby, each section on the travel route is expressed on an arrival prediction time axis from the current location. In other words, the physical distance on the travel route can be expressed on a time axis obtained by performing division by using the average speed in each corresponding section.

[0220] In the present embodiment, as Figure 10(d) is shown, all sections in which the travel section display is performed are divided into three sections, and the time axis of each section is changed. In other words, a first section from the current location to a first location (time tO, for example, about 10 minutes) is displayed as a time linear display nearest section on a first time axis. For example, the time tO is set as a necessary and sufficient period of time after the ordinary driver terminates the secondary task and before the ordinary driver resumes driving. The nearest section according to the travel approach has a visual intuitive effect equivalent to the indication effect on the map that looks to proceed at a certain speed, so the driver can start preparing for accurate driving resumption after approaching the event. There is an advantage that the point at which the resumption will start can be intuitively grasped in a slightly accurate manner. In other words, the purpose of displaying this section is to provide the user with start determination information of the accurate resumption point of the driver.

[0221] Further, a second section from the first location (time tO) to a second location (time tl, for example, about one hour) is displayed as a time inverse display section on a time axis that is sequentially changed from the first time axis to a second time axis that is reduced at a predetermined ratio with respect to the first time axis. The purpose of displaying this second section is mainly to accurately provide the driver with road conditions during a long period in a small display, because if displayed at a scale ratio equal to that of the first section described above, it is difficult to display a long period in a small display space. This enables the driver to easily grasp which location up to a certain section in front will not request driving intervention according to the travel, and has an advantage that it is possible to perform participation in a secondary task as planned. The intervention and non-intervention of driving are clearly separated from each other. Further, in a secondary task after communication with a third party or the like, it plays a role in presenting important information when planning to release the driver or the like from the secondary task.

[0222] Here, reference is made to Figure 10 (d) describes a method for setting the second display section. When the height of the triangle is assumed to be hO, the time t at the location where h exists before its end is obtained according to the formula (1) described below.

[0223] t = tO * hO / h (1)

[0224] Further, the second time axis at the second location (time tl) has been reduced at a ratio of hs / hO with respect to the first time axis. For example, in the case of hs = hO / 8, the reduction rate is 1 / 8.

[0225] When traveling at a constant speed, the second display interval corresponds to the display of a map with the travel straight line extended and the display interval tilted in the direction of travel, or when viewing the road surface from the front at an angle. In other words, the visual effect of this display interval can be described as intuitively understanding distance based on the height of the displayed image. This makes it easy to grasp the perceived distance even without the need for a precise location scale on the screen. While the size of distant intervals decreases, they are not immediately accessible during driving. Therefore, while a rough estimate is important, the driver does not need to have a firm grasp of arrival times, as is the case with nearby locations. This is preferable when planning secondary tasks.

[0226] Furthermore, the third section, from the second location (time t1) to the third location (time t2), is displayed as a time-linear display of the distant section on the second time axis (the rate of decrease of hs / h0). As described above, by displaying three divided sections, the driver can obtain detailed information about the most recent section in a limited display space, as well as information about the distant sections. Note that the display of the distant section in the second section falls below human visual resolution and further below the system's display resolution limit, making it impossible to distinguish information required for planning and determining secondary tasks, and rendering the display useless. Therefore, the most effective display is to complete the reduction of the display scale at a stage where the sense of the time section can be fully grasped intuitively, appropriately displaying the division between the sections requiring intervention and those not requiring intervention, and then returning the display scale to a fixed scale for subsequent sections.

[0227] Note that the vehicle control system 100 has default values ​​for times t0, t1, and t3. It is conceivable that the values ​​of times t0, t1, and t3 may vary between long-distance driving and short-distance driving. Therefore, the number of default values ​​is not limited to one. Multiple types of default values ​​may be used, and the driver (user) or the system may selectively use multiple types of default values ​​depending on the driving route. Furthermore, it is conceivable that the values ​​of times t0, t1, and t3 may be arbitrarily set by the driver (user).

[0228] Figure 11 (a) and 11(b) show examples of the driving interval display that is finally displayed. Note that the length of the arrow indicates whether the time axis is linear, and the change in the reduction rate of the time axis. Figure 11 In the case of (a), all the intervals, ie, the first interval, the second interval, and the third interval, are displayed to have the first width.

[0229] On the contrary, Figure 11In the case of (b), a first section from the current location to the first location (time t0) is displayed to have a first width, a second section from the first location (time t0) to the second location (time t1) is displayed to have a width sequentially changed from the first width to a second width narrower than the first width, and a third section from the second location (time t1) to the third location (time t2) is displayed to have the second width. This enables the driver to visually recognize the degree of reduction in the time axis of the second and third sections relative to the first section. In other words, Figure 10 The display form of (b) is a display that takes into account only the reduction rate in the direction of travel. By further artificially changing the width intersecting the direction of travel of the display information in accordance with perspective, the same perspective effect as the perspective effect facing the direction of infinity when traveling along a road or a map is exerted, and the distribution of the driving intervention necessary section can be intuitively and easily grasped compared to the case where the screen is gazed at. In particular, in the case where only the second section is rotated and not counterclockwise and is observed, in the case where the vehicle is traveling at a constant speed, the road width matching the front side of the road and the arrival time at each corresponding location look. Therefore, even if the correct position scale cannot be observed and judged, the feeling of actually arriving at each location can be intuitively grasped, and a display form in which time allocation can be performed can be realized.

[0230] Note that, for example, in a portion having a small reduction rate hs / h0 such as the third section, if a section having a short time length is displayed to have a time length that does not change, the section is displayed very thin, and it is expected that the driver will have difficulty in recognizing the section. Therefore, even in the case where the actual driver intervention section (manual driving section, takeover section, and cautious driving section) length is less than or equal to a certain time length, the driver intervention section is displayed in the certain time length. In this case, for example, in the case where the takeover section and the manual driving section are continuous, the display of the takeover section can be omitted. In Figure 11 In (a) and (b), the display of the first manual driving section Sb in the third section indicates such a state. Thereby, in the third section in which the size of the time axis is significantly reduced, the driver intervention necessary section having a short time length can be displayed as a section that the driver can recognize.

[0231] Further, in a portion having a small reduction rate hs / h0 such as the third section, in the case where the manual driving sections Sb are continuously connected at short intervals, all the manual driving sections Sb are continuously displayed. In this case, for example, in the case where the manual driving sections Sb are continuously connected at short intervals, the display of the manual driving sections Sb can be omitted. Figure 11 In (a) and (b), the display of the second manual driving section Sb in the third section indicates such a state of continuous display. Actually, as Figure 11(c) As shown, the manual driving section interval Sb shown in (b) includes, in addition to the manual driving section interval Sb, a takeover section interval Sd having a short period and an automatic driving available section interval Sa. Note that, as described later, detailed display can be performed, for example, by double-touching the corresponding place in a state in which the travel section interval display is performed on a tablet or the like.

[0232] The above travel section interval display on the travel route is updated based on the position information of the host vehicle and the acquired LDM update information. Thereby, the travel section interval display is scroll-displayed in a manner that each section approaches the host vehicle as time elapses. Figure 12 (a) to (d) show examples in which the travel section interval display changes as time elapses. The example indicates an example in which the second section is displayed in a tapered manner, but similar cases apply to a case in which all the sections are displayed to have the same width.

[0233] In this case, in the first section, each section moves quickly. Further, in the second section, the reduction in the size of the time axis becomes smaller at a place closer to the first section from the side of the third section, and thus each section moves faster. Further, in the third section, the reduction in the size of the time axis becomes larger, and thus each section moves slowly.

[0234] Figure 13 (a) and (b) show examples of the travel section interval display 181 on the travel route displayed on the screen of the tablet 182. Figure 13 (a) shows an example of a case in which the tablet 182 is used in a portrait long shape. In this case, the travel section interval display 181 is performed along the left-hand side and the upper side in a curved state, and is performed in parallel with a work window serving as an execution screen of a secondary task executed on the tablet 182. Figure 13 (b) shows an example of a case in which the tablet 182 is used in a landscape long shape. Similarly, in this case, the travel section interval display 181 is performed along the left-hand side and the upper side in a curved state, and is performed in parallel with a work window serving as an execution screen of a secondary task executed on the tablet 182. Note that, in the example shown, the travel section interval display 181 is arranged on the screen of the tablet 182 in a curved state. However, in a case in which a layout space can be sufficiently acquired, it is also conceivable that the travel section interval display 181 is arranged linearly.

[0235] Figure 14One example of a state in which the driver is performing a secondary task by actually using the tablet 182 is shown. In this example, the tablet 182 is used in a landscape long shape. On the screen of the tablet 182, the travel section display 181 is made along the left-hand side and the upper side in a curved state. Note that whether or not to make this travel section display 181 on the screen can be selected in accordance with the operation performed by the driver (user). In this case, for example, in a case where the travel section display 181 is not made on the screen, and in a case where the driver intervention necessary section enters and notifies the driver for a certain period of time, the travel section display 181 can be automatically made on the screen.

[0236] In a state in which the travel section display 181 is made on the screen of the tablet 182, in a case where a driver intervention necessary section is newly generated in the display section, the newly generated driver intervention necessary section is newly displayed. In this case, for example, the newly generated driver intervention necessary section is displayed as a blinking switch during a certain period of time, so that the newly generated driver intervention necessary section can be distinguished from another section. This blinking display can be accompanied by a warning alarm sound. Here, as a case where a newly generated driver intervention necessary section is generated, a case where a newly generated cautious travel section or a manual travel section is generated, and a case where a cautious travel section is changed to a manual travel section are included.

[0237] Figure 15 A state in which a cautious travel section Sd is newly generated in the second section and a warning is given to the driver by using a blinking display is shown. Note that in this case, the blinking, that is, the warning state, can be stopped by the driver touching the display part of the cautious travel section Sd displayed as a blinking switch. Alternatively, by the driver touching the display part of the cautious travel section Sd displayed as a blinking switch, a small window can be popped up, and the blinking, that is, the warning state, can be stopped by touching the screen to agree.

[0238] Further, in a state in which the travel section display 181 is made on the screen of the tablet 182, in a case where the driver (user) double touches and designates an arbitrary point, a small window, for example, as shown in Figure 16 , is popped up, and a display related to the point is made.

[0239] [5. Safety determination processing and manual driving recovery available time estimation processing performed by mobile device]

[0240] Next, safety determination processing and manual driving recovery available time estimation processing performed by the mobile device are described.

[0241] These processing are mainly performed by a safety determination unit 155 configured in the automatic driving controller 112 of the mobile device 100 described with reference to Figure 4 .

[0242] As described above, the safety determination unit 155 learns the optimal recovery timing in accordance with the driver's recovery action pattern, the vehicle characteristics, and the like, and provides the learned information to the situation recognizer 153 or the like. Thereby, for example, the optimal timing required for the driver to normally recover from the automatic driving to the manual driving by a predetermined certain rate or more can be presented to the driver.

[0243] Figure 17 is a diagram that illustrates a specific configuration example of the safety determination unit 155 and processing.

[0244] The left-hand side illustrates a mobile device 200 such as a car, and the right-hand side illustrates a server 300 that performs communication with the mobile device 100 via a network.

[0245] The mobile device 200 includes a driver information acquisition unit 201, an environment information acquisition unit 202, a driver personal identification information acquisition unit 203, a safety determination unit 210, a notification controller 221, a notification unit 222, a communication unit 230, and a storage 240.

[0246] Note that the safety determination unit 210 corresponds to the safety determination unit 155 configured in the situation analyzer 133 illustrated in Figure 4

[0247] The safety determination unit 210 includes a learning processing unit 211, a safety determination processing execution unit (manual driving recovery available time estimation unit) 212, and a personal authentication processing unit 213.

[0248] The server 300 includes a learning data acquisition or generation unit 301, a learning processing unit 302, a communication unit 303, and a storage 304.

[0249] First, the processing performed by each component of the mobile device 200 is described.

[0250] The driver information acquisition unit 201 includes a camera or various sensors, and acquires information related to the driver.

[0251] Examples of the driver information acquired by the driver information acquisition unit 201 include a line-of-sight direction, eye behaviors, and a pupil diameter acquired from an image including an eye region, a facial expression acquired from an image including a face region, and the like.

[0252] These pieces of information acquired by the driver information acquisition unit 201 are used as information for determining the driver's degree of alertness, for example.

[0253] ​Note that the above examples are simplified and described. The driver's detailed state cannot be determined simply by acquiring information related to the eyes, face, etc., and the determination of the driver's alertness involves performing a detailed behavioral analysis of the eyes, face, etc., and comprehensively performing a hierarchical determination of the driver's internal alertness state required for manual driving recovery based on observable values. The following describes examples of some of this biometric information used in the determination.

[0254] Examples of methods for observing a driver's alertness and identifying the activity of an object or specific part by observing active areas within the brain include large-scale measurement equipment such as electroencephalogram (EEG) monitoring by electrically detecting electrical potentials, infrared observation of blood flow on the surface of the head, and functional magnetic resonance imaging (fMRI) assessment of brain function. However, current technology is not suitable for normal and routine use by being installed in a vehicle, and there are limitations on obtaining correlations with other biological data such as heartbeat waveforms or EOG in experimental settings, as well as on their use in offline learning based on this biological information.

[0255] In addition, although it is difficult to directly determine the short-term wakefulness state in the stage where takeover is suddenly required, there is information that can be used to predict the recovery of wakefulness in advance by continuously pre-acquiring medium-term and long-term advance log information. The time required after receiving the recovery notification from the system and before recovery varies according to various factors (such as sleep depth, accumulated fatigue, or the time elapsed from the start of a nap at the time of receiving the notification) and individual characteristics. It is difficult for this advance information to give a direct recovery time, but it is used as an influencing factor for wakefulness recovery. Therefore, the available observable information is used as an influencing factor for estimating the wakefulness recovery time.

[0256] The log of the driver information acquired by the driver information acquisition unit 201 is acquired in accordance with these states of the driver, and the state of wakefulness is also observed based on such as reception of the recovery notification or approach to the trigger of the target takeover event. For example, in the case where the driver wears and uses an observable device such as a wristwatch, the observation of the long-term log can include the life log before entering the vehicle. In this case, information related to the pre-rest state or the excessive sleep or sleep deficiency state can also be used as a help for the determination. In normal use where the pre-log acquisition is not available, the user can enter the vehicle and the degree of loss of attention such as fatigue or drowsiness can be observed from various sensors installed on the vehicle, for example, by evaluating the Lorenz curve of the electrocardiogram, the tension state or the fatigue state can also be estimated from information related to the medium- and long-term change log of the heartbeat, the pulse wave or the blood flow, the blood pressure, and the respiration or the exhalation / body odor, and the state of the driver can also be estimated by analyzing the tone of voice or the response delay of the user in the verbal (questioning) response to the system. Furthermore, means for analyzing the action such as the facial expression, the face down, or the occurrence of the inattentive driving state can also be employed.

[0257] Furthermore, the state can be determined by the active evaluation indicating the response characteristic or the amount of execution activity determined by evaluating the speed of the gesture.

[0258] As described above, the log of the driver information acquired by the driver information acquisition unit 201 is observed in advance, and the wakefulness recovery influence factor is monitored in advance. In order to perform the final takeover determination, it is necessary to determine the internal wakefulness state with high precision. When the wakefulness of the driver and the recovery from the automatic driving to the manual driving are observed from another viewpoint, the state of the driver changes in time series according to the following order.

[0259] 1. Confirmation of the occurrence of the event requiring the takeover,

[0260] 2. Grasping and cognition of the necessary conditions accompanying the necessity of the takeover, and

[0261] 3. In the case of the sound notification, the grasping and cognition of the notification include the response to the notification, and the response is the detection of the action cognition using the switch, the response and the cognition using the gesture, the detection of the action cognition using the hand placed on the steering wheel, or the detection of the driver seat recovery.

[0262] 4. Generally, the driver who receives the recovery notification and finally returns to the recovery posture performs the process of grasping and checking the conditions necessary for the recovery to the manual driving before starting the operation of the steering device such as the steering wheel, the brake, and the accelerator, and starts the steering of the device, so one of the main information acquisition means is the acquisition of the visual information.

[0263] 5. When the driver can visually confirm the situation, the driver starts to operate the appropriate steering device. However, for safety, the system does not immediately take over the entire steering, but performs a process of gradually bringing forward the takeover to a stage at which the appropriate intervention of the driver in the steering and the muscle steering response situation can be confirmed.

[0264] The driver information acquired by the driver information acquisition unit 201 includes the function observed in this series of processes.

[0265] The driver information acquired by the driver information acquisition unit 201 has been described to further grasp the visual situation, and in the above-described process 4, the manual steering is prepared until the visual recognition, determination, and planning are further performed. By capturing only the information necessary to determine in the central field of view of the line of sight, the driver cannot determine and plan the steering action to be taken. Therefore, the object necessary for the risk determination of the line of sight is directed to the object, which is generally captured by the peripheral field of view, and the central field of view = the line of sight is directed in that direction. Thus, the rapid action of the eye swing due to the saccade. When the object is located in the central field of view of the line of sight, the fixation mainly starts in order to facilitate the understanding of the object, and when the determination of the object is completed by referring to the knowledge information in the brain while grasping the characteristics of the target, the understanding is caused. Then, when the understanding of the object is completed, the saccade action of the eye is performed in a new direction in which the grasping of the next object to be grasped is simultaneously and in parallel, or the tracking rotation for canceling the movement in the relative direction is performed in order to perform the determination of the action of the grasped object.

[0266] In other words, when the determination is performed in the cycle of the initial visual acquisition of the wide area saccade search occurring as the visual information search action of the eye, the tracking action of following the flow as the movement proceeds, the fixation micro movement of the line of sight staying until the cognitive of the object of interest to which the line of sight is directed, the range thereof, the fluctuation stability, the stay time of the saccade to the point of interest, and the like, and the confirmation fixation micro movement of the knowledge cognition and the cognitive determination, that is, the slight time delay causing the determination to occur is affected by the amount of activity in the brain, and there is also a phenomenon that this influence can be observed in a short period. Here, in particular, in the visual reflex action, it is assumed that individual drivers refer to their own visual memory from the soft information that has started to be grasped, start the risk determination, and supplement the additional supplemental information necessary to solve the determination while performing the fixation and slight movement in an unconscious state until the risk determination is performed. Therefore, the initial visual information immediately after the eye swing due to the saccade is intermittently repeatedly evaluated and the determination necessary to decide the continuation of the driving is carefully repeatedly performed in an unconscious state to determine the driving action, and the sequence of repeating the saccade to acquire the information to be checked next. Then, the task of acquiring the information necessary for the steering of the driver is an action under the wakefulness and cognition, and therefore this series of behaviors of the eye appears as a detailed behavior of the eye, which strongly reflects the recovery level of the internal wakefulness state that the driver needs to manually recover.

[0267] Further, the driver information acquisition unit 201 also includes an operation information acquisition unit that acquires driver operation information. These include, for example, a camera that captures an image of the face of the driver, an operation information acquisition unit of each operation unit (steering wheel, accelerator, brake, etc.), and the like.

[0268] The environment information acquisition unit 202 acquires each type of environment information.

[0269] Examples of the environment information acquired by the environment information acquisition unit 202 include an image from an imaging unit provided in the mobile device 200, depth information or three-dimensional structure information, terrain information from a sensor such as a LiDAR provided in the mobile body, position information from a GPS, information from a communication device provided in infrastructure such as a road (such as the state of a traffic light or information related to a sign), and the like.

[0270] The driver personal identification information acquisition unit 203 acquires information that can be employed in personal authentication such as iris authentication, fingerprint authentication, vein authentication, or voiceprint authentication, for example.

[0271] Note that, Figure 17 The configuration of the driver information acquisition unit 201, the environment information acquisition unit 202, and the driver personal identification information acquisition unit 203 shown in FIG. 1 corresponds to the respective configurations of the data acquisition unit 102 and the detection unit 131 in the configuration shown in FIG. 2. Figure 4

[0272] The safety determination unit 210 includes a learning processing unit 211, a safety determination processing execution unit (manual driving resumption available time estimation unit) 212, and a personal authentication processing unit 213.

[0273] The personal authentication processing unit 213 receives, as input, personal authentication information in iris authentication, fingerprint authentication, vein authentication, voiceprint authentication, and the like acquired by the driver personal identification information acquisition unit 203, for example, and executes personal authentication processing.

[0274] The registration information required in this authentication processing is acquired from the storage device 240 or the server 300.

[0275] For example, when the driver enters a car that serves as a mobile device, personal authentication is executed as personal identification processing. After personal identification, the driver information, time, driving input, and environment information of the road, and the like are constantly or periodically recorded and transmitted to the server 300.

[0276] ​Note that information input from the driver personal identification information acquisition unit 203 and that can be employed in personal authentication such as iris authentication, fingerprint authentication, vein authentication, or voiceprint authentication can be transmitted to the server 300, and the personal authentication processing can be performed by the server 300 by using information registered in the server 300.

[0277] The learning processing unit 211 receives, as input, driver information acquired by the driver information acquisition unit 201, environment information acquired by the environment information acquisition unit 202, and the like, and performs learning processing for safety determination using these input information.

[0278] Specifically, for example, a learner capable of calculating a "safety index value" or a "manual driving resumption available time estimation value" corresponding to "driver information" or "environment information" is constructed. Note that the learner not only performs learning to perform determination limited to driver-inherent characteristics, but also learns characteristics according to vehicle characteristics, environment characteristics, and the like. For example, when a driver identified as the same driver drives taking into account the steering characteristics of a passenger car used for commuting using a private car and the vehicle behavior characteristics of a vehicle connected with a heavy load in a job, the steering start point, the braking amount, and the like significantly vary depending on the travel situation. Therefore, by learning factors contributing to the safety of the same, a learner including optimization of the situation or the vehicle characteristics can be obtained.

[0279] Note that the learner refers to a collection of models, algorithms, parameters, and the like capable of learning input / output relationships by using machine learning. There are various names such as a statistical learner, a learning model, a machine learning model, a statistical learning model, a predictor, a prediction model, an estimator, or an estimation model, but they all have approximately the same meaning, and thus the name "learner" is adopted here. Depending on the specific purpose, names such as "classifier", "classification model", "recognition device", "recognition model", "regression learner", "regression learning model", or "generative model" can be used, but the name "learner" is a general concept including all these concepts.

[0280] The safety determination processing execution unit 212 receives, as input, driver information acquired by the driver information acquisition unit 201, environment information acquired by the environment information acquisition unit 202, and the like, and also uses a learner that is a result of learning performed by the learning processing unit 211 or a learner acquired from the server 300, and calculates a safety index value based on the driver information and the environment information of the current time.

[0281] Further, the time required until safe manual driving resumption (= manual driving resumption available time) is estimated.

[0282] Note that the time until the safe manual driving resumption is calculated based on the time when the driver starts the manual driving at a high rate before reaching the place where the takeover request is made, not the time calculated by directly evaluating the safety. The timing of requesting the preparation for the resumption to the manual driving is affected by various factors such as the motion characteristics of the corresponding vehicle, the road environment condition, or the weather, and thus, by taking these influences into account, the safety can be maximized.

[0283] The notification controller 221 controls the notification timing of the manual driving resumption request notification based on the manual driving resumption available time estimated by the safety determination processing execution unit 212, as described above with reference to Figure 2 The notification controller 221 controls the notification timing of the manual driving resumption request notification based on the manual driving resumption available time estimated by the safety determination processing execution unit 212, as described above with reference to

[0284] Note that the notification unit 222 includes, for example, an alarm, a display on a window, a vibration processing execution unit of a steering wheel or a seat, and the like.

[0285] The communication unit 230 performs communication with an external device such as the server 300.

[0286] Next, the processing performed by each component of the server 300 is described.

[0287] The server 300 includes a learning data acquisition or generation unit 301, a learning processing unit 302, a communication unit 303, and a storage device 304.

[0288] The learning data acquisition or generation unit 301 acquires or generates the learner input / output data required to construct a learner. Specifically, for example, "driver information" or "environment information" is acquired from various mobile devices connected to a network. Further, for example, data for learning based on each type of data that can be acquired such as map data, accident occurrence condition data, or traffic congestion data is generated. Note that the expression "learner input / output data" is used here, but this is data referred to as teaching data, training data, or the like in supervised learning. If only the desired input / output relationship can be learned, the learning possibility using other methods such as unsupervised learning, semi-supervised learning, or reinforcement learning is not excluded, and thus, the general name "learner input / output data" is used here.

[0289] These data are stored in the storage device 304.

[0290] The learning processing unit 303 performs a learning process using the learner input / output data acquired or generated by the learner input / output data acquisition or generation unit 301 and stored in the storage device 304.

[0291] As a result of this learning processing, for example, learner input / output data that enables calculation of a "safety index value" or a "manual driving resumption available time estimation value" corresponding to the "driver information" or the "environment information" is constituted.

[0292] Next, the details of the processing of the safety determination unit 210 of the mobile device 200 will be described.

[0293] The safety determination unit 210 includes a learning processing unit 211, a safety determination processing execution unit (manual driving resumption available time estimation unit) 212, and a personal authentication processing unit 213.

[0294] The learning processing unit 211 receives, as input, the driver information acquired by the driver information acquisition unit 201 and the environment information acquired by the environment information acquisition unit 202, and performs learning processing for safety determination using these input information.

[0295] The safety determination processing execution unit (manual driving resumption available time estimation unit) 212 receives, as input, the driver information acquired by the driver information acquisition unit 201, the environment information acquired by the environment information acquisition unit 202, and the like, and further uses a learner that is a result of learning of the learning processing unit 211 or a learner acquired from the server 300, and calculates a safety index value based on the driver information and the environment information at the current time.

[0296] Further, the time required until safe manual driving resumption (= manual driving resumption available time) is estimated.

[0297] Note that in an initial state in which input / output data does not exist and a learner after learning cannot be acquired from the learning processing unit 211, the safety determination processing execution unit (manual driving resumption available time estimation unit) 212 acquires, from the storage device 240, for example, a learner that has been learned using a large amount of unspecified data that has been acquired in advance through experiments, and performs safety index value calculation processing or manual driving resumption available time estimation processing using this learner.

[0298] Alternatively, a fixed manual driving resumption available time that is specified is output as an estimation value.

[0299] Note that it is also possible to acquire a learner generated in learning processing using a large amount of unspecified data from the server 300.

[0300] After the learning processing unit 211 has performed learning processing, safety index value calculation processing or manual driving resumption available time estimation processing based on driver information and environment information that have been input is performed using a learner.

[0301] The learning processing unit 211 executes learning processing for constructing a correspondence relationship between, for example, the following information (a) and (b):

[0302] (a) driver information acquired by the driver information acquisition unit 201 and environment information acquired by the environment information acquisition unit 202; and

[0303] (b) operation information indicating whether or not the driver can perform driving in a safe state after the notification of switching from the automatic driving mode to the manual driving mode.

[0304] Note that, as described above, the driver information acquisition unit 12 has a function of acquiring, for example, driver operation information (i.e., information related to operations performed on each operation unit (steering wheel, accelerator, brake, etc.)) in addition to the information acquisition processing for determining the degree of wakefulness of the driver.

[0305] In other words, the learning processing unit 211 acquires driver information and environment information, acquires driver operation information after the manual driving resumption request notification is issued at a certain timing, as described above with reference to Figure 2 and checks whether or not safe driving is being performed.

[0306] If safe driving is being performed, it is determined that the notification timing of the manual driving resumption request notification can be further delayed in a case where the same driver information and the same environment information are acquired.

[0307] On the contrary, if safe driving is not being performed, it is determined that the notification timing of the manual driving resumption request notification is to be advanced in a case where the same driver information and the same environment information are acquired.

[0308] In the learning processing executed by the learning processing unit 211, a learner for determining the notification timing of such a manual driving resumption request notification is constructed.

[0309] Note that, in the learning processing executed by the learning processing unit 211, in addition to data for determining the notification timing of the manual driving resumption request notification, a learner that can be employed in processing for determining whether or not the driver can perform safe manual driving in a case where certain driver information and environment information have been acquired, i.e., a learner in which "driver information and environment information" and "safety index value" are associated with each other, is also constructed.

[0310] By acquiring a transition of the pre-characteristic of the recovery notification occurring at the time of each event occurrence by the driver, and by further using (b) operation information indicating whether or not the driver is able to drive in a safe state as teaching data to perform success / failure determination or takeover quality evaluation, a system including a self-contained learner is realized, and as a feature of the learner, precision can be improved.

[0311] As described above, in the learning processing performed by the learning processing unit 211, safety index value data associated with the driver information acquired by the driver information acquisition unit 201 and the environmental information acquired by the environmental information acquisition unit 202 can be obtained.

[0312] Further, a manual driving recovery available time (time required until safe manual driving recovery) corresponding to the driver information acquired by the driver information acquisition unit 201 and the environmental information acquired by the environmental information acquisition unit 202 can be calculated.

[0313] Based on the manual driving recovery available time, a notification time of a manual driving start request notification issued to the driver can be determined.

[0314] Result data of the learning processing performed by the learning processing unit 211 is stored in the storage device 240.

[0315] The data (learner and / or learner input / output data) stored in the storage device 240 includes, for example, data described below.

[0316] The following data can be accumulated in the storage device 240:

[0317] Safety index value data associated with the driver information and the environmental information;

[0318] Manual driving recovery available time (time required until safe manual driving recovery) associated with the driver information and the environmental information (= optimal pre-notification time of the manual driving start request);

[0319] Abnormality of the driver information, abnormality of the environmental information, time required for recovery after the manual driving start request notification; and

[0320] Learner.

[0321] The data accumulated in the storage device 240 is periodically transmitted to the server 300.

[0322] The server 300 stores the data received from the mobile device 200 in the storage device 304, and further performs learning processing based on the received data in the learning data generation unit 301.

[0323] The server 300 can receive a large amount of data from a large number of mobile devices (cars) and can perform a learning process based on the large amount of data. As a result, input / output data (learner) with higher generalizability and higher accuracy can be obtained.

[0324] The server 300 performs a learning process based on a large amount of data to update the learner, and stores the learner in the storage device 304 of the server 300.

[0325] The learner stored in the storage device 304 of the server 300 includes data similar to the data stored in the storage device 240 of the mobile device 200 described above. In other words, the following data is stored in the storage device 304 of the server 300:

[0326] The learner;

[0327] Safety index value data associated with the driver information and the environment information;

[0328] Manual driving recovery available time (time required until safe manual driving recovery) associated with the driver information and the environment information (= optimal pre-notification time of the manual driving start request); and

[0329] Abnormality of the driver information, abnormality of the environment information, and time required for recovery after the manual driving start request notification.

[0330] These learners stored in the storage device 304 of the server 300 are provided to the mobile device 200 at any time, and are used in the safety judgment processing unit 212 of the safety determination unit 210 of the mobile device 200 to calculate the safety index value based on the driver information acquired by the driver information acquisition unit 201 and the environment information acquired by the environment information acquisition unit 202, the manual driving recovery available time, and the like.

[0331] Note that the learning processing unit 211 can generate a learner that enables prediction of a safety index value or a manual driving recovery available time associated with a person, a car (vehicle type, etc.), or a time by including personal identification information, vehicle type information, time information, and the like in the learning data.

[0332] The personal identification information can be acquired via the personal authentication processing unit 213.

[0333] The vehicle type information can be acquired from the environment information acquisition unit 202. Alternatively, a configuration that uses data stored in advance in the storage device can be employed.

[0334] The safety determination processing execution unit (manual driving resumption available time estimation unit) 212 acquires the manual driving resumption available time using the learner calculated as a result of the learning processing executed by the learning processing unit 211, and issues a manual driving resumption request notification based on the acquired manual driving resumption available time.

[0335] Further, the learning processing unit 211 acquires the driver's driving operation information from the driver information acquisition unit 201 and the environment information acquisition unit 202 after the notification, and determines whether normal driving is being executed.

[0336] According to this determination, the learner is updated and / or relearning is executed to achieve the output, for example, in a manner that the notification time is advanced in a case where an abnormality occurs with a high frequency and the notification time is delayed in a case where an abnormality occurs with a low frequency. This update / relearning processing also enables the notification time control.

[0337] Next, a specific example of the manual driving resumption available time estimation processing executed by the safety determination processing unit 212 of the safety determination unit 210 of the mobile device 200 is described.

[0338] As described above, during automatic driving, the driver can execute various types of processing (secondary tasks), and the degree of wakefulness changes significantly depending on the secondary task being executed.

[0339] For example, simple release of the steering wheel includes a case similar to when the driver gazes at the front of the car during driving, a case where the driver reads a book, or a case where the driver dozes off. The degree of wakefulness (level of consciousness) of the driver changes due to the difference between these processes.

[0340] For example, if the driver dozes off, the degree of wakefulness of the driver deteriorates. In other words, the level of consciousness has deteriorated. In a state where the degree of wakefulness has deteriorated, as described above, normal manual driving cannot be executed. If switching to the manual driving mode is executed in this state, in the worst case, it is possible that an accident occurs.

[0341] Therefore, in order to execute the manual driving resumption available time estimation processing executed by the safety determination processing unit 212, it is preferable that the driver information acquisition unit 201 acquires information that enables checking of the secondary task execution state of the driver during execution of automatic driving, and the learning processing unit 211 executes learning processing using these data.

[0342] It is preferable that the configuration of the driver information acquisition unit 201 for this purpose includes a camera provided to observe the inside of the vehicle, a pressure sensor provided in the steering wheel or the seat, a temperature sensor, and the like.

[0343] The learning processing unit 211 calculates the results of face authentication, head posture estimation, gaze estimation, body posture estimation, and action recognition by using images obtained from the camera, and uses this data in learning processing.

[0344] When performing the secondary task, management is performed with additional information added in a manner known to have acquired input information in the automatic driving state. Upon recovery from the secondary task, it is necessary to present information to the driver before the recovery available time calculated according to an arbitrary method in a manner in which the driver returns to the manual driving state. At a time point at which data for estimating the manual driving recovery available time is not sufficiently collected, it is necessary to issue a recovery notification at a time at which recovery is reliably achieved. Since the individual characteristics have not been learned, information is presented by using a fixed definition time that does not depend on information related to individual characteristics. As for this fixed definition time, for example, a definition time for recovery at a target success rate can be set based on statistical data obtained by statistically evaluating the use result data based on various driver populations, pre-acquiring a time period in which it is estimated that the user can take over at a certain rate, and pre-learning a driver population average value of the pre-collected recovery delay time, and the like.

[0345] Furthermore, at the time of using this periodic time, a user who uses the system for the first time is cautious, and thus an offset setting that takes this characteristic into consideration can be performed.

[0346] At this time, every time a takeover event occurs, the time period from information presentation to recovery to manual driving is measured, and this time period is recorded and determined as the manual driving recovery available time in the form of a log.

[0347] Furthermore, upon determining that recovery to manual driving has been performed, a pre-learned result can be referred to. If similar execution is repeated, data for estimating the recovery available time related to the specified individual and the specified vehicle can be collected.

[0348] In a case in which data for estimating the safety index value or estimating the manual driving recovery available time is sufficiently collected, a learner that predicts the safety index value or the optimal recovery available time can be constructed by performing a machine learning process using, for example, the following input / output data.

[0349] Input: driver information, environment information, and the like

[0350] Output: safety index value or manual driving recovery available time

[0351] Further, in the inherent recovery characteristic of the inherent authorization, in some cases, the recovery time changes depending on the driver's past driving experience or environmental conditions. For example, the time required for recovery after preparation for recovery changes depending on conditions such as a situation in which the visibility is poor due to rain, a night-time situation, a situation in which an obstacle of the visibility becomes backlight, a situation in which accumulated fatigue is accumulated in the morning or evening, an action in which the same driver uses a private car for commuting, a situation in which the vehicle characteristics of a large bus or a cargo loading vehicle in the driving, and the like. Hereinafter, these many condition factors are examples, and these factors are roughly divided into two groups, and are referred to as "driver information" or "environmental information".

[0352] Next, a sequence example of the learning processing performed by the learning processing unit 211 of the mobile device 200 and the learning processing unit 302 of the server 300 is described with reference to the flowchart shown in FIG. 6. Figure 18

[0353] As described above, the learning processing unit 211 of the mobile device 200 and the learning processing unit 302 of the server 300 constitute a learner that enables calculation of the "safety index value" or the "manual driving recovery available time estimation value" corresponding to the "driver information" or the "environmental information".

[0354] First, in step S21, the "driver information" or the "environmental information" is input.

[0355] Next, in step S22, the driver's operation information after the start of the manual driving due to the manual driving start request notification is acquired from the driver information acquisition unit, and the safety index value corresponding to whether or not a safe operation is being performed is calculated.

[0356] As described above, the driver information acquisition unit 12 has a function of acquiring the driver's operation information (i.e., information related to the operation performed on each operation unit (steering wheel, accelerator, brake, and the like)) in addition to the information acquisition processing for determining the driver's degree of alertness.

[0357] ​Note that in the safety index value calculation processing of step S22, the processing for calculating the safety index value can be executed using, for example, specific event response information or accident information in addition to the driver operation information. For example, a configuration can be adopted in which the safety index value is calculated using event response information corresponding to time or road information and the like in addition to "driver information" or "environment information". The event response information continuously monitors the driving steering condition as an accurate steering determination evaluation level of the driver, and a comprehensive steering safety evaluation is continuously executed on the basis of, for example, a change amount, that is, a change in the generation acceleration of the steering correction, such as whether the steering wheel operation for correction is executed at an appropriate timing if the vehicle starts to deviate from the travel lane, whether a delay is observed in stable observation, whether an overcorrection action occurs, or whether a correction cut-in interval is irregularly generated.

[0358] Once the system starts to make the driver control the vehicle system, the driver inputs a steering intention with the steering wheel, the brake, or the accelerator. Therefore, it is obvious from the above-described example of the correction of the steering that the system can start to evaluate the difference between the ideal speed travel control that the system has determined to be desired in the steering and the control performed by the driver, and thus can quantitatively evaluate the level of the driver's alertness recovery from the steering effectiveness of the driver. In the stage in which the system control has been handed over to the driver, the system has completed the steering intervention. Thereby, the system can acquire a log data set containing the history of so-called "driver information" and "environment information" such as the driver state, the vehicle state, and the environment state before the start of the takeover, and the success / failure, the actual delay time required for the normal takeover, the quality evaluation log of the steering at the start of the takeover, and the recovery behavior characteristic change profile required for the full recovery, and the like.

[0359] In a subsequent step S23, input / output data including correspondence data between the input information (driver information, environment information, and the like) and the safety index value is generated, and the learner is updated. The occurrence event is an event in which the takeover is scheduled in the travel plan or an event newly occurring during the travel time, and they do not always need to be distinguished from each other. A set of logs of the observed values acquired in advance before the recovery and the determination of the success / failure of the takeover result and the determination of the success, avoidance, and quality occurring due to the repeated use of the vehicle are acquired as a learning data set.

[0360] The processing of steps S21 to S23 is repeatedly executed, and the learner including the correspondence data between the input information (driver information, environment information, and the like) and the safety index value is sequentially updated by using a large amount of learning data.

[0361] The processing of step S24 is processing in a case in which the learner has been constructed by the learning processing unit 302 of the server 300.

[0362] In step S24, the latest learner generated by the server 300 is transmitted to the mobile device (car).

[0363] Note that, Figure 18 The flowchart illustrated is a flow of processing for generating a learner including correspondence data between "driver information, environmental information, etc." and "safety index value". Further, reference is made to Figure 19 The flowchart illustrated describes a processing sequence in the case of a learner of input / output data associated with "driver information, environmental information, etc." and "manual driving resumption available time".

[0364] First, in step S31, "driver information" or "environmental information" is input.

[0365] Next, in step S32, the time from the manual driving start request notification time to the manual driving start is measured.

[0366] Next, in step S33, the driver's operation information after the manual driving start is acquired from the driver information acquisition unit, and a safety index value corresponding to whether or not a safe operation is being performed is calculated.

[0367] Note that, in the safety index value calculation processing of step S33, processing for calculating a safety index value can be executed using specific accident information in addition to the driver's operation information, for example. For example, a configuration can be adopted in which a safety index value is calculated by using accident information corresponding to time or road information, etc. in addition to "driver information" or "environmental information".

[0368] In subsequent step S34, the elapsed time from the notification time to the manual driving start time (manual driving resumption available time) is calculated such that the safety index value based on the operation information after the manual driving start is greater than or equal to a certain value (safe).

[0369] In subsequent step S35, input / output data including correspondence data between the input information (driver information, environmental information, etc.), safety index value, and manual driving resumption available time is generated, and the learner is updated.

[0370] The processing of steps S31 to S35 is repeatedly executed, and the learner configured using correspondence data between input information (driver information, environmental information, etc.), safety index value, and manual driving resumption available time is sequentially updated by using a large amount of learning data.

[0371] The processing of step S36 is processing in the case where the learner has been constructed by the learning processing unit 302 of the server 300.

[0372] In step S36, the latest learner generated by the server 300 is transmitted to the mobile device (car).

[0373] In Figure 20 An example of input / output data of the learner generated in this learner construction processing is shown in Table 1.

[0374] Figure 20 The data shown is learner input / output data composed of input information (driver information, environmental information, etc.), a safety index value, and a manual driving recovery available time.

[0375] Note that the driver information or the environmental information includes a plurality of observation values (parameters) acquired by a plurality of sensors different from each other.

[0376] In other words, the driver information or the environmental information includes a set of a plurality of parameters.

[0377] Figure 20 The safety index value in the example shown is an example set by using a value of 0 to 1, 0 (dangerous) to 1 (safe).

[0378] This is an example, and as described above, a binary setting of 0 (dangerous) and 1 (safe) or other numerical value ranges can also be adopted.

[0379] The manual driving recovery available time estimate value is a time calculated in the processing of step S34 in the flow described above. Figure 19 In other words, this is a calculated value of the elapsed time from the notification time to the manual driving start time (manual driving recovery available time) required to make the safety index value based on the operation information after the manual driving start greater than or equal to a certain value (safe).

[0380] Figure 17 The safety determination processing execution unit (manual driving recovery available time estimation unit) 212 of the safety determination unit 210 shown uses such a learner, and estimates a safety index value or a manual driving recovery available time corresponding to input information such as driver information, environmental information, and the like.

[0381] For example, the manual driving recovery available time estimation processing employing the learner is executed as the following processing.

[0382] The safety determination processing execution unit (manual driving recovery available time estimation unit) 212 receives, as input, driver information acquired by the driver information acquisition unit 201, environmental information acquired by the environmental information acquisition unit 202, and the like.

[0383] Further, a learner using the result of learning performed by the learning processing unit 211 or a learner acquired from the server 300, that is, by using a learner having Figure 20 The learner of input / output data learning using the data configuration illustrated in FIG. 6 is used, and the time required until safe manual driving resumption (= manual driving resumption available time) is estimated based on the driver information and the environmental information of the current time.

[0384] Note that, Figure 20 The data illustrated in FIG. 6 includes correspondence data between "driver information and environmental information" and "safety index value and manual driving resumption available time", but this is an example. For example, by including personal identification information, vehicle type information, time information, and the like in the data, a learner including a safety index value or a manual driving resumption available time associated with a person, a car (vehicle type, and the like), or a time can be generated.

[0385] Further, similarly, in a case where the same person enters, for example, a vehicle of a different vehicle type, by providing the relevance of the input information of each vehicle, similar manual driving resumption available time estimation can be performed. By newly collecting information of the vehicle and performing additional learning, the accuracy of the prediction can be further improved. The present technology is applicable not only to individual resumption available time prediction but also to manual driving resumption available time prediction applicable to a vehicle.

[0386] Further, in a case where a plurality of persons enter one vehicle, a learner that calculates a resumption available time estimation value associated with members of the driver and the co-riders who enter the vehicle can be constructed.

[0387] By adopting such a configuration, the best manual driving resumption available time corresponding to the members of the driver and the co-riders who enter the vehicle can be predicted.

[0388] As described above, by performing processing by taking into account the driving history, the accident history, and the driving operation information used as information related to an individual driver, information related to a car type, and the like after the start of manual driving by the driver, or by performing learning processing by using information related to the number of passengers or the loaded cargo, a learner corresponding to various situations can be generated.

[0389] Note that, in a case where many types of information are used, a configuration in which many types of information are used in a state in which many types of information have been divided according to data characteristics for advanced learning and for online learning can be adopted.

[0390] [6. Sequence of processing performed by mobile device and information processing device according to the present disclosure]

[0391] Next, a sequence of processing performed by a mobile device and an information processing device according to the present disclosure is described.

[0392] Figure 21 The flowchart shown is a flowchart illustrating one example of a sequence of processing performed by a car serving as a mobile body according to the present disclosure when switching from an automatic driving mode to a manual driving mode.

[0393] In step S41, the mobile body observes an occurrence event of a request for switching from an automatic driving mode to a manual driving mode.

[0394] This observation information is acquired, for example, based on LDM information of a road environment.

[0395] In other words, the acquisition is performed based on so-called local dynamic map (LDM) information that is constantly updated based on high-density travel map information of a road on which the vehicle travels.

[0396] As a specific example, for example, in a case where a schedule of intruding from an expressway to a public road has been acquired based on LDM, determination of an occurrence event of a request for switching from an automatic driving mode to a manual driving mode is performed.

[0397] Next, in step S42, an observation value is obtained. This observation value acquisition processing is performed, for example, by Figure 17 the driver information acquisition unit 201, the environment information acquisition unit 202, and the driver personal identification information acquisition unit 203 shown in FIG. 2. Note that these configurations correspond to respective configurations of the data acquisition unit 102 and the detection unit 131 in the configuration shown in FIG. 1. Figure 4

[0398] As described above, the driver information acquisition unit 201 includes a camera or various sensors and acquires information of the driver, for example, information for determining the degree of alertness of the driver. Examples include a line-of-sight direction, eye behavior, and pupil diameter acquired from an image including an eye region, a facial expression acquired from an image including a face region, and the like. The driver information acquisition unit 201 also acquires information related to operations performed by the driver on each operation unit (steering wheel, accelerator, brake, and the like).

[0399] The environment information acquisition unit 202 acquires, for example, an image from an imaging unit provided in the mobile body, depth information or three-dimensional structure information, topographical information from a sensor such as a LiDAR provided in the mobile body, position information from a GPS, information from a communication device provided in infrastructure such as a road, such as a state of a traffic light or information related to a sign, and the like.

[0400] Next, in step S43, a manual driving recovery available time is calculated based on the acquired observation value.

[0401] This processing is performed by Figure 17 ​The process shown is executed by the safety determination process execution unit (manual driving resumption available time estimation unit) 212 of the safety determination unit 210.

[0402] The safety determination processing execution unit (manual driving resumption available time estimation unit) 212 receives as input the driver information acquired by the driver information acquisition unit 201, the environmental information acquired by the environmental information acquisition unit 202, and the like. Furthermore, the unit estimates the time required until safe manual driving is resumed (=manual driving resumption available time) based on the current driver information and environmental information using a learner obtained as a result of learning performed by the learning processing unit 211 or a learner obtained from the server 300.

[0403] Here, the learner used is, for example, Figure 20 A learner that learns based on the input / output data described by it.

[0404] Note that, as described above, the data shown in 12 is learner input / output data including correspondence data between “driver information and environmental information” and “safety index value and manual drive resumption available time”.

[0405] Figure 20 An example of a table formed by combining specific parameters based on a combination of driver and environmental information, assuming a combination of input information, is shown. However, in reality, the frequency of occurrence of inherent combination events is not very high. Each event occurs as a combination of various situations or conditions, so information is input multidimensionally, and a log of information indicating the driver's progress is not always available. Furthermore, even with the same observed biometric information, the delay in performing determinations based on acquired visual information varies from person to person due to individual's inherent constitution, gestures, experience, memory, and other factors.

[0406] To make more accurate decisions under various conditions, a data set comprising a collection of driver information and safety indicators is used, and the safety determination device is updated through learning. Examples of learning methods include support vector machines, boosting, convolutional neural networks, deep belief networks, and long short-term memory. During updates, newly acquired data sets can be added to the data set used for pre-learning, and re-learning can be performed, or online learning can be used.

[0407] The safety determination processing execution unit (manual driving recovery available time estimation unit) 212 uses the Figure 20 The learner is trained on the data shown in the figure and performs prediction by taking the driver information and the environment information as input. Alternatively, it is also possible to use Figure 20The data shown selects an entry including the parameters closest to the driver information acquired by the driver information acquisition unit 201 and the environmental information acquired by the environmental information acquisition unit 202 from among the plurality of entries, and acquires a simple method of the manual driving recovery available time estimation value set in the selected entry.

[0408] Note that the learner used in the process of estimating the manual driving recovery available time can be set for each driver, or can also be set to include the type of secondary task during the execution of automatic driving in the observation information.

[0409] In this case, the process of using the personal identification information of the driver who is executing driving and the information related to the type of the current secondary task as observation information is executed (manual driving recovery available time estimation process).

[0410] Figure 22 (a) shows one example of the distribution of a plurality of pieces of relationship information (observation chart) between the observable evaluation value corresponding to the observation value and the recovery delay time (= manual driving recovery available time). This example corresponds to a specific secondary task type of a specific driver. In order to calculate the recovery delay time from the plurality of pieces of relationship information (observation chart), the relationship information (observation chart) in a region having a fixed width in the direction of the evaluation value corresponding to the acquired observation value is extracted (shown in a dotted rectangular frame). The dotted line C in the figure indicates the observation of the recovery delay time at which the recovery success rate is 0.95 under different observation values of the driver Figure 22 The boundary line at the recovery delay time at which the recovery success rate is 0.95 described later in (b).

[0411] By giving the automatic-to-manual recovery notification or warning to the driver at a grace time longer than the dotted line C (i.e., an earlier grace time), an area in which the automatic-to-manual recovery of the driver is ensured to be successful at a rate of 0.95 or higher is obtained. Note that the target value of the normal recovery of the driver from automatic driving to manual driving at each corresponding time (request for recovery rate) is specified, for example, according to the necessity of the infrastructure of the roadside, and is provided to the individual intervals by the vehicle.

[0412] Note that in a case where the stop of the vehicle on the traveling road does not lead to an obstacle to the surroundings, it is only necessary to cope by stopping the vehicle and reducing the speed to a speed at which the system can cope. In general, in many cases, it is not desirable to stop on the traveling road. Therefore, as a default setting, a high recovery rate is desirable. In particular, on a designated route such as an urban highway in a large city, in some cases, even if the update information is not intentionally provided from the infrastructure, a very high recovery success rate is obtained by default.

[0413] Figure 22(b) shows the relationship between the recovery delay time and the recovery success rate obtained from the extracted multiple pieces of relationship information (observation graphs). Here, curve a represents the individual success rate at each recovery delay time, and curve b represents the cumulative success rate at each recovery delay time. In this case, the recovery delay time t1 is calculated based on curve b so as to achieve a specified success rate, which is 0.95 in the example shown.

[0414] This calculation process is performed by referring to, for example, Figure 17 The calculation process is performed by the safety determination unit 210. For example, the calculation process is performed by the safety determination unit 210 using distribution information of a plurality of relationship information (observation maps) between observable evaluation values ​​and restoration delay times that are stored in the storage device 240 and acquired in the past.

[0415] Figure 23 This is a diagram illustrating the time available for resuming manual driving according to the type of processing (secondary task) executed in a state where the driver is disengaged from the driving steering task in the automatic driving mode.

[0416] like Figure 22 As shown in (b), the individual distribution profile corresponds to the curve a predicted based on the observed values ​​(i.e., the driver's state). In other words, when the takeover point from automatic driving to manual driving is completed with the required recovery probability, the observed values ​​detected at each stage and capable of evaluating the driver's alertness are referenced to the past characteristics required for the driver's recovery, and based on the recovery curve ( Figure 22 At time t1 when the recovery success rate curve (b) has the desired value, it is monitored whether the actual state required for recovery is achieved in each recovery stage until the takeover is completed.

[0417] For example, the initial curve for a nap is a cumulative average distribution obtained by estimating sleep levels based on observational information such as respiration or pulse waves passively monitored during nap periods in autonomous driving, and observing the corresponding driver's recovery delay characteristics after the issuance of a wake-up warning. Each intermediate distribution is determined based on the driver's state observed during the mobile recovery process after waking up. Observing the "6. Napping Case" diagram, the right-hand timing for issuing a wake-up warning is determined, and the subsequent intermediate processes represent the distribution of recovery times within the recovery budget predicted based on the observed driver state assessment values ​​at the predicted intermediate points.

[0418] The non-violation of the remaining takeover limitation (i.e., the time limit for sequentially reducing until the takeover) is continued in the middle, and in the case where there is a risk of violation, the speed is reduced, and time delay generation and the like are performed. Note that, for example, in the case where the recovery is resumed from the "4. Irregular rotation in non-driving posture sitting" without the steps "6. Napping situation" and "5. Sitting", the recovery process is resumed from the first situation awareness. Therefore, the posture of the same item "4. Irregular rotation in non-driving posture sitting" as the state in the middle process from the "6. Napping situation" is the same, but the thinking process is in the conscious recovery process. In the case where the situation awareness from the posture of "4. Irregular rotation in non-driving posture sitting" is started from the beginning, the time for situation awareness is required, and a longer time is taken.

[0419] Note that, in some cases, the relationship information between the observable evaluation value of the driver in the driving and the recovery delay time is not sufficiently stored in the storage device. In this case, the recovery delay time tl can be calculated by using the assumed distribution information of the recovery as the recovery characteristic information generated based on, for example, information collected from a population of drivers of the same age, which is provided in the storage device in advance. Since the characteristics inherent to the driver have not been sufficiently learned, the recovery information can be used with the same recovery probability based on this information, or a higher recovery success rate can be set. Note that, from the viewpoint of ergonomics, an inexperienced user is more cautious. Therefore, early recovery is expected in the early stage of use, and as the driver becomes accustomed to the use, the driver himself gradually coordinates the action in synchronization with the notification from the system. Note that, in the distribution business of driving a large number of vehicles, the operation business of a bus, a taxi, and the like, and in the case where different vehicles are used as a shared car or a rental car, the driver can be individually authenticated, the observable information and the recovery characteristic of the driving can be centrally or distributively managed or learned by a remote server or the like, and a remote learning process or holding can also be performed without always holding the data related to the recovery characteristic in the individual vehicle.

[0420] Further, the notification timing is important, and therefore, the description is provided under the assumption that the recovery success rate is a uniform time before success / failure. However, the determination extended to the recovery takeover quality can be further performed without limiting the success / failure from the manual driving of the automatic driving to a binary success / failure. In other words, the delay time of the recovery process transition to the actual recovery confirmation, the recovery start delay in response to the notification, the holding of the intermediate recovery action, and the like are recoveries within a tolerable time, and can be further input to the learner as a recovery quality evaluation value.

[0421] Now, returning to Figure 21In step S44, at a notification timing specified by the recovery delay time calculated in step S43, that is, at a timing at which the takeover target event (the takeover interval from automatic driving to manual driving or the cautious driving interval from automatic driving to cautious driving) becomes closer to the recovery delay time, a notification is issued to prompt the driver to resume driving. The notification is issued, for example, as display processing, as described above with reference to Figure 2 Alternatively, the notification can be issued as a sound output or vibration of the steering wheel or seat. For example, in the case where the driver is dozing off, a notification method for waking up the driver from the sleep state is selected.

[0422] Next, in step S45, the driver's recovery transition is monitored. Then, in step S46, it is determined whether or not the driving resumption can be made within the recovery delay time based on the monitoring result in step S45. When it is determined that the driving resumption can be made, the driver makes the driving resumption in step S47. Thereafter, in step S48, the learning data is updated. In other words, one sample value of the relationship information (observation map) between the observable evaluation value with respect to the type of the initial secondary task of the driver at the time of the above-described driving resumption and the actual recovery delay time is added. Thereafter, in step S50, the processing is terminated. Note that, in the present example, a description has been provided that only the plot data generated in each event is learned. However, in reality, the learning depends largely on the previous state (history) before the event occurs. Therefore, by performing multi-dimensional learning, it is possible to further improve the estimation accuracy of the time required for the recovery delay according to the state observation value of the driver.

[0423] Further, when it is determined in step S46 that the driving resumption will not be made, the start of the sequence of deceleration, slow travel, and backing up to a stop of the vehicle is executed in step S51. Next, in step S52, a record of a penalty for the takeover deficiency event is issued, and the processing is terminated. Note that the record of the penalty is stored in the storage device.

[0424] [7. Configuration example of information processing device]

[0425] The above-described processing can be executed with reference to the configuration of the mobile device described above. Figure 4 However, it is also possible to perform a part of the processing, for example, by an information processing device mounted on the mobile device.

[0426] Reference is made to Figure 24 for a description of a hardware configuration example of such an information processing device.

[0427] Figure 24 is a diagram showing a hardware configuration example of an information processing device.

[0428] A central processing unit (CPU) 501 functions as a data processing unit that performs each type of processing in accordance with a program stored in a read only memory (ROM) 502 or a storage device 508. For example, processing in accordance with the sequence described in the above-described example is performed.

[0429] A random access memory (RAM) 503 stores a program, data, and the like executed by the CPU 501. The above-described CPU 501, ROM 502, and RAM 503 are connected to each other through a bus 504.

[0430] The CPU 501 is connected to an input / output interface 505 via the bus 504, and the input / output interface 505 is connected to an input unit 506 including various switches, a keyboard, a touch panel, a mouse, a microphone, a situation data acquisition unit such as a sensor, a camera, or a GPS, and the like, and an output unit 507 including a display, a speaker, and the like.

[0431] Note that input information from the sensor 521 is also input to the input unit 506.

[0432] Further, the output unit 507 outputs drive information to a drive unit 522 of the mobile device.

[0433] The CPU 501 receives a command, situation data, and the like input from the input unit 506 as input, performs each type of processing, and outputs a processing result to, for example, the output unit 507.

[0434] The storage device 508 connected to the input / output interface 505 includes, for example, a hard disk or the like, and stores a program executed by the CPU 501 or each type of data. A communication unit 509 functions as a transmission / reception unit of data communication via a network such as the Internet or a local area network, and performs communication with an external device.

[0435] The drive 510 connected to the input / output interface 505 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory (for example, a memory card), and records or reads data.

[0436] [8. Summary of Configuration According to the Present Disclosure]

[0437] The example of the present disclosure has been described in detail above with reference to specific examples. However, it is obvious that a person skilled in the art can modify or replace the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples, and should not be interpreted as limiting. In order to determine the gist of the present disclosure, the claims should be considered.

[0438] Note that the technology disclosed here can employ the configuration described below.

[0439] (1) An information processing apparatus comprising:

[0440] a safety determination unit that receives, as input, driver information and environment information, and calculates a safety index value indicating whether a driver in a vehicle during automated driving is in a state in which safe manual driving can be performed.

[0441] (2) The information processing apparatus according to (1),

[0442] wherein the safety determination unit performs:

[0443] receiving, as input, driver information and environment information, and estimating a manual driving resumption available time including a time until a driver in a vehicle during automated driving can start safe manual driving.

[0444] (3) The information processing apparatus according to (1) or (2),

[0445] wherein the safety determination unit:

[0446] performs a learning process, and generates a learner in which driver information and environment information are associated with a safety index value; and

[0447] adopts the generated learner, and calculates a safety index value.

[0448] (4) The information processing apparatus according to (2),

[0449] wherein the safety determination unit:

[0450] performs a learning process, and generates a learner in which driver information and environment information are associated with a manual driving resumption available time; and

[0451] adopts the generated learner, and estimates a manual driving resumption available time.

[0452] (5) The information processing apparatus according to any one of (1) to (4), further comprising:

[0453] a communication unit that performs communication with a server,

[0454] wherein the information processing apparatus acquires, from the server, a learner in which data (a) and (b) below are associated with each other:

[0455] (a) driver information and environment information; and

[0456] (b) at least a safety index value or a manual driving resumption available time including a time until a driver in a vehicle during automated driving can start safe manual driving.

[0457] (6) The information processing device according to any one of (1) to (5),

[0458] The driver information includes information that can be used to calculate the driver's alertness.

[0459] (7) The information processing device according to any one of (1) to (6),

[0460] The driver information includes at least one of gaze direction, eye behavior, pupil diameter and facial expression, wherein the gaze direction, eye behavior and pupil diameter are obtained from an image including an eye area, and the facial expression is obtained from an image including a facial area.

[0461] (8) The information processing device according to any one of (1) to (7),

[0462] Among them, the driver information includes information including the driver's operation information after the mode is switched from the automatic driving mode to the manual driving mode.

[0463] (9) The information processing device according to any one of (1) to (8),

[0464] The environmental information includes at least one of captured images, depth information, three-dimensional structure information, LiDAR acquisition information, terrain information and location information.

[0465] (10) The information processing device according to any one of (1) to (9),

[0466] Wherein, the security determination unit:

[0467] performing a learning process and generating a learner in which driver information and environmental information are associated with a manual driving resumption available time; and

[0468] Take the generated learner and calculate the time available to resume manual driving, and

[0469] The information processing device further includes:

[0470] A notification controller executes notification processing for issuing a manual driving resumption request notification to the driver via a notification unit at a timing determined based on the manual driving resumption available time calculated by the safety determination unit.

[0471] (11) The information processing device according to (10),

[0472] The notification unit includes at least one of a display unit, a sound output unit, and a vibrator.

[0473] (12) A mobile device comprising:

[0474] a driver information acquisition unit that acquires driver information of the mobile device;

[0475] an environment information acquisition unit that acquires environment information of the mobile device; and

[0476] a safety determination unit that receives the driver information and the environment information as inputs, and calculates a safety index value indicating whether a driver in the mobile device during automated driving is in a state in which safe manual driving can be performed.

[0477] (13) The mobile device according to (12),

[0478] wherein the safety determination unit performs:

[0479] receiving the driver information and the environment information as inputs, and estimating a manual driving resumption available time including a time until the driver in the mobile device during automated driving can start safe manual driving.

[0480] (14) The mobile device according to (12) or (13),

[0481] wherein the safety determination unit:

[0482] performs a learning process, and generates a learner in which the driver information and the environment information are associated with the safety index value; and

[0483] adopts the generated learner, and calculates the safety index value.

[0484] (15) The mobile device according to (13),

[0485] wherein the safety determination unit:

[0486] performs a learning process, and generates a learner in which the driver information and the environment information are associated with the manual driving resumption available time; and

[0487] adopts the generated learner, and estimates the manual driving resumption available time.

[0488] (16) The mobile device according to (15), further comprising:

[0489] a notification controller that performs a notification process for issuing a manual driving resumption request notification to the driver via a notification unit at a timing determined based on the manual driving resumption available time calculated by the safety determination unit.

[0490] (17) The mobile device according to (16),

[0491] wherein the notification unit includes at least one of a display unit, a sound output unit, and a vibrator.

[0492] (18) An information processing method executed by an information processing device, the information processing method comprising:

[0493] The safety determination unit receives the driver information and the environment information as inputs, and executes processing for calculating a safety index value indicating whether a driver in a vehicle during automated driving is in a state capable of performing safe manual driving.

[0494] (19) An information processing method executed by a mobile device, the information processing method comprising:

[0495] The driver information acquisition unit acquires driver information of a driver of the mobile device in a driver information acquisition step;

[0496] The environment information acquisition unit acquires environment information of the mobile device in an environment information acquisition step; and

[0497] The safety determination unit receives the driver information and the environment information as inputs, and calculates a safety index value indicating whether a driver in the mobile device during automated driving is in a state capable of performing safe manual driving.

[0498] (20) A program causing an information processing device to execute an information processing, the information processing comprising:

[0499] The safety determination unit receives the driver information and the environment information as inputs, and calculates a safety index value indicating whether a driver in a vehicle during automated driving is in a state capable of performing safe manual driving.

[0500] Further, a series of processes described in the specification can be executed by hardware or software, or a combination of hardware and software. In the case where a series of processes is executed by software, the series of processes can be executed by installing a program recording the processing sequence in a memory in a computer incorporated with a dedicated hardware, or can be executed by installing the program in a general-purpose computer capable of executing various types of processes. For example, the program can be recorded in a recording medium in advance. The program can be installed in a computer from the recording medium, or the program can be received via a network such as a local area network (LAN) or the Internet, and can be installed in a recording medium such as a built-in hard disk.

[0501] Note that each type of processing described in this specification can be executed in order according to the specification, or can be executed in parallel or individually according to the processing capacity of the device that executes the processing or as needed. Further, the system described here is a configuration of a logical set including a plurality of devices, and does not always include devices with respective configurations in the same housing.

[0502] Industrial applicability

[0503] As described above, by adopting the configuration in one example of the present disclosure, a configuration is realized in which driver information and environmental information are input, and a safety index value or a manual driving recovery available time indicating whether a driver who is performing automatic driving is in a state in which safe manual driving can be performed is estimated.

[0504] Specifically, for example, the configuration includes a driver information acquisition unit that acquires driver information of a mobile device such as a car, an environmental information acquisition unit that acquires environmental information of the mobile device, and a safety determination unit that receives driver information and environmental information as input, and learns and calculates a safety index value indicating whether a driver in the mobile device during automatic driving is in a state in which safe manual driving can be performed. The safety determination unit also estimates a manual driving recovery available time including a time required until the driver in the mobile device during automatic driving can start safe manual driving.

[0505] By adopting this configuration, a configuration is realized in which driver information and environmental information are input, and a safety index value or a manual driving recovery available time indicating whether a driver who is performing automatic driving is in a state in which safe manual driving can be performed is estimated.

[0506] List of reference numerals

[0507] 10 car

[0508] 11 data processing unit

[0509] 12 driver information acquisition unit

[0510] 13 environmental information acquisition unit

[0511] 14 communication unit

[0512] 15 notification unit

[0513] 20 driver

[0514] 30 server

[0515] 100 mobile device

[0516] 101 input unit

[0517] 102 data acquisition unit

[0518] 103 communication unit

[0519] 104 in-vehicle device

[0520] 105 output controller

[0521] 106 output unit

[0522] 107 drive system controller

[0523] 108 drive system

[0524] 109 body system controller

[0525] 110 body system

[0526] 111 storage device

[0527] 112 autonomous driving controller

[0528] 121 communication network

[0529] 131 detection unit

[0530] 132 self-position estimation unit

[0531] 133 situation analyzer

[0532] 134 planning unit

[0533] 135 action controller

[0534] 141 outside-vehicle information detection unit

[0535] 142 in-vehicle information detection unit

[0536] 143 vehicle state detection unit

[0537] 151 map analyzer

[0538] 152 traffic rule recognizer

[0539] 153 situation recognizer

[0540] 154 situation prediction unit

[0541] 155 safety determination unit

[0542] 161 route planning unit

[0543] 162 action planning unit

[0544] 163 action planning unit

[0545] 171 emergency avoidance unit

[0546] 172 acceleration / deceleration controller

[0547] 173 direction controller

[0548] 201 mobile device

[0549] 181 travel section display

[0550] 182 tablet

[0551] 201 driver information acquisition unit

[0552] 202 environment information acquisition unit

[0553] 203 driver personal identification information acquisition unit

[0554] 210 safety determination unit

[0555] 211 learning processing unit

[0556] 211 safety determination processing execution unit (manual driving resumption available time estimation unit)

[0557] 213 personal authentication processing unit

[0558] 221 notification controller

[0559] 222 notification unit

[0560] 230 communication unit

[0561] 240 storage device

[0562] 300 server

[0563] 301 learning data acquisition or generation unit

[0564] 302 learning processing unit

[0565] 303 communication unit

[0566] 304 storage device

[0567] 501 CPU

[0568] 502 ROM

[0569] 503 RAM

[0570] 504 bus

[0571] 505 input / output interface

[0572] 506 input unit

[0573] 507 output unit

[0574] 508 storage device

[0575] 509 Communication Unit

[0576] 510 Driver

[0577] 511 Removable Media

[0578] 521 sensor

[0579] 522 drive unit

Claims

1. An information processing device, comprising: A safety determination unit receives driver information and environmental information as input and calculates a manual driving resumption available time.

2. The information processing apparatus according to claim 1, wherein The safety determination unit also calculates a safety index value indicating whether a driver in the vehicle is in a state capable of performing safe manual driving during automatic driving.

3. The information processing apparatus according to claim 2, wherein The safety determination unit performs: receiving driver information and environmental information as input, The manual driving resumption available time includes the time required until the driver in the vehicle becomes able to start safe manual driving during automatic driving.

4. The information processing device according to claim 2, wherein: The safety determination unit performs the steps of: performing a learning process and generating a learner in which driver information and environmental information are associated with a safety index value; and employing the generated learner and calculating a safety index value. The information processing device according to claim 1 , wherein: The safety determination unit performs the steps of: performing a learning process and generating a learner in which driver information and environmental information are associated with a manual driving resumption available time; and employing the generated learner and estimating the manual driving resumption available time.

6. The information processing apparatus according to claim 3, further comprising: A communication unit that performs communication with a server, wherein the information processing device obtains from the server a learner in which the following data (a) and (b) are associated with each other: (a) driver information and environmental information; and (b) at least a safety index value or a manual driving resumption available time, wherein the manual driving resumption available time includes the time required until the driver in the vehicle becomes able to start safe manual driving during automatic driving.

7. The information processing apparatus according to claim 1, wherein: The driver information includes information that enables calculation of the driver's alertness.

8. The information processing apparatus according to claim 1, wherein: The driver information includes at least one of gaze direction, eyeball behavior, pupil diameter and facial expression, wherein the gaze direction, eyeball behavior and pupil diameter are obtained from an image including an eyeball area, and the facial expression is obtained from an image including a face area.

9. The information processing apparatus according to claim 1, wherein: The driver information includes information including operation information of the driver after the mode is switched from the automatic driving mode to the manual driving mode.

10. The information processing apparatus according to claim 1, wherein: The safety determination unit performs the following operations: executing a learning process and generating a learner in which driver information and environmental information are associated with a manual driving resumption available time; as well as The generated learner is used to calculate the manual driving recovery available time, and the information processing device further includes: A notification controller executes notification processing for issuing a manual driving resumption request notification to the driver via a notification unit at a timing determined based on the manual driving resumption available time calculated by the safety determination unit. The information processing apparatus according to claim 10 , wherein: The notification unit includes at least one of a display unit, a sound output unit, and a vibrator.

12. The information processing apparatus according to claim 1, further comprising: A notification controller executes notification processing for issuing a manual driving resumption request notification to the driver via a notification unit at a timing determined based on the manual driving resumption available time calculated by the safety determination unit.

13. An information processing system comprising: a driver information acquiring unit configured to acquire driver information of a mobile device; an environmental information acquiring unit, configured to acquire environmental information of the mobile device; as well as A safety determination unit receives driver information and environmental information as input and calculates a manual driving resumption available time.

14. The information processing system according to claim 13, wherein: The safety determination unit performs the following operations: receiving driver information and environmental information as input, and calculating a safety index value indicating whether the driver in the mobile device is in a state capable of performing safe manual driving during automatic driving, wherein the manual driving recovery available time includes a time required until the driver in the mobile device becomes capable of starting safe manual driving during automatic driving.

15. The information processing system according to claim 14, wherein: The safety determination unit performs the following operations: performs a learning process and generates a learner in which driver information and environmental information are associated with a safety index value; and employs the generated learner and calculates a safety index value.

16. The information processing system according to claim 14, wherein: The safety determination unit performs the following operations: performs a learning process and generates a learner in which driver information and environmental information are associated with a manual driving resumption available time; and employs the generated learner and estimates the manual driving resumption available time.

17. The information processing system according to claim 16, wherein: The notification unit includes at least one of a display unit, a sound output unit, and a vibrator.

18. An information processing method performed by an information processing device, the information processing method comprising: The safety determination unit receives driver information and environmental information as input and performs processing for calculating a manual driving resumption available time.

19. An information processing method performed by an information processing system, the information processing method comprising: A driver information acquiring step in which a driver information acquiring unit acquires driver information of a mobile device; An environmental information acquisition step in which an environmental information acquisition unit acquires environmental information of the mobile device; as well as The safety determination unit receives the driver information and the environment information as input and calculates a safety determination step of a manual driving resumption available time.

20. A program for causing an information processing device to execute information processing, the information processing comprising: The safety determination unit receives driver information and environmental information as input and calculates a manual driving resumption available time.

Images