Information processing device, mobile device, method and program

Abstract

Provided is a configuration for generating driving interval display data that enables at least an automatic driving allowed interval and a manual driving interval on a driving route of a vehicle to be identified, and presenting the driving interval display data to a user (driver). The configuration includes a data processing unit that generates driving interval display data that enables at least an automatic driving allowed interval and a manual driving interval on a driving route of a vehicle to be identified. For a short-distance interval and a long-distance interval, the data processing unit generates time-linear display data that represents time (t) and display position (h) in a proportional relationship. The data processing unit generates display data in which the distance per unit display length of the long-distance interval is made greater than the distance per unit display length of the short-distance interval. With respect to a medium-distance interval between the short-distance interval and the long-distance interval, the data processing unit generates display data having a time axis that gradually changes at a predetermined ratio from a first time axis corresponding to the short-distance interval to a second time axis corresponding to the long-distance interval.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, a mobile device, a method, and a program. More specifically, the present disclosure relates to an information processing device, a mobile device, a method, and a program for performing processing to generate various display data for a vehicle that performs switching control between automatic driving and manual driving. Background Art

[0002] In recent years, active technology development related to autonomous driving is underway.

[0003] Autonomous driving technology is a technology that enables autonomous driving on roads by using various sensors, such as position detection devices installed on vehicles (cars), and is predicted to be widely used in the near future.

[0004] However, in the current state, since autonomous driving is in the development stage, it is considered that it will take some time before fully autonomous driving becomes possible, and during this period, the car is expected to be driven by switching between autonomous driving and manual driving by the driver as appropriate.

[0005] For example, it is expected that mode switching is necessary when driving in automatic driving mode on a straight road with sufficient road width such as an expressway, when the car has exited the expressway and wants to be parked at a desired location in a parking lot, when the car is traveling on a mountain road with a narrow road width, or in other similar situations, it is necessary that the driving mode is switched to manual driving mode and the car is driven by the driver.

[0006] While the vehicle is performing autonomous driving, the driver does not have to direct his / her gaze forward, which is the direction of travel of the vehicle, but is allowed to engage in free activities such as taking a nap, watching TV, reading, or sitting back and talking with people in the back seat.

[0007] In a vehicle driven in a driving mode that switches between automatic driving and manual driving, when it is necessary to switch from the automatic driving mode to the manual driving mode, it is necessary to cause the driver to start manual driving.

[0008] In order for the driver to start manual driving, notification must be given at an appropriate time. If the driver switches to manual driving mode inappropriately without complete notification, normal manual driving cannot be started, and in the worst case, there is a possibility of causing an accident.

[0009] In order to ensure driving safety, it is necessary to reliably request the driver to resume manual driving at an appropriate timing.

[0010] In addition, it is also important to make the driver realize that he is approaching the manual driving zone. This is because a predetermined preparation process is required to start manual driving, and the preparation needs to be performed sufficiently before starting manual driving. If there is not enough time before resuming manual driving, there is a possibility of an accident, which is very dangerous.

[0011] In order to avoid such danger, for example, the vehicle may be automatically emergency stopped or decelerated, or moved to a side parking position, but such processing may lead to problems such as traffic jams.

[0012] For example, PTL 1 (Japanese Patent Publication No. 2016-139204) discloses a technology for displaying the degree of danger of multiple dangers on a predetermined travel road of a user's vehicle. In addition, for example, PTL 2 (Japanese Patent Publication No. 2016-090274) discloses a technology for notifying a driver whose attention is focused on a mobile terminal that he / she is in a situation where manual driving should be started by displaying information on the screen of the mobile terminal when the driver is about to start manual driving during automatic driving.

[0013] [Citation list]

[0014] [Patent Document]

[0015] [PTL 1] Japanese Patent Publication No. 2016-139204

[0016] [PTL 2] Japanese Patent Publication No. 2016-090274 Summary of the invention

[0017] [Technical issues]

[0018] There are known technologies in which notification of specific information is performed as disclosed in PTL 1 and PTL 2. However, the state of the driving route of the autonomous driving vehicle changes at any time. Specifically, there is a possibility of changes in the settings of the autonomous driving allowed interval and the manual driving interval. For example, the interval setting is expected to change due to the occurrence of an accident or construction. Therefore, it is important to reliably notify the user (driver) of the latest information at an early stage, but there is no prior art that discloses a structure that enables such information provision processing.

[0019] The present disclosure has been created in view of problems such as the above, and its purpose is to provide an information processing device, a mobile device, a method and a program that enable various information to be presented to a user (driver) of a vehicle that switches between automatic driving and manual driving, wherein the various information includes interval information of automatic driving permitted intervals and manual driving intervals on a driving route.

[0020] [Solution to the problem]

[0021] According to a first aspect of the present disclosure, an information processing device includes a data processing unit, which generates driving interval display data, and the driving interval display data enables at least an automatic driving permitted interval and a manual driving interval on a vehicle's driving route to be identified, wherein the data processing unit generates a piece of driving interval display data, and the driving interval display data is generated by connecting a plurality of display data having time axes that differ according to a predetermined time period for driving of the vehicle.

[0022] In addition, according to a second aspect of the present disclosure, a mobile device capable of switching between automatic driving and manual driving, the mobile device including a driver information acquisition unit, which acquires driver information of a driver of the mobile device; an environment information acquisition unit, which acquires surrounding environment information of the mobile device; and a data processing unit, which generates driving interval display data, the driving interval display data enabling at least identification of automatic driving permitted intervals and manual driving intervals on a driving route of the vehicle, wherein the data processing unit generates a piece of driving interval display data, the driving interval display data being driving interval display data enabling at least identification of automatic driving permitted intervals and manual driving intervals on a driving route of the vehicle, and being generated by connecting a plurality of display data having time axes that differ according to a predetermined time period of driving of the vehicle.

[0023] In addition, according to a third aspect of the present disclosure, an information processing method is performed in an information processing device, wherein a data processing unit generates a driving interval display data, the driving interval display data being driving interval display data that enables at least identification of an automatic driving permitted interval and a manual driving interval on a driving route of a vehicle, and is generated by connecting a plurality of display data having a time axis that differs according to a predetermined time period of driving of the vehicle.

[0024] In addition, according to a fourth aspect of the present disclosure, there is an information processing method performed in a mobile device, wherein the mobile device is capable of switching between automatic driving and manual driving, the information processing method includes a driver information acquisition step performed by a driver information acquisition unit to acquire driver information of a driver of the mobile device, and an environmental information acquisition step performed by an environmental information acquisition unit to acquire surrounding environment information of the mobile device, and a data processing unit generates a driving interval display data, the driving interval display data being driving interval display data that enables at least identification of an automatic driving permitted interval and a manual driving interval on a driving route of the vehicle, and is generated by connecting a plurality of display data having a time axis that differs according to a predetermined time period of driving of the vehicle.

[0025] In addition, according to a fifth aspect of the present disclosure, there is a program for executing information processing in an information processing device, wherein a data processing unit is caused to generate a driving interval display data, wherein the driving interval display data is driving interval display data that enables at least identification of an automatic driving permitted interval and a manual driving interval on a driving route of a vehicle, and is generated by connecting a plurality of display data having a time axis that differs according to a predetermined time period of driving of the vehicle.

[0026] Note that, for example, the program according to the present disclosure is a program that can be provided by a storage medium or a communication medium, and the storage medium or the communication medium provides the program in a computer-readable format to an information processing device or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing according to the program is implemented on the information processing device or the computer system.

[0027] Other purposes, features and advantages of the present disclosure will become apparent from a more detailed description based on the embodiments of the present disclosure and the accompanying drawings described below. Note that the system in this specification has a logical set of multiple devices and is not limited to a system including various devices contained in a single housing.

[0028] According to a configuration of one embodiment of the present disclosure, a configuration is implemented to generate travel interval display data, and present the travel interval display data to a user (driver), wherein the travel interval display data enables at least an automatic driving permitted section and a manual driving section on a vehicle's travel route to be identified.

[0029] Specifically, for example, a data processing unit is included that generates driving interval display data, wherein the driving interval display data enables at least the automatic driving allowed interval and the manual driving interval on the driving route of the vehicle to be identified. For the short-distance interval and the long-distance interval, the data processing unit generates time-linear display data that represents time (t) and display position (h) in a proportional relationship. The data processing unit generates display data in which the distance per unit display length of the long-distance interval is made greater than the distance per unit display length of the short-distance interval. As for the medium-distance interval between the short-distance interval and the long-distance interval, the data processing unit generates display data having a time axis that gradually changes at a predetermined ratio from a first time axis corresponding to the short-distance interval to a second time axis corresponding to the long-distance interval.

[0030] This configuration realizes a configuration in which travel section display data is generated and presented to a user (driver), the travel section display data enabling identification of at least an automatic driving permitted section and a manual driving section on a travel route of the vehicle.

[0031] Note that the advantages described in this specification are presented for illustrative purposes only and are not intended to limit the advantages of the present disclosure. Additional advantages may exist. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 1 is a diagram for explaining a configuration example of a mobile device according to the present disclosure.

[0033] Figure 2 is a diagram for explaining an example of data displayed on the display section of the mobile device according to the present disclosure.

[0034] Figure 3 Diagrams for explaining processes performed by a mobile device according to the present disclosure are described.

[0035] Figure 4 Diagrams for explaining processes performed by a mobile device according to the present disclosure are described.

[0036] Figure 5 Diagrams for explaining examples of actions of a driver in a mobile device according to the present disclosure are described.

[0037] Figure 6 1 is a diagram for explaining an example of data display on a terminal held by a driver.

[0038] Figure 7 1 is a diagram for explaining an example of data display on a terminal held by a driver.

[0039] Figure 8 2 is a diagram for explaining a configuration example of a mobile device according to the present disclosure.

[0040] Fig. 9 2 is a diagram for explaining a configuration example of a mobile device according to the present disclosure.

[0041] Fig.10 1 is a diagram for explaining an example of a sensor configuration of a mobile device according to the present disclosure.

[0042] Fig.11 is a diagram describing an example of a mode switching sequence from an automatic driving mode to a manual driving mode performed by a mobile device according to the present disclosure.

[0043] Fig.12 is a diagram describing a flowchart for explaining an example of an operation sequence of automatic driving.

[0044] Fig.13 is a diagram describing an example of a travel route determined when a driver sets a destination, with automatic driving permitted sections and prohibited sections that are sporadically set or appear.

[0045] Fig.14 Diagrams for explaining information processing for display of driving sections on a travel route are described.

[0046] Fig.15 Description: A diagram illustrating an example of a driving interval display to be finally displayed.

[0047] Fig.16 Description: A diagram illustrating an example (scrolling example) of changes in a driving interval display seen over time.

[0048] Fig.17 Description Diagrams illustrating an example of display of driving sections on a travel route displayed on a screen of a tablet terminal device (hereinafter simply referred to as a “tablet”) are described.

[0049] Fig.18 The following is a diagram describing an example of a state in which a driver actually performs a secondary task by using a tablet computer.

[0050] Fig.19 1 is a diagram for describing a state in which a cautious driving zone Sd newly appears in the second zone and the driver is warned by a flashing display.

[0051] Fig. 20 A diagram depicting a state in which a small window is displayed as a pop-up display on the screen of a tablet computer.

[0052] Fig.21 FIG. 1 is a diagram for explaining a usage configuration example of a wristwatch-type information terminal according to the present disclosure.

[0053] Fig. 22 FIG. 1 is a diagram for explaining a usage configuration example of a wristwatch-type information terminal according to the present disclosure.

[0054] Fig.23 FIG. 1 is a diagram for explaining a usage configuration example of a wristwatch-type information terminal according to the present disclosure.

[0055] Fig.24 is a diagram used to illustrate the concept of neurotransmission in which combined / layered information provides an enhanced level of regression consciousness.

[0056] Fig.25 It is a diagram for explaining the detailed structure of the travel section display data.

[0057] Fig.26 It is a diagram for explaining the detailed structure of the travel section display data.

[0058] Fig. 27 It is a diagram for explaining an example of updating of the travel section display data over time.

[0059] Fig.28 This is a diagram for explaining an example of updating of travel section display data on a wristwatch terminal as time passes.

[0060] Fig.29 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data.

[0061] Fig.30 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data.

[0062] Fig.31 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data.

[0063] Fig.32 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data.

[0064] Fig.33 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data.

[0065] Fig.34 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data.

[0066] Fig.35 1 is a diagram for explaining an additional display example of additional information with respect to travel section display data displayed on a wristwatch-type terminal.

[0067] Fig.36 It is a diagram for explaining an example of displaying data using animation.

[0068] Fig.37 Diagrams for explaining examples of displaying data using animation are described.

[0069] Fig.38 Diagrams for explaining examples of displaying data using animation are described.

[0070] Fig.39 Diagrams for explaining examples of display data on a terminal such as a smartphone are described.

[0071] Fig.40 Diagrams for explaining examples of display data on a terminal such as a smartphone are described.

[0072] Fig.41 Diagrams for explaining examples of display data on a terminal such as a smartphone are described.

[0073] Fig.42Diagrams for explaining examples of display data on a terminal such as a smartphone are described.

[0074] Fig.43 Diagrams for explaining examples of display data on a terminal such as a smartphone are described.

[0075] Fig.44 It is a diagram for explaining an example of the hardware configuration of the information processing device. DETAILED DESCRIPTION

[0076] The following describes details of the information processing device, mobile device, method, and program according to the present disclosure with reference to the accompanying drawings. Note that the description is made according to the following items.

[0077] 1. Overview of the configuration and processing of mobile devices and information processing devices

[0078] 2. Specific configuration and processing examples of mobile devices

[0079] 3. Mode switching sequence from autonomous driving mode to manual driving mode

[0080] 4. Example of an operation sequence for autonomous driving

[0081] 5. Information provided to users (drivers) and notification processing to users

[0082] 5-1. Detailed structure of travel section display data and process of updating travel section display data

[0083] 5-2. Example of display of additional information for travel section display data

[0084] 5-3. Examples of notification and warning processing using animation

[0085] 5-4. Other display processing examples

[0086] 6. Configuration example of information processing device

[0087] 7. Summary of the present disclosure

[0088] [1. Overview of the configuration and processing of mobile devices and information processing devices]

[0089] First reference Figure 1 and subsequent figures, illustrate an overview of the configuration and processing of a mobile device and an information processing device.

[0090] The mobile device according to the present disclosure is, for example, a car that can be driven in a driving mode that switches between automatic driving and manual driving.

[0091] When it is necessary to switch from the automatic driving mode to the manual driving mode in such a car, the driver needs to start manual driving.

[0092] However, there are various processes (secondary tasks) that the driver performs during execution of autonomous driving.

[0093] For example, in some cases, the drivers simply took their hands off the steering wheel and stared at the space in front of the car as they would when driving the car, while in some other cases, they were reading a book or taking a nap.

[0094] Due to the differences between these processes, the driver's awakening (level of consciousness) also varies.

[0095] For example, if the driver falls asleep, the driver's alertness decreases. That is, the level of consciousness becomes low. In this state of reduced alertness, normal manual driving cannot be performed, and if the driving mode is switched to the manual driving mode in this state, there is a possibility that the driver causes an accident in the worst case.

[0096] In order to ensure driving safety, it is necessary to allow the driver to start manual driving when the driver's alertness is high, that is, when the driver has a high level of awareness.

[0097] For this purpose, it is necessary to change the notification timing of requesting switching from the automatic driving to the manual driving according to the driver's alertness during the execution of the automatic driving.

[0098] For example, in a case where the driver looks forward and stares at the road during execution of autonomous driving, the driver's alertness is high, that is, the driver can start manual driving at any time.

[0099] In this case, the notification of switching to manual driving can be issued at a timing immediately before the time when manual driving is required. This is because the driver can immediately start safe manual driving.

[0100] However, in the case where the driver falls asleep during the execution of the autonomous driving, the driver's alertness is extremely low.

[0101] In this case, if the notification of switching to manual driving is issued at a timing immediately before the time when manual driving is required, the driver will inevitably have to start manual driving in a state where the driver has a low level of consciousness. As a result, the possibility of the driver causing an accident increases. Therefore, in such a situation where the awakening degree is low, it is necessary to issue a notification of switching to manual driving at an earlier stage.

[0102] The mobile device according to the present disclosure or the information processing device according to the present disclosure that can be installed in the mobile device controls the timing of notifying the switch to manual driving according to the driver's alertness, for example.

[0103] refer to Figure 1 and the subsequent drawings illustrate the configuration and processing of a mobile device according to the present disclosure and an information processing device according to the present disclosure that can be installed on the mobile device.

[0104] Figure 1 1 is a diagram describing one configuration example of a car 10 as an example of a mobile device according to the present disclosure.

[0105] The information processing device according to the present disclosure is installed on Figure 1 On the car 10 shown in .

[0106] Figure 1 The car 10 shown in FIG. 1 is a car that can be driven in two driving modes, which are a manual driving mode and an automatic driving mode.

[0107] In the manual driving mode, driving is performed based on the operation of the driver 20 , that is, the operation of the steering wheel (steering) and the operation of the accelerator, brake, and the like.

[0108] On the other hand, in the automatic driving mode, no operation by the driver 20 is required, and driving is performed based on sensor information such as position sensors and other surrounding environment information detection sensors.

[0109] Position sensors include, for example, GPS receivers, and surrounding environment information detection sensors include, for example, cameras, ultrasonic sensors, radars, LiDAR (light detection and ranging, laser imaging detection and ranging), sonars, and the like.

[0110] Notice, Figure 1 Detailed description of the present invention is given in the following sections.

[0111] like Figure 1 As shown in FIG. 1 , the automobile 10 has a data processing section 11 , a driver information acquisition section 12 , an environment information acquisition section 13 , a communication section 14 , and a notification section 15 .

[0112] The driver information acquisition unit 12 acquires, for example, information for determining the driver's awakening degree, the driver's operation information, etc. Specifically, for example, the driver information acquisition unit 12 includes a camera for capturing a facial image of the driver, an operation information acquisition unit for acquiring various operation information about the operation parts (steering wheel, accelerator, brake, etc.), etc.

[0113] The environment information acquisition unit 13 acquires the driving environment information of the automobile 10. For example, the driving environment information includes information on images of the space in front, behind, left, and right of the automobile; location information from GPS; information on surrounding obstacles from LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), sonar, etc.; and the like.

[0114] The data processing unit 11 receives the driver information acquired by the driver information acquisition unit 12 and the environmental information acquired by the environmental information acquisition unit 13 as input, and calculates a safety index value, which indicates whether the driver in the vehicle performing automatic driving is able to perform safe manual driving, and further, whether the driver who is performing manual driving is performing safe driving, etc.

[0115] Furthermore, for example, when it is necessary to switch from the automatic driving mode to the manual driving mode, the data processing unit 11 performs processing for notifying the switching to the manual driving mode via the notification unit 15 .

[0116] For example, the timing of the notification process is an optimal timing calculated by receiving inputs from the driver information acquisition section 12 and the environment information acquisition section 13 .

[0117] That is, the timing is a timing that allows the driver 20 to start safe manual driving.

[0118] Specifically, the processing is performed in such a manner that when the driver's alertness is high, the notification is issued immediately before the time to start manual driving, for example, 5 seconds before the time, and when the driver's alertness is low, the notification is issued 20 seconds before the time to start manual driving, thereby leaving extra time, or in other similar manners. The specific calculation of the optimal timing of the notification is described later.

[0119] The notification unit 15 includes a display unit, a sound output unit, or a vibrator on a steering wheel or a seat that issues a notification.

[0120] Figure 2 An example of a warning display on the display section included in the notification section 15 is described in FIG.

[0121] like Figure 2 As shown in , the display section 30 presents the following information as the display.

[0122] Driving mode information = "Autonomous driving"

[0123] Warning display = "Switch to manual driving"

[0124] In the display area of ​​the driving mode information, “Automatic Driving” is displayed when the automatic driving mode is executed, and “Manual Driving” is displayed when the manual driving mode is executed.

[0125] The display area of ​​the warning display information is a display area that displays the following information when the car performs automatic driving in the automatic driving mode. Note that although the entire display screen is used in this embodiment, a part of the screen may be used for the display.

[0126] "Switch to manual driving"

[0127] Note that although the description is explicitly stated in sentences in this example, the display is not limited to such a text display, and may include symbols such as pictograms, for example.

[0128] Note that Figure 1 As shown in , the car 10 is configured to be able to communicate with the server 30 via the communication unit 14 .

[0129] For example, a part of the process of calculating the appropriate time of the notification output by the data processing unit 11 , specifically, the learning process, may be performed by the server 30 .

[0130] A specific example of this point will be described later.

[0131] Figure 3 Diagrams describing specific examples of processing performed by a mobile device and an information processing device according to the present disclosure.

[0132] Figure 3 Description of diagrams illustrating setting examples of appropriate timing of notification requesting switching to manual driving while a car is performing automatic driving in an automatic driving mode, notification processing examples in the following two examples are described.

[0133] (a) Notification processing when the driver's alertness is high during the execution of autonomous driving

[0134] (b) Notification processing when the driver's alertness is low during the execution of autonomous driving

[0135] The example of (a) is an example in which the driver looks ahead and stares at the road during the execution of the autonomous driving. In this case, the driver's alertness is high, that is, the driver can start manual driving at any time.

[0136] In this case, even if the notification of switching to manual driving is issued at the timing immediately before the time when manual driving is required, the driver can immediately start safe manual driving.

[0137] In the example of (b), in the case where the driver falls asleep during the execution of the autonomous driving, the driver's alertness is extremely low.

[0138] In this case, if the notification of switching to manual driving is issued at a timing immediately before the time when manual driving is required, the driver will inevitably start manual driving in a state where the driver has a low level of consciousness, and the possibility of the driver causing an accident increases. Therefore, in such a state where the awakening level is low, it is necessary to issue a notification of switching to manual driving at an earlier stage.

[0139] Furthermore, in the case where the vehicle is a transport vehicle such as a vehicle having a luggage compartment and allowing the driver to move to the luggage compartment to work during the execution of the autonomous driving. Figure 4 As shown in , the notification timing for requesting switching to manual driving is set in different ways depending on the following three situations.

[0140] (a) Notification processing when the driver’s alertness is high during the execution of autonomous driving

[0141] (b) Notification processing when the driver's alertness is low during the execution of autonomous driving

[0142] (c) Notification processing when the driver leaves the driver's seat during the execution of autonomous driving

[0143] The example of (a) is an example in which the driver looks ahead and stares at the road during the execution of the autonomous driving. In this case, the driver's alertness is high, that is, the driver can start manual driving at any time.

[0144] In this case, even if the notification of switching to manual driving is issued at the timing immediately before the time when manual driving is required, the driver can immediately start safe manual driving.

[0145] In the example of (b), the driver fell asleep during the execution of the autonomous driving, in which case the driver's alertness was extremely low.

[0146] In this case, if the notification of switching to manual driving is issued at a timing immediately before the time when manual driving is required, the driver will inevitably start manual driving in a state where the driver has a low level of consciousness, and the possibility of the driver causing an accident increases. Therefore, in such a state where the awakening level is low, it is necessary to issue a notification of switching to manual driving at an earlier stage.

[0147] In the example of (c), the driver leaves the driver's seat to perform work during the execution of the autonomous driving, in which case it takes time for the driver to return to the driver's seat.

[0148] In this case, if the notification of switching to manual driving is issued at a timing immediately before the time when manual driving is required, there is a possibility that the vehicle unexpectedly enters the manual driving zone before the driver returns to the driver's seat. Therefore, in the case where the driver leaves the driver's seat, it is necessary to issue a notification of switching to manual driving at an earlier stage.

[0149] In addition, when the driver 20 in the car 10 leaves the driver's seat, even if the Figure 2 In the above-described method, the notification is displayed on the notification unit (display unit) 15 of the driver's seat, and the driver 20 cannot notice the notification.

[0150] In order to solve this problem, the information terminal 50 worn on the arm of the driver 20, for example, Figure 5 The watch-type information terminal 50 shown in FIG. 1 is used to provide notification and information. The information terminal 50 is used to display and refer to Figure 2 The display data is processed similarly to the display data described above, and also displays the approach information about the section (automatic driving section, manual driving section) of the road on which the car 10 is traveling. In addition, the information terminal 50 has alarm output, sound output and vibration functions to notify the driver 20 of various notifications and warnings.

[0151] Figure 5 A diagram illustrating an example of use of the information terminal 50 is described. When the car 10 is performing automatic driving, the driver 20 in the car 10 is in the driver's seat in some cases, such as Figure 4 (a), but in some other cases, leaving the driver's seat and working in the luggage compartment, such as Figure 4 (b). However, the driver 20 always wears the information terminal 50 on his / her arm, so that the driver can notice the display information displayed on the information terminal 50, as well as the output sound, alarm, vibration, etc. In addition, such a wearable information terminal 50 operates together with the driver information acquisition unit 12 of the vehicle, and has a response input function as a confirmation means. This configuration makes it possible to respond to notifications and warnings. In addition, a rejection notification for early resumption of manual driving described later can also be sent to the vehicle system.

[0152] Figure 6 2 is a diagram for describing an example of display information of the information terminal 50 . Figure 6 The examples described in the illustrations are similar to those in the previous references Figure 2 The information terminal 50 displays the following information.

[0153] Driving mode information = "Autonomous driving"

[0154] Warning display = "Switch to manual driving"

[0155] The driver 20 can check the display wherever he is.

[0156] Furthermore, in another possible configuration, the following may be used: Figure 7 The tablet-type information terminal 50 shown in FIG. 1 is used to perform notification and information provision.

[0157] [2. Specific configuration and processing example of mobile device]

[0158] Reference below Figure 8 and the subsequent figures illustrate specific configuration and processing examples of a mobile device according to the present disclosure.

[0159] Figure 8 A configuration example of the mobile device 100 is described. Note that hereinafter, in the case of distinguishing a vehicle provided with the mobile device 100 from other vehicles, the former is referred to as a user's car or a user's vehicle.

[0160] The mobile device 100 includes an input unit 101, a data acquisition unit 102, a communication unit 103, an in-vehicle device 104, an output control unit 105, an output unit 106, a drive system control unit 107, a drive system 108, a body system control unit 109, a body system 110, a storage unit 111 and an automatic driving control unit 112.

[0161] The input section 101, the data acquisition section 102, the communication section 103, the output control section 105, the drive system control section 107, the body system control section 109, the storage section 111, and the automatic driving control section 112 are interconnected via a communication network 121. The communication network 121 includes, for example, an in-vehicle communication network conforming to a certain standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark), a bus, etc. Note that in some cases, the respective sections of the mobile device 100 are directly connected without using the communication network 121.

[0162] Note that, hereinafter, in the case where the various parts of the mobile device 100 communicate via the communication network 121, the description about the communication network 121 is omitted. For example, in the case where the input part 101 and the automatic driving control part 112 communicate via the communication network 121, it is simply described that the input part 101 and the automatic driving control part 112 communicate.

[0163] The input unit 101 includes a device for the occupant to input various data, instructions, etc. For example, the input unit 101 includes an operating device such as a touch panel, a button, a microphone, a switch, or a joystick; an operating device by which information can be input by a method other than manual operation, by voice, gesture, etc.; and the like. In addition, for example, the input unit 101 may be a remote control device using infrared or other radio waves, or an external connection device that supports the operation of the mobile device 100, such as a mobile device or a wearable device. The input unit 101 generates an input signal based on the data, instructions, etc. input by the occupant, and provides the input signal to each part of the mobile device 100.

[0164] The data acquisition unit 102 includes various sensors and the like that acquire data for processing by the mobile device 100 and provide the acquired data to various units of the mobile device 100 .

[0165] For example, the data acquisition unit 102 includes various sensors for detecting the state of the user's car, etc. Specifically, for example, the data acquisition unit 102 includes a gyro sensor; an acceleration sensor; an inertial measurement unit (IMU); a sensor for detecting an accelerator pedal operation amount, a brake pedal operation amount, a steering wheel steering angle, an engine speed, a motor speed, a wheel speed, etc.; and the like.

[0166] In addition, for example, the data acquisition unit 102 includes various sensors for detecting information about the outside of the user's car. Specifically, for example, the data acquisition unit 102 includes an image capture device such as a ToF (time of flight) camera, a stereo camera, a monocular camera, an infrared camera, or other types of cameras. In addition, for example, the data acquisition unit 102 includes an environmental sensor for detecting meteorological conditions, atmospheric phenomena, etc.; and a surrounding environment information detection sensor for detecting objects around the user's car. Environmental sensors include, for example, raindrop sensors, fog sensors, sunshine sensors, snow sensors, etc. The surrounding environment information detection sensor includes, for example, an ultrasonic sensor, a radar, a LiDAR (light detection and ranging, laser imaging detection and ranging), a sonar, etc.

[0167] For example, Fig. 9 Describes an installation example of various sensors for detecting information about the outside of the user's car. Image capture devices 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, and rear door of the vehicle 7900, and the upper portion of the windshield inside the vehicle.

[0168] The image capturing device 7910 provided at the front nose and the image capturing unit 7918 provided at the upper portion of the windshield inside the vehicle mainly acquire images of the space in front of the vehicle 7900. The image capturing devices 7912 and 7914 provided at the side mirrors mainly acquire images of the space to the side of the vehicle 7900. The image capturing device 7916 provided at the rear bumper or the rear door mainly acquires images of the space behind the vehicle 7900. The image capturing device 7918 provided at the upper portion of the windshield inside the vehicle is mainly used to detect the preceding vehicle, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc. In addition, in the near future of autonomous driving, the use of the image capturing device can be expanded to cover a wider range when the vehicle turns right or left, to cover pedestrians crossing the road ahead when the vehicle turns right or left, and to further cover the range of objects approaching the crosswalk.

[0169] Notice, Fig. 9 An example of the image capturing ranges of the image capturing devices 7910, 7912, 7914, and 7916 is described. The image capturing range a represents the image capturing range of the image capturing device 7910 disposed at the front nose, the image capturing ranges b and c represent the image capturing ranges of the image capturing devices 7912 and 7914 disposed at the side mirrors, and the image capturing range d represents the image capturing range of the image capturing device 7916 disposed at the rear bumper or the rear door. For example, by superimposing the pieces of data of the images captured by the image capturing devices 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above is obtained, and then, an omnidirectional stereoscopic display image around the periphery of the vehicle in a curved plane, etc.

[0170] Sensors 7920, 7922, 7924, 7926, 7928, and 7930 disposed at the front, rear, sides, and corners of the vehicle 7900, and at the upper portion of the windshield inside the vehicle, may be, for example, ultrasonic sensors or radars. Sensors 7920, 7926, and 7930 disposed at the front nose, rear bumper, and rear door of the vehicle 7900, and at the upper portion of the windshield inside the vehicle, may be, for example, LiDARs. Sensors 7920 to 7930 are mainly used to detect the vehicle ahead, pedestrians, obstacles, and the like. The detection results may also be applied to stereoscopic display or omnidirectional stereoscopic display that improves the bird's-eye view display.

[0171] return Figure 8 , the constituent elements are further described. The data acquisition unit 102 includes various sensors for detecting the current position of the user's car. Specifically, for example, the data acquisition unit 102 includes a GNSS (Global Navigation Satellite System) receiver that receives GNSS signals from GNSS satellites, etc.

[0172] In addition, for example, the data acquisition unit 102 includes various sensors for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 102 includes an image capture device that captures an image of the driver; a life sensor that detects vital information of the driver; a microphone that collects sounds inside the vehicle, etc. For example, the life sensor is set on a seat surface, a steering wheel, etc., and detects the sitting state of an occupant sitting on the seat, or the vital information of a driver holding the steering wheel. The vital signals that can be used include various observable data, such as heart rate, pulse rate, blood flow, breathing, mind-body correlation, visual stimulation, brain waves, sweating state, head posture behavior, eyes, staring, blinking, saccade, microsaccade, gaze, drift, staring, or iris pupil reaction. The life activity observable information reflecting the observable driving state is aggregated into an observable evaluation value estimated from the observation, and is used to calculate the recovery notification timing in the safety determination unit 155 described below based on the recovery delay time characteristics associated with the log of the evaluation value as a characteristic characteristic of the recovery delay situation of the relevant driver.

[0173] Fig.10 Examples of various sensors included in the data acquisition unit 102 and used to obtain information about the driver in the vehicle are described. For example, the data acquisition unit 102 includes a ToF camera, a stereo camera, a seat strain gauge (Seat Strain Gauge), etc. as detectors for detecting the position and posture of the driver. In addition, the data acquisition unit 102 includes a face recognizer (face (head) recognition), a driver eye tracker (Driver Eye Tracker), a driver head tracker (Driver Head Tracker), etc. as detectors for detecting observable information of the driver's life activities.

[0174] In addition, the data acquisition unit 102 includes a vital signal detector as a detector for acquiring observable information of the driver's vital activities. In addition, the data acquisition unit 102 includes a driver authentication unit. Note that in addition to knowledge-based authentication using a password, PIN number, etc., examples of authentication methods that can be used include life authentication using a face, fingerprint, iris of the pupil, voiceprint, etc.

[0175] The communication unit 103 communicates with the in-vehicle device 104 and various devices outside the vehicle, a server, a base station, etc., transmits data supplied from various parts of the mobile device 100, and provides received data to each part of the mobile device 100. Note that the communication protocol supported by the communication unit 103 is not particularly limited to any type of communication protocol, and the communication unit 103 can also support multiple types of communication protocols.

[0176] For example, the communication unit 103 performs wireless communication with the in-vehicle device 104 via wireless LAN, Bluetooth (registered trademark), NFC (near field communication), WUSB (wireless USB), etc. In addition, for example, the communication unit 103 performs wired communication with the in-vehicle device 104 via USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), MHL (Mobile High-Definition Link), etc. via a connection terminal (and a cable, if necessary) not shown in the figure.

[0177] In addition, for example, the communication unit 103 communicates with a device (e.g., an application server or a control server) on an external network (e.g., the Internet, a cloud network, or a private network of an enterprise operator) via a base station or an access point. In addition, for example, the communication unit 103 communicates with a terminal (e.g., a terminal of a pedestrian or a store, or an MTC (machine type communication) terminal) near the user's car by using a P2P (peer-to-peer) technology.

[0178] In addition, for example, the communication unit 103 performs V2X communication such as vehicle to vehicle communication, vehicle to infrastructure communication, vehicle to home communication, or vehicle to pedestrian communication. In addition, for example, the communication unit 103 includes a beacon receiving unit that receives radio waves or electromagnetic waves emitted from a wireless station installed on the road, etc., thereby acquiring information about the current position, traffic congestion, traffic control, required time, etc. Note that the communication unit can be used to pair with a vehicle that is ahead of the user vehicle and can be a leading vehicle traveling in the section, and can acquire information acquired from a data acquisition unit installed on the leading vehicle as advance driving information and used to supplement the data of the data acquisition unit 102 of the user car. In particular, this serves as a means of ensuring higher safety for the following vehicles when they are traveling in a queue under the guidance of the leading vehicle or in other similar situations.

[0179] For example, the in-vehicle device 104 includes a mobile device (tablet computer, smart phone, etc.) or a wearable device carried by the occupant; an information device brought into or attached to the user's car; a navigation device that searches for routes to certain destinations, etc. Note that, considering that if autonomous driving is more widely used, the occupants do not necessarily sit in a fixed position, in the future, the in-vehicle device 104 that can be used can be expanded to video players, game consoles, and other devices that can be attached to and detached from the mounting position in the vehicle. Although in the example described in this embodiment, information presentation about the geographical point where the driver's intervention becomes necessary is performed only for the relevant driver, when the vehicles are traveling in a queue or in other similar situations, the information can also be provided for the following vehicles. In addition, by always providing information to the operation management center of passenger sharing buses and long-distance logistics commercial vehicles, it can be appropriately used in combination with remote driving assistance.

[0180] The output control section 105 controls the output of various information to the occupants of the user's car or to the outside of the vehicle. For example, the output control section 105 generates an output signal including visual information (e.g., image data) and / or auditory information (e.g., sound data), and supplies them to the output section 106, thereby controlling the output of the visual information and the auditory information from the output section 106. Specifically, for example, the output control section 105 synthesizes various pieces of data of images captured by different image capture devices of the data acquisition section 102, generates a bird's-eye view image, a panoramic image, etc., and supplies the output signal including the generated image to the output section 106. In addition, for example, the output control section 105 generates sound data including a buzzer sound, a warning message, etc. regarding dangers such as collision, contact, or entering a danger zone, and supplies the output signal including the generated sound data to the output section 106.

[0181] The output unit 106 includes a device capable of outputting visual information or auditory information to the occupants of the user's car or to the outside of the vehicle. For example, the output unit 106 includes a display device; an instrument panel; an audio speaker; headphones; a wearable device such as a glasses-type display worn by the occupants; a projector; a lamp, etc. In addition to a device with a normal display, for example, the display device included in the output unit 106 can be a device that displays visual information in the driver's field of view, such as a head-up display, a transmissive display, or a device with an AR (augmented reality) display function.

[0182] The drive system control unit 107 generates various control signals and supplies the control signals to the drive system 108, thereby controlling the drive system 108. In addition, the drive system control unit 107 supplies control signals to various parts other than the drive system 108, and gives a notification about the control state of the drive system 108, etc., as necessary.

[0183] The drive system 108 includes various devices related to the drive system of the user's car. For example, the drive system 108 includes a drive force generating device such as an internal combustion engine or a drive motor for generating a drive force; a drive force transmitting mechanism for transmitting the drive force to the wheels; a steering mechanism for adjusting a steering angle; a brake device for generating a braking force; ABS (anti-lock braking system); ESC (electronic stability control); an electric power steering device, etc.

[0184] The body system control unit 109 generates various control signals and supplies them to the body system 110, thereby controlling the body system 110. In addition, the body system control unit 109 supplies control signals to various parts other than the body system 110, and gives notifications on the control status of the body system 110, etc., as necessary.

[0185] The body system 110 includes various devices related to the body system installed on the body. For example, the body system 110 includes a keyless entry system; a smart key system; an electric window device; an electric seat; a steering wheel; an air conditioning device; various lights (such as headlights, taillights, brake lights, flashing lights, fog lights, etc.), etc.

[0186] For example, the storage unit 111 includes a magnetic storage device such as a ROM (read only memory), a RAM (random access memory), or an HDD (hard disk drive); a semiconductor storage device; an optical storage device; a magneto-optical storage device, etc. The storage unit 111 stores various programs, data, etc. used by various parts of the mobile device 100. For example, the storage unit 111 stores map data, such as a three-dimensional high-precision map such as a dynamic map, a global map with lower precision than a high-precision map but covering a larger area, or a local map including information about the area around the user's car.

[0187] The automatic driving control unit 112 performs control related to automatic driving such as autonomous driving or driving assistance. Specifically, for example, the automatic driving control unit 112 performs cooperative control aimed at realizing the functions of ADAS (Advanced Driver Assistance System), and the functions of ADAS (Advanced Driver Assistance System) include collision avoidance or impact reduction of the user's car, follow-up driving based on the distance between vehicles, vehicle constant speed driving, collision warning of the user's car, lane departure warning of the user's car, etc. In addition, for example, the automatic driving control unit 112 performs cooperative control of automatic driving aimed at autonomous driving without relying on the driver's operation. The automatic driving control unit 112 includes a detection unit 131, a current position estimation unit 132, a situation analysis unit 133, a planning unit 134, and an operation control unit 135.

[0188] The detection unit 131 detects various information required for the control of the automatic driving. The detection unit 131 includes a vehicle exterior information detection unit 141 , a vehicle interior information detection unit 142 , and a vehicle state detection unit 143 .

[0189] The vehicle exterior information detection unit 141 performs a process of detecting information about the exterior of the user's car based on data or signals from various units of the mobile device 100. For example, the vehicle exterior information detection unit 141 performs a process of detecting, identifying, and tracking objects around the user's car, and a process of detecting the distance to and relative speed of the objects. For example, objects that are detection targets include vehicles, people, obstacles, buildings, roads, traffic lights, traffic signs, road markings, and the like.

[0190] In addition, for example, the vehicle exterior information detection unit 141 performs a process of detecting the environment around the user's car. For example, the environment around the user's car as a detection target includes weather, temperature, humidity, brightness, road conditions, etc. The vehicle exterior information detection unit 141 provides data indicating the result of the detection process to the current position estimation unit 132, the map analysis unit 151 of the situation analysis unit 133, the traffic regulation recognition unit 152 and the situation recognition unit 153, and the emergency avoidance unit 171 of the operation control unit 135.

[0191] If the user's car is traveling in a section where the local dynamic map (LDM) that is continuously updated is supplied from the infrastructure as a section where autonomous driving is allowed first, then the information acquired by the vehicle external information detection unit 141 can be mainly provided by the infrastructure. Alternatively, in some other possible cases, the user's car can be traveling while continuously receiving updated information in advance from a vehicle or a vehicle group that traveled earlier in the relevant section before entering the section. In addition, in the case where the infrastructure does not continuously update the latest local dynamic map or in other similar cases, in order to obtain road information immediately before the entrance section to improve safety, especially when driving in a queue, the road environment information obtained from the leading vehicle that has entered the section can be further supplemented. In many cases, whether autonomous driving is allowed in a certain section is determined based on whether there is advance information provided from the infrastructure. The updated latest local dynamic map (LDM) included in the information provided from the infrastructure indicating whether autonomous driving is allowed or prohibited on the route is equivalent to providing an invisible track that is generally referred to as "information". Note that although the vehicle exterior information detection unit 141 is described as being mounted on the user's vehicle as a premise for convenience, the predictability while traveling can be further enhanced by using information captured by the preceding vehicle as "information".

[0192] The vehicle interior information detection unit 142 performs a process of detecting information about the interior of the vehicle based on data or signals from various units of the mobile device 100. For example, the vehicle interior information detection unit 142 performs a process of identifying and authenticating the driver, a process of detecting the driver's state, a process of detecting passengers, a process of detecting the environment inside the vehicle, etc. For example, the state of the driver as a detection target includes physical condition, awakeness, concentration, fatigue, line of sight direction, detailed eye behavior, etc.

[0193] Furthermore, it is expected that in future uses of automated driving, the driver will completely disengage from the driving control work, the driver will temporarily doze off, or begin to perform other tasks, making it necessary for the system to identify to what extent the awakening degree of consciousness necessary for resuming driving has been restored. That is, in the driver monitoring system traditionally considered, the detection device is mainly responsible for detecting the decline of consciousness, such as drowsiness, but in the near future, the driver will not intervene in the driving control at all. Therefore, the system will no longer have the means to directly observe the degree of driver intervention when driving from the control stability of the control device, etc., so that the system must observe the transition from the state where the driver's accurate state of consciousness is unknown to the recovery of consciousness required for driving, and intervene in the control transition from automated driving to manual driving by identifying the driver's accurate internal state of awakening.

[0194] In view of this, the vehicle interior information detection unit 142 mainly plays a role in two main stages. The first role is the passive monitoring of the driver's state during automatic driving, and the second role is the detection and judgment of the driver's peripheral cognition, perception, judgment and control device driving ability after the recovery request is issued from the system until the driver reaches the level of manual driving before reaching the attention driving interval. As a control, self-diagnosis of the fault of the entire vehicle can also be performed, and similarly, in the case of a decline in the automatic driving function due to a partial functional failure of the automatic driving, the driver can also be urged to resume manual driving as soon as possible. The passive monitoring mentioned here means a detection method that does not require the driver to respond and react consciously, but does not exclude the object of emitting physical radio waves, light, etc. from the device and detecting the response signal. That is, the passive method means the state monitoring of the unconscious driver when dozing, etc., and does not involve the classification of the driver's cognitive response and reaction. It does not exclude the active response device that analyzes and evaluates the reflected or diffused signals as a result of emitting radio waves, infrared rays, etc. On the contrary, monitoring that requires the driver to respond and react consciously is called active monitoring.

[0195] For example, the environment inside the vehicle as the detection target includes temperature, humidity, brightness, smell, etc. The vehicle interior information detection unit 142 provides data representing the result of the detection process to the situation recognition unit 153 of the situation analysis unit 133 and the operation control unit 135. Note that, in the case where it is found that the driver cannot achieve manual driving within an appropriate period of time after the system issues an instruction to the driver to resume driving, and it is determined that even if an attempt is made to obtain additional time by performing deceleration control while maintaining automatic driving, the conversion will be too late, an instruction is issued to the emergency avoidance unit 171 of the system, etc., to start the process of decelerating, driving to the roadside, and stopping to allow the vehicle to pull over. That is, even in the case where the conversion is also too late as an initial state, by decelerating the vehicle early, the arrival time to the conversion limit can be made longer. By making the arrival time to the conversion limit longer, extra time is generated for the system to process events, and measures can be taken to ensure safety. However, it should be noted that the application of this method is limited because, as described below, excessive deceleration or slowing down of the car increases the risk of traffic congestion triggers and rear-end collisions.

[0196] The vehicle state detection unit 143 performs a process of detecting the state of the user's car based on the data or signals from each unit of the mobile device 100. For example, the state of the user's car as the detection target includes speed; acceleration; steering angle; presence or absence of abnormality; content of abnormality; state of driving operation; position and tilt of the power seat; state of the door lock; state of other in-car equipment, etc. The vehicle state detection unit 143 provides data indicating the result of the detection process to the state recognition unit 153 of the state analysis unit 133, the emergency avoidance unit 171 of the operation control unit 135, etc.

[0197] The current position estimation unit 132 performs processing to estimate the position and posture of the user's car based on data or signals from various units of the mobile device 100, such as the vehicle external information detection unit 141 and the situation recognition unit 153 of the situation analysis unit 133. In addition, if necessary, the current position estimation unit 132 generates a local map for estimating the current position (hereinafter referred to as a current position estimation map).

[0198] For example, the current position estimation map is a highly accurate map using a technique such as SLAM (Simultaneous Localization and Mapping). The current position estimation unit 132 provides data indicating the result of the estimation process to the map analysis unit 151, the traffic regulation recognition unit 152, and the situation recognition unit 153 of the situation analysis unit 133. In addition, the current position estimation unit 132 causes the storage unit 111 to store the current position estimation map.

[0199] The situation analysis unit 133 performs a process of analyzing the situation of the user's car and its surroundings. The situation analysis unit 133 includes a map analysis unit 151 , a traffic regulation recognition unit 152 , a situation recognition unit 153 , a situation prediction unit 154 , and a safety determination unit 155 .

[0200] The map analysis unit 151 performs a process of analyzing various maps stored in the storage unit 111 and constructs a map including information required for the process of autonomous driving, while using data or signals from various units of the mobile device 100, such as the current position estimation unit 132 or the vehicle external information detection unit 141, as needed. The map analysis unit 151 provides the constructed map to the traffic regulation recognition unit 152, the situation recognition unit 153, the situation prediction unit 154, and the route planning unit 161, the action planning unit 162, and the operation planning unit 163 of the planning unit 134.

[0201] The traffic regulation recognition unit 152 performs a process of recognizing traffic regulations around the user's car based on data or signals from various units of the mobile device 100, such as the current position estimation unit 132, the vehicle external information detection unit 141, or the map analysis unit 151. Through this recognition process, for example, the position around the user's car and the state of the signal, the content of the traffic control around the user's car, the lane in which the user's car is allowed to travel, etc. are recognized. The traffic regulation recognition unit 152 provides data indicating the result of the recognition process to the situation prediction unit 154 and the like.

[0202] The situation recognition unit 153 performs a process of recognizing a situation related to the user's car based on data or signals from various units of the mobile device 100, such as the current position estimation unit 132, the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, the vehicle state detection unit 143, or the map analysis unit 151. For example, the situation recognition unit 153 performs a process of recognizing the situation of the user's car, the situation around the user's car, the situation of the driver of the user's car, and the like. In addition, if necessary, the situation recognition unit 153 generates a local map (hereinafter referred to as a situation recognition map) for recognizing the situation around the user's car. For example, the situation recognition map is an occupancy grid map.

[0203] For example, the conditions of the user's car as the identification target include the position, posture and movement (such as speed, acceleration, direction of movement, etc.) of the user's car; vehicle-specific conditions that determine the motion characteristics of the user's car and cargo loading-specific conditions, such as cargo loading amount, movement of the center of gravity of the vehicle body due to cargo loading, tire pressure, braking distance movement due to the wear condition of the brake pads, maximum allowable deceleration braking to prevent cargo movement resulting in load braking, centrifugal force reduction limit speed when driving along a curve with a liquid load, etc.; and road-specific conditions, such as the friction coefficient of the road surface, the curve of the road, or the slope of the road. Since the recovery start timing required for control varies depending on the characteristics of the vehicle itself and the load, etc. even if the road environment is exactly the same, it is necessary to collect and learn various conditions so that the optimal timing for control reflects these various conditions. When determining the control timing based on the type and load of the vehicle, it is not enough to simply observe and monitor whether the user's vehicle has abnormalities, the content of the abnormalities, etc. Parameters for determining the additional extra time until recovery that is desirable to ensure a certain level of safety may be pre-set to fixed values ​​according to load-specific characteristics in the logistics industry, etc., and may not necessarily adopt a method in which all notification timing determination conditions are uniformly set through self-accumulative learning.

[0204] For example, the conditions around the user's car as the recognition target include the type and position of the surrounding stationary objects; the type, position and movement (such as speed, acceleration, moving direction, etc.) of the surrounding moving objects; the layout and road conditions of the surrounding roads; the weather, temperature, humidity and brightness of the space around the user's car; etc. For example, the state of the driver as the recognition target includes physical condition, awakening, concentration, fatigue, movement of sight, driving operation, etc. For the safe driving of the vehicle, according to different vehicle-specific states, such as loading amount, chassis fixing state of the carrying part, eccentric center of gravity state, maximum deceleration possible acceleration value, maximum load possible centrifugal force, recovery response delay amount according to the driver's state, etc., the control start point that needs to be processed is very different.

[0205] The situation recognition unit 153 supplies data indicating the result of the recognition process (including a situation recognition map, if necessary) to the current position estimation unit 132, the situation prediction unit 154, etc. In addition, the situation recognition unit 153 causes the storage unit 111 to store a situation recognition map.

[0206] The situation prediction unit 154 performs a process of predicting a situation related to the user's car based on data or signals from various units of the mobile device 100, such as the map analysis unit 151, the traffic regulation recognition unit 152, or the situation recognition unit 153. For example, the situation prediction unit 154 performs a process of predicting the situation of the user's car, the situation around the user's car, the driver's situation, etc.

[0207] For example, the condition of the user's car as a prediction target includes the behavior of the user's car, the occurrence of abnormalities, the drivable distance, etc. For example, the condition around the user's car as a prediction target includes the behavior of moving objects around the user's car, changes in the state of signals, changes in the environment such as weather, etc. For example, the condition of the driver as a prediction target includes the driver's behavior and physical condition, etc.

[0208] The situation prediction unit 154 provides data indicating the result of the prediction processing together with data from the traffic regulation recognition unit 152 and the situation recognition unit 153 to the route planning unit 161 , the action planning unit 162 , the operation planning unit 163 and the like of the planning unit 134 .

[0209] The safety determination unit 155 learns the optimal restoration timing according to the driver's restoration action pattern, vehicle characteristics, etc., and provides the learned information to the situation recognition unit 153, etc. As a result, for example, the driver can be presented with the statistically determined optimal timing required for the driver to normally restore from automatic driving to manual driving at a predetermined rate or higher.

[0210] The route planning unit 161 plans a route to a destination based on data or signals from various units of the mobile device 100, such as the map analysis unit 151 or the condition prediction unit 154. For example, based on a global map, the route planning unit 161 sets a route from a current location to a designated destination. In addition, for example, the route planning unit 161 changes the route as appropriate based on traffic congestion, accidents, traffic control, conditions such as construction, the driver's physical condition, etc. The route planning unit 161 provides data representing the planned route to the action planning unit 162, etc.

[0211] Based on data or signals from various parts of the mobile device 100, such as the map analysis part 151 or the situation prediction part 154, the action planning part 162 plans the action of the user's car for safely traveling on the route planned by the route planning part 161 within the planned time. For example, the action planning part 162 plans starting; stopping; traveling direction (e.g., forward, backward, left turn, right turn, change of direction, etc.); driving lane; driving speed; overtaking, etc. The action planning part 162 provides data representing the planned action of the user's car to the operation planning part 163, etc.

[0212] Based on data or signals from various parts of the mobile device 100, such as the map analysis part 151 or the situation prediction part 154, the operation planning part 163 plans the operation of the user's car for realizing the action planned by the action planning part 162. For example, the operation planning part 163 plans acceleration; deceleration; driving trajectory, etc. The operation planning part 163 provides data representing the planned operation of the user's car to the acceleration / deceleration control part 172 and the direction control part 173 of the operation control part 135, etc.

[0213] The operation control unit 135 controls the operation of the user's car and includes an emergency avoidance unit 171 , an acceleration / deceleration control unit 172 , and a direction control unit 173 .

[0214] Based on the detection results of the vehicle external information detection unit 141, the vehicle internal information detection unit 142, and the vehicle state detection unit 143, the emergency avoidance unit 171 performs processing to detect emergency situations such as collision, contact, entry into a dangerous area, abnormality of the driver, or abnormality of the vehicle. When the occurrence of an emergency situation is detected, the emergency avoidance unit 171 plans the operation of the user's car to avoid the emergency situation, such as an emergency stop or a sharp turn. The emergency avoidance unit 171 provides data indicating the planned operation of the user's car to the acceleration / deceleration control unit 172, the direction control unit 173, etc.

[0215] The acceleration / deceleration control unit 172 performs acceleration / deceleration control for implementing the operation of the user's car planned by the operation planning unit 163 or the emergency avoidance unit 171. For example, the acceleration / deceleration control unit 172 calculates the control target value of the driving force generating device or the braking device for implementing the planned acceleration, deceleration or emergency stop, and provides a control command representing the calculated control target value to the drive system control unit 107. Note that there are two main situations in which an emergency situation may occur. That is, one of them is a situation in which an unpredictable accident occurs due to an unexpected reason during automatic driving on a road (the road is initially indicated as a safe road by a local dynamic map or the like acquired from the infrastructure on the driving route during automatic driving), and the driver's emergency recovery will be too late. Another situation is a situation in which it is difficult for the driver to correctly resume manual driving from automatic driving.

[0216] The direction control unit 173 performs direction control for realizing the operation of the user's car planned by the operation planning unit 163 or the emergency avoidance unit 171. For example, the direction control unit 173 calculates a control target value of a steering mechanism for realizing the driving track or sharp turn planned by the operation planning unit 163 or the emergency avoidance unit 171, and provides a control command representing the calculated control target value to the drive system control unit 107.

[0217] [3. Mode switching sequence from automatic driving mode to manual driving mode]

[0218] Next, the transition sequence from the automatic driving mode to the manual driving mode is explained.

[0219] Fig.11 An example of a mode switching sequence from the automatic driving mode to the manual driving mode in the automatic driving control section 112 is schematically described.

[0220] In step S1, the driver is completely out of driving control. In this state, for example, the driver can perform secondary tasks, such as taking a nap, watching a video, concentrating on playing a game, or engaging in work using a visual tool such as a tablet or a smart phone. For example, in some possible cases, work using a visual tool such as a tablet or a smart phone can be performed in a state where the driver's seat is displaced, or in a state where the driver is sitting in a seat other than the driver's seat.

[0221] Depending on the state of the driver like this, it is expected that when the car approaches the section on the route where the car is required to resume manual driving, the time required for the driver to resume manual driving varies greatly according to the content of the work the driver is doing at that time. In some cases, if the notification is issued before the event is about to occur, there is not enough time for the driver to resume manual driving, or in some other cases, in the case of trying to leave extra time before the event and issuing the notification too early, the time until the timing when the driver is actually required to resume manual driving becomes too long. As a result, if the situation of not issuing the notification at the correct timing occurs repeatedly, the driver believes that the notification timing of the system is unreliable, and thus becomes less attentive to the notification. This causes the driver to become negligent in responding correctly; as a result, the risk of not making a good transition increases, and at the same time, this also becomes a factor that hinders the safe performance of secondary tasks. In view of this, the system needs to optimize the notification timing so that the driver can respond to the notification and start the correct preparation for resuming driving.

[0222] Step S2 corresponds to the above reference Figure 2 The timing of the notification of the request to resume manual driving is the same as described above. The driver is notified of the request to resume driving in a dynamic and tactile manner using vibration or the like, or in a visual or auditory manner. For example, the automatic driving control unit 112 monitors the driver's stable state, identifies the timing of issuing a notification, and issues a notification at an appropriate timing. That is, it is desirable that the state of the driver performing a secondary task is always passively monitored during the previous passive monitoring period, the system is able to calculate the optimal timing for notifying the optimal timing, the passive monitoring in the period of step S1 is continuously performed at all times, and the resumption timing and resumption notification are determined according to the recovery characteristics specific to the driver.

[0223] That is, it is desirable to learn the optimal resumption timing corresponding to the driver's resumption action pattern, vehicle characteristics, etc., and present the driver with the optimal timing required to normally resume manual driving from automatic driving at a certain predetermined rate or higher, which is statistically determined by the driver. In this case, in the case where the driver fails to respond to the notification within a certain length of time, a warning is issued by an alarm, etc.

[0224] In step S3, it is checked whether the driver has resumed the seated state. In step S4, the driver's internal arousal state is checked based on facial or eye behavior analysis regarding saccades, etc. In step S5, the stability of the driver's actual control status is monitored. Subsequently, in step S6, the transition from automatic driving to manual driving is completed.

[0225] [4. Example of an operation sequence for autonomous driving]

[0226] Next, refer to Fig.12 The flowchart shown in FIG. 1 illustrates an example of an operation sequence for autonomous driving.

[0227] Fig.12 The flowchart shown in is a flowchart for explaining the operation sequence of the automatic driving performed by the mobile device 100.

[0228] First, in step S11, driver authentication is performed. The driver authentication is performed by using knowledge-based authentication using a password, PIN number, etc., life authentication using a face, fingerprint, iris of the pupil, voiceprint, etc., or a combination of knowledge-based authentication and life authentication. By performing driver authentication in this way, even when multiple drivers drive the same vehicle, information for determining notification timing can be accumulated in association with each driver.

[0229] Next, in step S12, the driver operates the input unit 101 to set a destination. In this case, the driver's input operation is performed based on the display of the instrument panel.

[0230] Note that although in the example used as the case for explanation of the present embodiment, it is assumed that the trip is set after the driver gets on the vehicle, the driver can make an advance reservation remotely via a smartphone before getting on the vehicle or via a personal computer before leaving his / her home, and the like. In addition, the vehicle system can make pre-planned settings based on the driver's arrangements expected according to the schedule, and can update and acquire LDM information of the road environment, i.e., what is commonly referred to as local dynamic map (LDM) information, which is high-density and continuously updated driving map information of the roads on which the vehicle will travel, and in addition, display suggestions for actual driving like a concierge when or before the driver gets on the vehicle.

[0231] Then, in step S13, the display of the driving interval on the driving route is started. In addition to being displayed on the instrument panel, the driving interval display is also displayed on a tablet computer or the like where the driver performs secondary tasks, for example, next to the work window. This enables the driver who is working on the work window to easily identify the driver intervention required interval and the automatic driving allowed interval on the driving route along the arrival prediction time axis from the current location.

[0232] In the driving interval display, future schedules and information about each approaching location are presented. In this driving interval display, the driver intervention required interval and the automatic driving allowed interval on the driving route are displayed along the arrival prediction time axis from the current location. In addition, the driver intervention required interval includes a manual driving interval, an interval for switching from automatic driving to manual driving, and a cautious driving interval after resuming from automatic driving. The details of this driving interval display are described later.

[0233] Next, in step S14, acquisition of LDM update information is started. Together with the acquisition of the LDM update information, the content of the driving interval display can be changed to the latest state. Then, in step S15, driving is started. Then, in step S16, the display of the driving interval display is continuously updated based on the position information of the user's car and the acquired LDM update information. As a result, the driving interval display is scrolled and displayed in association with driving, as if each interval is approaching the user's vehicle.

[0234] The means of presenting information about the relatively close front driving environment associated with driving and the required timing of the transition in the relevant user vehicle, etc. is not limited to the scrolling means, and as other methods, it is preferable to present the approaching time interval in an intuitive and clear manner, and to make the driver know when she / he should start to resume driving with less misunderstanding. For example, it can be performed by a method of presenting time by using an hourglass-like image, or by a means of directly presenting the remaining time of the transition to the device worn by the user in a watch mode of a timer format.

[0235] Next, in step S17, the driver's state is monitored. Subsequently, in step S18, event change response processing is performed. The event change response processing includes mode switching processing for the case where the vehicle is approaching a switching point or a cautious driving section between the automatic driving mode and the manual driving mode that already exists on the driving route; event occurrence processing for the case where a mode switching point or a driver intervention-required section of the cautious driving section newly appears on the driving route, etc. After that, the processing of steps S16 to S18 is repeated as appropriate.

[0236] "Details of driving range display"

[0237] Fig.13An example of a driving route determined when a driver sets a destination is described. On the driving route, there are an automatic driving permission interval Sa, a manual driving interval Sb, an interval Sc for switching from automatic driving to manual driving, and a careful driving interval Sd after automatic driving. Here, there is always a switching interval Sc immediately before the manual driving interval Sb, and the driver is required to be in a posture ready to resume manual driving in the switching interval Sc. In addition, the careful driving interval Sd is an interval in which the vehicle can be driven at a reduced speed while the automatic driving is still being performed under the careful monitoring of the driver who is in a posture ready to resume manual driving. Note that these automatic driving permission intervals Sa, manual driving intervals Sb, intervals Sc for switching from automatic driving to manual driving, and careful driving intervals Sd after automatic driving are divisions for convenient use in accordance with the description of the disclosed technology, and they can be further subdivided in combination with various applications of future automatic driving, or can be divided in a simplified manner. In addition, these divisions are not uniquely defined for road environments. These are sections or lanes determined based on a combination of factors such as differences in usage regarding how they are divided into lanes, characteristics of the associated vehicles, road conditions, weather conditions, the vehicle's own autonomous driving performance due to partial functional degradation of autonomous driving performance based on the vehicle's self-diagnostic information, etc. These are not limited to uniquely defined sections, such as the sections of the present disclosure. Fig.13 A fixed interval on the map described in .

[0238] In the example shown in the figure, the automatic driving permission zone Sa is indicated in green, the manual driving zone Sb is indicated in red, and the conversion zone Sc and the cautious driving zone Sd are indicated in yellow. Note that the colors are indicated in different patterns for convenience.

[0239] On the driving interval display on the center information display or a display device such as a tablet, each interval on the driving route as described above is displayed along the arrival prediction time axis from the current location. The data processing unit, such as Figure 1 The data processing unit 11 in the structure shown in FIG. Figure 8 The data processing unit in the automatic driving control unit 112 in the configuration shown in FIG. 1 , for example, the detection unit 131, the situation analysis unit 133, the planning unit 134, and the output control unit 105 perform information generation processing for displaying the driving section on the driving route based on the driving route information and the traffic information. Under the control of the output control unit 105, the display data obtained as a result of the information processing is displayed on the display unit included in the output unit 106.

[0240] Fig.14 (a) Each section of the travel route is represented with a constant scale along the axis of travel distance from the current location. Fig.14(b) represents the average road traffic flow speed v(t) at each location. Fig.14 (c) represents each section represented along the travel distance axis after conversion into the time axis by using the speed v(t). As a result, each section on the travel route is represented along the arrival prediction time axis from the current location. That is, each section can be represented along the time axis corresponding to the quotient of the physical distance of the travel route divided by the average speed of the relevant section.

[0241] In this embodiment, all the zones in the driving zone display are divided into the following Fig.14 The three intervals shown in (d) are arranged so that the time axis of each interval is different from each other. That is, the first interval from the current location to the first location (time t0; for example, about 10 minutes) is displayed along the first time axis as a time-linear display of the nearest interval. For example, time t0 is set to a sufficient length of time required for a general driver to finish a secondary task and resume driving. The nearest interval approaching as driving provides an intuitive visual effect equivalent to a representation on a map as if the car is moving on the map at a constant speed. Therefore, its advantage is that the driver can start preparing to resume driving correctly as the event approaches, and can accurately and intuitively identify the point at which to start resuming driving to a certain extent. That is, the display purpose of this interval is to correctly provide the user with information on the start determination of the driver's correct resumption point.

[0242] In addition, as a time countdown display interval, a second interval from a first location (time t0) to a second location (time t1; for example, about 1 hour) is displayed along a time axis that gradually changes from the first time axis to the second time axis, which is reduced from the first time axis at a predetermined ratio. Regarding the display purpose of this second interval, mainly, if it is displayed with the same scale factor as the first interval mentioned above, it becomes difficult to display for a long time in a narrow display space, so this is a strategy to accurately provide the driver with road conditions for a longer period of time with a narrow display. In this way, the driver can easily recognize which location in the front interval that is approaching as the vehicle travels before the driver is required to intervene in driving, which has the advantage of enabling the driver to engage in his / her secondary tasks in a planned manner. This allows the driver to separate the secondary task from the driving intervention and maintain a balance between them, and plays a role in presenting information that is important for secondary tasks that are planned to be separated from the driver in secondary tasks accompanying communication with a third party, for example.

[0243] Here, reference Fig.14 (d) A method of setting the second display interval is described. If the height of the triangle is h0, the time t at a point at a distance h before the vertex of the triangle is determined by the following equation (1).

[0244] t = t0*h0 / h ... (1)

[0245] In addition, the second time axis at the second point (time t1) is reduced from the first time axis at a ratio of hs / h0. For example, this means that when hs=h0 / 8 holds, the reduction rate is 1 / 8.

[0246] The display of the second display interval described above is equivalent to the state of expanding the display interval by looking at the driving straight line on the map in a direction inclined relative to the traveling direction, or looking at the front of the road plane in an inclined direction when the vehicle is traveling at a constant driving speed. In other words, the visual effect of the display interval allows intuitive recognition of the distance between far and near distances through the display image height position, so that this display can be said to be a display that enables easy recognition of the perceived distance even if the scale for displaying the precise position is not displayed on the screen. In addition, the distant interval is reduced, but the distant interval is not a place that is immediately reached as a result of driving. Therefore, a rough prediction is important, but the driver does not need to intuitively recognize the arrival time information of the distant interval as strictly as for the nearby point, so that this is also suitable for making plans for the execution of secondary tasks.

[0247] In addition, along the second time axis (reduction rate hs / h0), the third section from the second location (time t1) to the third location (time t2) is displayed as a time-linear display of the distant section. By displaying the three divided sections in this way, the driver can grasp the information about the section closest in time in detail and obtain information about the section farther in time in a limited display space. Note that if the distant part is displayed while maintaining the display mode of the second section, it becomes equal to or lower than the human visual resolution, or even the display resolution limit of the system, and thus becomes unrecognizable as information required for planning determination of the secondary task, and the meaning of the display function is lost. In view of this, the most effective display is to allow the driver to intuitively fully grasp the time sense of the section, end the reduction of the display scale at the stage of appropriately displaying the necessary division between the intervention section and the unnecessary section, and display the subsequent section at a constant scale again.

[0248] Note that the vehicle control system 100 includes default values ​​for the times t0, t1, and t3. Since different values ​​for the times t0, t1, and t3 can be used for long-distance driving and short-distance driving in a possible manner, the number of each default value is not limited to 1, and the system may include multiple default values, which the driver (user) or the system may selectively use according to the driving route. In addition, the values ​​of the times t0, t1, and t3 may be arbitrarily set by the driver (user) in a possible manner.

[0249] Fig.15(a) and (b) describe examples of driving interval displays to be displayed in the end. These are straight driving interval display data extending in one direction according to the predetermined driving time period of the vehicle. Note that the arrows described in the lower part indicate the same as those in the Fig.15 The driving interval display data described in (a) and (b) are of the same length in which the driving time (unit time) of the same vehicle is equivalent. The first interval is associated with a smaller number of long arrows, and the third interval is associated with a larger number of short arrows. This means that the first interval is a shorter interval that the vehicle can pass through in a short time, while the third interval is a longer interval that the vehicle needs a longer time to pass. The second interval is associated with an arrow that becomes longer as it approaches the vehicle, which means that the driving display data of the second interval is displayed so that it changes to correspond to a distance that the vehicle needs less time to pass as the second interval approaches the vehicle.

[0250] Note that Fig.15 In the case of (a), all of the first section, the second section, and the third section are displayed with the first display width.

[0251] On the other hand, Fig.15 In the case of (b), one item of travel section display data is generated by connecting a plurality of display data items having different time axes corresponding to the scheduled travel time period of the vehicle.

[0252] That is, the first section from the current location to the first location (time t0) of the vehicle's scheduled travel time period, which is the earliest, is displayed with the first display width, the second section from the first location (time t0) to the second location (time t1) is displayed with a display width that gradually changes from the first display width to a second display width that is narrower than the first display width, and the third section from the second location (time T1) to the third location (time T2) of the vehicle's scheduled travel time period, which is the last, is displayed with the second display width. As a result, the driver can visually recognize the degree of reduction of the time axis of the second and third sections relative to the first section. That is, although Fig.14The display mode in is a display that only considers the reduction rate of the direction of travel, but by artificially changing the display width of the road according to the distance in the direction of travel of the displayed information, a visual perspective effect is obtained that is the same as the effect obtained when moving along the road or map and looking in the direction of infinity, so that the distribution of the driving intervention required intervals can be more easily and intuitively identified by only looking at the screen for a moment. In particular, if the second interval is viewed in a state where the second interval is rotated 90° counterclockwise, the display width of the road ahead becomes narrower as the distance increases. This is a display format that provides a visual effect similar to the visual effect obtained when the driver looks at the actual road in front of the vehicle, and the driver can view the display data with the same sense of distance as when the driver looks at the actual road. Therefore, this can be said to be a display mode that allows the driver to intuitively grasp the reality of arriving at various locations even without looking at the precise location memory to make a judgment, and can perform time allocation.

[0253] Note that if, in a portion where the reduction rate hs / h0 is small, such as in the third interval, intervals with shorter time lengths are directly displayed using time lengths, these intervals are displayed as very thin intervals, and it is expected that the driver will have difficulty recognizing these intervals. Therefore, even in the case where the driver intervention intervals (manual driving intervals, transition intervals, careful driving intervals) actually correspond to a time length equal to or less than a certain time length, they are displayed as corresponding to the certain time length. In this case, for example, in the case where there are continuous transition intervals and manual driving intervals, the display of the transition intervals is omitted in some cases. Fig.15 In (a) and (b), the display of the first manual driving section Sb in the third section indicates such a state. As a result, the third section with a significantly reduced time axis can be displayed so that the driver can recognize the shorter driver intervention required section.

[0254] In addition, when the manual driving sections Sb appear continuously intermittently at short intervals in a portion where the reduction rate hs / h0 is small, such as in the third section, they are all displayed as continuous manual driving sections Sb. Fig.15 In (a) and (b), the display of the second manual driving zone Sb in the third zone represents such a continuous display state. The manual driving zone Sb displayed in this way is actually as follows. Fig.15 As shown in (c), in addition to the manual driving zone Sb, a conversion zone Sd and a shorter automatic driving permission zone Sa are included. Note that, as described later, when the driving zone display is displayed on a tablet or the like, for example, by double-clicking the location, details can be displayed.

[0255] The display of the driving interval on the driving route mentioned above is updated based on the position information of the user's car and the acquired LDM update information. Thus, as time passes, the driving interval display is scrolled and displayed as if each interval approaches the car. Fig.16 (a)-(d) describe an example of changes in the display of driving intervals seen over time. Although this example describes a case where the second interval is displayed to gradually become smaller, this also applies to a case where all intervals are displayed with the same width.

[0256] In this case, the movement within each interval in the first interval is faster. In addition, the movement within each interval in the second interval becomes faster as each interval is closer to the first interval than to the third interval, because the reduction of the time axis becomes less obvious. In addition, the movement within each interval in the third interval is slower because the reduction of the time axis is significant.

[0257] Fig.17 (a) and (b) describe an example of a driving interval display 181 of driving intervals on a travel route displayed on the screen of a tablet computer 182 . Fig.17 (a) is an example in which the tablet computer 182 is used in a vertically long state. In this case, the driving range display 181 is displayed in a state of being bent from the left side to the upper side and along the upper side, and the driving range display 181 is displayed next to the work window that is the execution screen of the secondary task performed using the tablet computer 182. Fig.17 (b) is an example in which the tablet computer 182 is used in a horizontally long state. In this case, the driving interval display 181 is also displayed in a state of being bent from the left to the upper side and along the upper side, and the driving interval display 181 is displayed next to the work window, which is the execution screen of the secondary task performed using the tablet computer 182. Note that although in the example shown in the figure, the driving interval display 181 is arranged in a curved manner on the screen of the tablet computer 182, if there is sufficient arrangement space, the driving interval display 181 can also be arranged in a straight line in a possible manner.

[0258] Fig.18 An example of a state in which the driver actually performs a secondary task using the tablet computer 182 is described. In this example, the tablet computer 182 is used in a horizontally long state. On the screen of the tablet computer 182, the driving interval display 181 is displayed in a state of bending from the left to the upper side and along the upper side. Note that the driver (user) can select whether to make the driving interval display 181 appear on the screen through operation. In this case, for example, when the driving interval display 181 is not on the screen, when the driver intervention required zone is entered within a certain time and the driver is notified, the driving interval display 181 can automatically appear on the screen.

[0259] When a driver intervention required zone newly appears in the currently displayed zone while the driving zone display 181 is displayed on the screen of the tablet computer 182, a display of the newly appeared driver intervention required zone newly appears. In this case, the newly appeared driver intervention required zone is displayed as a flashing display for a certain period of time so that the driver intervention required zone and other zones can be distinguished. This flashing display may be accompanied by an alarm sound to attract the driver's attention. Here, in addition to the case where a cautious driving zone or a manual driving zone newly appears, the case where a driver intervention required zone newly appears also includes the case where a cautious driving zone becomes a manual driving zone.

[0260] Fig.19 The state in which the cautious driving zone Sd newly appears in the second zone and the driver is warned by flashing display is described. Note that in this case, when the driver touches the display portion of the cautious driving zone Sd displayed as a flashing display, the flashing, i.e., warning state can be stopped. Alternatively, when the driver touches the approval screen that is displayed as a small window pop-up when the driver touches the display portion of the cautious driving zone Sd displayed as a flashing display, the flashing, i.e., warning state can be stopped.

[0261] In addition, when the driving range display 181 is displayed on the screen of the tablet computer 182, when the driver (user) double-clicks and specifies a certain point, for example, Fig. 20 As shown in , a small window pops up and displays the information associated with the place.

[0262] In addition, if Fig.21 As shown in, in another possible configuration, the Figure 5 and 6 The watch-type information terminal 50 shown in the description is displayed with reference to Figure 14-20 The similar display using different colors to represent different intervals.

[0263] Fig.21 The example shown in is an example of displaying information with respect to a section such as an automatic driving section or a manual driving section on a road corresponding to a driving route determined when the driver sets a destination. The numerical values ​​0 to 9 around the display portion represent the elapsed time 0 to 9 minutes after the current time. The entire display area represents the section information of the driving schedule of the car from the current time (0) to 10 minutes later.

[0264] On the driving route, there are automatic driving sections Sa and manual driving sections Sb, and there are also sections Sc for switching from automatic driving to manual driving, and careful driving sections Sd set in the automatic driving section Sa, etc. The circular driving section display data extending circumferentially according to the scheduled driving time period of the vehicle is displayed.

[0265] exist Fig.21 In the display example shown in , the following three different display areas are set on the display unit of the information terminal 50.

[0266] (Sa)Automatic driving area Sa (=green display)

[0267] (Sc) Change driving zone Sc (=yellow display)

[0268] (Sb) Manual driving zone Sb (= red display)

[0269] Fig.21 The display example shown in shows the travel schedule of the car 10 minutes from now. That is, the display example shows

[0270] (Sa) The car is scheduled to travel through the autonomous driving zone Sa (= green display) from the current time to the next 6 minutes and 10 seconds, and then

[0271] (Sc) The car is scheduled to travel through the transition driving zone Sc (=yellow display) from 6 minutes and 10 seconds after the current time to 8 minutes and 40 seconds after the current time, and then

[0272] (Sb) The car is scheduled to travel through the manual driving zone (=red display) 8 minutes and 40 seconds after the current time.

[0273] Note that the information terminal 50 receives, via the communication unit, the scheduled travel time information for each section calculated by the data processing unit 11 in the car based on information about the travel speed of the car 10 or the average speed of vehicles on the travel route of the car 10, and displays the scheduled travel time information.

[0274] The driver 20 can check the time remaining before the car enters the manual driving zone by viewing the display information of the information terminal 50, and return to the driver's seat after terminating the work in the luggage compartment 21 so that he / she is ready at that time.

[0275] Note that Fig.21 In the example shown in , based on the display on the information terminal 50, the driver 20 can recognize that the car enters the manual driving zone approximately 8 minutes and 40 seconds after the current time, and continues to work in order to return to the driver's seat before then.

[0276] In addition, Fig.21 The information terminal 50 shown in FIG. 1 also displays an arrow. The arrow is a notification time corresponding to the request to resume manual driving, for example, switching to Figure 5The time of display shown in , or the time of notifying the driver 20 of a warning by performing an alarm output or a vibration start is set. This notification timing is determined, for example, on the mobile device (car 10) side, based on information about at least one of the driver's awakening level and the driver's position. The determined notification timing information is sent to the information terminal 50. The time display of the notification by using an arrow is only an example, and the means need not be limited to arrows, as long as the time interval can be visually presented by giving a means having a function of visually prompting the driver's attention using different colors, grid display, flashing, etc. Note that the process of calculating the optimal notification timing performed on the mobile device is described later.

[0277] The information terminal 50 performs the notification of the request for restoration of manual driving at the notification timing received from the mobile device. That is, at the notification timing determined by the mobile device, the information terminal 50 performs the above reference Figure 5 At least one of the display processing, alarm output and vibration activation of the description is performed. Note that the notification setting can be changed or stopped by the user.

[0278] Note that although the display portion of the information terminal 50 is described as a circular display portion in the present embodiment, this is an example, and the shape of the display portion is not limited to a circle, and may be variously set to an ellipse, a rectangle, or the like.

[0279] Fig. 22 is a diagram describing an example of changes in display data of an information terminal with the passage of time. Fig. 22 An example of display information of the information terminal 50 at time t1 and display information of the information terminal 50 at subsequent time t2 is described.

[0280] As an example of display data at time t2, display data at about 4 minutes after time t1 is described.

[0281] The display data at time t2 indicates that the car is scheduled to travel through the manual driving zone Sb (=red display) approximately 3 minutes and 50 seconds after the current time.

[0282] At time t2, the driver 20 can recognize that the driver is approaching the manual driving zone Sb (=red display) based on the display of the information terminal 50, and quickly finish the work and start preparing to return to the driver's seat.

[0283] Note that the information terminal 50 has an alarm output function or a vibration function, and when, for example, the remaining time until the manual driving zone Sb (=red display) becomes a predetermined time length (e.g., 1 minute), an alarm output or vibration activation is performed, and an alarm notification is issued to the driver 20. Note that the activation time of the alarm or vibration can be changed or stopped according to user settings.

[0284] Note that although reference Fig.21 and Fig. 22 The illustrated information terminal 50 is configured to display display data that rotates counterclockwise as time passes, but the rotation direction of the display data may be set so that, for example, Fig.23 As shown in , the displayed data rotates clockwise.

[0285] [5. Information provided to the user (driver) and notification processing to the user]

[0286] Next, information provided to the user (driver) and notification processing to the user will be described in detail.

[0287] Next, each of the items described below is explained in the following paragraphs.

[0288] 5-1. Detailed structure of travel section display data and process of updating travel section display data

[0289] 5-2. Example of display of additional information for travel section display data

[0290] 5-3. Examples of notification and warning processing using animation

[0291] 5-4. Other display processing examples

[0292] First, before explaining each of the above items, the importance of providing information to a user (driver) and a notification process to the user will be explained.

[0293] As described above, in the automatic driving permission section in which the use of automatic driving is permitted, the driver may not perform driving manipulation but may engage in secondary tasks other than driving manipulation.

[0294] However, in the case where the autonomous driving permission interval ends and the vehicle enters the manual driving interval, the driver needs to normally resume manual driving control from the secondary task before the vehicle enters the manual driving interval.

[0295] For example, the determination as to whether the driver is in a state where he / she can normally resume manual driving control from a secondary task is made by the vehicle's autonomous driving control system. However, it is not uncommon for autonomous driving control systems to have difficulty identifying the driver's state with a high degree of accuracy.

[0296] There are dangers associated with allowing the driver to resume manual driving in a state where the automated driving control system is unable to recognize the driver's proper state of alertness.

[0297] In this situation, the automatic driving control system of the vehicle starts a process of the vehicle aimed at avoiding an accident (called MRM (Minimum Risk Maneuver)), specifically, a pull-over action such as deceleration or stopping of the vehicle to pull over.

[0298] However, if many vehicles perform deceleration or stop processing based on MRM control every time the automated driving allowed section ends, there will be many decelerated or stopped vehicles at the end of the automated driving allowed section. As a result, traffic congestion occurs on the road. In addition, if the road is too narrow for vehicles to park close together, like a single-lane road, there is a possibility that the road will be completely blocked, which will cause serious negative social impacts.

[0299] As a measure to deal with such a problem of MRM, there is a measure to set up a pull-over lane before the end of the autonomous driving allowed section to allow slowed vehicles and stopped vehicles to stop. However, such infrastructure improvement requires a lot of cost to acquire land, etc., and there is not always land near the end of all autonomous driving allowed sections where parking spaces can be built.

[0300] Note that the automated driving function is still in the development stage and it will take time before fully automated driving is possible. Before then, it is expected that frequent switching between automated driving and manual driving with driver monitoring will occur. At this stage, the degree of driver disengagement is considered limited.

[0301] However, even in this situation, that is, even in a situation where frequent switching between automatic driving and manual driving is expected, it is necessary to check whether the driver is able to perform normal manual driving before entering the manual driving zone. In the case where the switching is expected to be difficult, it is required to suggest the driver to accept early abandonment, that is, prompt the user to agree to abandon the resumption of manual driving, and based on this agreement, the vehicle's automatic driving control system takes a pull-over action that does not hinder other following vehicles.

[0302] In the future, it is expected that the autonomous driving function will be improved and the autonomous driving permission range will gradually become longer. In other words, the driver, as the user of the autonomous driving, can use the autonomous driving function stably for a long time. However, it is expected that in this case, the driver will over-rely on the autonomous driving function and will not feel the need to return to manual driving from autonomous driving.

[0303] That is, it is expected that if the driver gets used to the use of the automated driving control system, and in addition, the number of requests from the system to resume manual driving gradually decreases, the driver's awareness when using automated driving decreases, and the degree of caution required to resume manual driving also gradually decreases. In addition, there is a possibility that if manual driving is not performed for a long period of time, such as several months or a year, and only automated driving is used, the driver will not even anticipate that they will have to resume manual driving.

[0304] As described above, although a notification or warning is issued in the case where the driver is required to resume manual driving, if unnecessary notifications are continuously and repeatedly issued from the system in the case where manual driving is not required or the vehicle is traveling under the driver's attention, there is a possibility that the driver undesirably learns that the notifications performed by the system are not indispensable for actual action judgment. If the driver learns such a thing, he / she will not react to the "notification", and until the driver receives a stronger stimulus such as a switching warning, there is a possibility that the switching event recognition and its response may be too late.

[0305] Therefore, it is preferable to perform notification processing that strikes a balance between not being troublesome for the driver and drawing the driver's attention.

[0306] When the driver starts the secondary task during the execution of the automatic driving, thereafter, without information about the switching point of the manual driving, the driver cannot know when the request to resume the manual driving will be issued, so the driver needs to perform the secondary task in his / her tense or uneasy state. However, the tense or uneasy mental state is temporary, and there is a possibility that the driver will gradually immerse himself / herself in the secondary task as the risk awareness decreases.

[0307] This state of meaningless tension and concentration on secondary tasks in an area close to the transition point to manual driving cannot be said to be a preferred state.

[0308] In order to prevent this situation, it is ergonomically preferable that the automatic driving control system presents information related to the manual driving transition to the driver in advance, so that the driver can prepare to start manual driving before the vehicle approaches the manual driving transition location, and start the immediate recovery process. That is, it is preferable that the driver performs the secondary task while acquiring information about the near future that can be known. By also acquiring preparatory information about the future during the execution of the secondary task, the driver can smoothly resume manual driving after receiving the notification of resuming manual driving.

[0309] Not only humans, but also many organisms that have evolved to have improved judgment of action necessary for survival have become able to prepare themselves mentally in advance so that they can start to act under a final stimulus. The nerves that make the final judgment of action are activated, and the action is started.

[0310] Many of these mental preparation processes are performed unconsciously. Many organisms have enhanced the sense of identifying targets to pay close attention to from various information, such as environmental sounds, smells, wind direction, movement of clouds in the sky, cold, behavior of surrounding animals, etc., and take action. In a similar way, in order to make a final action judgment faster, it is important to obtain supplementary information to make decisions more multidimensionally. In particular, visual and auditory information are effective. Humans use different memories for different types of information to make judgments on various risks and make final action judgments. In addition, humans usually check the possibility of information used for risk judgment. This processing also provides the effect of reducing negligence and response delays.

[0311] Examples of how humans use future predictions to make action decisions include, for example, examples of how humans rely on prophecies to make action decisions. In action decisions, it is important to maintain a balance of risks, and by assuming a psychological attitude that gives weight to factors that have a greater impact on the judgment and using that psychological attitude to make judgments, it is possible to make quick and correct action decisions for the situation at the time.

[0312] With the introduction of automated driving, if the user is completely focused on secondary tasks during automated driving, cut off from any future prediction information, it will be difficult for the driver to follow the correct action judgment criteria when the driver resumes manual driving. As a result, even if the driver eventually receives the notification to resume manual driving, they will not be able to make correct judgments about the situation and may even suffer from panic disorder.

[0313] Regarding Level 3 automated driving as defined by the SAE (Society of Automotive Engineers), the driver is required to always be in a state where they can resume manual driving. However, having to be in a tense state for a long time where the driver keeps his / her hands on the steering wheel just to wait for the resumption of manual driving, and actually neither focuses on other secondary tasks nor on driving, can be considered a torture.

[0314] In view of this, it is preferable to provide a system that continuously provides information to the driver in advance and allows the driver to check the timing when the driver needs to resume manual driving. For example, in a preferable mode, information that allows prediction of the switching point from the automatic driving allowed section to the manual driving section is provided as appropriate, and the driver can receive this information while leisurely engaging in secondary tasks.

[0315] However, it should be noted that even if the time remaining before the end of the interval where autonomous driving is allowed and the end of the interval where autonomous driving is allowed with the driver's attention are uniformly displayed as information to assist the driver in making action decisions to resume manual driving, this cannot necessarily be said to be an effective measure for the driver to be fully mentally prepared.

[0316] This is because the psychology of human action judgment is the result of various risk balances. Humans use information about the status of each of many combined factors, the accumulation of risk balance stimuli, etc. to judge the best action. For these judgments, in some cases, humans also conduct detailed checks and re-checks based on initial information. If the risk is approaching, then a set of various neural synapses triggered by the judgment for risk aversion will be set to their more sensitive standby state.

[0317] Fig.24 is a diagram used to illustrate the concept of neurotransmission in which combined / layered information provides an enhanced level of regression consciousness.

[0318] In other words, the driver makes a judgment on the best action based on a balance between risk information unique to the driver and predicted outcomes about the future that will accompany the selected action.

[0319] However, since such action judgments based on various types of information are not made consciously in various daily action judgments, humans do not feel that various types of information are helpful for action judgments.

[0320] In order to allow the driver to make a favorable transition from automated driving to manual driving when a switch occurs, predictive information related to the actual driving action before the driver begins to perform the action plays an important role in a smooth transition.

[0321] For example, on a highway, if a vehicle does not exit the highway at an highway entrance / exit (IC) at appropriate timing, the vehicle cannot exit the highway until the next IC. If the vehicle cannot exit at appropriate timing, this will result in unnecessary highway travel and create the risk of a detour.

[0322] In order to reduce such risk, it is preferred to move the vehicle to the best lane for leaving the highway some time before the exit rather than just before the exit, and prepare to exit at the exit while checking the deceleration start point before the exit by checking the exit signs, etc.

[0323] However, if the advance information required for taking such an optimal action is insufficient, then it actually becomes too late to make the action decision to take the exit, resulting in the inability to take the exit and inevitably having to continue driving on the highway.

[0324] Note that people who have made wrong judgments in the past and failed to exit at the appropriate time should accumulate information necessary for making the best judgment by checking exit signs in advance, etc., in order to make the right judgment earlier the next time they use the expressway.

[0325] The information that allows the driver to make his own decision to end the secondary task and attempt to resume manual driving during the execution of the secondary task during autonomous driving includes the timing of event occurrence, risk information for each selection information, and many other information that affects the action.

[0326] Each of these information becomes an additional neural stimulus that prompts the driver to make individual decisions hierarchically, and the sum of these various neural stimulations results in the firing of neurons that control the decision. The best action decision is then executed.

[0327] From the perspective of ergonomics, an action decision is made based on the balance of many unconscious risks. If there is a large amount of foreseeable advance risk information, then a more appropriate action decision can be made based on the large amount of information. In addition, the weighting of risks is also an important factor affecting the action decision.

[0328] Human action judgments are made unconsciously, not only regarding judgments along a simple timeline, but also regarding judgments about the risks of taking action and responding to a situation, not taking action and not responding to a situation, and other similar situations when there are multiple action options.

[0329] Thus, in order to allow the driver to make appropriate action decisions, it is important to provide information to the driver by using an interface that presents various risks to the driver, that is, by using an HMI (Human Machine Interface).

[0330] However, ideally, these types of predictive judgment information are not unified information, but information corresponding to each driver individually. For this reason, in a preferred configuration, individual identification of the driver is performed, and after converting the arrival time based on the predicted driving speed in each road section, a recovery notification or warning point corresponding to the length of time required to safely resume manual driving for each road section in which the vehicle is traveling, avoidance selection information, etc., is presented to each driver in accordance with the characteristics of each individual required for safe driving of the vehicle being driven.

[0331] By using the HMI as an interface for such information presentation, it is possible to allow the driver to appropriately determine the secondary tasks that can be performed during autonomous driving, and also to provide the driver with the information necessary for risk judgment at each point during autonomous driving.

[0332] The information provided as the vehicle approaches the manual driving zone serves as information for the driver to prepare for a cautious state of mind required for action judgment, and is expected to cause the driver to make appropriate switching responses individually for each situation and to achieve an appropriate balance.

[0333] One of the important information provided to the driver is the selection information for each risk of the response required at each manual driving transition point. Depending on the response performed by the driver, the system sometimes has to perform an emergency stop or emergency pull-over operation of the vehicle. In this case, the passage of subsequent traffic will be hindered.

[0334] In addition, depending on the response performed by the driver, various levels of situations may occur in the system corresponding to situations such as changing the route to a lower speed route that deviates from the planned route, pulling over at a rest area, or imposing a fine.

[0335] If the driver understands the various risks based on the information provided to the driver, the driver can take the correct preparatory actions to avoid conversion failure.

[0336] Note that the preparatory action is not necessarily limited to physical action, and may be a conscious preparation required for the transition. The transition point from the automatic driving use section to the manual driving section may change, for example, depending on the road conditions, the driving dynamics of the user's vehicle, etc. In addition, the timing of the manual driving resumption request notification to the driver and the risk at the time of resumption may also change. Therefore, the driver is required to respond to various situations according to judgment criteria that vary depending on the situation.

[0337] As information required for consciously preparing action judgments, the information obtained by acquiring prediction information is important. However, it should be noted that, for example, in the case where the road information or the like provided by the system is information lacking in credibility, the prediction information loses its meaning.

[0338] In view of this, there is a need to effectively provide dynamic information based on information on the road and traffic ahead of the travel route obtained from the LDM or a group of leading vehicles for each section in a form suitable for the driver as appropriate.

[0339] Humans use many sensory organs to obtain information about their surroundings. However, if humans try to send all the information input from all these sensory organs simultaneously and in parallel to the brain without weighing it, and process it, information overload will immediately occur, and humans will panic and even be unable to transmit information through nerves.

[0340] To prevent this, humans have evolved the ability to selectively filter information as a self-defense function as a way to prevent delays in action decisions caused by information overload.

[0341] Thus, for example, if much of the information provided from the system is information that lacks credibility when using a vehicle, the provided information is classified as unnecessary information that confuses judgment, even if the information is important as a judgment factor of action risk. That is, the information becomes the cleanup target information for filtering, and the information provision loses its meaning.

[0342] Thus, the transition-related information to be provided to the driver is preferably provided together with additional symbol-based information that is simple, does not require thoughtful interpretation, and is associated with risk-action decisions.

[0343] In addition, as the timing to make a decision becomes closer, the details of the switching action become more and more important. Thus, for example, if all information is presented in the planning stage of determining the driving schedule before starting driving, the driver will also be confused because he cannot keep up with the thinking.

[0344] Therefore, it is preferred to provide information in the form of providing relevant information of a few minutes to tens of minutes during driving at any time. Relevant information beyond this time range is presented in a simplified form. For example, only the information of each interval and the summary information of the importance of the conversion are presented in a scaled and compressed manner. Note that in another possible configuration, a temporary enlarged display or detailed display can be displayed according to the input of the driver as needed.

[0345] In addition, in the case where the automatic driving allowed section, the manual driving section, and the switching point between sections are displayed as display information, it is preferable to present index information serving as a milestone on the map together with displaying each point on a scale of arrival time calculated by simply dividing the road average traffic speed of each road section. Through such display processing, it is easier for the driver to associate the information with the memory information, which is effective for enhancing the awareness of the necessity of switching in the driver's consciousness. In particular, if the vehicle is traveling on a familiar road, even if the importance of the switching required at each point varies depending on the condition of traffic congestion on the road, the memory information complements each other. Thereby, the level of awareness about the time / point / place where the switching is required can be kept high.

[0346] As described above, in order to successfully perform the transition from automatic driving to manual driving with a high probability, the driver's transition preparation awareness needs to be raised to the level required for manual driving in advance before the point where the transition is required, that is, before the end of the automatic driving use permission zone.

[0347] In order to successfully perform the transition with a high probability, it is necessary to effectively prepare the driver to start manual driving before reaching the transition start point. For this purpose, for example, as an HMI technology, a technology such as providing symbol information that enables action risk judgment along the arrival time axis is effective.

[0348] In summary, when the driver uses the vehicle in the automatic driving mode and performs a secondary task that can be performed during this time, it is effective to perform processing based on the following information as a process to appropriately generate the driver's return awareness and maintain the awareness.

[0349] (Information 1) Driver-specific recovery delay time information obtained through self-learning of driver-specific status required for recovery (recovery time, actions to confirm recovery notification, actual recovery behavior / action history, etc.)

[0350] (Information 2) Information on appropriate recovery locations affected by road section-specific characteristics (expected conversion success rate (RRR: request recovery rate), risk level of turns, etc., avoidance options, penalties when avoidance is unavoidable, etc.) and vehicle-specific characteristics

[0351] (Message 3) Select message in case of irreversibility

[0352] Based on these types of information, the timing of the manual resume notification to the driver is determined.

[0353] In addition, if the driver does not initiate appropriate actions to resume manual driving in response to the resumption notification, if the initiation or its process is late, or if the system performs preventive actions such as deceleration and accident avoidance due to poor resumption quality, a penalty is generated and recorded for the driver.

[0354] In addition, multiple layers of information to which risk can be associated are presented, which are information on the selection of recovery judgment actions that the driver can take before the end of the automatic driving use interval. Through this information presentation processing, the driver can be provided with knowledge information required for risk judgment for different recovery actions. That is, the driver is not provided with sudden transition notifications, but information on different risk levels is provided at any time.

[0355] In addition, together with the above information, information about the work risks that occur when continuing the secondary task is also presented to enhance the driver's awareness of returning as early as possible. If the secondary task is continued, the quality of the resumed manual driving after the conversion notification will be affected. Therefore, the work risks that occur when continuing the secondary task are intentionally provided. For example, in the case where the driver forcibly continues the secondary task performed using an information input device such as a tablet terminal, visual interference with the work data being executed or loss of information being processed occurs. By providing such information associated with a penalty, the driver makes an early risk judgment to avoid the loss of information, and the habit of improving awareness that prioritizes recovery can be gradually cultivated in the driver's mind.

[0356] By presenting information that intuitively links an operation that degrades the quality of resumption (e.g., resumption delay of manual driving) with a penalty imposed by the influence thereof in this manner, the user (driver) of the automated driving can consciously acquire information about the start of manual driving even at a stage where he / she does not receive a direct notification of a request to resume manual driving. As a result, instead of starting the preparation for switching for the first time after receiving the notification of a request to switch to manual driving, the driver can acquire priority information about an action judgment for the notification in advance based on selection information predicted in advance, make an action judgment, and start preparations for switching to manual driving earlier.

[0357] In addition, humans reconfirm the risk according to the severity of the advance information that the driver needs to pay attention to, for information that may be a risk based on experience and advance information up to that point in time. If something suddenly happens when a person has no advance information at all, the brain cannot make an immediate judgment and can only take simple reflexive avoidance actions. In view of this, in order to avoid such sudden information input, humans habitually reconfirm the information that may be approaching in daily life actions appropriately and regularly, and make action predictions. That is, whether a person obtains the information required to predict the upcoming risk depends on the frequency and confirmation method of reconfirming the information according to the necessity of advance information caused by the risk. Although information such as LDM obtained from the environment of the road on which the vehicle is traveling itself includes information that affects risk judgment, by further giving a penalty to the secondary task work content as a risk caused by human intervention when the response to the switch to manual driving is delayed, the necessity of preconfirmation can be virtually generated. In this way, by actively providing a variety of risk information to the driver, the driver reconfirms the risk as the situation that requires manual driving to be restored approaches, and the quality of recovery caused by recovery delays can be prevented, and a safer recovery can be achieved.

[0358] By combining consideration of road conditions, the driver's state, recovery capabilities, the vehicle's dynamic behavior, etc., and further providing applicable risk action judgment information according to the driver's unique recovery characteristics, the driver can make appropriate risk judgments and make appropriate action judgments.

[0359] From a micro perspective, the advance information and penalty information provided as predictive information have the effect of lowering the threshold for starting actions to return to manual driving when the driver makes information judgments in his brain, thereby enabling the driver to make necessary response preparations in advance.

[0360] For example, by providing advance information that allows the driver to easily predict the time remaining before starting manual driving, the driver can also keep the level of awareness of resuming manual driving at a high level during the execution of the secondary task. In addition, by using the penalty function in combination, the user begins to develop a habit of paying attention to confirming the provided information. As a result, the driver is expected to voluntarily recognize the situation even if he does not receive a system notification. That is, the driver's cautious psychological cycle is expected to be continuously activated, thereby avoiding delays in response to neglect or switching.

[0361] Note that the HMI technology according to the present disclosure provides the driver with hierarchical information about the various options associated with the transition location required for resumption. Note that after a manual resumption notification, if the driver does not resume manual driving appropriately, a penalty is imposed to encourage the driver to improve the quality of resumption.

[0362] Note that in order for the penalty to work effectively, the system needs to correctly identify the quality of recovery in response to the notification and quantitatively assess the quality of recovery.

[0363] As such evaluation processing, for example, with the driver sitting in the driver's seat and in a posture in which he / she can drive, state recognition is performed, for example, from the driver's awake state to the line of sight before the driver normally performs actual manual driving manipulation.

[0364] In addition, in the case where the driver's seating posture is not a posture in which he / she can perform normal manual driving, restoration of the normal posture or tracking of the posture is detected using a 2D camera or a device capable of three-dimensional posture detection, and the posture is evaluated.

[0365] In addition, in the case where the driver leaves the seat, in the same manner, for the action of recovering from the state where the driver leaves the seat, a 2D camera and a tracking device capable of three-dimensional posture detection are used to detect the transition of posture recovery, measure the predicted time before recovery, and evaluate the delayed recovery relative to the time of good recovery.

[0366] In this way, according to the driver's condition, the tracking information including the recovery action is observed, and whether the recovery quality is low is evaluated. Note that in another possible configuration, in order to quantitatively evaluate the recovery quality, each driver can be identified as an individual, the driver's unique recovery characteristics can be learned, and the evaluation process can be performed by using the learned data.

[0367] For example, the learned data for each individual is compared with the observed data of the recovery action at each event, for example, it is determined whether the observed data indicates a delay compared to the prediction based on the learned data, and the recovery quality is evaluated according to the characteristics of the individual.

[0368] Note that the content of the secondary tasks of the autonomous driving user is diverse, and it is difficult to unconditionally and uniquely define the recovery transition, which is preferably evaluated by using individual-specific learning data. In particular, the technology that tracks the driver's recovery actions when he / she leaves the seat requires a certain degree of flexibility.

[0369] Delays in resuming normal driving have a significant impact. Therefore, the system makes a request notification for resuming manual driving in advance, taking into account the predicted time required for resumption. In addition, the delay time before the actual resumption time is also considered as an evaluation target.

[0370] In the event that the driver does not take immediate recovery action after notification, and the start of recovery is delayed, for example because he / she did not pause but continued with a secondary task, etc., the driver is expected to attempt to perform some or all recovery actions quickly in order to minimize penalties.

[0371] This hasty action to make up for the delay can become the cause of an accident.

[0372] The appropriateness of such recovery actions also needs to be evaluated. When tracking the driver's actions, for example, the system uses an observation device such as a ToF (Time of Flight) sensor that enables three-dimensional observation of the speed of an object's operation to observe the driver's actions. The system self-learns the action history and generates a learning dictionary for recovery actions.

[0373] Actions taken hastily when the driver resumed manual driving in a delayed situation after notification of resumption of manual driving were compared with actions taken quickly and calmly after notification, and resumption quality was quantitatively recorded.

[0374] For quality evaluation in the case where the driver resumes manual driving after he / she leaves the driver's seat, for example, a belt or clock-type device worn by the driver can be used. For example, information acquired by a device having an in-vehicle position measuring device capable of identifying the in-vehicle position of the driver is used to acquire and evaluate information about the driver's movement in the vehicle.

[0375] In addition, as a detection point of the recovery quality, a seating sensor such as a driver's seat can be used to detect the fastening of a seat belt or the gripping of a steering wheel, or a recovery milestone (turning point) such as a pedal contact.

[0376] In addition, the action assessment information that can be used includes whether and how the driver gets up from a bed such as a nap space away from the driver's seat; whether and how the driver gets up from the driver's seat, etc.; whether and how the driver gets up from the non-driver's seat, etc.; whether and how the driver moves from one seat to another; whether and how the driver walks; whether and how the driver turns the body or seat, or slides the body or seat sideways; whether and how the driver pauses work; whether and how a change in foot placement to a manual driving mode is detected; whether and how the driver takes a visual action to move his or her eyes to a driving information presentation device; whether and how a specified confirmation behavior is detected; and so on.

[0377] Note that as an additional benefit of processing the history obtained when the driver performs impulse recovery by converting the history into a recovery quality evaluation value, it is no longer necessary to retain the user information as simple captured image data of the user, and it can be recorded without privacy-related recording restrictions. As a result, it becomes easier to operate and manage inappropriate control actions, etc.

[0378] [5-1. Detailed Structure of Traveling Section Display Data and Processing for Updating Traveling Section Display Data]

[0379] Next, the detailed structure of the travel segment display data and the process of updating the travel segment display data will be described.

[0380] As described above, in the autonomous driving permission section where the use of autonomous driving is permitted, the driver may not be performing driving maneuvers but may be engaged in secondary tasks other than driving maneuvers.

[0381] However, when the autonomous driving permission range ends and the vehicle enters the manual driving range, the driver needs to normally resume manual driving control from the secondary task.

[0382] It is necessary to reliably notify the driver that the vehicle is approaching the manual driving zone.

[0383] For example, on an information processing device such as a tablet terminal while the driver is performing a secondary task, a display such as the reference Figure 14-16The driving interval display data includes, for example, driving interval display data including different colors for different intervals along the time axis, such as green for the automatic driving allowed interval Sa, red for the manual driving interval Sb, and yellow for the conversion interval Sc and the careful driving interval Sd.

[0384] For example, in the data processing department, such as Figure 1 The data processing unit 11 in the structure shown in FIG. Figure 8 The data processing units in the automatic driving control unit 112 in the configuration shown in the figure, such as the detection unit 131, the situation analysis unit 133, and the planning unit 134, and the output control unit 105, perform acquisition of data required for displaying these data and display data generation processing. The display data is sent to a tablet terminal or the like used by the user via the communication unit 103, for example, and is displayed thereon. Note that the display data can also be displayed on the display unit included in the output unit 106 under the control of the output control unit 105.

[0385] As previously referenced Fig.14 (d) Fig.15 and Fig.16 For example, the driving interval display data is displayed in three display intervals, such as Fig.14 (d), and make the time axis of each interval different from each other. Fig.25 The driving interval display data 70 is shown in the middle portion of the figure. Although in the following display example, a device such as a tablet computer directly operated by the driver is mainly described, the information can also be directly displayed on an in-vehicle device such as a center console panel of the vehicle. In this case, the information can be actually displayed based on the relative position relationship of the driver, taking into account the visibility and inclination of the display.

[0386] Specifically, display processing such as that described later is performed.

[0387] The purpose of the display examples in this article is to provide information as a notification of the time to arrive at various locations. In the case of displaying a display corresponding to the time to arrive at various locations, the time to arrive at each location is calculated by dividing the moving distance by the driving speed along the route. The time to arrive at each location needs to be replaced by a value obtained by converting from the driving speed formed by the traffic flow that varies depending on the driving conditions of each road section. However, it should be noted that since the content of describing the predicted arrival notification time based on the current time is relatively complicated, the conversion is omitted in the description of the following processing, and the display processing is instead described in detail taking into account that when the vehicle is traveling at a constant fixed speed, the arrival time is proportional to or equal to the "distance" traveled by the vehicle.

[0388] A close distance interval adjacent to the user's car position is displayed along the first time axis as a time linear display of the nearest interval in which time (t) and position (h) are displayed in a proportional relationship.

[0389] Similar to the near distance section, the far distance section at a distance is displayed along the second time axis as a time linear display of the far distance section in which the time (t) and the display position (h) are displayed in a proportional relationship.

[0390] However, it should be noted that the long-distance interval is set so that the time per unit display length is longer than that of the short-distance interval, that is, the longer the "distance per unit display length / elapsed time". For example, it is set so that when the display length of 1 cm in the short-distance interval corresponds to the length of time required to travel 1 km, the display length of 1 cm in the long-distance interval corresponds to the length of time required to travel 5 km, etc.

[0391] The interval represented by the first time axis represents the travel distance when the vehicle is traveling at a constant speed as viewed from a high altitude, and the interval represented by the third time axis represents the travel distance when the vehicle is traveling at a constant speed as viewed from a point far from the interval. Therefore, this is equivalent to a display zoomed out from the display along the first time axis.

[0392] Since the section displayed on the second time axis is displayed as viewed from above in an oblique direction, the display shrinks as the distance increases in the infinite distance direction, which allows the driver to intuitively recognize the long and short distances relative to the section.

[0393] As the time countdown display section, a middle distance section between the short distance section and the long distance section is displayed along a time axis gradually changing at a predetermined ratio from a first time axis corresponding to the short distance section to a second time axis corresponding to the long distance section.

[0394] See also Fig.25 , indicating that Fig.25 The details of the generation and update process of the travel section display data 70 shown in FIG.

[0395] Fig.25 The right side of is a graph (time / display position correspondence graph) in which the current time to the predicted arrival time (t:min) is set along the horizontal axis and the display position (h:mm) of the travel interval display data 80 in the display is set along the vertical axis. The display position (h:mm) along the vertical axis corresponds to the travel interval display data 70 shown in the middle.

[0396] like Fig.25 As shown in FIG. , the driving interval display data 70 includes three intervals:

[0397] Close range;

[0398] Middle distance interval; and

[0399] Long distance interval.

[0400] The short-distance interval is an interval close to the position of the user's car, and is displayed as a time linear display of the nearest interval along the first time axis.

[0401] The section from the current position (O) to (P) on the graph shown on the right is equivalent to the short distance section. This section is a section in which the display time is displayed linearly, that is, the display position is determined in proportion to the passage of time.

[0402] As shown in the time / display position correspondence graph, in the short distance zone, the display position h is set in proportion to the time t. That is, the display position h of each driving zone (Sa to Sd) in the travel zone display data 70 is determined according to the following (Formula 1).

[0403] h=A×t...(Formula 1)

[0404] In the above (Formula 1), A is a coefficient that determines the slope of the straight line in the section OP in the time / display position correspondence graph.

[0405] Note that the slope A has a pre-assigned value.

[0406] Furthermore, a branch point P between the short distance section and the middle distance section, and a branch point Q between the middle distance section and the long distance section are designated in advance.

[0407] For example, the short distance interval (O-P) is from the current time (time = 0) to the time = (t ref ) period. The vehicle is in this period (t = 0 ~ t ref ) is set as a short-distance section (h = 0 ~ h ref ). Note that this processing is performed under the assumption that the vehicle is traveling at a constant speed.

[0408] h ref =A×t ref

[0409] The middle distance section (P to Q) before the branch point P between the short distance section and the middle distance section is a time countdown display section, and a process is performed to display the distance represented by the predetermined display height of the short distance section while gradually increasing the distance from P to Q. The display of the middle distance section is equivalent to the situation of seeing the road plane extending straight ahead, or the situation of seeing the road plane of a straight road on a map tilted in the direction of advance, which is equivalent to the display that the distance decreases toward the infinite point.

[0410] exist Fig.25The two dotted lines L1 and L2 shown in the travel section display data shown in the center are lines drawn so that they converge from the branch point P between the short-distance section and the medium-distance section toward a distant point. The convergence point is defined as a virtual infinity point node I.

[0411] The display position (display height) h of the virtual infinity point node I in the travel section display data 70 is defined as h=h0.

[0412] At this time, the relationship between the display position h of each section (Sa to Sd) in the middle distance section (P to Q) and the time t elapsed from the current time (t=0) can be expressed by the following (Formula 2a).

[0413] h-h0=-B×(1 / t)...(Formula 2a)

[0414] Note that B is a coefficient and is related to Fig.25 The slopes of the lines L1 and L2 shown in FIG. 1 correspond to the values. However, it should be noted that in order to smoothly and continuously connect the short-distance interval (O to P) and the medium-distance interval (P to Q) at the branch point P between the short-distance interval and the medium-distance interval, the coefficient B and the coefficient A need to be set so as to satisfy the conditional expression described later.

[0415] From Fig.25 It can be seen from the graph shown on the right side that in the medium distance interval (P~Q), the line of the time / display position corresponding graph is a curve in which the change of the display position (h) gradually decreases with the passage of time (t).

[0416] The slope at each position of the curve, that is, (dh / dt) can be calculated by differentiating the time t based on the above (Formula 2a) and is expressed by the following (Formula 2b).

[0417] (dh / dt) = B × (1 / t) 2 ...(Formula 2b)

[0418] Note that in order to smoothly connect the short-distance interval (O to P) and the medium-distance interval (P to Q) at the branch point P between the short-distance interval and the medium-distance interval,

[0419] In the time / display position relationship expression (th relationship expression) of the short distance interval represented by the above (Formula 1), that is,

[0420] h=A×t...(Formula 1)

[0421] and the time / display position relationship expression (th relationship expression) of the middle distance interval represented by the above (Formula 2a), that is,

[0422] h-h0=-B×(1 / t)...(Formula 2a)

[0423] In these two time / display position relationship expressions (th relationship expressions), at time t ref The timing, that is, the position h and slope of the timing at the branch point P need to match.

[0424] First, the slope at the branch point P is checked.

[0425] The following two values ​​need to match

[0426] The slope of the short distance interval represented by the above (Equation 1) = A, and

[0427] The slope calculated according to the above (Equation 2b)

[0428] (dh / dt) = B × (1 / t) 2 ...(Formula 2b).

[0429] Since the time t of the branch point P between the short-distance interval and the medium-distance interval is

[0430] t=t ref

[0431] As a condition for slope matching, it is necessary to satisfy

[0432] B×(1 / t ref ) 2 =A...(conditional expression a)

[0433] In addition, regarding the display position h of the branch point P between the short distance interval and the medium distance interval, the position h calculated according to the time / display position relationship expression (th relationship expression) of the short distance interval (O~P) and the position h calculated according to the time / display position relationship expression (th relationship expression) of the medium distance interval (P~Q) need to match.

[0434] The branch point P represented by the above (Equation 1) at time t ref The display position h is

[0435] h=A×t ref

[0436] At the same time, the branch point P represented by the above (Equation 2a) is at time t ref The display position h is

[0437] h=-B×(1 / t ref )+h0

[0438] In order for them to match,

[0439] A×tref =-B×(1 / t ref )+h0...(conditional expression b)

[0440] If these two conditional expressions a and b are satisfied, the short distance section (O to P) and the middle distance section (P to Q) are smoothly connected at the branch point P between the short distance section and the middle distance section. The coefficients A and B are set to values ​​satisfying the two conditional expressions a and b.

[0441] Notice, Fig.25 The middle distance interval shown in , that is, the inclined display interval ends before the virtual infinity point node I (ends at Q), and then, as a long distance interval (Q and beyond), the time linear display is displayed, that is, the display of the proportional relationship between time and display position. However, it should be noted that Fig.25 As can be understood from the graph shown in , the long-distance section (Q and beyond) is set so that the increase in the display position per unit time is small.

[0442] That is, in the long distance section (Q and longer) of the travel section display data 70, data for a long time is displayed in a shorter display area, that is, data corresponding to a longer distance is displayed in a shorter display area.

[0443] The long distance interval (Q and beyond) starts at a predetermined position (R) measured from the displayed position (h0) of the virtual infinity point node I.

[0444] R is set to a value that satisfies conditional expressions c and d described later.

[0445] As in Fig.25 As shown in the graph, the time (t R / H ) can be expressed in the following way.

[0446] t R / H =(H / R)×t ref

[0447] Note that in order to smoothly connect the medium-distance interval (P~Q) and the long-distance interval (Q and beyond) at point Q, it is necessary to match the display position (h) and slope (dh / dt) of the medium-distance interval (P~Q) and the long-distance interval (Q and beyond) at point Q.

[0448] First, the slope (dh / dt) of the long distance section (Q and beyond) needs to be set to the slope (dh / dt) of the branch point Q between the medium distance section (P to Q) and the long distance section (Q and beyond).

[0449] The slope (dh / dt) of the mid-range interval (P to Q) is calculated by the above definition (Equation 2b), that is,

[0450] (dh / dt) = B × (1 / t) 2 ...(Formula 2b)

[0451] The time t of the branch point Q is, that is,

[0452] t R / H =(H / R)×t ref

[0453] Substitute the value of into equation (2b).

[0454] (dh / dt) = B × (1 / ((H / R) × t ref )) 2 ...(Formula 3a)

[0455] The slope (dh / dt) in the long-distance interval (Q and beyond) is the slope according to the above (Equation 3a).

[0456] Note that by based on the (conditional expression a) described above, that is,

[0457] B×(1 / t ref ) 2 =A...(conditional expression a),

[0458] By transforming the above (Formula 3a), the slope (dh / dt) in the long-distance section (Q and beyond) is defined by the following (Formula 3b).

[0459] (dh / dt) = A × (H / R) 2 ...(Formula 3b)

[0460] Note that H = h ref =A×t ref Established.

[0461] In addition, in order to satisfy the condition of matching the display position (h) in the middle distance section (P to Q) and the long distance section (Q and beyond) at point Q, the following (conditional expression c) needs to be satisfied.

[0462] A×(R / H) 2 ×(t-(H / R)×t ref )+A×t ref ×(2-(R / H))=A(R / H) 2 ×t+2×A×t ref ×(2-(R / H))...(conditional expression c)

[0463] By setting R so that the conditional expression c is satisfied, the medium distance section (P to Q) and the long distance section (Q and beyond) can be smoothly connected.

[0464] Next, refer to Fig.26 , illustrating the display width W of each section of the travel section display data 70. The display width of the short-distance section (O to P) and the long-distance section (Q and beyond) is equivalent to a display as a bird's-eye view of the road width from a high altitude, and the display width of the medium-distance section (P to Q) is equivalent to the relevant road width from a road width having a certain width represented by parallel lines, converging to a point toward an infinite point according to the long and short distances. Together with the reduction of the display in the vertical direction, this provides an effect that allows the driver to intuitively sense the arrival time.

[0465] First, the display width W of the travel section display data 70 in the short distance section (O to P) is a width W defined by the following (Formula 4).

[0466] W=W ref ...(Formula 4)

[0467] Display width W ref Can be set as needed.

[0468] The display width W from the branch point P between the short distance section (O to P) and the medium distance section (P to Q) to the medium distance section (P to Q) is set so that it becomes narrower as the distance from the current position increases.

[0469] Specifically, the display width W of the middle distance section (P to Q) is a width W defined by the following (Formula 5).

[0470] W=W ref ×(t ref / t)...(Formula 5)

[0471] Note that t ref It is the length of time required for the vehicle to reach the branch point P between the short-distance section (O~P) and the medium-distance section (P~Q).

[0472] At time t ref The display width W of the branch point P between the short distance interval (O~P) and the medium distance interval (P~Q) is

[0473] W=W ref

[0474] Thereafter, in the middle distance section (P to Q), the display width W gradually decreases as time passes according to the above (Formula 5).

[0475] As mentioned above, the time t required for the vehicle to reach the branch point Q between the medium-distance section (P~Q) and the long-distance section (Q and beyond) is

[0476] t=t R / H=(H / R)×t ref

[0477] The display width at this position can be calculated according to the following formula based on the above (Formula 5).

[0478] W=W ref ×(t ref / ((H / R)×t ref ))

[0479] =W ref ×(R / H)

[0480] The display width W calculated according to the above formula is equivalent to the display width W of the long-distance interval (Q and beyond) R / H That is, the display width W of the long-distance interval (Q and beyond) R / H It is expressed by the following (Formula 6).

[0481] W R / H =W ref ×(R / H)...(Formula 6)

[0482] Note that based on the above (Formula 6), the time / display position relationship expression (th relationship expression) of the middle distance interval described above, that is,

[0483] h-h0=-B×(1 / t)...(Formula 2a),

[0484] And the definition of the display width W of the middle distance interval (P~Q) (Formula 5), ​​that is,

[0485] W=W ref ×(t ref / t)...(Formula 5)

[0486] The following relational expression (Formula 7) can be obtained.

[0487] W=W ref ×(t ref / t)

[0488] =-(W ref ×(t ref / B)×(h-h0)...(Formula 7)

[0489] The above (Formula 7) proves the following relationship.

[0490] The short-distance interval (O to P) and long-distance interval (Q and beyond), which are time-linear display intervals in which time (t) and display position (h) are displayed in a proportional relationship, have a display width displayed in a linearly reduced relationship with respect to the display position h, that is,

[0491] Display width W of the short distance range (O~P) ref ,and

[0492] Display width W for long-distance range (Q and beyond) R / H .

[0493] By displaying the travel section display data 70 in such a display mode, the user (driver) can intuitively recognize that each section is approaching as the vehicle travels without any sense of disobedience due to the combined effect of the visual perspective of the width.

[0494] Reference below Fig. 27 , an example of a process of updating the travel section display data 70 as time passes will be described.

[0495] Fig. 27 The change in the display position of the display section of the travel section display data 70 over time is described.

[0496] The five travel section display data 70 at time t1 to t5 are five data displayed at a constant time interval after t1.

[0497] Note that, in order to make it easier to understand, only one section data is shown in each of the short distance section, the middle distance section, and the long distance section.

[0498] The section data (Sa to Sd) displayed in each of the short distance section, the middle distance section, and the long distance section is updated to approach the vehicle as time passes. Note that it is assumed that the vehicle travels at a constant speed.

[0499] The short-distance interval is an interval in which time is linearly displayed, that is, an interval in which the display position is determined in proportion to the passage of time. Fig.25 As described above, the display position h of the section data (Sa to Sd) displayed in the short distance section is determined according to the following (Formula 1).

[0500] h=A×t...(Formula 1)

[0501] Therefore, the slope (dh / dt) of the dotted line showing the time transition of the interval data (Sa to Sd) in the short distance interval shown in the figure is

[0502] (dh / dt) = A

[0503] The long-distance interval is also an interval in which time is linearly displayed, that is, an interval in which the display position is determined in proportion to the passage of time. Fig.25 As described above, the slope (dh / dt) of the dotted line indicating the time transition of the section data (Sa to Sd) of the long-distance section is represented by the following equation (3b).

[0504] (dh / dt) = A × (H / R) 2 ...(Formula 3b)

[0505] Note that since the slope A of the short-distance interval is greater than the slope A×(H / R) of the long-distance interval 2 , therefore, the section data (Sa~Sd) of the short-distance section is updated so that it approaches the position of the user's car faster than the section data (Sa~Sd) of the long-distance section.

[0506] In addition, the middle distance section is not a section in which time is linearly displayed, and the slope (dh / dt) is expressed by the following (Formula 2b) described above.

[0507] (dh / dt) = B × (1 / t) 2 ...(Formula 2b)

[0508] The middle distance section is updated so that the closer the area is to the vehicle position, the faster the area approaches the vehicle position. Therefore, the display length of the section data (Sa to Sd) of the middle distance section becomes longer as the display position approaches the vehicle position over time.

[0509] The order of the speed at which the displayed position of the section data (Sa to Sd) approaches the vehicle position is

[0510] Long distance interval < medium distance interval < short distance interval

[0511] And the speed in the middle distance interval increases as the display position gets closer to the vehicle position.

[0512] Note that although in reference Figure 25-27 In the example described, the travel range display data 70 is displayed as straight line data, but in another possible configuration, as described above with reference to FIG. Figure 21-23 As described above, the segment data (Sa to Sd) may be displayed on the wristwatch-type information terminal 50 using different colors.

[0513] Fig.28 An example thereof is shown.

[0514] Fig.28 The lower part describes an example of updating display data with the elapse of time t1, t2, and t3.

[0515] Note that the displayed data can be moved counterclockwise as time passes. In this configuration, the order of the speed at which the displayed position of the interval data (Sa to Sd) approaches the vehicle position is

[0516] Long distance interval < medium distance interval < short distance interval

[0517] [5-2. Display example of additional information for travel section display data]

[0518] Next, a display example of additional information with respect to the travel section display data will be described.

[0519] refer to Figure 25-28 The display examples described are display examples in which only the driving interval display data 70 is displayed. That is, they are display examples of the driving interval display data 70 including only the position information of the automatic driving permission zone Sa, the manual driving zone Sb, the zone Sc for switching from the automatic driving to the manual driving, and the cautious driving zone Sd.

[0520] Data processing department, such as Fig.11 The data processing unit 11 in the structure shown in FIG. Figure 8 The data processing unit in the automatic driving control unit 112 in the structure shown in , for example, the detection unit 131, the situation analysis unit 133 and the planning unit 134, and the output control unit 105 also generate information useful to the user (driver) as additional information for the driving range display data 70.

[0521] refer to Fig.29 Specific examples of display examples of additional information are described in the following drawings.

[0522] Fig.29 The following additional information display example is described.

[0523] Additional information a: Low-speed detour road section provided when manual driving cannot be resumed (local roads parallel to the expressway, etc.)

[0524] Additional information b: There is a risk of running out of the section, and the section needs to be driven manually with caution

[0525] Additional information c: Areas where abandonment of resumption of manual driving is prohibited (penalty target sections) (e.g. sections where traffic congestion or road closures may occur due to lane reduction, etc.)

[0526] Additional information d: Intervals whose lengths may vary greatly, such as traffic congestion intervals

[0527] For example, this additional information is displayed as Fig.29 A combination of a specific color area or icon and descriptive information as shown in the .

[0528] Through this information, the user (driver) can confirm detailed information on the vehicle's traveling direction.

[0529] Note that only the icon may be displayed, and in the case where the user performs a process of touching or clicking the icon, the description may be displayed.

[0530] In addition, if Fig.30 As shown in , in another possible configuration, signboards set up on the road, information about gas stations, service areas, construction, etc., detour route information, etc. can be displayed. Fig.30 The following additional information display example is described.

[0531] Additional information e1~e5: sign information, gas station information

[0532] Additional information f: Detour route location guidance information

[0533] Notice, Fig.30 The additional information e1 and additional information f shown in FIG. 1 are, for example, detour route information about the section before entering the manual driving section, and are display information that can be used when other automatic driving sections can be entered through the detour route and in other cases.

[0534] In addition, if Fig.31 As shown in the additional information g in FIG. 1 , information on the time to arrive at each section (Sa to Sd) displayed in the travel section display data 70 can be displayed.

[0535] In addition, if Fig.31 As shown in the additional information h in FIG. 1 , the distance ratio information of each display section (short distance section to long distance section) can also be displayed.

[0536] In the example shown in the figure, the distance ratio of the short distance interval is 1:1, while the distance ratio of the long distance interval is 1:25. This means that, for example, when the actual interval length of the predetermined unit length of the short distance interval is 1 km, the actual interval length of the same predetermined unit length of the long distance interval is 25 km.

[0537] For medium distance intervals, it means that the actual interval length of the same predetermined unit length is t×1Km.

[0538] In addition, if Fig.32 As shown in , various timing information can be additionally displayed as additional information. Fig.32 The following additional information display example is described.

[0539] Additional information m: MRM (Minimum Risk Maneuver) starting point position (timing to start emergency deceleration, parking, or pulling over when the start of the manual driving resumption process cannot be detected)

[0540] Additional information n1: Notification point (notification point of notification of resumption of manual driving request)

[0541] Additional information n2: Warning point (Warning point for resuming manual driving request)

[0542] Additional information p: The remaining time before the notification point

[0543] Based on these types of additional information, the user (driver) can confirm detailed information about various notifications and warnings to be issued, automatic driving control processing, and the like.

[0544] Note that the display of these types of additional information may be displayed at different display positions or display timings based on the information about the actions of the user (driver).

[0545] For example, in the case where the user (driver) is in a state where he / she is sitting in the driver's seat and can immediately resume manual driving, Fig.32 In the case where the user (driver) is not in a state where he / she can immediately resume manual driving because he / she leaves the driver's seat, is sleeping, etc., the display mode shown in FIG. Fig.33 Display mode shown in .

[0546] Despite Fig.33 In the display Fig.32 Similar additional information is shown in the table below, but the displayed location and time information of each of the following information are close to the current location and current time.

[0547] Additional information n1: Notification point (notification point of notification of resumption of manual driving request)

[0548] Additional information n2: Warning point (Warning point for resuming manual driving request)

[0549] Additional information p: The remaining time before the notification point

[0550] By performing such a display, it is possible to make the user (driver) realize that he / she is in a highly urgent state, and to make the user (driver) start preparing for resuming manual driving at an early stage.

[0551] Fig.34 is a diagram describing a process of updating additional information over time.

[0552] Specifically, an example of a process of updating display data having the following additional information as time (t1 to t3) elapses is described.

[0553] Additional information m: MRM (Minimum Risk Maneuver) starting point position (timing to start emergency deceleration, parking, or pulling over when the start of the manual driving process cannot be detected)

[0554] Additional information n1: Notification point (notification point of notification of resumption of manual driving request)

[0555] Additional information p: the remaining time before the notification point

[0556] Additional information e: Signs (signs indicating detour routes)

[0557] Additional information f: Detour route location guidance information

[0558] The display examples at time t1 and time t2 are examples of display data displayed in a state where the detour route can be entered.

[0559] These display data include the following additional information.

[0560] Additional information e: Signs (signs indicating detour routes)

[0561] Additional information f: Detour route location guidance information

[0562] The display example at time t3 is an example of display data displayed at a time when the detour route cannot be entered.

[0563] The display data at time t3 is updated so that the following additional information is not displayed.

[0564] Additional information e: Signs (signs indicating detour routes)

[0565] Additional information f: Detour route location guidance information

[0566] In this manner, the data processing section performs display control to select and display only information available to the user (driver) at each time.

[0567] Note that in Fig.35 As shown in , in the case where a circular display is displayed on a wristwatch or the like, display of additional information is similarly performed.

[0568] Note that Fig.28 In the example of the display mode in which a circular display is displayed on a watch or the like, the information related to the progress of time rotates counterclockwise. In this case, information about the most recent approach timing is displayed in the quadrants from 3 o'clock to 0 o'clock, and from the first quadrant to the second quadrant, information about the timing immediately before the user makes a switch can be intuitively obtained.

[0569] On the other hand, if we consider the human perception of analog clocks, humans intuitively perceive the remaining time from the remaining rotation angle of the second hand moving toward 0 o'clock / 12 o'clock, etc. Fig.35As shown in , it is also effective to present the information by rotating the animation for presenting the information clockwise. In addition, in a circular display mode such as a circular display mode on a watch, when a circular display such as a display of the arrival time of the MRM boundary location time, a display of the arrival time of the detour possible boundary location time, a display of the arrival time of the conversion notification point, etc. is displayed on a watch, additional information is similarly performed.

[0570] [5-3. Example of notification and warning processing using animation]

[0571] Next, an example of processing of notification and warning using animation will be described.

[0572] Data processing department, such as Figure 1 The data processing unit 11 in the structure shown in FIG. Figure 8 The data processing unit in the automatic driving control unit 112 in the configuration shown in , for example, the detection unit 131, the situation analysis unit 133, the planning unit 134, and the output control unit 105, not only performs the processing of generating the driving range display data 70, but also performs the processing of further generating and using the animation display data to notify the user (driver) of the time remaining before the start of the manual driving request notification to resume the manual driving of the user (driver), or the start of the manual driving, and issuing a warning about the remaining time.

[0573] refer to Fig.36 and subsequent figures illustrate examples of displaying data using animation.

[0574] Fig.36 Describes an information terminal on which a user (driver) performs secondary tasks. Assume that the user is using the "secondary task execution screen area" of the terminal shown in the figure to perform various secondary tasks, for example, secondary tasks such as watching moving images, creating documents, or playing games.

[0575] On the upper part of the display unit of the terminal, under the control of the data processing unit, animation information 80 for notifying the remaining time until the manual driving switching point is displayed.

[0576] The animation information 80 is animation information for notifying the user (driver) who is performing the secondary task of the time remaining until a manual driving switch request notification is issued, or the length of time remaining until a manual driving switch point is reached.

[0577] The animation information 80 indicates that there is coffee in the cup.

[0578] Figure 1 The data processing unit 11 in the configuration shown in FIG. 1 , or Figure 8The data processing unit in the automatic driving control unit 112 shown in FIG. 1 updates the animation information 80 as time passes.

[0579] refer to Fig.37 and Fig.38 , which illustrates a specific processing example of the update.

[0580] Figure 37-Figure 38 An example of updating the animation information 80 with the passage of time (t1 to t8) is described.

[0581] As in Fig.37 As shown in the display data of time t1 to t4 in FIG. 1 , the data processing unit performs display control to display a state in which the amount of coffee in the cup gradually increases with the passage of time.

[0582] This is a display process for making the user (driver) notice that the length of time remaining before the manual driving changes places is becoming shorter and shorter as time passes.

[0583] In addition, as in Fig.38 As shown in the display data at time t5 to t8, the data processing unit performs display control to display a state in which the amount of coffee in the cup further increases with the passage of time, the coffee overflows the cup, and the secondary task execution screen area is filled with coffee.

[0584] This is a display process for forcing the user (driver) to stop the secondary task and resume manual driving by making the user (driver) notice that the time remaining before the manual driving switching point becomes extremely short as time passes.

[0585] By performing such display control, even when the user (driver) is concentrating on a secondary task, the user (driver) can be forced to stop the secondary task and resume manual driving.

[0586] [5-4. Other display processing examples]

[0587] Next, other display processing examples will be described.

[0588] The travel section display data and other display data are not limited to the above-mentioned display examples, and may be subjected to various display processes.

[0589] Data processing department, such as Figure 1 The data processing unit 11 in the structure shown in FIG. Figure 8The data processing unit in the automatic driving control unit 112 in the structure shown in the figure, for example, the detection unit 131, the situation analysis unit 133, the planning unit 134, and the output control unit 105 generate driving range display data and other display data, and generate display data for a display unit such as a display unit in the vehicle, or a terminal carried by a user (driver), for example, a PC, a tablet terminal or a smartphone.

[0590] refer to Fig.39 and the accompanying drawings, illustrate examples of display data other than the display data mentioned above.

[0591] Fig.39 This is an example of outputting display data to a smartphone or tablet terminal carried by a user (driver).

[0592] The figure shows a display example of the travel section display data 70 when the smartphone or tablet terminal is held in a vertically long state.

[0593] The area in the central portion of the terminal is a secondary task execution screen area used by the user. The user uses this area as an area for various secondary tasks, such as a document creation area, or a moving image display area.

[0594] The travel section display data 70 is displayed in the peripheral area of ​​the secondary task execution screen area.

[0595] Assuming that the lower left portion of the terminal shown in the figure corresponds to the current position of the vehicle, the driving interval information (Sa~Sd, etc.) of the short-distance interval, medium-distance interval and long-distance interval is displayed, so that the displayed information circles from the left end portion through the upper end portion and the right end portion to the lower end portion.

[0596] Notice, Fig.39 The following two driving range display data display examples are described.

[0597] (a) Driving range display data display example 1

[0598] (b) Example 2 of displaying data in driving range

[0599] (a) Traveling Section Display Data Display Example 1 is an example in which the display widths of the traveling section display data on the upper side, the lower side, the left side, and the right side are uniformly set.

[0600] On the other hand, (b) traveling interval display data display example 2 is an example in which the display widths of the traveling interval display data on the upper side, lower side, left side, and right side are not uniformly set, and the data width on the left side corresponding to the short distance interval is increased.

[0601] In addition, if Fig.40As shown in , in another possible configuration, the length of time remaining before the notification point of the manual driving start request or before the manual driving switching point may be displayed.

[0602] Fig.40 A display example at time ( t1 ) and a display example at the subsequent time ( t2 ) are described.

[0603] At time (t1), the remaining time is displayed as 3 minutes and 40 seconds. By confirming this time, the user (driver) can start preparing for manual driving.

[0604] In the display of time (t2), the remaining time is displayed as 40 seconds. In addition, risk information 81 = (Trap-in Pit) is displayed as additional information.

[0605] The risk information 81 = (Trap-in Pit) is information on the risk that occurs when the manual driving transition fails, and is display data for notifying the user that the vehicle enters the pull-in parking area and is forced to stop when the manual driving transition fails.

[0606] Fig.41 The example shown in FIG. 8 is an example in which manual driving switch request messages 82 a and 82 b are displayed in the secondary task execution screen area.

[0607] As time passes, the manual driving transition request message 82 is displayed in increasing size.

[0608] In addition, the fallen leaves are caused to fall in the area used by the user as the secondary task execution area, thereby hindering the user's secondary task. The number of fallen leaves increases with the passage of time, thereby trying to make the user give up continuing the secondary task.

[0609] Fig.42 The example shown in FIG. 1 is an example of displaying an hourglass as animations 83 a and 83 b , which enable intuitive recognition of the length of time remaining until reaching the notification point or the manual driving changeover point.

[0610] As time passes, the sand in the hourglass falls, and when the upper space becomes empty, the user can intuitively recognize that remaining time=0, which produces an effect of prompting the user to return to manual driving earlier.

[0611] Fig.43 The example shown in is an example of animations 84a and 84b imitating annoying rotating fireworks in the secondary task execution screen area. As time goes by, the rotating fireworks unfold and run around in the working area on the screen, which produces an effect of causing the user to give up continuing the secondary task.

[0612] In addition to these displays that trigger the abandonment of secondary tasks described as examples in the accompanying drawings, favorite customized settings for issuing notifications, notification marks for welcoming pets, marks indicating prayer times rooted in the unique culture of the area of ​​use or its culture, marks for being chased by the devil, etc. also produce psychological effects that induce coping, and are effective as functions to encourage users to return as soon as possible.

[0613] In addition, an icon character service can be used in which the user can select an icon character through the network. Whatever is set is effective as long as the effect of prompting the driver to suspend the secondary task is obtained in addition to the healing effect, such as cat paw prints, local unique characters, shark tail fins, police car and police motorcycle characters, and devil marks and servant characters that forcibly suggest that the user start preparing for the transformation.

[0614] [6. Configuration example of information processing device]

[0615] Although the above treatment can be applied by reference Figure 8 The present invention is performed by the configuration of the mobile device described above, but, for example, some of the processing may be performed in an information processing device that is attachable to and detachable from the mobile device.

[0616] refer to Fig.44 , an example of the hardware configuration of such an information processing device is described.

[0617] Fig.44 It is a diagram for explaining an example of the hardware configuration of an information processing device.

[0618] A CPU (Central Processing Unit) 501 functions as a data processing unit that executes various processes according to a program stored in a ROM (Read Only Memory) 502 or a storage unit 508. For example, the CPU 501 executes processes corresponding to the sequence described in the above embodiment.

[0619] A RAM (Random Access Memory) 503 stores programs executed by the CPU 501 , data, and the like. The CPU 501 , the ROM 502 , and the RAM 503 are interconnected by a bus 504 .

[0620] The CPU 501 is connected to the input / output interface 505 via the bus 504, and the input / output interface 505 is connected to the input unit 506 and the output unit 507. The input unit 506 includes various switches, a keyboard, a touch panel, a mouse, a microphone, and a status data acquisition unit such as a sensor, a camera, a GPS or an in-vehicle position measurement device, and the output unit 507 includes a display, a speaker, etc.

[0621] Note that the input section 506 receives input information from the sensor 521 as input.

[0622] In addition, the output unit 507 also outputs driving information of the driving unit 522 for moving the device.

[0623] For example, the CPU 501 receives commands, status data, and the like input from the input unit 506 as input, executes various processes, and outputs the process results to the output unit 507 .

[0624] The storage unit 508 connected to the input / output interface 505 includes, for example, a hard disk, etc., and stores programs executed by the CPU 501 and various data. The communication unit 509 functions as a transmission / reception part for data communication via a network such as the Internet or a local area network, and communicates with an external device.

[0625] The drive 510 connected to the input / output interface 505 drives a magnetic disk, an optical disk, a magneto-optical disk, or a removable medium 511 such as a semiconductor memory (eg, a memory card), and performs recording or reading of data.

[0626] [7. Summary of the configuration according to the present disclosure]

[0627] Thus far, embodiments according to the present disclosure have been described in detail with reference to specific embodiments. However, various modifications or substitutions of the embodiments may be conceived by those skilled in the art without departing from the gist of the present disclosure. That is, the present invention is disclosed in an illustrative manner and should not be interpreted in a limiting manner. To determine the gist of the present disclosure, reference should be made to the claims.

[0628] Note that the technology disclosed in this specification can have the configuration described below.

[0629] (1) An information processing device comprising:

[0630] a data processing unit that generates driving section display data that enables at least an automatic driving permitted section and a manual driving section on a driving route of the vehicle to be identified;

[0631] The data processing section generates one piece of travel section display data generated by connecting a plurality of pieces of display data having time axes that differ according to a scheduled travel time period of the vehicle.

[0632] (2) The information processing device according to (1), wherein

[0633] The data processing unit is configured to

[0634] For a short distance section where the scheduled travel time period of the vehicle comes earlier, time-linear display data is generated along a first time axis in which time (t) and display position (h) are set in a proportional relationship, and

[0635] For a long-distance section where the vehicle's scheduled travel time period arrives later, time-linear display data is generated along a second time axis in which time (t) and display position (h) are set in a proportional relationship, and

[0636] The data processing section generates display data in which a distance per unit display length of the display data of the long distance section is made larger than a distance per unit display length of the display data of the short distance section.

[0637] (3) The information processing device according to (2), wherein

[0638] The data processing unit generates display data of a driving section in which a medium distance section is set between the short distance section and the long distance section, and

[0639] The display data of the middle distance section includes display data having a time axis that gradually changes at a predetermined ratio from a first time axis corresponding to the short distance section to a second time axis corresponding to the long distance section.

[0640] (4) The information processing device according to any one of (1) to (3), wherein

[0641] The data processing unit generates travel segment display data in which a display width (W) of a short distance segment in which a scheduled travel time period of the vehicle arrives earlier is set to be larger than a display width (W) of a long distance segment in which a scheduled travel time period of the vehicle arrives later.

[0642] (5) The information processing device according to (4), wherein

[0643] The data processing unit generates display data in which a display width (W) of a middle distance interval between the short distance interval and the long distance interval is made the same as the width of the short distance interval in a portion connected to the short distance interval, then decreases as the distance increases, and becomes the same as the width of the long distance interval in a portion connected to the long distance interval.

[0644] (6) The information processing device according to any one of (1) to (5), wherein

[0645] The data processing unit generates driving section display data including an automatic driving permission section Sa, a manual driving section Sb, a section Sc for switching from automatic driving to manual driving, and a cautious driving section Sd.

[0646] (7) The information processing device according to any one of (1) to (6), wherein

[0647] The data processing unit generates linear travel section display data extending in one direction according to a scheduled travel time period of the vehicle.

[0648] (8) The information processing device according to any one of (1) to (6), wherein

[0649] The data processing unit generates circular travel segment display data extending in a circumferential shape according to a scheduled travel time period of the vehicle.

[0650] (9) The information processing device according to any one of (1) to (6), wherein

[0651] The data processing unit generates travel section display data set along a partial area around the display unit.

[0652] (10) The information processing device according to any one of (1) to (9), wherein

[0653] The data processing section generates display data by adding additional information to the travel section display data.

[0654] (11) The information processing device according to (10), wherein

[0655] The additional information includes at least any one of detour route information, sign information, and information on arrival time at each location.

[0656] (12) The information processing device according to (10) or (11), wherein

[0657] The additional information includes information indicating an MRM (Minimum Risk Management) start point, which is a start point of risk avoidance processing.

[0658] (13) The information processing device according to any one of (1) to (12), wherein

[0659] The data processing unit generates animation information that enables identification of a remaining time length before notification of a manual driving request is resumed or before a manual driving section.

[0660] (14) The information processing device according to (13), wherein

[0661] The animation information includes an animation that changes over time to cover a secondary task execution screen area used by a user.

[0662] (15) A mobile device capable of switching between automatic driving and manual driving, the mobile device comprising:

[0663] a driver information acquisition unit, the driver information acquisition unit acquiring driver information of a driver of the mobile device;

[0664] an environment information acquisition unit, the environment information acquisition unit acquiring surrounding environment information of the mobile device; and

[0665] a data processing unit that generates driving section display data that enables at least an automatic driving permitted section and a manual driving section on a driving route of the vehicle to be identified;

[0666] The data processing unit generates a piece of driving interval display data, which is driving interval display data that enables at least an automatic driving allowed section and a manual driving section on a driving route of the vehicle to be identified, and is generated by connecting a plurality of display data having time axes that differ according to a scheduled time period of driving of the vehicle.

[0667] (16) The mobile device according to (15), wherein

[0668] The data processing unit is configured to

[0669] For a short distance section where the scheduled travel time period of the vehicle comes earlier, time-linear display data is generated along a first time axis in which time (t) and display position (h) are set in a proportional relationship, and

[0670] For a long-distance section where the scheduled travel time period of the vehicle arrives later, time-linear display data is generated along a second time axis in which time (t) and display position (h) are set in a proportional relationship, and

[0671] The data processing section generates display data in which a distance per unit display length of the display data of the long distance section is made larger than a distance per unit display length of the display data of the short distance section.

[0672] (17) The mobile device according to (16), wherein

[0673] The data processing unit generates a piece of travel section display data in which a medium distance section is set between the short distance section and the long distance section, and

[0674] The display data of the middle distance section includes display data having a time axis that gradually changes at a predetermined ratio from a first time axis corresponding to the short distance section to a second time axis corresponding to the long distance section.

[0675] (18) A method of executing 4 in an information processing device, wherein

[0676] The data processing unit generates a piece of travel interval display data, which is travel interval display data that enables at least an automatic driving permitted section and a manual driving section on a travel route of the vehicle to be identified, and is generated by connecting a plurality of display data having time axes that differ according to a scheduled travel time period of the vehicle.

[0677] (19) An information processing method executed in a mobile device, wherein

[0678] The mobile device is a mobile device capable of switching between automatic driving and manual driving,

[0679] The information processing method comprises

[0680] a driver information acquiring step of acquiring driver information of a driver of the mobile device by a driver information acquiring unit, and

[0681] an environmental information acquisition step of acquiring the surrounding environmental information of the mobile device by an environmental information acquisition unit, and

[0682] The data processing unit generates a piece of travel interval display data, which is travel interval display data that enables at least an automatic driving permitted section and a manual driving section on a travel route of the vehicle to be identified, and is generated by connecting a plurality of display data having time axes that differ according to a scheduled travel time period of the vehicle.

[0683] (20) A program for causing information processing to be executed in an information processing device, wherein

[0684] A data processing unit is caused to generate one piece of driving interval display data, wherein the driving interval display data is driving interval display data that enables at least an automatic driving permitted section and a manual driving section on a driving route of the vehicle to be identified, and is generated by connecting a plurality of display data having time axes that differ according to a scheduled driving time period of the vehicle.

[0685] In addition, the series of processes described in this specification can be performed by hardware, software, or a combination of hardware and software. In the case where the process is performed by software, the program in which the process sequence is recorded can be installed in the memory of a computer incorporated into the dedicated hardware and executed on the computer, or the program can be installed on a general-purpose computer capable of performing various types of processes and executed on the general-purpose computer. For example, the program can be recorded in advance on a recording medium. In addition to being installed from a recording medium to a computer, the program can also be received via a network such as a LAN (local area network) or the Internet, and the program can be installed on a built-in recording medium such as a hard disk.

[0686] Note that the various processes described in this specification can be executed not only in the order described, but also in parallel or individually as needed or according to the processing capabilities of the device executing the process. In addition, the system in this specification has a logical set of multiple devices and is not limited to a system including various devices contained in a single housing.

[0687] [Industrial Applicability]

[0688] As described above, according to a configuration of one embodiment of the present disclosure, a configuration is implemented to generate travel interval display data that enables at least identification of an automatic driving permitted section and a manual driving section on a vehicle's travel route, and present the travel interval display data to a user (driver).

[0689] Specifically, for example, a data processing unit is included that generates driving interval display data that enables at least identification of an automatic driving allowed interval and a manual driving interval on a driving route of a vehicle. For a short distance interval and a long distance interval, the data processing unit generates time linear display data that represents time (t) and display position (h) in a proportional relationship. The data processing unit generates display data in which the distance per unit display length of the long distance interval is made greater than the distance per unit display length of the short distance interval. Regarding a medium distance interval between the short distance interval and the long distance interval, the data processing unit generates display data having a time axis that gradually changes at a predetermined ratio from a first time axis corresponding to the short distance interval to a second time axis corresponding to the long distance interval. In addition, by displaying information including penalties for risk factors as appropriate along with approach information, the driver naturally learns to be careful to avoid risks and pays attention to confirmation; as a result, it can be expected that the user of the vehicle in the automatic driving mode can resume the high degree of caution and consciousness recovery required for manual driving according to the driving conditions.

[0690] This configuration realizes a configuration in which travel section display data that enables at least an automatic driving permitted section and a manual driving section on a travel route of a vehicle to be identified is generated and presented to a user (driver).

[0691] [reference numerals list]

[0692] 10: Cars

[0693] 11: Data Processing Department

[0694] 12: Driver information acquisition unit

[0695] 13: Environmental Information Acquisition Department

[0696] 14: Ministry of Communications

[0697] 15: Notification Department

[0698] 20: Driver

[0699] 30: Server

[0700] 50: Information Terminal

[0701] 70: Driving range display data

[0702] 80: Animation information

[0703] 100: Mobile device

[0704] 101: Input section

[0705] 102: Data acquisition unit

[0706] 103: Ministry of Communications

[0707] 104:In-car equipment

[0708] 105: Output control unit

[0709] 106: Output unit

[0710] 107: Drive system control unit

[0711] 108: Drive system

[0712] 109: Body system control unit

[0713] 110:Body system

[0714] 111: Storage

[0715] 112:Automatic driving control unit

[0716] 121: Communication Network

[0717] 131: Detection Department

[0718] 132: Current position estimation unit

[0719] 133: Situation Analysis Department

[0720] 134: Planning Department

[0721] 135: Operation Control Unit

[0722] 141: Vehicle external information detection unit

[0723] 142: Vehicle internal information detection unit

[0724] 143: Vehicle status detection unit

[0725] 151: Map Analysis Department

[0726] 152: Traffic rules recognition unit

[0727] 153: Situation Identification Unit

[0728] 154: Situation Forecast Department

[0729] 155: Safety determination unit

[0730] 161: Route Planning Department

[0731] 162: Action Planning Department

[0732] 163: Operation Planning Department

[0733] 171: Emergency Evacuation Department

[0734] 172: Acceleration / deceleration control unit

[0735] 173: Direction Control Unit

[0736] 501:CPU

[0737] 502:ROM

[0738] 503: RAM

[0739] 504: Bus

[0740] 505: Input / output interface

[0741] 506: Input unit

[0742] 507: Output unit

[0743] 508: Storage

[0744] 509: Ministry of Communications

[0745] 510:Driver

[0746] 511: Removable Media

[0747] 521: Sensor

[0748] 522: Driving unit

Claims

1. An information processing device, include: A data processing unit configured to generating driving section display data that enables identification of at least an automatic driving permitted section and a manual driving section on a driving route of the vehicle; By connecting a plurality of display data having different time axes according to the scheduled travel time period of the vehicle, a travel section display data is generated, wherein, in the one piece of driving interval display data, a first display width of a short distance interval in which the vehicle's scheduled driving time period arrives earlier is set to be larger than a second display width of a long distance interval in which the vehicle's scheduled driving time period arrives later, and wherein, in the one driving interval display data, the display width of the middle distance interval between the short distance interval and the long distance interval is made equal to the first display width of the short distance interval at the portion connected to the short distance interval, then gradually decreases as the distance increases, and is made equal to the second display width of the long distance interval at the portion connected to the long distance interval; generating driving section display data including an automatic driving allowed section Sa, a manual driving section Sb, a section Sc for switching from automatic driving to manual driving, and a cautious driving section Sd in one or more of the short distance section, the long distance section, and the medium distance section; and generating a notification to the driver of a scheduled transition from the automated driving permitted zone to the manual driving zone based on the received indication of the driver's level of alertness, Here, the time for generating the notification to the driver before the predetermined transition is longer when the driver's awakening level is low than the time for generating the notification to the driver before the predetermined transition when the driver's awakening level is high.

2. The information processing device according to claim 1, wherein The data processing unit is configured to For a short distance section where the scheduled travel time period of the vehicle comes earlier, time-linear display data is generated along a first time axis in which time (t) and display position (h) are set in a proportional relationship, and For a long-distance section where the vehicle's scheduled travel time period arrives later, time-linear display data is generated along a second time axis in which time (t) and display position (h) are set in a proportional relationship, and The data processing section generates display data in which a distance per unit display length of the display data of the long distance section is made larger than a distance per unit display length of the display data of the short distance section.

3. The information processing device according to claim 2, wherein The data processing unit is configured to generate a piece of travel section display data in which a medium distance section is set between the short distance section and the long distance section, and The display data of the middle distance section includes display data having a time axis that gradually changes at a predetermined ratio from a first time axis corresponding to the short distance section to a second time axis corresponding to the long distance section.

4. The information processing device according to claim 1, wherein The data processing unit is configured to generate straight travel section display data extending in one direction according to a predetermined travel time period of the vehicle.

5. The information processing device according to claim 1, wherein The data processing unit is configured to generate circular travel section display data extending in a circumferential shape according to a scheduled travel time period of the vehicle.

6. The information processing device according to claim 1, wherein The data processing unit is configured to generate travel section display data set along a partial area around the display unit.

7. The information processing device according to claim 1, wherein The data processing section is configured to generate display data generated by adding additional information to the travel section display data.

8. The information processing device according to claim 7, wherein The additional information includes at least any one of detour route information, sign information, and information on arrival time at each location.

9. The information processing device according to claim 7, wherein The additional information includes information indicating a minimum risk operation MRM starting point, where the MRM starting point is a starting point of a risk avoidance process.

10. The information processing apparatus according to claim 1, wherein The data processing section is configured to generate animation information that enables identification of a length of time remaining before notification of a manual driving request is resumed or before a manual driving interval.

11. The information processing apparatus according to claim 10, wherein The animation information includes an animation that changes over time to cover a secondary task execution screen area used by a user.

12. A mobile device capable of switching between automatic driving and manual driving, the mobile device include: a driver information acquisition unit configured to acquire driver information of a driver of the mobile device; an environment information acquisition unit, the environment information acquisition unit being configured to acquire surrounding environment information of the mobile device; and An information processing device configured as an information processing device according to any of the preceding claims.

13. An information processing method executed in an information processing device, the information processing method include: generating a piece of driving section display data, the driving section display data being driving section display data that enables at least an automatic driving permitted section and a manual driving section on a driving route of the vehicle to be identified; By connecting a plurality of display data having different time axes according to the scheduled travel time period of the vehicle, a travel section display data is generated, wherein, in the one piece of driving interval display data, a first display width of a short distance interval in which the vehicle's scheduled driving time period arrives earlier is set to be larger than a second display width of a long distance interval in which the vehicle's scheduled driving time period arrives later, and wherein, in the one driving interval display data, the display width of the middle distance interval between the short distance interval and the long distance interval is made equal to the first display width of the short distance interval at the portion connected to the short distance interval, then gradually decreases as the distance increases, and is made equal to the second display width of the long distance interval at the portion connected to the long distance interval; generating driving section display data including an automatic driving allowed section Sa, a manual driving section Sb, a section Sc for switching from automatic driving to manual driving, and a cautious driving section Sd in one or more of the short distance section, the long distance section, and the medium distance section; and generating a notification to the driver of a scheduled transition from the automated driving permitted zone to the manual driving zone based on the received indication of the driver's level of alertness, Here, the time for generating the notification to the driver before the predetermined transition is longer when the driver's awakening level is low than the time for generating the notification to the driver before the predetermined transition when the driver's awakening level is high.

14. An information processing method executed in a mobile device, wherein The mobile device is a mobile device capable of switching between automatic driving and manual driving, The information processing method include: a driver information acquisition step of acquiring driver information of a driver of the mobile device by a driver information acquisition unit, an environmental information acquisition step of acquiring the surrounding environmental information of the mobile device by an environmental information acquisition unit, and The information processing method according to claim 13.

15. A computer program product comprising a program which, when executed by a processor, causes the processor to execute the information processing method according to claim 13.

Images