Information processing equipment, information processing systems and information processing methods
By obtaining the level of autonomous driving and notifying the driver that activity is permitted, the problem of insufficient notification of activity changes in autonomous vehicles is solved, thus improving driving safety.
Patent Information
- Application Number
- CN202180077949.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-11-27
- Filing Date
- 2021-10-20
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-10-20
AI Technical Summary
In autonomous vehicles, the driver's permitted activities vary depending on the level of automation, but existing technologies have failed to effectively notify the driver of changes in activity, leading to safety risks.
By obtaining the autonomous driving level of the mobile device and referring to the list of permitted activities on an external server or storage unit, the driver is notified of the corresponding permitted activities and the activity restrictions are updated in a timely manner.
It enables timely notification of changes in driver activity when the level of autonomous driving changes, thereby improving driving safety and driver cooperation.
Smart Images

Figure CN116490911B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to information processing apparatus, information processing system, and information processing method. More specifically, this disclosure relates to information processing apparatus, information processing system, and information processing method that enable safe autonomous driving by notifying the driver of driver permission for the vehicle during autonomous driving. Background Technology
[0002] In recent years, various sensors have been installed on vehicles, and functions for driver assistance are being introduced. Furthermore, active development of technologies related to autonomous driving is underway.
[0003] Autonomous driving technology is a technology that enables autonomous driving on roads by using various sensors, such as position detection units set in vehicles (cars), and this technology is expected to become widespread rapidly in the future.
[0004] For example, the Institute of Electrical and Electronics Engineers (IEEE), an academic body for electrical and information engineering, proposed a communication system for vehicle-to-vehicle communication based on the IEEE 802.11p communication standard, known as Dedicated Short Range Communication (DSRC).
[0005] Furthermore, in the 3rd Generation Partnership Project (3GPP), the standard specification for third-generation (3G) mobile communication systems, version 14 (Rel.14) was developed as Celler-V2X (C-V2X), a communication standard based on Long Term Evolution (LTE) D2D (device-to-device) communication.
[0006] Celler-V2X (C-V2X) is a communication standard that includes vehicle-to-vehicle (V2V) communication, vehicle-to-infrastructure (V2I) communication, and more.
[0007] Furthermore, advanced driver assistance systems (ADAS), which utilize vehicle-to-vehicle (V2V) communication and a sensor hybrid system employing various sensors mounted on the vehicle, are being introduced. Additionally, the development of autonomous driving systems that utilize artificial intelligence (AI), machine learning (ML), or deep learning (DL) without human intervention is also underway.
[0008] Note that the Society of Automotive Engineers (SAE), the automotive technology council in the United States, defines six levels of autonomous driving, from Level 0 (manual driving) to Level 5 (autonomous driving).
[0009] Countries around the world, including Japan, adopt this SAE definition of autonomous driving levels.
[0010] For example, automakers around the world typically specify which level of autonomous driving, from 0 to 5, each vehicle is capable of.
[0011] In addition, traffic regulations in various countries have adopted measures such as establishing rules that allow Level 3 autonomous driving on certain sections of highways.
[0012] For example, Level 5 autonomous driving means the driver does not need to perform any driving or monitoring operations and can engage in any activity. For instance, the user can do his or her favorite activities, such as taking a nap, reading, or playing games.
[0013] However, for example, Level 3 of automated driving is defined as a level where the driver needs to continuously monitor the road and revert to manual driving in emergencies, and the driver can only perform activities while monitoring the road ahead. For example, taking a nap is not permitted.
[0014] Currently, autonomous driving is still in the development stage, and it is believed that Level 5 autonomous driving, which is considered 100% autonomous driving on all road sections, will take time. For some time, it is anticipated that the system will need to handle the switching of autonomous driving levels according to road sections and conditions.
[0015] For example, on straight roads with sufficient width, such as highways, driving at levels 4 and 5 of autonomous driving is permitted. However, in cases where the vehicle is expected to leave the highway and park in a parking lot, or on narrow mountain roads, it is necessary to switch to levels 0 to 3 of autonomous driving to drive under the supervision or operation of the driver.
[0016] Note that examples of publicly disclosed technologies related to autonomous driving include Patent Document 1 (Japanese Patent Application Publication No. 2020-029238), Patent Document 2 (Japanese Patent Application Publication No. 2019-018842), and Patent Document 3 (Japanese Patent Application Publication No. 2019-185246), etc.
[0017] Patent Document 1 discloses a technology for changing the driving mode according to the driving environment during driving. Patent Document 2 discloses a configuration for detecting design defects in a driver assistance system. Furthermore, Patent Document 3 describes countermeasures for malfunctions in automated driving control devices.
[0018] As mentioned above, the Society of Automotive Engineers (SAE), the automotive technology council of the United States, defines six levels of autonomous driving, from Level 0 (manual driving) to Level 5 (autonomous driving), and currently predicts that the processing of driving will need to switch between autonomous driving levels according to road segments and conditions.
[0019] As mentioned above, the activities permitted to the driver vary greatly depending on the level of automation. For example, Level 5 automation is autonomous driving, where the driver does not need to perform any driving or monitoring operations and can engage in any activity. For instance, the user can perform his or her favorite activities, such as taking a nap, reading, or playing games.
[0020] However, for example, Level 3 of automated driving is defined as a level where the driver needs to continuously monitor the road and revert to manual driving in emergencies, and the driver can only perform activities while monitoring the road ahead. For example, taking a nap is not permitted.
[0021] Thus, depending on the level of autonomous driving, the activities that the driver is allowed to engage in vary greatly, but there is a problem that the driver cannot confirm which activities are permitted for each level.
[0022] Citation List
[0023] Patent documents
[0024] Patent Document 1: Japanese Patent Application Publication No. 2020-029238
[0025] Patent Document 2: Japanese Patent Application Publication No. 2019-018842
[0026] Patent Document 3: Japanese Patent Application Publication No. 2019-185246 Summary of the Invention
[0027] The problem to be solved by the present invention
[0028] In existing manually driven vehicles, the driver is required to focus on driving operations. Therefore, for drivers of manually driven vehicles, activities other than driving operations, such as conversation, are permitted, but other activities are not allowed.
[0029] However, in autonomous vehicles, the activities permitted for the driver are expected to increase as the driver is freed from driving operations.
[0030] At present, vehicles that achieve autonomous driving in which the driver does not need to participate in driving operations at all (e.g., SAE-defined Level 5 of autonomous driving) have not yet been put into actual use, but vehicles corresponding to various levels of autonomous driving, such as Level 3 of autonomous driving, have already appeared.
[0031] Before all vehicles are replaced with Level 5 autonomous driving vehicles, there is a possibility that vehicles corresponding to various levels of autonomous driving may coexist on public roads. During the phase where various levels of autonomous driving are mixed and put into practical use, it is assumed that the permitted activities for each level of autonomous driving will differ. That is, it is necessary to define the driver's permitted and prohibited activities according to the level of autonomous driving.
[0032] Furthermore, when the level of autonomous driving is switched, the permitted and prohibited activities also change. Therefore, when the level of autonomous driving is switched, the driver needs to be notified of the change in permitted activities to maintain driving safety.
[0033] For example, this disclosure was made in view of the above-mentioned problems, and the purpose of this disclosure is to provide an information processing device, information processing system, and information processing method that enables safe autonomous driving by notifying the driver of activities that authorize the vehicle during autonomous driving.
[0034] Solution to the problem
[0035] A first aspect of this disclosure is an information processing device comprising a control unit that acquires the level of autonomous driving currently being performed by a mobile device and notifies the driver of permitted activities corresponding to the acquired level of autonomous driving.
[0036] Furthermore, a second aspect of this disclosure is an information processing system, which includes a mobile device and external devices.
[0037] The mobile device obtains the level of autonomous driving it is performing.
[0038] The external device provides the mobile device with a list of permitted activities corresponding to autonomous driving levels, where autonomous driving levels and permitted activities are correlated.
[0039] The mobile device refers to the list of permitted activities corresponding to the autonomous driving level, obtains the permitted activities corresponding to the autonomous driving level that the mobile device is currently executing, and notifies the driver of the obtained permitted activities.
[0040] Furthermore, a third aspect of this disclosure is an information processing method executed in an information processing device.
[0041] The control unit obtains the autonomous driving level of the mobile device being executed and notifies the driver of the driver's permitted activities corresponding to the obtained autonomous driving level.
[0042] Furthermore, a fourth aspect of this disclosure is a program that causes an information processing device to perform information processing, the program causing a control unit to:
[0043] Obtain the level of autonomous driving that the mobile device is currently performing, and
[0044] Inform the driver of permitted activities corresponding to the acquired level of autonomous driving.
[0045] Note that the program disclosed herein may be provided, for example, to an information processing device or computer system capable of executing various program codes via a storage medium or communication medium provided in a computer-readable format. By providing such a program in a computer-readable format, processing corresponding to the program is implemented on the information processing device or computer system.
[0046] Other objects, features, and advantages of this disclosure will become apparent from the more detailed description based on the embodiments and accompanying drawings described below. Note that in this specification, a system is a logical collection of multiple devices and is not limited to a system in which each of the constituent devices is housed in the same housing.
[0047] According to one embodiment of the present disclosure, a configuration for notifying a driver of permitted activities corresponding to an autonomous driving level is implemented.
[0048] Specifically, for example, the system obtains the autonomous driving level currently being performed by the mobile device and notifies the driver of the permitted activities corresponding to the obtained autonomous driving level. The control unit retrieves a list of permitted activities corresponding to autonomous driving levels from an external server or storage unit, where autonomous driving levels and permitted activities are correlated. Referring to this list, the control unit retrieves the permitted activities corresponding to the autonomous driving level and notifies the driver of the retrieved permitted activities. Furthermore, if the autonomous driving level of the mobile device changes, the system analyzes the differences between the permitted activities corresponding to the change in autonomous driving level and, if differences exist, notifies the driver of the discrepancy.
[0049] This configuration provides a way to notify the driver of permitted activities corresponding to the level of autonomous driving.
[0050] Note that the effects described in this specification are merely examples and are not limited, and may have additional effects. Attached Figure Description
[0051] Figure 1 This is a diagram illustrating an example of the configuration of the information processing system disclosed herein.
[0052] Figure 2 It is a diagram used to illustrate levels of autonomous driving.
[0053] Figure 3 It is a diagram used to illustrate Local Dynamic Map (LDM).
[0054] Figure 4 This is a diagram illustrating an example of the configuration of the information processing system disclosed herein.
[0055] Figure 5 This is a diagram illustrating an example of the configuration of the information processing apparatus of this disclosure.
[0056] Figure 6 This is a diagram illustrating an example of the configuration of a sensor unit in the control unit of the information processing device of this disclosure.
[0057] Figure 7 It is a diagram illustrating a process sequence performed by the information processing apparatus of this disclosure.
[0058] Figure 8 This is an example diagram illustrating the list of permitted activities corresponding to different levels of autonomous driving, which includes the autonomous driving level and corresponding data for the activities permitted to be performed by the driver as a user.
[0059] Figure 9 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0060] Figure 10 It is a diagram illustrating a process sequence performed by the information processing apparatus of this disclosure.
[0061] Figure 11 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0062] Figure 12 It is a diagram illustrating a process sequence performed by the information processing apparatus of this disclosure.
[0063] Figure 13 This is a diagram illustrating an example of the configuration of a sensor unit in the control unit of the information processing device of this disclosure.
[0064] Figure 14 It is a diagram illustrating a process sequence performed by the information processing apparatus of this disclosure.
[0065] Figure 15 This is a diagram illustrating an example of processing performed by the information processing device of this disclosure using a list of permitted activities corresponding to an autonomous driving level.
[0066] Figure 16This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0067] Figure 17 This is a diagram illustrating an example of processing performed by the information processing device of this disclosure using a list of permitted activities corresponding to an autonomous driving level.
[0068] Figure 18 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0069] Figure 19 This is a diagram illustrating an example of processing performed by the information processing device of this disclosure using a list of permitted activities corresponding to an autonomous driving level.
[0070] Figure 20 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0071] Figure 21 It is a diagram illustrating a process sequence performed by the information processing apparatus of this disclosure.
[0072] Figure 22 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0073] Figure 23 It is a diagram illustrating a process sequence performed by the information processing apparatus of this disclosure.
[0074] Figure 24 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0075] Figure 25 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0076] Figure 26 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0077] Figure 27 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0078] Figure 28 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0079] Figure 29 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0080] Figure 30 This is a diagram illustrating an example of display data output by the information processing device of this disclosure.
[0081] Figure 31 This is a diagram illustrating an example of the hardware configuration of the information processing device disclosed herein. Detailed Implementation
[0082] The information processing apparatus, information processing system, and information processing method of this disclosure will be described in detail below with reference to the accompanying drawings. Note that the description will be carried out according to the following items.
[0083] 1. Summary of examples of the processing and configuration of the information processing system disclosed herein
[0084] 2. Examples of the configuration of an information processing device in a mobile device according to this disclosure
[0085] 3. The basic sequence of processes performed by the information processing equipment of this disclosure
[0086] 4. Regarding the analysis of the driver's confirmation status of the notification of permitted activity, and the processing sequence for emergency response in the absence of confirmation notification.
[0087] 5. Regarding the procedure for monitoring the driver's actions after being notified of permitted activity and for taking emergency response measures when the driver is performing prohibited activities.
[0088] 6. Processing sequence for detecting changes in autonomous driving levels and issuing change notifications for permitted activities.
[0089] 7. Regarding the analysis of the driver's confirmation status of notifications regarding differences in permitted activities associated with changes in automated driving levels, and the processing sequence for emergency response in the absence of confirmation notifications.
[0090] 8. Regarding the procedure for monitoring driver actions following notifications of discrepancies in permitted activities associated with a shift in autonomous driving level, and for taking emergency response measures when the driver is performing prohibited activities.
[0091] 9. Examples of other processing performed by the information processing equipment of this disclosure
[0092] 9-a. (Processing Example a) Selecting and Notifying the User of Allowed Activities in Accordance with Their Request
[0093] 9-b. (Processing Example b) A processing example that considers the duration of the autonomous driving level and notifies the user that activity is permitted.
[0094] 10. Examples of the hardware configuration of the information processing device according to this disclosure
[0095] 11. Summary of the composition of this disclosure
[0096] [1. A summary of examples of the components of the information processing system disclosed herein]
[0097] refer to Figure 1 This section will provide a summary of examples of the processing configurations of the information processing system disclosed herein.
[0098] Figure 1 This is a diagram illustrating an example of the configuration of the information processing system disclosed herein.
[0099] Figure 1 The diagram illustrates a mobile device (autonomous vehicle) 10 traveling on a road, a roadside communication unit (Roadside Unit: RSU) 20 as road infrastructure (Road Facility), a base station 30, and a control device 50.
[0100] For example, control device 50 is a device that constitutes the core network and performs the control and management required for communication between mobile device (autonomous vehicle) 10 and base station 30 or between mobile device (autonomous vehicle) 10 and roadside communication unit (RSU) 20.
[0101] Base station 30 or roadside communication unit (RSU) 20 provides traffic-related information in the area to mobile device (autonomous vehicle) 10. Here, the traffic-related information can be all of a high-precision three-dimensional map, referred to as a dynamic map, or it can be some information that constitutes the dynamic map, such as dynamic information, semi-dynamic information, semi-static information, or static information. For example, the roadside communication unit (RSU) 20 is installed on traffic lights, streetlights, etc.
[0102] In addition, the base station 30 or the roadside communication unit (RSU) 20 receives V2X messages sent from the mobile device (autonomous vehicle) 10. Here, for example, the V2X message is a message called a Distributed Environment Notification Message (DENM), a Cooperative Awareness Message (CAM), or a Basic Safety Message (BSM).
[0103] The control device 50 generates dynamic or semi-dynamic information, such as constituting a dynamic map, by utilizing information included in V2X messages acquired from multiple mobile devices (autonomous vehicles) 10.
[0104] In addition, the base station 30 or roadside communication unit (RSU) 20 is equipped with a camera module, such as an image sensor, and acquires image information of the surroundings. The image information acquired by the camera module is provided to the mobile device (autonomous vehicle) 10 via V2X communication.
[0105] Furthermore, the base station 30 or the roadside communication unit (RSU) 20 can generate dynamic information, semi-dynamic information, or V2X messages such as DENM, CAM, or BSM that constitute a dynamic map from the output obtained by inputting image information acquired by the camera module into artificial intelligence (AI), such as machine learning or neural network models generated through deep learning. Note that the base station 30 or the roadside communication unit (RSU) 20 can enable external devices such as cloud servers to perform AI-related processing.
[0106] Furthermore, the mobile device (autonomous vehicle) 10 conducts vehicle-to-vehicle communication with other mobile devices (autonomous vehicles) 10 via V2X communication. In vehicle-to-vehicle communication, in addition to V2X messages such as DENM, CAM, or BSM, information acquired by sensors installed on the mobile device (autonomous vehicle) 10 is shared, such as information related to speed, acceleration, angular velocity, position, and images.
[0107] Mobile device (autonomous vehicle) 10 is an autonomous vehicle.
[0108] Note that, as mentioned above, the Society of Automotive Engineers (SAE), the US automotive technology council, defines six levels of automated driving, from Level 0 (manual driving) to Level 5 (autonomous driving). Many countries around the world, including Japan, have adopted this SAE definition of automated driving levels.
[0109] refer to Figure 2 This explains the SAE's definition of autonomous driving levels.
[0110] As a level of autonomous driving, SAE defines Figure 2 The diagram shows the six levels of automated driving, from Level 0 (manual driving) to Level 5 (autonomous driving).
[0111] The definitions (driving states) of each level of autonomous driving are as follows.
[0112] Level 0 = Driver performs all driving operations (= Manual driving)
[0113] Level 1 = The automated driving system performs accelerator and brake operations or steering wheel operations.
[0114] Level 2 = The automated driving system performs accelerator, brake, and steering wheel operations.
[0115] Level 3 = The automated driving system performs all automated driving under specified conditions (e.g., a predetermined highway section). However, the driver needs to monitor the system at all times and revert to manual driving in emergencies.
[0116] Level 4 = The automated driving system performs all automated driving functions under specified conditions.
[0117] Level 5 = The automated driving system unconditionally performs all automated driving functions.
[0118] SAE defines six levels of automated driving, from Level 0 (manual driving) to Level 5 (autonomous driving).
[0119] Note that the conditions specified for Level 3 or Level 4 are, for example, conditions for driving in specific locations. Specifically, these conditions include driving on highways, driving in areas with low traffic and good visibility, such as sparsely populated areas, or driving in areas with relatively simple driving environments, such as university campuses or airport facilities.
[0120] Figure 1 The mobile device (autonomous vehicle) 10 shown in the diagram is a vehicle capable of switching between at least several levels of autonomous driving, which are defined as SAE levels. For example, such a vehicle is capable of switching from Level 0 (manual driving) to Level 2 (partial autonomous driving), from Level 0 (manual driving) to Level 4 (semi-autonomous driving), or from Level 0 (manual driving) to Level 5 (autonomous driving).
[0121] return Figure 1 The following will further describe the composition of the information processing system disclosed herein.
[0122] The mobile device 10 can communicate with each other and with the roadside communication unit (Roadside Unit: RSU) 20.
[0123] In addition, the mobile device 10 and the roadside communication unit (RSU) 20 can communicate with the control device 50 via a communication network such as the base station 30.
[0124] As described above, for example, the control device 50 is a device that constitutes the core network and performs the control and management required for communication between the mobile device (autonomous vehicle) 10 and the base station 30 or between the mobile device (autonomous vehicle) 10 and the roadside communication unit (RSU) 20.
[0125] For example, control device 50 acquires various data from mobile device 10, such as settings for autonomous driving levels, data indicating driving conditions, sensor detection information, and control information for the powertrain, braking, and steering systems installed on mobile device 10, and performs management related to the movement of mobile device 10, i.e., driving. Note that driving-related management refers to the management of autonomous driving control within the driver assistance processing unit of the information processing device installed on mobile device 10. Here, control device 50 may be an external device connected to the network functions (NFs) constituting the core network via a service-based interface.
[0126] Furthermore, the control device 50 performs data processing such as machine learning (ML) or deep learning (DL), resulting in the generation of learning models such as neural network models. For example, it generates and manages learning models for each level of autonomous driving.
[0127] In addition, the control device 50 provides map information to the mobile device 10. Note that a map information providing server different from the control device 50 can be used to provide map information to the mobile device 10.
[0128] For example, the control device 50 and the map information providing server generate a so-called Local Dynamic Map (LDM) and provide the generated Local Dynamic Map to the mobile device 10. The Local Dynamic Map is continuously updated with high density of driving map information of the road on which the vehicle travels.
[0129] refer to Figure 3 This section provides an overview of Local Dynamic Maps (LDMs). Local Dynamic Maps (LDMs) are one of the types of information required for autonomous driving. Here, LDM refers, for example, to high-resolution 3D map information.
[0130] like Figure 3 As illustrated in the diagram, an LDM comprises hierarchical information groups of various types. For example, an LDM includes the following four types of information:
[0131] Type 1 = Static Data
[0132] Type 2 = Semi-static data
[0133] Type 3 = Semi-dynamic data
[0134] Type 4 = Dynamic Data
[0135] Type 1 = Static data, such as map information generated based on medium- to long-term updated maps from the Japan Geospatial Information Agency.
[0136] Type 2 = Semi-static data, such as buildings, trees, or signs, which do not change much in the short term but change in the long term.
[0137] Type 3 = Semi-dynamic data includes data such as signals, traffic jams, or accidents that can change within a certain time unit.
[0138] Type 4 = Dynamic data, such as traffic information for vehicles, people, etc., and includes data that changes sequentially.
[0139] The control device 50 and the map information providing server send a Local Dynamic Map (LDM) containing this data to the mobile device 10. The mobile device 10 can analyze the LDM and use it for autonomous driving control, such as setting driving routes and controlling driving speed and lanes.
[0140] Note that the control device 50 and the map information providing server continuously perform local dynamic map (LDM) update processing based on the latest information, and each mobile device 10 can obtain the latest information from the server and use the latest information when using LDM.
[0141] However, currently, autonomous vehicles can only use LDM information for autonomous driving in limited areas, such as sections of highways, leaving many areas requiring manual driving by the driver. Furthermore, there are situations where the current LDM is not updated in a timely manner, in which case the map information from the old LDM may differ from the current map information. This disclosure enables reliable handling even in such cases.
[0142] refer to Figure 4 This describes an example of the overall structure and individual components of the information processing system disclosed herein.
[0143] Figure 4 This is a diagram illustrating an example of the configuration of the information processing system 70 of this disclosure.
[0144] like Figure 4 As shown in the diagram, the information processing system 70 of this disclosure includes mobile devices (autonomous driving vehicles) 10a and 10b, roadside communication units (RSUs) 20a and 20b, and control equipment 50. Furthermore, the information processing system may include a map information providing server 50b.
[0145] These devices can communicate with each other via a communication network 40.
[0146] Notice, Figure 4The diagram illustrates two mobile devices (autonomous vehicles) 10a and 10b and two roadside communication units (RSUs) 20a and 20b, but a large number of mobile devices (autonomous vehicles) 10 and roadside communication units (RSUs) 20 can exist. Furthermore, multiple control devices 50 and map information providing servers 50b can exist, as well as various other servers.
[0147] The communication network 40 includes, for example, multiple base stations. Here, the mobile device (autonomous vehicle) 10 selects nearby devices from multiple control devices 50 and a map information providing server 50b based on location information.
[0148] As described above, the mobile device (autonomous vehicle) 10 is a vehicle capable of switching between at least several levels of autonomous driving, which are defined as SAE levels.
[0149] The mobile device 10 can communicate with each other and with the roadside communication unit (Roadside Unit: RSU) 20.
[0150] Note that communication between vehicles is called vehicle-to-vehicle communication (V2V). Furthermore, communication between vehicles and infrastructure devices such as roadside communication units (RSUs) is called vehicle-to-infrastructure communication (V2I). These communications are collectively referred to as V2X communication. V2X communication includes communication between vehicles, between vehicles and pedestrians (V2P), between vehicles and networks (V2N), between vehicles and infrastructure devices, and between vehicles and servers, among others.
[0151] Mobile device 10 is a vehicle capable of performing the aforementioned V2X communication.
[0152] Mobile device 10 and roadside communication unit (RSU) 20 can communicate with control device 50 and map information provider server 50b via communication network 40 such as base station 30.
[0153] As described above, the control device 50 acquires various data from the mobile device 10, such as settings for the autonomous driving level, data indicating driving conditions, sensor detection information, and control information for the powertrain, braking, and steering systems installed on the mobile device 10, and performs management related to the movement of the mobile device 10, i.e., driving. Note that the management related to driving refers to the autonomous driving control management within the driver assistance processing unit of the information processing device installed on the mobile device 10.
[0154] Furthermore, the control device 50 performs data processing such as machine learning (ML) or deep learning (DL) using data acquired from the mobile device 10, resulting in the generation of learning models such as neural network models. For example, it generates and manages learning models for each level of autonomous driving.
[0155] Control device 50 and map information providing server 50b provide map information to mobile device 10. For example, generating map information with reference... Figure 3 The description of the composition of the Local Dynamic Map (LDM) is provided to the mobile device 10.
[0156] Mobile device 10 can analyze LDM and use LDM for autonomous driving control, such as setting driving routes and controlling driving speed and lanes.
[0157] [2. Examples of the configuration of information processing devices in mobile devices according to this disclosure]
[0158] Next, an example of the configuration of an information processing device installed on a mobile device disclosed herein will be described.
[0159] Figure 5 This is a diagram illustrating an example of the configuration of an information processing device 100 installed on a mobile device 10 of this disclosure.
[0160] like Figure 5 As shown in the diagram, the information processing device 100 includes a communication unit 110, a control unit 120, a sensor unit 131, a storage unit 132, and an input and output unit 133.
[0161] Notice, Figure 5 The configuration shown in the diagram is an example; the hardware configuration is not limited to this. Figure 5 The configuration shown in the diagram can be implemented in various different ways. Furthermore, the functions of the information processing device 100 installed on the mobile device 10 can be distributed across multiple physically separate configurations.
[0162] The communication unit 110 includes a receiving unit 111 and a transmitting unit 112.
[0163] The control unit 120 includes a data collection unit 121, a driving assistance processing unit 122, a decision unit 123, and a communication control unit 124.
[0164] Figure 5 The configuration shown in the diagram is a partial configuration of the information processing device 100 installed on the mobile device 10, and only the main configuration used for the processing disclosed herein is illustrated.
[0165] Communication unit 110 communicates with external devices under the control of communication control unit 124 of control unit 120. For example, with devices such as... Figure 4 The device communicates with external devices such as other mobile devices 10, base station 30, roadside communication unit (RSU) 20, control device 50 and map information providing server 50b shown in the diagram.
[0166] Communication unit 110 supports one or more wireless communication access methods. For example, communication unit 110 supports both Long Term Evolution (LTE) and 5G Radio (NR) methods. In addition, communication unit 110 can be configured to support various communication methods such as W-CDMA and CDMA2000 in addition to LTE and 5G NR methods.
[0167] The communication unit 110 includes a receiving unit 111, a transmitting unit 112, and an antenna 113. Note that the communication unit 110 may include multiple receiving units 111, transmitting units 112, and antennas 113. For example, when the communication unit 110 supports multiple wireless access methods, each unit of the communication unit 110 may be configured separately for each wireless access method. For example, the receiving unit 111 and the transmitting unit 112 may be configured in LTE and 5G NR modes, respectively.
[0168] The sensor unit 131 includes multiple different sensors.
[0169] The sensor detection information of sensor unit 131 is acquired by data collection unit 121 of control unit 120 and stored in driving assistance processing unit 122 or accessible storage unit 132 of driving assistance processing unit 122. In addition, the sensor detection information of sensor unit 131 is provided to external devices, such as control device 50, via communication unit 110 as needed.
[0170] Figure 6 The diagram illustrates a detailed example of the configuration of sensor unit 131.
[0171] like Figure 6 As shown in the diagram, the sensor unit 131 includes a position information sensor 151, a camera module (including an image sensor) 152, a light detection and ranging or laser imaging detection and ranging (LiDAR) 153, a radar 154, a sensor 155, etc.
[0172] Location information sensors 151 include, for example, Global Navigation Satellite Systems (GNSS), represented by the Global Positioning System (GPS).
[0173] Alternatively, the location information sensor may be a sensor that uses positioning technology that uses signals transmitted and received via a communication unit 110 that supports odometer, LTE, 4G or 5G cellular systems.
[0174] The camera module 152 is equipped with multiple image sensors to acquire image information about the exterior of the mobile device (vehicle) and image information about the interior of the vehicle, including the driver's movements and expressions.
[0175] LiDAR 153 and Radar 154 detect various objects such as other vehicles and obstacles around mobile devices and acquire the data needed to measure the distance to the detected objects.
[0176] Sensor 155 includes, for example, an inertial measurement unit (IMU), which is a unit that integrates an accelerometer, a rotational angular acceleration sensor, a gyroscope sensor, a magnetic field sensor, a barometric pressure sensor, a temperature sensor, etc.
[0177] As described above, the information acquired by the sensor unit 131 is provided to the driving assistance processing unit 122 via the data collection unit 121 of the control unit 120, and is stored in the accessible storage unit 132 of the driving assistance processing unit 122. Furthermore, the information acquired by the sensor unit 131 may be provided to external devices, such as the control device 50, via the communication unit 110 as needed.
[0178] For example, storage unit 132 is a storage device such as DRAM, SRAM, flash memory, or hard disk capable of reading and writing data. Storage unit 132 serves as the storage section of information processing device 100.
[0179] For example, the input and output unit 133 is an interface for exchanging information with a user such as a driver. For example, the input and output unit 133 serves as an interface that can be operated by the driver, such as operation keys and a touch panel.
[0180] The input and output unit 133 can be configured as a display device such as a liquid crystal display or an organic electroluminescent (EL) display. Furthermore, the input and output unit 133 can be a sound device such as a speaker, microphone, or buzzer. Additionally, the input and output unit 133 can be a lighting device such as a light-emitting diode (LED) lamp. The input and output unit 133 functions as the input and output section (input section, output section, operation section, or notification section) of the information processing device 100.
[0181] Control unit 120 is a controller that controls the various units of information processing device 100. For example, control unit 120 includes processors such as CPU, MPU, and GPU, as well as memories such as RAM and ROM. The processor of control unit 120 uses memory such as RAM as its working area to execute various programs stored in storage unit 132 and perform various processes required by information processing device 100.
[0182] Note that the control unit 120 can be implemented using integrated circuits such as ASICs or FPGAs. Any of a CPU, MPU, GPU, ASIC, and FPGA can be considered a controller.
[0183] like Figure 5 As shown in the diagram, the control unit 120 includes a data collection unit 121, a driving assistance processing unit 122, a decision unit 123, and a communication control unit 124.
[0184] Each block constituting the control unit 120 (data collection unit 121 to communication control unit 124) is a function block that indicates the function of the control unit 120. These function blocks can be software blocks or hardware blocks.
[0185] For example, each of the above-mentioned functional blocks can be a software module implemented through software (including microprograms), or a circuit block on a semiconductor chip (wafer). Furthermore, each functional block can be a processor or an integrated circuit. The configuration of the functional blocks is arbitrary. Note that the control unit 120 can be composed of functional units different from the functional blocks described above.
[0186] Notice, Figure 5 The information processing device 100 shown in the diagram is a device installed in the mobile device 10, and for example, the driver assistance processing unit 122 of the control unit 120 controls various components of the vehicle such as the engine, accelerator, brakes, steering wheel (steering gear) and transmission, and performs automatic driving control.
[0187] The data collection unit 121 acquires sensor detection information from the sensor unit 131. The sensor detection information collected by the data collection unit 121 is provided to the driver assistance processing unit 122 and stored in the accessible storage unit 132 of the driver assistance processing unit 122. Furthermore, the sensor detection information from the sensor unit 131 may be provided to external devices, such as the control device 50, via the communication unit 110 as needed.
[0188] As described above, the driver assistance processing unit 122 controls various components of the vehicle, such as the engine, accelerator, brakes, steering wheel (steering gear) and transmission of the mobile device 10, and performs autonomous driving control.
[0189] The driver assistance processing unit 122 performs autonomous driving control by controlling the driving steering unit of the mobile device 10 based on the output obtained by inputting the data collected by the data collection unit 121 into an artificial intelligence (AI) model, such as machine learning or a neural network model generated by deep learning.
[0190] For example, implementing automated driving control via advanced driver assistance systems (ADAS) or autonomous driving (AD).
[0191] The driver assistance processing unit 122 performs, for example, the processing of providing control information such as acceleration and deceleration of the mobile device 10 to the power system, providing control information such as deceleration and stopping to the braking device, and providing control information such as left xx [cm] and right yy [cm] to the steering device (steering gear).
[0192] Note that the neural network model (learning model) used by the driver assistance processing unit 122 for autonomous driving control can be pre-installed in the driver assistance processing unit 122, or it can be obtained from the external control device 50 or other external servers via the base station 30, roadside communication unit (RSU) 20, etc., stored in the storage unit 132, and updated as appropriate.
[0193] Furthermore, the neural network model (learning model) used by the driver assistance processing unit 122 for autonomous driving control can be a single neural network model, or it can consist of multiple neural network models. These multiple neural network models can be neural network models prepared for each sensor as so-called edge AI, or neural network models prepared for each control function of the control unit 120.
[0194] The information processing device 100 installed on the mobile device 10 according to this embodiment detects whether there is an anomaly during automatic or manual driving based on information obtained via the data collection unit 121 or the communication unit 110 of the control unit 120. Furthermore, if an anomaly is detected, an appropriate response is taken corresponding to the detected anomaly, and processing for maintaining safe driving is executed.
[0195] Furthermore, when the mobile device 10 is in autonomous driving mode, the information processing device 100 installed on the mobile device 10 according to this embodiment notifies the driver of permitted activities as actions permitted by the driver, according to the level of autonomous driving being performed.
[0196] For example, the system can notify the user of permitted activities based on the level of autonomous driving being performed, such as being able to make a phone call at level 2 and being able to read or take a nap at level 4.
[0197] By informing the driver of such permitted activities corresponding to the level of autonomous driving, the driver can confirm the permitted activities and only perform the permitted activities.
[0198] For example, in the case of Level 2 autonomous driving, it is possible to prevent accidents caused by taking a nap, which is an activity that is not permitted, thereby enabling safe autonomous driving.
[0199] The specific processing will be explained below.
[0200] [3. Basic sequence of processes performed by the information processing equipment of this disclosure]
[0201] Next, the basic sequence of processes performed by the information processing device of this disclosure will be described.
[0202] refer to Figure 7 The flowchart shown in the figure will explain the process from the reference. Figure 5 The information processing device 100 of this disclosure describes the basic sequence of processes performed by the information processing device 100 installed on the mobile device 10.
[0203] Note that the processing described below can be executed according to a program stored in, for example, the storage unit 132 of the information processing device 100. For example, the processing can be executed under the control of a data processing unit (control unit) including a CPU with program execution capabilities.
[0204] The following will explain in sequence. Figure 7 The flowchart shown in the middle illustrates the processing of each step.
[0205] (Step S101)
[0206] First, in step S101, the control unit 120 of the information processing device 100 specifies the level of autonomous driving that the mobile device 10 is currently performing.
[0207] The processing is performed by the data collection unit 121, communication unit 110, and driving assistance processing unit 122 in the control unit 120 of the information processing device 100.
[0208] For example, data collected by data collection unit 121 and data obtained from external sources by communication unit 110 are used for the specified processing of autonomous driving level.
[0209] The driver assistance processing unit 122 inputs data collected by the data collection unit 121 and data obtained from the outside by the communication unit 110, and specifies the level of autonomous driving that the mobile device 10 is currently performing based on this data.
[0210] Data collection unit 121 collects sensor detection information from sensor unit 131. (See above reference.) Figure 6 The sensor unit 131 includes a location information sensor 151, a camera module 152, a LiDAR 153, a radar 154, and a sensor 155.
[0211] The data collection unit 121 acquires detection information from the various sensors of the sensor unit 131. The information acquired by the data collection unit 121 includes, for example, its own position information, images captured by the camera, and object distance information.
[0212] The information acquired by the data collection unit 121 is input into the driving assistance processing unit 122.
[0213] In addition, the communication unit 110 inputs information obtained from other mobile devices, roadside infrastructure such as base station 30 and roadside communication unit (RSU) 20, and external servers such as control device 50 and map information providing server 50b into the driver assistance processing unit 122.
[0214] For example, while the control device 50 maintains the autonomous driving management data of the mobile device 10, information about the level of autonomous driving currently being performed by the mobile device 10 can be obtained from the control device 50.
[0215] In addition, the permissible autonomous driving level information for each road segment registered in the map information provided by the map information providing server 50b can be obtained, and the permissible autonomous driving level information can be used to determine the level of autonomous driving currently being performed by the mobile device 10.
[0216] Furthermore, if the level of autonomous driving currently being performed by the mobile device 10 is set or recorded in the storage unit 131 or memory within the mobile device 10, this information can be obtained.
[0217] For example, the driver assistance processing unit 122 can specify the level of autonomous driving that the mobile device 10 is currently performing based on any of the methods described above.
[0218] As described above, in step S101, the driving assistance processing unit 122 inputs the data collected by the data collection unit 121 and the data obtained from the outside by the communication unit 110, and specifies the level of autonomous driving that the mobile device 10 is currently performing based on this data.
[0219] (Step S102)
[0220] Next, in step S102, the control unit 120 of the information processing device 100 obtains a list of activities (actions) permitted to the driver that are set as list data corresponding to the level of autonomous driving, that is, a list of permitted activities associated with the level of autonomous driving.
[0221] The allowed activity list is used as the list data setting for the above reference. Figure 2 This describes a list of activities that the driver is allowed to perform at each level of automated driving (levels 0-5).
[0222] For example, the driver assistance processing unit 122 of the control unit 120 of the information processing device 100 receives the list of permitted activities from the control device 50 via the communication unit 110.
[0223] Note that the list of permitted activities associated with the autonomous driving level can also be stored in the storage unit 131 within the information processing device 100 of the mobile device 10. In this case, the driving support processing unit 122 of the control unit 120 of the information processing device 100 retrieves the list of permitted activities from the storage unit 131.
[0224] Figure 8 This diagram illustrates an example of a list of permitted activities associated with a level of autonomous driving.
[0225] like Figure 8 As shown in the diagram, the list of permitted activities is a list that includes the autonomous driving level (0-5) and the corresponding data for the activities permitted for the driver as a user at each autonomous driving level (0-5).
[0226] like Figure 8 As shown in the diagram, the activities permitted for the driver vary depending on the level of automation (0-5).
[0227] exist Figure 8 In the example list shown in the diagram, there are no activities permitted for the driver at Autopilot Level 0 (manual driving) and Level 1.
[0228] At Level 2 of autonomous driving, the only activity permitted for the driver is making a phone call.
[0229] At Level 3 of autonomous driving, the activities permitted by the driver are:
[0230] (1) Eat, (2) Make a phone call, (3) Check and reply to emails, (4) Read a book, and (5) Play a game.
[0231] At Level 4 of autonomous driving, the activities permitted by the driver are:
[0232] (1) Eat, (2) Make a phone call, (3) Check and reply to emails, (4) Read a book, (5) Play a game, (6) Make a video call, and (7) Take a nap.
[0233] At Level 5 of autonomous driving, the activities permitted for the driver are:
[0234] (1) eating, (2) making phone calls, (3) checking and replying to emails, (4) reading, (5) playing games, (6) making video calls, (7) taking naps, and (8) all other activities.
[0235] The list of permitted activities consists of data that, as described above, correlates the level of autonomous driving with the activities permitted for the driver.
[0236] In step S102, the control unit 120 of the information processing device 100 obtains such as Figure 8 The diagram shows a list of permitted activities associated with each level of autonomous driving.
[0237] (Step S103)
[0238] Next, in step S103, the control unit 120 of the information processing device 100 selects, from the list of permitted activities corresponding to the autonomous driving level obtained in step S102, the permitted activities for the driver associated with the autonomous driving level currently being executed by the mobile device 10.
[0239] This processing is performed by the driver assistance processing unit 122 in the control unit 120 of the information processing device 100.
[0240] (Step S104)
[0241] Finally, in step S104, the control unit 120 of the information processing device 100 outputs the driver-permitted activities obtained from the list of permitted activities in step S103, that is, the driver-permitted activities associated with the current autonomous driving level being performed by the mobile device 10, to the input and output unit 133 of the mobile device 10.
[0242] For example, a message explaining the driver's permitted activities can be output via a display unit or voice output unit that constitutes the input and output unit 133.
[0243] Note that it is possible to notify the driver of activities that cannot be performed. Alternatively, a unit can be set up to determine the activities that the driver can and cannot perform, and notifications for both types of activities can be provided together.
[0244] refer to Figure 9 This section will illustrate an example of a notification using a display unit that corresponds to a permitted activity level for autonomous driving.
[0245] like Figure 9 As shown in the diagram, a message containing a list of permitted activities corresponding to the level of autonomous driving is displayed on a display unit of a mobile device that is visible to the driver.
[0246] Figure 9 The example of the displayed information shown in the diagram is an example of the following message.
[0247] "Currently, the autonomous driving level is Level 3. At Level 3, the following activities are permitted for the driver:"
[0248] (1) Eating
[0249] (2) Making a phone call
[0250] (3) View and reply to emails
[0251] (4) Reading books
[0252] (5) Playing games
[0253] As a user, the driver can see the message displayed on the display unit to confirm that the current autonomous driving level of the mobile device 10 is Level 3. In addition, at the current autonomous driving level 3, the driver is allowed to engage in activities such as (1) eating, (2) making phone calls, (3) viewing and replying to emails, (4) reading books, and (5) playing games.
[0254] Users can confirm the permitted activities corresponding to Level 3 of autonomous driving displayed on the display unit and strive to act within the scope of the displayed permitted activities.
[0255] Note that notification processing for permitted activities corresponding to autonomous driving levels is not limited to, for example... Figure 9 The display processing on the display unit shown in the diagram can, for example, be notified via voice information.
[0256] [4. Regarding the analysis of the driver's confirmation status of the notification of permission to move, and the processing sequence for emergency response in the absence of confirmation notification]
[0257] Next, we will explain the process of analyzing the driver's confirmation status of the notification that permission to move is granted, and the sequence of actions to respond to an emergency in the absence of a confirmation notification.
[0258] Reference Figure 7 The flowchart shown in the diagram illustrates the processing sequence in which only notifications of permitted activities corresponding to the level of autonomous driving are made, without performing analysis on whether the driver has acknowledged the notification.
[0259] The process described below analyzes the driver's confirmation status of the notification allowing activity and performs an emergency response if the notification is not confirmed.
[0260] Reference Figure 10 The flowchart shown in the diagram illustrates the sequence of processes.
[0261] Figure 10 The flowchart shown in the diagram is in which the above Figure 7 The flowchart shown in the diagram adds steps S121 and S122 after step S104.
[0262] The following will explain Figure 10 The flowchart shown in the middle illustrates the processing of each step.
[0263] (Steps S101 to S104)
[0264] The processing in steps S101 to S104 is similar to that in the reference. Figure 7 The processing of the description.
[0265] These processes will be briefly explained.
[0266] First, in step S101, the control unit 120 of the information processing device 100 specifies the level of autonomous driving that the mobile device 10 is currently performing.
[0267] Next, in step S102, the list of activities (actions) allowed for the driver corresponding to the autonomous driving level is obtained as list data. That is, the list of allowed activities corresponding to the autonomous driving level.
[0268] Furthermore, in step S103, the control unit 120 of the information processing device 100 selects, from the list of permitted activities corresponding to the autonomous driving level obtained in step S102, the permitted activities for the driver associated with the autonomous driving level currently being executed by the mobile device 10.
[0269] Next, in step S104, the control unit 120 of the information processing device 100 outputs the driver-permitted activities obtained from the list of permitted activities in step S103, that is, the driver-permitted activities associated with the current autonomous driving level being performed by the mobile device 10, to the input and output unit 133 of the mobile device 10.
[0270] The processing up to this point is similar to the reference. Figure 7 The processing of the description.
[0271] (Step S121)
[0272] Next, in step S121, the control unit 120 of the information processing device 100 determines whether it has detected confirmation input from the driver for the permitted activities corresponding to the autonomous driving level that were notified to the driver in step S104.
[0273] Figure 11 The diagram illustrates an example of a user interface that enables the detection of confirmation input from the driver.
[0274] Figure 11 The diagram illustrates how to refer to the above. Figure 9 The example of a notification that allows activity corresponding to the described level of autonomous driving is similar to the display data obtained by adding user input via the confirmation button 201.
[0275] The display unit is a touch panel, and when the user (driver) touches the confirmation button 201, a detection signal indicating that the user has confirmed is input to the control unit 120 of the information processing device 100.
[0276] exist Figure 10 In step S121 of the process shown in the diagram, the presence or absence of a detection signal is determined.
[0277] When the user (driver) touches the confirmation button 201 and inputs a detection signal to the control unit 120 of the information processing device 100, the determination in step S121 is "yes". That is, it is determined that the driver has confirmed the permitted activity data corresponding to the autonomous driving level displayed on the display unit, and then the process ends.
[0278] On the other hand, if the user (driver) does not touch the confirmation button 201 within a predetermined time (e.g., 10 seconds) and does not input a detection signal to the control unit 120 of the information processing device 100, the determination in step S121 is "No". That is, it is determined that the driver has not confirmed the permitted activity data corresponding to the autonomous driving level displayed on the display unit.
[0279] In this case, the process proceeds to step S122.
[0280] (Step S122)
[0281] In the determination process of step S121, if it is determined that the driver has not confirmed the permitted activity data corresponding to the autonomous driving level displayed on the display unit, the process proceeds to step S122.
[0282] In this situation, the control unit 120 of the information processing device 100 performs emergency response processing in step S122.
[0283] For example, emergency response handling is any of the following:
[0284] (a) Emergency stop of mobile device 10;
[0285] (b) Switch mobile device 10 to slow driving mode; and
[0286] (c) Output a warning sound.
[0287] Or it can be obtained by combining multiple processes.
[0288] For example, if it is determined that there is no valid data on the display unit corresponding to the autonomous driving level, the driver assistance processing unit 122 of the control unit 120 performs an emergency stop on the mobile device 10 and outputs a warning sound as an emergency response.
[0289] By implementing such an emergency response, the situation where the mobile device 10 is driven while the driver is engaged in an activity that is not permitted can be avoided, thereby preventing accidents in advance.
[0290] [5. The procedure for monitoring the driver's actions after notification of permitted activity and for emergency response when the driver is performing prohibited activity]
[0291] Next, we will describe the sequence of actions to monitor the driver's behavior after the notification that the activity is permitted, and the emergency response procedures to take when the driver is performing an activity that is not permitted.
[0292] Reference Figure 12 The flowchart shown in the diagram illustrates the sequence of processes.
[0293] Figure 12 The flowchart shown in the diagram is one of the above. Figure 7 The flowchart shown in the middle diagram adds steps S141 to S143 after step S104.
[0294] The following will explain Figure 12 The flowchart shown in the middle illustrates the processing of each step.
[0295] (Steps S101 to S104)
[0296] The processing in steps S101 to S104 is similar to that in the reference. Figure 7 The processing of the description.
[0297] These processes will be briefly explained.
[0298] First, in step S101, the control unit 120 of the information processing device 100 specifies the level of autonomous driving that the mobile device 10 is currently performing.
[0299] Next, in step S102, the list of activities (actions) allowed for the driver corresponding to the autonomous driving level is obtained as list data. That is, the list of allowed activities corresponding to the autonomous driving level.
[0300] Furthermore, in step S103, the control unit 120 of the information processing device 100 selects, from the list of permitted activities corresponding to the autonomous driving level obtained in step S102, the permitted activities for the driver associated with the autonomous driving level currently being executed by the mobile device 10.
[0301] Next, in step S104, the control unit 120 of the information processing device 100 outputs the driver-permitted activities obtained from the list of permitted activities in step S103, that is, the driver-permitted activities associated with the current autonomous driving level being performed by the mobile device 10, to the input and output unit 133 of the mobile device 10.
[0302] The processing up to this point is similar to the reference. Figure 7 The processing of the description.
[0303] (Step S141)
[0304] Next, in step S141, the control unit 120 of the information processing device 100 analyzes the driver's actions based on sensor detection information.
[0305] Note that in performing the processing of this embodiment, sensors such as cameras are required to acquire information for analyzing the driver's actions inside the vehicle.
[0306] Figure 13 The illustration shows an example of the configuration of the sensor unit 131 in this embodiment.
[0307] Figure 13 The sensor unit 131 shown in the diagram has a reference to the upper part. Figure 6 The sensor unit 131 described herein includes a configuration in which a driver monitoring sensor 156 is added.
[0308] The driver monitoring sensor 156 is a sensor that acquires information for analyzing the driver's actions inside the vehicle, and includes, for example, a camera, microphone, distance sensor, odor sensor, etc.
[0309] The data collection unit 121 of the control unit 120 inputs the information acquired by the driver monitoring sensor 156 and inputs the acquired information to the decision unit 123.
[0310] The decision unit 123 analyzes the information acquired by the driver monitoring sensor 156 and estimates the driver's ongoing activities. This estimation is based on the output obtained by inputting the information acquired by the driver monitoring sensor 156 into a neural network model generated through AI, such as machine learning or deep learning.
[0311] (Step S142)
[0312] In step S142, it is determined whether the driver's activity estimated in step S141 is a permitted activity corresponding to the current level of autonomous driving.
[0313] This process is executed by the decision unit 123 of the control unit 120.
[0314] The determination unit 123 determines whether the estimated driver activity is a permitted activity corresponding to the current level of autonomous driving.
[0315] The process ends when it is determined that the driver's activity is a permitted activity corresponding to the current level of autonomous driving.
[0316] On the other hand, if it is determined that the driver's activity is not a permitted activity corresponding to the current level of autonomous driving, the process proceeds to step S143.
[0317] (Step S143)
[0318] When it is determined in step S142 that the driver is performing an activity that is not a permitted activity corresponding to the current level of autonomous driving, the process proceeds to step S143.
[0319] In this situation, the control unit 120 of the information processing device 100 performs emergency response processing in step S143.
[0320] For example, emergency response handling is any of the following:
[0321] (a) Emergency stop of mobile device 10;
[0322] (b) Switch mobile device 10 to slow driving mode; and
[0323] (c) Output a warning sound.
[0324] Or it can be obtained by combining multiple processes.
[0325] For example, when it is determined that the driver is performing an activity that is not permitted under the current level of autonomous driving, the driver assistance processing unit 122 of the control unit 120 performs an emergency stop of the mobile device 10 and outputs a warning sound as an emergency response.
[0326] By implementing such an emergency response, the situation where the mobile device 10 is driven while the driver is engaged in an activity that is not permitted can be avoided, thereby preventing accidents in advance.
[0327] [6. Processing sequence for detecting changes in autonomous driving levels and issuing change notifications for permitted activities]
[0328] Next, as an example of processing performed by the information processing device of this disclosure, a processing sequence for detecting a change in the level of autonomous driving and executing a change notification that allows activity will be described.
[0329] Reference Figure 14 The flowchart shown in the diagram is used for explanation.
[0330] Note that the program can be executed according to the program stored in, for example, the storage unit 132 of the information processing device 100. Figure 14 The process is illustrated in the diagram. For example, the process is executed under the control of a data processing unit (control unit), including a CPU with program execution capabilities.
[0331] The following will explain in sequence. Figure 14 The flowchart shown in the middle illustrates the processing of each step.
[0332] (Step S201)
[0333] First, in step S201, the control unit 120 of the information processing device 100 specifies the level of autonomous driving that the mobile device 10 is currently performing.
[0334] The processing is performed by the data collection unit 121, communication unit 110, and driving assistance processing unit 122 in the control unit 120 of the information processing device 100.
[0335] For example, data collected by data collection unit 121 and data obtained from external sources by communication unit 110 are used for the specified processing of autonomous driving level.
[0336] The driver assistance processing unit 122 inputs data collected by the data collection unit 121 and data obtained from the outside by the communication unit 110, and specifies the level of autonomous driving that the mobile device 10 is currently performing based on this data.
[0337] Data collection unit 121 collects sensor detection information from sensor unit 131. (See above reference.) Figure 6 The sensor unit 131 includes a location information sensor 151, a camera module 152, a LiDAR 153, a radar 154, and a sensor 155.
[0338] The data collection unit 121 acquires detection information from the various sensors of the sensor unit 131. The information acquired by the data collection unit 121 includes, for example, its own position information, images captured by the camera, and object distance information.
[0339] The information acquired by the data collection unit 121 is input into the driving assistance processing unit 122.
[0340] In addition, the communication unit 110 inputs information obtained from other mobile devices, roadside infrastructure such as base station 30 and roadside communication unit (RSU) 20, and external servers such as control device 50 and map information providing server 50b into the driver assistance processing unit 122.
[0341] For example, while the control device 50 maintains the autonomous driving management data of the mobile device 10, information about the level of autonomous driving currently being performed by the mobile device 10 can be obtained from the control device 50.
[0342] In addition, the permitted autonomous driving level information for each road segment registered in the map information providing server 50b can be obtained, and the permitted autonomous driving level information can be used to determine the level of autonomous driving currently being performed by the mobile device 10.
[0343] Furthermore, if the level of autonomous driving currently being performed by the mobile device 10 is set or recorded in the storage unit 131 or memory within the mobile device 10, this information can be obtained.
[0344] For example, the driver assistance processing unit 122 can specify the level of autonomous driving that the mobile device 10 is currently performing based on any of the methods described above.
[0345] As described above, in step S201, the driving assistance processing unit 122 inputs the data collected by the data collection unit 121 and the data obtained from the outside by the communication unit 110, and based on this data, specifies the level of autonomous driving that the mobile device 10 is currently performing.
[0346] (Step S202)
[0347] Next, in step S202, the control unit 120 of the information processing device 100 determines whether there is a change (transformation) in the autonomous driving level being performed by the mobile device 10.
[0348] This process is performed by the decision unit 123 of the control unit 120 of the information processing device 100.
[0349] In step S203, the determination unit 123 of the control unit 120 of the information processing device 100 determines whether there is a change (transformation) in the autonomous driving level being performed by the mobile device 10.
[0350] The change (modification) of the autonomous driving level occurs due to, for example, a change in the permitted autonomous driving level set for the road segment corresponding to the road in which the mobile device 10 is traveling.
[0351] For example, if the permitted level of autonomous driving is level 3 in road segment A to B of the road on which the mobile device 10 is traveling, and the permitted level of autonomous driving is level 4 in the next road segment B to C, the mobile device 10 will automatically change the autonomous driving level from level 3 to level 4.
[0352] Note that the process of changing the autonomous driving level in the mobile device 10 is performed under the control of the driving assistance processing unit 122 of the control unit 120 of the information processing device 100 in the mobile device 10.
[0353] The driver assistance processing unit 122 performs a change in the level of autonomous driving, for example, based on instructions from the control device 50.
[0354] Alternatively, the autonomous driving level can be changed based on the permitted autonomous driving level information for each road segment recorded in the map information received from the map information provider server 50b.
[0355] If the determination unit 123 of the control unit 120 of the information processing device 100 determines in step S202 that a change (modification) in the autonomous driving level of the mobile device 10 has been detected, the processing proceeds to step S203.
[0356] On the other hand, in step S202, if the determination unit 123 determines that no change (modification) of the autonomous driving level of the mobile device 10 is detected, the process returns to step S201 and continues the process of step S201.
[0357] (Step S203)
[0358] When it is determined in step S202 that a change (modification) in the autonomous driving level of the mobile device 10 is detected, in step S203, a list of permitted activities in which the autonomous driving level is associated with the activities permitted for the driver is obtained.
[0359] As per the above reference Figure 8 The permitted activity list is a table that associates the level of autonomous driving with the activities permitted for the driver.
[0360] The process of obtaining the list of permitted activities corresponding to the autonomous driving level is performed by the driving assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0361] In step S203, the driving assistance processing unit 122 of the control unit 120 of the information processing device 100 obtains a list of permitted activities associated with the autonomous driving level.
[0362] The list of allowed activities is one of those referenced above. Figure 2 The descriptions of the various levels of autonomous driving (levels 0-5) are accompanied by a list of activities permitted for the driver.
[0363] For example, the driver assistance processing unit 122 of the control unit 120 of the information processing device 100 receives the list of permitted activities corresponding to the autonomous driving level from the control device 50 via the communication unit 110.
[0364] Note that the list of permitted activities corresponding to the autonomous driving level can also be stored in the storage unit 131 of the information processing device 100 in the mobile device 10. In this case, the driving assistance processing unit 122 of the control unit 120 of the information processing device 100 can obtain the list of permitted activities corresponding to the autonomous driving level from the storage unit 131.
[0365] In step S203, the control unit 120 of the information processing device 100 obtains such as Figure 8 The diagram shows the association between the autonomous driving level and the list of permitted activities corresponding to the autonomous driving level.
[0366] (Step S204)
[0367] Next, in step S204, the control unit 120 of the information processing device 100 refers to the list of permitted activities corresponding to the autonomous driving level obtained in step S203 and analyzes the difference between the permitted activities before and after the change of the autonomous driving level of the mobile device 10 detected in step S202.
[0368] This processing is performed by the driver assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0369] The driver assistance processing unit 122 of the control unit 120 of the information processing device 100 analyzes the difference between permitted activities before and after the transition to a new level of autonomous driving.
[0370] For example, if the autonomous driving level before the transition is Level 3 and the autonomous driving level after the transition is Level 4, analyze the differences between the permitted activities corresponding to Level 3 and Level 4.
[0371] (Step 205)
[0372] Next, in step S205, based on the difference analysis results in step S204, the control unit 120 of the information processing device 100 determines whether there is a difference in the permitted activities before and after the change in the autonomous driving level.
[0373] This processing is performed by the driver assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0374] If the driver assistance processing unit 122 of the control unit 120 of the information processing device 100 determines in step S205 that there is a difference between the permitted activities before and after the change of the autonomous driving level, the processing proceeds to step S206.
[0375] On the other hand, when it is determined in step S205 that there is no difference between the permitted activities before and after the transition of the autonomous driving level, the process ends.
[0376] (Step S206)
[0377] When it is determined in step S205 that there is a difference between the permitted activities before and after the transition of the autonomous driving level, the processing in step S206 is performed.
[0378] This processing is performed by the driver assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0379] If, in step S205, the driver assistance processing unit 122 of the control unit 120 of the information processing device 100 determines that there is a difference between the permitted activities before and after the change of the autonomous driving level, in step S206, the permitted activities after the change (modification) of the autonomous driving level, that is, the permitted activities that clearly indicate the difference caused by the change of the autonomous driving level, are output to the input and output unit 133 of the mobile device 10.
[0380] For example, a message explaining the driver's permitted activities can be output via a display unit or voice output unit that constitutes the input and output unit 133.
[0381] Note that it is preferable to notify the difference in permitted activities before and after the transition of the autonomous driving level, so that the difference can be easily understood.
[0382] refer to Figure 15 The accompanying drawings will illustrate examples of notifications of permitted activities corresponding to the level of autonomous driving using a display unit.
[0383] Note that the handling of changes in autonomous driving levels has the following two distinct aspects.
[0384] (a) Level change processing for upgrading the level of automated driving, i.e., level change processing regarding the proportion of reduced human intervention in driving operations, and
[0385] (b) Level change processing for downgrading the level of autonomous driving, i.e., level change processing for increasing the proportion of human intervention in driving operations.
[0386] The following will explain the different situations:
[0387] (a) Processing of Level Changes for Upgrading Automated Driving Levels
[0388] (b) Processing of Level Changes for Downgrading Automated Driving Level
[0389] When the level of autonomous driving is increased and new permitted activities are added, the driver needs to be notified of the new permitted activities at least once.
[0390] First, refer to Figure 15 and Figure 16 This illustrates an example of the display data of the display unit and the processing of the driver assistance processing unit 122 when the autonomous driving level before the transition is level 3 and the autonomous driving level after the transition is level 4.
[0391] Figure 15 It is a diagrammatic explanation. Figure 14 The diagram illustrates steps S204 and S205 of the process, which are examples of processing performed by the driving assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0392] In steps S204 and S205, the driver assistance processing unit 122 refers to the list of permitted activities corresponding to the autonomous driving level obtained in step S203 and analyzes the differences between the permitted activities before and after the change in the autonomous driving level of the mobile device 10 detected in step S202.
[0393] The driver assistance processing unit 122 analyzes the differences between permitted activities before and after the transition to a new level of autonomous driving.
[0394] Figure 15 The diagram illustrates the permitted activities for Level 3 and Level 4 of Automated Driving, as recorded in the list of permitted activities corresponding to each Automated Driving Level.
[0395] like Figure 15 As shown in the diagram, the permitted activities for Automated Driving Level 3, recorded in the list of permitted activities corresponding to each Automated Driving Level, are the following five activities:
[0396] (1) Eating
[0397] (2) Making a phone call
[0398] (3) View and reply to emails
[0399] (4) Reading books
[0400] (5) Playing games
[0401] On the other hand, the permitted activities for Automated Driving Level 4, which are recorded in the list of permitted activities corresponding to the Automated Driving Level, are the following 7 activities:
[0402] (1) Eating
[0403] (2) Making a phone call
[0404] (3) View and reply to emails
[0405] (4) Reading books
[0406] (5) Playing games
[0407] (6) Video call
[0408] (7) Nap
[0409] exist Figure 14 In steps S204 and S205 of the process shown in the diagram, the driver assistance processing unit 122 analyzes the differences between the permitted activities of autonomous driving level 3 and autonomous driving level 4.
[0410] from Figure 15 As can be understood from the data shown in the diagram, the difference between the permitted activities at Level 3 and Level 4 of Automated Driving is the following two activities added as permitted activities at Level 4 of Automated Driving.
[0411] (6) Video call
[0412] (7) Nap
[0413] The driver assistance processing unit 122 determines that there is a difference between the permissible activities before and after the transition of the autonomous driving level, thereby... Figure 14 In the process illustrated in the diagram, step S205 is determined to be "yes", and then the process proceeds to step S206.
[0414] In step S206, the driver assistance processing unit 122 outputs the permitted activities after the change (modification) of the autonomous driving level, that is, the activities that the driver is allowed to perform in relation to the current autonomous driving level being performed by the mobile device 10, to the display unit of the mobile device 10.
[0415] For example, the driver assistance processing unit 122 generates display data that facilitates understanding of the differences in permitted activities before and after the transition to a new level of autonomous driving, and displays the display data on the display unit.
[0416] Figure 16 This is an illustration illustrating an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0417] Figure 16 The diagram shows an example of the following messages being displayed.
[0418] "The autonomous driving level has been changed from Level 3 to Level 4. As a Level 4 driver permitted to engage in activities, the following (6) and (7) have been added."
[0419] (1) Eating
[0420] (2) Making a phone call
[0421] (3) View and reply to emails
[0422] (4) Reading books
[0423] (5) Playing games
[0424] (6) Video call
[0425] (7) Nap
[0426] like Figure 16 The diagram in the middle highlights the two newly added activities.
[0427] (6) Video calls, and
[0428] (7) Nap
[0429] For example, by outputting the two activities in a different color such as red, or by changing their size and displaying them at a larger size, these two activities are made more eye-catching than the other activities (1) to (5). That is, different visual features are added to the display of these two activities.
[0430] For example, such as Figure 16 As shown in the diagram, the driver assistance processing unit 122 generates and outputs display data that is easy to understand regarding the differences in permitted activities before and after the level transition.
[0431] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 has changed from level 3 to level 4. In addition, as for the activities allowed for the driver in the autonomous driving level 4 after the change, in addition to the activities allowed in level 3 (1) eating, (2) making phone calls, (3) checking and replying to emails, (4) reading books and (5) playing games, new activities (6) video calls and (7) taking a nap are also allowed.
[0432] Users can confirm the permitted activities corresponding to Level 4 of autonomous driving displayed on the display unit and strive to act within the scope of the displayed permitted activities.
[0433] Note that notification processing for permitted activities corresponding to autonomous driving levels is not limited to, for example... Figure 16 The display processing on the display unit shown in the diagram can, for example, be notified via voice information.
[0434] Notice, Figure 16 The example of display data shown in the diagram is just one example; other different display data can be generated and output.
[0435] Reference Figure 17 and Figure 18 This explains how it generates and displays data. Figure 16 The diagram shows examples of different display data.
[0436] Similar to Figure 15 and Figure 16 , Figure 17 and Figure 18 The example shown in the diagram is also an example of the display data of the display unit and the processing of the driving assistance processing unit 122 when the level of autonomous driving before the transition is level 3 and the level of autonomous driving after the transition is level 4.
[0437] Figure 17 It is a diagrammatic explanation. Figure 14 The diagram illustrates steps S204 and S205 of the process, which are examples of processing performed by the driving assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0438] In steps S204 and S205, the driver assistance processing unit 122 refers to the list of permitted activities associated with the autonomous driving level obtained in step S203 and analyzes the differences between permitted activities before and after the change in the autonomous driving level of the mobile device 10 detected in step S202.
[0439] The driver assistance processing unit 122 analyzes the differences between permitted activities before and after the transition to a new level of autonomous driving.
[0440] although Figure 17 The diagram illustrates the permitted activities for Level 3 and Level 4 of Automated Driving, recorded in the list of permitted activities associated with each level of Automated Driving. However, it differs from the reference above. Figure 15 The examples described are different. The activities that are not included in the permitted activities of Level 3 of autonomous driving, but are only added to the permitted activities of Level 4 of autonomous driving ((1) video call, (2) nap) are set as the beginning data of the permitted activities of Level 4 of autonomous driving.
[0441] Thus, after setting the difference data at the beginning, the driver assistance processing unit 122... Figure 14 In step S206 of the process illustrated in the diagram, display data is generated to facilitate understanding of the differences in permitted activities before and after the transition of the autonomous driving level, and the display data is displayed on the display unit.
[0442] Figure 18 This is an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0443] Figure 18 The diagram shows an example of the following messages being displayed.
[0444] "The autonomous driving level has been changed from Level 3 to Level 4. As a Level 4 driver permitted to engage in activities, the following (1) and (2) have been added."
[0445] (1) Video call
[0446] (2) Nap
[0447] (3) Eating
[0448] (4) Making a phone call
[0449] (5) View and reply to emails
[0450] (6) Reading books
[0451] (7) Playing games
[0452] like Figure 18 As shown in the diagram, the two new activities
[0453] (1) Video calls, and
[0454] (2) Nap
[0455] The two activities are displayed in a position before other activities (3) to (7), and are also highlighted, for example, by outputting the two activities in a different color such as red, or by changing the size and displaying the two activities in a larger size, making the two activities stand out.
[0456] For example, such as Figure 18 As shown in the diagram, the driver assistance processing unit 122 generates and outputs display data that is easy to understand regarding the differences in permitted activities before and after the level transition.
[0457] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 has changed from level 3 to level 4. In addition, as for the activities allowed for the driver after the change to autonomous driving level 4, in addition to the activities allowed in level 3, new activities are allowed: (1) video calls and (2) naps.
[0458] Users can confirm the permitted activities corresponding to Level 4 of autonomous driving displayed on the display unit and strive to act within the scope of the displayed permitted activities.
[0459] Note that notification processing for permitted activities corresponding to autonomous driving levels is not limited to, for example... Figure 18 The display processing on the display unit shown in the diagram can, for example, be notified via voice information.
[0460] refer to Figures 15-18 The illustrated processing example is the display data of the display unit and the processing of the driver assistance processing unit 122 when the autonomous driving level before the transition is Level 3 and the autonomous driving level after the transition is Level 4. Furthermore, "video call" and "nap" are examples of new activities, and other activities are also possible.
[0461] Next, refer to Figure 19 and Figure 20 This illustrates an example of the display data of the display unit and the processing of the driving assistance processing unit 122 when the autonomous driving level before the transition is level 4 and the autonomous driving level after the transition is level 3, that is, when the autonomous driving level is transferred to a lower level.
[0462] Figure 19 It is a diagrammatic explanation. Figure 14 The diagram illustrates steps S204 and S205 of the process, which are examples of processing performed by the driving assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0463] In steps S204 and S205, the driver assistance processing unit 122 refers to the list of permitted activities associated with the autonomous driving level obtained in step S203 and analyzes the differences between permitted activities before and after the change in the autonomous driving level of the mobile device 10 detected in step S202.
[0464] The driver assistance processing unit 122 analyzes the differences between permitted activities before and after the transition to a new level of autonomous driving.
[0465] Figure 19 The diagram illustrates the permitted activities at Level 4 of Automated Driving before the transition to a new Level of Automated Driving and at Level 3 of Automated Driving after the transition.
[0466] The permitted activities of Level 3 of Automated Driving after the transition do not include (6) video calls and (7) naps, which are included in the permitted activities of Level 4 of Automated Driving before the transition, and thus these activities are extracted as differences.
[0467] exist Figure 14 In step S206 of the process illustrated in the diagram, the driver assistance processing unit 122 generates display data that is easy to understand the difference data, and displays the generated display data on the display unit.
[0468] Figure 20 This is an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0469] Figure 20 The diagram shows an example of the following messages being displayed.
[0470] "The autonomous driving level has been changed from Level 4 to Level 3. As a result of the change from Level 4 to Level 3, the following activities (NG) have been removed from driver-permitted activities."
[0471] (1) Eating
[0472] (2) Making a phone call
[0473] (3) View and reply to emails
[0474] (4) Reading books
[0475] (5) Playing games
[0476] (NG) Video call
[0477] (NG) nap
[0478] like Figure 20 As shown in the diagram, the following two permitted activities were removed due to a downgrade in the level of autonomous driving.
[0479] (NG) video call, and
[0480] (NG) Afternoon nap
[0481] It is displayed together with other activities (1) to (5) and is displayed as a notification to the user (driver) that these two activities are not permitted at the current level of autonomous driving.
[0482] For example, such as Figure 20 As illustrated in the diagram, the driver assistance processing unit 122 generates and outputs display data that is easy to understand regarding the differences in permitted activities before and after a level transition. For example, the difference data is emphasized by outputting it in a different color, such as red, or by changing its size to display it at a larger scale. In other words, different visual features are added to the display of the difference data.
[0483] Note that in Figure 20 In the example shown in the diagram, the illustration depicts an activity that is no longer permitted for the driver as (NG), but in addition to (NG), a cross mark (X) is also shown. Alternatively, it is possible to have various markings that allow the driver to identify prohibited activities, such as a setting where a red circle and a diagonal line, which are usually used to indicate prohibition, are written together.
[0484] As a user, the driver can see the displayed data on the display unit to confirm that the autonomous driving level of the mobile device 10 has been downgraded from level 4 to level 3. Furthermore, it confirms the types of activities permitted to the driver and the prohibited activities at the new autonomous driving level 3.
[0485] (NG) Video call
[0486] (NG) nap
[0487] The prohibited activities are those that are no longer permitted for the driver due to the shift from Level 4 to Level 3.
[0488] Users can confirm the permitted activities corresponding to Level 3 of autonomous driving displayed on the display unit and strive to act within the displayed permitted activity range.
[0489] Note that notification processing for permitted activities corresponding to autonomous driving levels is not limited to, for example... Figure 20 The display processing on the display unit shown in the diagram can, for example, be notified via voice information.
[0490] [7. Regarding the analysis of the driver's confirmation status of notifications regarding differences in permitted activities associated with the transition of autonomous driving levels, and the processing sequence for emergency response in the absence of confirmation notifications]
[0491] Next, we will explain the process of analyzing the driver's confirmation status of notifications regarding differences in permitted activities associated with changes in the level of autonomous driving, and the sequence of actions for emergency response in the absence of confirmation notifications.
[0492] refer to Figure 14 The flowchart shown in the diagram illustrates the processing sequence for informing the driver of the differences between permitted activities corresponding to the change in the level of autonomous driving, in order to make it easy to understand.
[0493] However, no analysis was performed regarding whether the driver acknowledged the notification.
[0494] The following describes the process of analyzing the driver's confirmation status of the notification of the difference in permitted activities and the handling of emergency response in the absence of a confirmation notification.
[0495] Reference Figure 21 The flowchart shown in the diagram illustrates the sequence of processes.
[0496] Figure 21 The flowchart shown in the diagram is one of the above. Figure 14 The flowchart shown in the diagram adds steps S221 and S222 after step S206.
[0497] The following will explain Figure 21 The flowchart shown in the middle illustrates the processing of each step.
[0498] (Steps S201 to S206)
[0499] The processing of steps S201 to S206 is similar to that in the reference. Figure 14 The processing of the description.
[0500] These processes will be briefly explained.
[0501] First, in step S201, the control unit 120 of the information processing device 100 specifies the level of autonomous driving that the mobile device 10 is currently performing.
[0502] Next, in step S202, it is determined whether there is a change (transformation) in the autonomous driving level being performed by the mobile device 10.
[0503] If the determination unit 123 of the control unit 120 of the information processing device 100 determines in step S202 that a change (modification) in the autonomous driving level of the mobile device 10 has been detected, the processing proceeds to step S203.
[0504] On the other hand, in step S202, if the determination unit 123 determines that no change (modification) of the autonomous driving level of the mobile device 10 is detected, the process returns to step S201 and continues the process of step S201.
[0505] When it is determined in step S202 that a change (modification) in the autonomous driving level of the mobile device 10 is detected, in step S203, a list of permitted activities in which the autonomous driving level is associated with the activities permitted for the driver is obtained.
[0506] Next, in step S204, the control unit 120 of the information processing device 100 refers to the list of permitted activities obtained in step S203 and analyzes the differences between permitted activities before and after the change in the autonomous driving level of the mobile device 10 detected in step S202.
[0507] Next, in step S205, based on the difference analysis results in step S204, the control unit 120 of the information processing device 100 determines whether there is a difference in the permitted activities before and after the change in the autonomous driving level.
[0508] If it is determined that there is a difference in permitted activities before and after the transition to the autonomous driving level, the process proceeds to step S206.
[0509] On the other hand, when it is determined in step S205 that there is no difference between the permitted activities before and after the transition of the autonomous driving level, the process ends.
[0510] When it is determined in step S205 that there is a difference between the permitted activities before and after the transition of the autonomous driving level, the processing in step S206 is performed.
[0511] This processing is performed by the driver assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0512] If, in step S205, the driver assistance processing unit 122 of the control unit 120 of the information processing device 100 determines that there is a difference between the permitted activities before and after the change of the autonomous driving level, in step S206, the permitted activities after the change (modification) of the autonomous driving level, that is, the permitted activities that clearly indicate the difference caused by the change of the autonomous driving level, are output to the input and output unit 133 of the mobile device 10.
[0513] The processing up to this point is similar to the reference. Figure 14 The processing of the description.
[0514] (Step S221)
[0515] Next, in step S221, the control unit 120 of the information processing device 100 determines whether it detects confirmation input from the driver regarding the permitted activities caused by the difference in the change of the autonomous driving level, which was clearly indicated to the driver in step S206.
[0516] Figure 22 The diagram illustrates an example of a user interface that enables the detection of confirmation input from the driver.
[0517] Figure 22 The diagram illustrates how to refer to the above. Figure 16 The example of a notification explaining the difference between permitted activities caused by a change in the level of autonomous driving is similar to the display data obtained by adding user input via the confirmation button 201.
[0518] The display unit is a touch panel, and when the user (driver) touches the confirmation button 201, a detection signal indicating that the user has confirmed is input to the control unit 120 of the information processing device 100.
[0519] exist Figure 21 In step S221 of the process shown in the diagram, the presence or absence of a detection signal is determined.
[0520] When the user (driver) touches the confirmation button 201 and inputs a detection signal to the control unit 120 of the information processing device 100, the determination in step S221 is "yes". That is, it is determined that the driver has confirmed the permitted activity caused by the difference in the change of the autonomous driving level, which is clearly indicated on the display unit, and then the process ends.
[0521] On the other hand, if the user (driver) does not touch the confirmation button 201 within a predetermined time (e.g., 10 seconds) and does not input a detection signal to the control unit 120 of the information processing device 100, the determination in step S221 is "No". That is, it is determined that the driver has not confirmed the permitted activity due to the difference caused by the change in the autonomous driving level, which is clearly indicated on the display unit.
[0522] In this case, the process proceeds to step S222.
[0523] (Step S222)
[0524] In the determination process of step S221, if it is determined that the driver has not confirmed the permitted activities due to the difference caused by the change in the autonomous driving level as clearly indicated on the display unit, the process proceeds to step S222.
[0525] In this situation, the control unit 120 of the information processing device 100 performs emergency response processing in step S222.
[0526] For example, emergency response handling is any of the following:
[0527] (a) Emergency stop of mobile device 10;
[0528] (b) Switch mobile device 10 to slow driving mode; and
[0529] (c) Output a warning sound.
[0530] Or it can be obtained by combining multiple processes.
[0531] For example, if it is determined that there is no clear indication on the display unit that the activity is permitted due to the difference caused by the change in the level of autonomous driving, the driver assistance processing unit 122 of the control unit 120 performs an emergency stop of the mobile device 10 and outputs a warning sound as an emergency response.
[0532] By implementing such an emergency response, the situation where the mobile device 10 is driven while the driver is engaged in an activity that is not permitted can be avoided, thereby preventing accidents in advance.
[0533] [8. A sequence of procedures for monitoring driver actions following notifications of differences in permitted activities associated with a shift in autonomous driving levels, and for taking emergency response measures when the driver is performing prohibited activities.]
[0534] Next, we will explain the sequence of actions to monitor the driver's behavior after receiving a notification of a difference in permitted activities associated with a change in the level of autonomous driving, and to take emergency response measures when the driver is performing an indisallowed activity.
[0535] Reference Figure 23 The flowchart shown in the diagram illustrates the sequence of processes.
[0536] Figure 23 The flowchart shown in the diagram is one of the above. Figure 14 The flowchart shown in the middle diagram adds steps S241 to S243 after step S206.
[0537] The following will explain Figure 23 The flowchart shown in the middle illustrates the processing of each step.
[0538] (Steps S201 to S206)
[0539] The processing of steps S201 to S206 is similar to that in the reference. Figure 14 The processing of the description.
[0540] These processes will be briefly explained.
[0541] First, in step S201, the control unit 120 of the information processing device 100 specifies the level of autonomous driving currently being performed by the mobile device 10.
[0542] Next, in step S202, it is determined whether there is a change (transformation) in the autonomous driving level being performed by the mobile device 10.
[0543] If the determination unit 123 of the control unit 120 of the information processing device 100 determines in step S202 that a change (modification) in the autonomous driving level of the mobile device 10 has been detected, the processing proceeds to step S203.
[0544] On the other hand, in step S202, if the determination unit 123 determines that no change (modification) of the autonomous driving level of the mobile device 10 is detected, the process returns to step S201 and continues the process of step S201.
[0545] When it is determined in step S202 that a change (modification) in the autonomous driving level of the mobile device 10 is detected, in step S203, a list of permitted activities in which the autonomous driving level is associated with the activities permitted for the driver is obtained.
[0546] Next, in step S204, the control unit 120 of the information processing device 100 refers to the list of permitted activities obtained in step S203 and analyzes the differences between permitted activities before and after the change in the autonomous driving level of the mobile device 10 detected in step S202.
[0547] Next, in step S205, based on the difference analysis results in step S204, the control unit 120 of the information processing device 100 determines whether there is a difference in the permitted activities before and after the change in the autonomous driving level.
[0548] When it is determined that there is a difference between the permitted activities before and after the transition of the autonomous driving level, the process proceeds to step S206.
[0549] On the other hand, when it is determined in step S205 that there is no difference between the permitted activities before and after the change in autonomous driving level, the process ends.
[0550] When it is determined in step S205 that there is a difference between the permitted activities before and after the transition of the autonomous driving level, the processing in step S206 is executed.
[0551] This processing is performed by the driver assistance processing unit 122 of the control unit 120 of the information processing device 100.
[0552] If, in step S205, the driver assistance processing unit 122 of the control unit 120 of the information processing device 100 determines that there is a difference between the permitted activities before and after the change of the autonomous driving level, in step S206, the permitted activities after the change (modification) of the autonomous driving level, that is, the permitted activities that clearly indicate the difference caused by the change of the autonomous driving level, are output to the input and output unit 133 of the mobile device 10.
[0553] The processing up to this point is similar to the reference. Figure 14 The processing of the description.
[0554] (Step S241)
[0555] Next, in step S241, the control unit 120 of the information processing device 100 analyzes the driver's actions based on sensor detection information.
[0556] Note that in performing the processing of this embodiment, sensors such as cameras are required to acquire information for analyzing the driver's actions inside the vehicle.
[0557] In this embodiment, the configuration of the sensor unit 131 is as described above. Figure 13 The configuration is described below. Specifically, the sensor unit 131 has a component that references upwards. Figure 6 The sensor unit 131 described herein includes a configuration in which a driver monitoring sensor 156 is added.
[0558] The driver monitoring sensor 156 is a sensor that acquires information for analyzing the driver's actions inside the vehicle, such as a camera, microphone, odor sensor, distance sensor, etc.
[0559] The data collection unit 121 of the control unit 120 inputs the information acquired by the driver monitoring sensor 156 and inputs the acquired information to the decision unit 123.
[0560] The decision unit 123 analyzes the information acquired by the driver monitoring sensor 156 and estimates the driver's ongoing activities. This estimation can be based on the output obtained by inputting the information acquired by the driver monitoring sensor 156 into a neural network model generated through AI, such as machine learning or deep learning.
[0561] (Step S242)
[0562] In step S242, it is determined whether the driver's activity estimated in step S241 is a permitted activity corresponding to the current level of autonomous driving.
[0563] This process is executed by the decision unit 123 of the control unit 120.
[0564] The determination unit 123 determines whether the estimated driver activity is a permitted activity corresponding to the current level of autonomous driving.
[0565] The process ends when it is determined that the driver's activity is a permitted activity corresponding to the current level of autonomous driving.
[0566] On the other hand, if it is determined that the driver's activity is not a permitted activity corresponding to the current level of autonomous driving, the process proceeds to step S243.
[0567] (Step S243)
[0568] When it is determined in step S242 that the driver is performing an activity that is not a permitted activity corresponding to the current level of autonomous driving, the process proceeds to step S243.
[0569] In this situation, the control unit 120 of the information processing device 100 performs emergency response processing in step S243.
[0570] For example, emergency response handling is any of the following:
[0571] (a) Emergency stop of mobile device 10;
[0572] (b) Switch mobile device 10 to slow driving mode; and
[0573] (c) Output a warning sound.
[0574] Or it can be obtained by combining multiple processes.
[0575] For example, when it is determined that the driver is performing an activity that is not permitted under the current level of autonomous driving, as an emergency response, the driver assistance processing unit 122 of the control unit 120 performs an emergency stop of the mobile device 10 and outputs a warning sound.
[0576] By implementing such an emergency response, the situation where the mobile device 10 is driven while the driver is engaged in an activity that is not permitted can be avoided, thereby preventing accidents in advance.
[0577] [9. Examples of other processing performed by the information processing equipment of this disclosure]
[0578] Next, other examples of processing performed by the information processing device of this disclosure will be described.
[0579] The following two processing examples will be explained.
[0580] (Processing Example a) Selecting and Notifying Allowed Activities Corresponding to the User's Request
[0581] (Processing Example b) Considering the duration of the autonomous driving level selection and notifying the processing example that the activity is permitted.
[0582] (9-a. (Processing Example a) Selecting and Notifying Allowed Activities in Accordance with User Request)
[0583] First, as (processing example a), a processing example will be described, which selects and notifies the user of the permitted activities in accordance with their request.
[0584] As per the above reference Figure 8 The autonomous driving level-corresponding permitted activity list, which associates autonomous driving levels with permitted activities for the driver, records all permitted activities associated with each autonomous driving level.
[0585] For example, the permitted activities at Level 3 of autonomous driving are the following five activities:
[0586] (1) Eating
[0587] (2) Making a phone call
[0588] (3) View and reply to emails
[0589] (4) Reading books
[0590] (5) Playing games
[0591] However, it is difficult for the driver, as a user, to perform all five activities.
[0592] If a driver wants to perform one of these five permitted activities, such as "eating," it is sufficient to provide a notification that "eating" is a permitted activity; no information about other permitted activities is required.
[0593] For example, desired activities can be displayed based on driver preferences. The driver pre-sets desired actions, and when displaying activities permitted at a certain level of autonomous driving, only those activities that fall under the pre-set actions are shown. For instance, at Autonomous Driving Level 3, permitted activities for the driver include eating, making phone calls, replying to emails, reading, and playing games.
[0594] Here, by having the driver pre-set the desired actions, only the activities that belong to the driver's desired actions are displayed, provided that the activities are within the range allowed by the driver at the current level of autonomous driving.
[0595] For example, if the desired action is "driver eats", then at Autopilot Level 3, only "eating" will be displayed as an activity permitted by the driver. Or, if the desired action is "driver contacts someone", then at Autopilot Level 3, only "making a phone call and replying to an email" will be displayed as an activity permitted by the driver.
[0596] Alternatively, if "driver plays" is set as the desired action, and the autopilot level is 3, only "playing games" will be displayed as an activity permitted by the driver.
[0597] Note that when setting the desired actions for the driver, a priority order can be set. When displaying permitted activities for the driver, these activities can be displayed based on the set priority order of the desired actions.
[0598] Reference Figure 24 Specific examples are illustrated in the accompanying drawings. The driver of the mobile device 10 speaks, for example, "I want to eat," into the input unit 133 of the information processing device 100, such as through a microphone.
[0599] The driver assistance processing unit 122 of the control unit 120 of the information processing device 100 performs speech recognition and speech semantic analysis processing of the user's speech, and interprets it as the user (driver) wanting to eat.
[0600] The driver assistance processing unit 122 outputs a message to the display unit based on the analysis results. The message indicates the current level of autonomous driving and whether eating is permitted at that level.
[0601] Figure 24 The diagram illustrates an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0602] Figure 24 The diagram shows an example of the following messages being displayed.
[0603] "Currently, the autonomous driving level is Level 3. At Level 3, drivers are allowed to 'eat'."
[0604] For example, such as Figure 24 As shown in the diagram, the driver assistance processing unit 122 generates and outputs a message indicating whether the requested activity (=eating) is allowed.
[0605] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 is Level 3, and that "eating" is an activity permitted for the driver at autonomous driving level 3.
[0606] Users can confirm the permitted activities corresponding to Level 3 of autonomous driving, as displayed on the display unit, including "eating," and thus can confidently begin "eating."
[0607] Note that in Figure 24 In the example shown in the diagram, "eating" is permitted because the autonomous driving level of mobile device 10 is Level 3. However, for example, if the autonomous driving level of mobile device 10 is Level 2 or lower, "eating" is not a permitted activity.
[0608] Reference Figure 25 Here's an example illustrating how data is displayed in this situation.
[0609] Figure 25 This is an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0610] Figure 25 The diagram shows an example of the following messages being displayed.
[0611] "Currently, the autonomous driving level is Level 2. At Level 2, drivers are not allowed to eat."
[0612] For example, such as Figure 25 As shown in the diagram, the driver assistance processing unit 122 generates and outputs a message indicating that the activity requested by the user (=eating) is not allowed.
[0613] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 is Level 2, and that "eating" is not an activity permitted for the driver at Autonomous Driving Level 2.
[0614] Users can confirm that "eating" is not included as a permitted activity corresponding to Level 2 of autonomous driving displayed on the display unit, and thus can stop "eating".
[0615] Note that the means of notifying whether a user is allowed to participate in the activity requested (e.g., eating) is not limited to displaying on the display unit; it can also be a notification via voice information.
[0616] Reference Figure 26 and Figure 27 This section provides an example of the data displayed when the driver's request is not "eating".
[0617] Figure 26The example shown in the diagram is an example of the displayed data when the driver of the mobile device 10 faces the input unit of the input and output unit 133 of the information processing device 100, such as the microphone, and says something like "I want to contact someone".
[0618] The driver assistance processing unit 122 of the control unit 120 of the information processing device 100 performs speech recognition and speech semantic analysis processing of the user's speech, and interprets it as the user (driver) wanting to contact someone.
[0619] The driver assistance processing unit 122 outputs a message to the display unit based on the analysis results. The message indicates the current level of autonomous driving and whether it is possible to "contact someone" at that level.
[0620] Figure 26 The diagram illustrates an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0621] Figure 26 The diagram shows an example of the following messages being displayed.
[0622] "Currently, the autonomous driving level is Level 3. At Level 3, the driver is allowed to '(1) make a phone call and (2) view and reply to emails'."
[0623] For example, such as Figure 26 As shown in the diagram, the driver assistance processing unit 122 generates and outputs a message indicating whether the user's requested activity (i.e., contacting someone) is allowed.
[0624] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 is Level 3. Moreover, at the autonomous driving level 3, "(1) making a phone call and (2) viewing and replying to emails" are activities permitted for the driver.
[0625] Users can confirm the permitted activities corresponding to Level 3 of autonomous driving as displayed on the display unit, including "(1) making a phone call and (2) viewing and replying to emails", so they can confidently start making a phone call or viewing and replying to emails.
[0626] Note that the means of notifying whether a user is allowed to participate in the activity requested (i.e., contact someone) is not limited to displaying on a display unit; it can also be a notification via voice information.
[0627] Figure 27 The example shown in the diagram is an example of the displayed data when the driver of the mobile device 10 faces the input unit of the input and output unit 133 of the information processing device 100, for example, when the driver speaks a phrase such as "I want to play" through the microphone.
[0628] The driver assistance processing unit 122 of the control unit 120 of the information processing device 100 performs speech recognition and speech semantic analysis processing of the user's speech, and interprets it as the user (driver) wanting to play.
[0629] The driver assistance processing unit 122 outputs a message to the display unit based on the analysis results. The message indicates the current level of autonomous driving and whether it is possible to "play" at that level.
[0630] Figure 27 The diagram illustrates an example of display data generated by the driver assistance processing unit 122 and displayed on the display unit.
[0631] Figure 27 The diagram shows an example of the following messages being displayed.
[0632] "Currently, the autonomous driving level is Level 3. At Level 3, drivers are allowed to 'play games'."
[0633] For example, such as Figure 26 As shown in the diagram, the driver assistance processing unit 122 generates and outputs a message indicating whether the user is allowed to participate in the requested activity (i.e., play).
[0634] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 is Level 3, and that "playing games" is an activity permitted by the driver at Autonomous Driving Level 3.
[0635] Users can confirm the permitted activities corresponding to Level 3 of autonomous driving as displayed on the display unit, including "playing games," and can then confidently start playing games.
[0636] Note that the means of notifying whether a user is allowed to participate in the requested activity (i.e., play) is not limited to displaying on the display unit; it can also be a notification via voice information.
[0637] (9-b. (Processing Example b) Processing example considering the duration of the autonomous driving level and notifying that activity is permitted)
[0638] Next, as (processing example b), a processing example will be described that takes into account the duration of the autonomous driving level and notifies that the activity is permitted.
[0639] Even within the same level of automation, the activities permitted by the driver can vary depending on the duration of the automation level. For example, at Level 4 automation, permitted activities include eating, making phone calls, replying to emails, reading, playing games, holding conference calls, and taking naps. However, at shorter Level 4 automation times, even if permitted activities exist, they may not be actually performed, or if they are, safe driving may not be possible.
[0640] The processing example described below addresses this issue by considering the duration selection of the autonomous driving level and notifying that activity is permitted.
[0641] As described above, the autonomous driving level changes depending on the driving range when the mobile device 10 is in motion. However, the same level may remain for a period of time, such as about an hour.
[0642] Note that the duration of the same level of autonomous driving is determined by the driver assistance processing unit 122 of the mobile device 10 from external sources or calculated based on the information obtained.
[0643] For example, it is obtained from the control device 50 that manages the autonomous driving of the mobile device 10. Alternatively, it can be processed by performing calculations based on map information provided by the map information providing server 50b and the speed information of the mobile device 10.
[0644] The driver assistance processing unit 122 of the mobile device 10 changes the permitted activities and notifies the driver of the permitted activities according to the duration of the same level of autonomous driving level obtained from the outside or calculated based on the obtained information.
[0645] This is because even when the level of autonomous driving is the same, such as level 4, it is believed that the activities a driver can perform will vary depending on the duration of level 4.
[0646] For example, as referenced above Figure 8 As described in the list of permitted activities associated with the autonomous driving level, the permitted activities at autonomous driving level 4 are the following seven activities:
[0647] (1) Eating
[0648] (2) Making a phone call
[0649] (3) View and reply to emails
[0650] (4) Reading books
[0651] (5) Playing games
[0652] (6) Video call
[0653] (7) Nap
[0654] However, for example, when the driving range of mobile device 10 is short, and mobile device 10 is driving on a road where the autonomous driving level will immediately change to level 3, among the seven activities mentioned above, the activity that is not allowed at autonomous driving level 3 must be canceled when the autonomous driving level changes to level 3.
[0655] To address this issue, the driver assistance processing unit 122 of the mobile device 10 calculates or obtains the duration of the autonomous driving level and notifies the driver of recommended and non-recommended activities corresponding to the calculated or obtained duration.
[0656] Note that the duration of an autonomous driving level is calculated by the control unit 120 based on route information set for the destination input by the driver via the input and output unit 133 of the information processing device 100.
[0657] Furthermore, for example, when there is a level change in the route, such as between autonomous driving level 3 and autonomous driving level 4, the duration of activities permitted under both autonomous driving levels, such as eating, making phone calls, replying to emails, reading, and playing games, is calculated assuming that the duration continues after the change in autonomous driving level.
[0658] Furthermore, the driver can select the activity to be performed from the permitted activities at the current level of autonomous driving and input the selected activity into the information processing device 100. For example, the driver can use a touch panel display that constitutes the input and output unit 133 to select and input the activity to be performed from the displayed activities. The driver assistance processing unit 122 of the control unit 120 determines whether the selected activity is permitted based on the selection information and notifies the driver, as the user, of the determination.
[0659] In addition, the driver assistance processing unit 122 can detect the activity selected by the driver based on the information obtained by the data collection unit 121 from the sensor unit 131.
[0660] For example, artificial intelligence (AI) can be used to analyze imaging information from cameras that capture moving images, including those of the driver, to determine which activity the driver is performing, such as eating, operating a smartphone, reading, playing games, or taking a nap. Furthermore, by using AI to analyze information obtained from the speaker and microphone of sensor unit 131, it is also possible to detect whether the driver is performing activities such as making phone calls or holding conference calls.
[0661] Note that the driver assistance processing unit 122 inputs the sensor information acquired by the data collection unit into a neural network model trained by methods such as machine learning, deep learning, or reinforcement learning, and specifies the activity being performed by the driver.
[0662] As described above, the driver assistance processing unit 122 analyzes the activities the driver is performing, and can issue a warning if the driver is performing an activity that is not permitted at the new autonomous driving level when the autonomous driving level changes. Note that the warning notification can be provided via a warning display on the display unit or via voice. Furthermore, if the driver fails to end the activity that is not permitted at the changed autonomous driving level within a set time period, the driver assistance processing unit 122 can perform emergency procedures such as stopping the mobile device 10.
[0663] refer to Figure 28 The accompanying figures will illustrate specific examples of processing that take into account the duration selection of the autonomous driving level and notify the user of permitted activities.
[0664] The driver assistance processing unit 122 takes into account the duration of the autonomous driving level and notifies the driver of the non-recommended and recommended activities selected according to the duration of the autonomous driving level.
[0665] Figure 28 This is an example of data generated by the driver assistance processing unit 122 and displayed on the display unit when the driver assistance processing unit 122 determines that the autonomous driving level 4 will last for two hours from the current time.
[0666] Figure 28 The data displayed in the diagram is as follows.
[0667] "The current level of autonomous driving is Level 4, and the duration of Level 4 is expected to be approximately 2 hours. Under the conditions of Level 4 and a duration of 2 hours, the permitted activities for the driver are as follows:"
[0668] (1) Eating
[0669] (2) Making a phone call
[0670] (3) View and reply to emails
[0671] (4) Reading books
[0672] (5) Playing games
[0673] (6) Video call
[0674] (7) Nap
[0675] For example, such as Figure 28As shown in the diagram, the driver assistance processing unit 122 informs the user of the current level of autonomous driving (level 4) and the duration of the same level 4 in the future (approximately 2 hours), and selects and displays permitted activities that are deemed sufficient for the user (driver) to perform under the conditions of level 4 and duration = 2 hours.
[0676] exist Figure 28 In the example shown in the diagram, since the current autonomous driving level (level 4) lasts for about 2 hours, all permitted activities corresponding to autonomous driving level 4 are displayed in the list of permitted activities associated with the autonomous driving level.
[0677] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 is level 4, and that the autonomous driving at level 4 will continue for about 2 hours in the future. In addition, there are activities (1) to (7) as activities permitted by the driver at autonomous driving level 4.
[0678] Users can confirm the permitted activities corresponding to Level 4 of autonomous driving as displayed on the display unit and strive to act within the displayed permitted activity range.
[0679] Note that the means of notifying the driver of recommended and non-recommended activities selected according to the duration of the autonomous driving level are not limited to display on the display unit, but can be notification via voice information.
[0680] Next, we will explain an example of the displayed data when the current autonomous driving level (Level 4) is short.
[0681] Figure 29 This is an example of data generated by the driver assistance processing unit 122 and displayed on the display unit when the driver assistance processing unit 122 determines that the autonomous driving level 4 will last for one hour from the current time.
[0682] Figure 29 The data displayed in the diagram is as follows.
[0683] "The current level of autonomous driving is Level 4, and the duration of Level 4 is expected to be approximately 1 hour in the future. Under the conditions of Level 4 and duration = 1 hour, the following (1) to (6) are activities that are permitted for the driver. (NG) is permitted but not recommended."
[0684] (1) Eating
[0685] (2) Making a phone call
[0686] (3) View and reply to emails
[0687] (4) Reading books
[0688] (5) Playing games
[0689] (6) Video call
[0690] (NG) nap
[0691] For example, such as Figure 29 As shown in the diagram, the driver assistance processing unit 122 informs the current level of autonomous driving (level 4) and the duration of the same level 4 in the future (approximately 1 hour), and under the condition of level 4 and duration = 1 hour, displays in a distinguishable manner the permitted activities (1) to (6) and permitted but not recommended activities (NG) that are considered sufficient to be performed by the user (driver).
[0692] That is, activities (1) to (6) are allowed to be displayed as recommended activities, while activities (NG) are displayed as not recommended activities.
[0693] Note that in Figure 29 The example shown in the diagram illustrates an instance where a non-recommended activity for the driver is set as (NG), but in addition to (NG), a cross mark (X) is also displayed. Alternatively, it is possible to have various markings that allow the driver to identify non-recommended activities, such as a setting where a red circle and a diagonal line, which are usually signs of prohibition, are written together.
[0694] exist Figure 29 In the example shown in the diagram, the current autonomous driving level (Level 4) lasts for approximately one hour, and as a recommended permitted activity, the following six activities will be notified.
[0695] (1) Eating
[0696] (2) Making a phone call
[0697] (3) View and reply to emails
[0698] (4) Reading books
[0699] (5) Playing games
[0700] (6) Video call.
[0701] On the other hand, the activity of (NG) napping is displayed in a distinguishable way as a permitted but not recommended activity (NG).
[0702] As a user, the driver can see the display data on the display unit to confirm that the autonomous driving level of the mobile device 10 is level 4, and that the autonomous driving at level 4 will continue for about 1 hour in the future. In addition, there are (1) to (6) and (NG) activities as permitted activities for the driver at autonomous driving level 4, and (NG) nap is a permitted but not recommended activity.
[0703] Users can confirm the permitted and unrecommended activities corresponding to autonomous driving level 4 displayed on the display unit, and can strive to act within the range of activities (1) to (6) excluded from the displayed permitted activities (1) to (6) and (NG).
[0704] Note that the means of notifying the driver of recommended and non-recommended activities selected according to the duration of the autonomous driving level are not limited to display on the display unit, but can be notification via voice information.
[0705] Figure 30 This is an example of data generated by the driver assistance processing unit 122 and displayed on the display unit when the driver assistance processing unit 122 determines that the autonomous driving level 4 will last for 30 minutes from the current time.
[0706] Figure 30 The data displayed in the diagram is as follows.
[0707] "The current level of autonomous driving is Level 4, and the duration of Level 4 is expected to be approximately 30 minutes in the future. Under the conditions of Level 4 and duration = 30 minutes, the following (2) and (3) are activities that are permitted for the driver. (NG) is permitted but not recommended."
[0708] (NG) eating
[0709] (2) Making a phone call
[0710] (3) View and reply to emails
[0711] (NG) Reading
[0712] (NG) Playing games
[0713] (NG) Video call
[0714] (NG) nap
[0715] For example, such as Figure 30As shown in the diagram, the driver assistance processing unit 122 informs the current autonomous driving level (level 4) and the duration (approximately 30 minutes) of the same level 4 in the future, and under the conditions of level 4 and duration = 30 minutes, displays the permitted activities (2) and (3) that are considered sufficient to be performed by the user (driver) as recommended activities in a distinguishable manner, and displays the permitted but not recommended activities (NG) as non-recommended activities.
[0716] exist Figure 30 In the example shown in the diagram, the current autonomous driving level (Level 4) lasts for approximately 30 minutes, thus, as a recommended permitted activity, the following two activities will be notified.
[0717] (2) Making a phone call
[0718] (3) View and reply to emails.
[0719] on the other hand,
[0720] (NG) eating
[0721] (NG) Reading
[0722] (NG) Playing games
[0723] (NG) Video call
[0724] (NG) nap
[0725] These activities are displayed in a distinguishable manner as permitted but not recommended (NG) activities.
[0726] Note that it is still in Figure 30 In this context, recommended activities for drivers are also set as (NG), but similar to those described above... Figure 29 In addition to (NG), a cross mark (X) may also be displayed. Alternatively, it is possible to have various markings that allow the driver to identify non-recommended activities, such as a setting where a red circle and a diagonal line are written together as a sign that usually indicates prohibition.
[0727] As a user, the driver can see the data displayed on the display unit to confirm that the autonomous driving level of the mobile device 10 is Level 4, and that Level 4 autonomous driving will continue for about 30 minutes. Moreover, there are activities (2), (3) and (NG) as permitted activities for the driver at Level 4 autonomous driving, and activities (NG) are permitted but not recommended activities.
[0728] Users can confirm the permitted and unrecommended activities corresponding to Level 4 of autonomous driving displayed on the display unit, and can strive to act within the range of activities (2) or (3) that are excluded from the displayed permitted activities (2), (3) and (NG).
[0729] Note that the means of notifying the driver of recommended and non-recommended activities selected according to the duration of the autonomous driving level are not limited to display on the display unit, but can be notification via voice information.
[0730] [10. Examples of the hardware configuration of the information processing device according to this disclosure]
[0731] Next, refer to Figure 31 This document describes specific hardware configuration examples of the information processing device disclosed herein.
[0732] Figure 31 The above is a diagram for reference. Figure 5 A diagram illustrating an example of the hardware configuration of the information processing device 100 of this disclosure.
[0733] The following will explain Figure 31 The diagram shows the various components of the hardware.
[0734] The central processing unit (CPU) 301 functions as a data processing unit that executes various processes according to programs stored in the read-only memory (ROM) 302 or the storage unit 308. For example, it executes processes according to the sequence described in the above embodiments. The random access memory (RAM) 303 stores programs, data, etc., executed by the CPU 301. These CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304.
[0735] CPU 301 is connected to input and output interface 305 via bus 304, including input unit 306 which includes various switches, keyboard, touch panel, mouse, microphone, and status data acquisition units such as sensors, cameras and GPS, and output unit 307 which includes display, speaker, etc., connected to input and output interface 305.
[0736] Note that the input information from sensor 321 is also input to input unit 306.
[0737] In addition, the output unit 307 also outputs the driving information of the driving unit 322 of the mobile device.
[0738] The CPU 301 receives instructions, status data, etc. from the input unit 306, performs various processes, and outputs the processing results to, for example, the output unit 307.
[0739] The storage unit 308, connected to the input and output interface 305, includes, for example, a hard disk and stores programs executed by the CPU 301 and various data. The communication unit 309 functions as a data transmission and reception unit for data communication via networks such as the Internet, local area networks, and wide area networks, and communicates with external devices.
[0740] In addition to or in place of the CPU, a graphics processing unit (GPU) can be configured. For example, the GPU processes image information input from a camera. That is, the CPU or GPU can be selectively used depending on the input information to be processed and the data processing performed according to the program.
[0741] The driver 310, connected to the input and output interface 305, drives the removable medium 311, such as a disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and performs data recording or reading.
[0742] [11. Summary of the composition of this disclosure]
[0743] The embodiments of this disclosure have been described in detail above with reference to specific examples. However, it will be apparent to those skilled in the art that modifications or substitutions can be made to the embodiments without departing from the scope of this disclosure. That is, the invention has been disclosed by way of example and should not be construed as restrictive. The claims should be considered in determining the scope of this disclosure.
[0744] Note that the technology disclosed in this specification may have the following configuration.
[0745] (1) An information processing device, comprising a control unit, the control unit acquiring the autonomous driving level of an autonomous driving system being performed by a mobile device and notifying the driver of permitted activities corresponding to the acquired autonomous driving level.
[0746] (2) The information processing equipment according to (1),
[0747] The control unit performs at least any of the processes in either the process of displaying the permitted activity on the display unit or the process of outputting voice.
[0748] (3) The information processing equipment according to (1) or (2),
[0749] The control unit refers to a list of permitted activities corresponding to autonomous driving levels, in which autonomous driving levels are associated with permitted activities, to obtain the permitted activities corresponding to the autonomous driving level being performed by the mobile device, and notifies the driver of the obtained permitted activities.
[0750] (4) The information processing device according to any one of (1) to (3),
[0751] The control unit obtains a list of permitted activities corresponding to autonomous driving levels from an external server or storage unit, in which autonomous driving levels are associated with permitted activities. Referring to the obtained list of permitted activities corresponding to autonomous driving levels, it obtains the permitted activities corresponding to the autonomous driving level being executed by the mobile device and notifies the driver of the obtained permitted activities.
[0752] (5) The information processing device according to any one of (1) to (4),
[0753] In the case of changes in the autonomous driving level of mobile devices,
[0754] The control unit analyzes the differences between permitted activities corresponding to changes in the level of autonomous driving, and if differences exist, the control unit notifies the driver of the difference information.
[0755] (6) The information processing equipment according to (5),
[0756] The control unit generates display data in which the difference information is highlighted, and outputs the display data on the display unit.
[0757] (7) The information processing device according to (5) or (6),
[0758] In the case of a reduction in the autonomous driving level of the mobile device,
[0759] The control unit generates display data that highlights permitted activities that have been removed due to a reduction in the level of autonomous driving, and outputs the display data on the display unit.
[0760] (8) The information processing device according to any one of (1) to (7),
[0761] The control unit analyzes whether there is a confirmation response from the driver regarding the notified permitted activity, and
[0762] Perform emergency response procedures in the absence of a confirmed response.
[0763] (9) The information processing device according to (8), wherein the emergency response processing is the process of stopping the mobile device.
[0764] (10) The information processing device according to any one of (1) to (9),
[0765] The control unit estimates the driver's actions based on sensor detection information.
[0766] Determine whether the estimated driver-performed activity is a driver-permitted activity corresponding to the mobile device's level of autonomous driving, and
[0767] If the estimated activity performed by the driver is not a driver-permitted activity corresponding to the autonomous driving level of the mobile device, an emergency response procedure is executed.
[0768] (11) The information processing device according to (10), wherein the emergency response processing is the process of stopping the mobile device.
[0769] (12) The information processing device according to any one of (1) to (11),
[0770] The control unit specifies the level of autonomous driving that the mobile device is performing based on information obtained from at least one of the mobile device's storage unit or sensors, an external server, a base station, or a roadside communication unit.
[0771] (13) The information processing device according to any one of (1) to (12),
[0772] The control unit specifies the level of autonomous driving that the mobile device is performing based on the permitted autonomous driving level information for each road segment registered in the map information obtained from the map information provider server.
[0773] (14) The information processing device according to any one of (1) to (13),
[0774] The control unit performs the process of informing the driver whether the action corresponding to the request from the driver of the mobile device is a permitted activity corresponding to the level of autonomous driving that the mobile device is performing.
[0775] (15) The information processing device according to any one of (1) to (14),
[0776] The control unit calculates or obtains the duration for which the autonomous driving level of the mobile device is the same, determines the non-recommended activities corresponding to the calculated or obtained duration, and notifies the driver of the determined non-recommended activities.
[0777] (16) The information processing device according to any one of (1) to (15),
[0778] The control unit calculates or obtains the duration for which the mobile device's autonomous driving level is the same, and notifies the driver of recommended and non-recommended activities corresponding to the calculated or obtained duration.
[0779] (17) An information processing system, comprising: a mobile device; and a control device,
[0780] The mobile device obtains the level of autonomous driving it is performing.
[0781] The control device provides the mobile device with a list of permitted activities corresponding to autonomous driving levels, wherein autonomous driving levels and permitted activities are correlated.
[0782] The mobile device refers to the list of permitted activities corresponding to the autonomous driving level, obtains the permitted activities corresponding to the autonomous driving level that the mobile device is currently executing, and notifies the driver of the obtained permitted activities.
[0783] (18) The information processing system according to (17),
[0784] The mobile device obtains the level of autonomous driving that it is performing from the control device.
[0785] (19) An information processing method executed in an information processing device.
[0786] The control unit obtains the autonomous driving level of the mobile device being executed and notifies the driver of the driver's permitted activities corresponding to the obtained autonomous driving level.
[0787] (20) A program that causes an information processing device to perform information processing, the program causing a control unit to:
[0788] Obtain the level of autonomous driving that the mobile device is currently performing, and
[0789] Inform the driver of permitted activities corresponding to the acquired level of autonomous driving.
[0790] Furthermore, the series of processes described in the specification can be executed by hardware, software, or a combination of both. In the case of software-executed processing, the program recording the processing sequence can be executed by memory installed within a computer incorporated into dedicated hardware, or the program can be executed by a general-purpose computer capable of performing various processes. For example, the program can be pre-recorded on a recording medium. Besides installing the program from the recording medium into the computer, the program can also be received via a network such as a local area network (LAN), the Internet, or a wide area network (WAN) including cellular networks, and installed on a recording medium such as an internal hard drive.
[0791] Note that the various processes described in this specification can be executed not only in the described sequence of events, but also in parallel or individually, depending on the processing capacity of the device performing the process or as needed. Furthermore, in this specification, the system is a logical collection of multiple devices, and is not limited to a system in which various constituent devices are housed in the same housing.
[0792] Industrial applicability
[0793] As described above, according to one embodiment of the present disclosure, a configuration is implemented for notifying a driver of permitted activities corresponding to the level of autonomous driving.
[0794] Specifically, for example, the system obtains the autonomous driving level currently being performed by the mobile device and notifies the driver of the permitted activities corresponding to the obtained autonomous driving level. The control unit retrieves a list of permitted activities corresponding to autonomous driving levels from an external server or storage unit, which associates autonomous driving levels with permitted activities. Referring to this list, the control unit obtains the permitted activities corresponding to the autonomous driving level and notifies the driver of the obtained permitted activities. Furthermore, if the autonomous driving level of the mobile device changes, the system analyzes the differences between the permitted activities corresponding to the change in autonomous driving level and, if differences exist, notifies the driver of the discrepancy.
[0795] This configuration provides a way to notify the driver of permitted activities corresponding to the level of autonomous driving.
[0796] List of reference numerals
[0797] 10 mobile devices
[0798] 20 Roadside Communication Units (RSUs)
[0799] 30 base stations
[0800] 40. Communication Network
[0801] 50 Control equipment
[0802] 50b Map information providing server 70 Information processing system
[0803] 100 Information processing equipment
[0804] 110 Communication Unit
[0805] 111 Receiving Unit
[0806] 112 Transmitting Unit
[0807] 120 Control Unit
[0808] 121 Data Collection Unit
[0809] 122 Driving Assistance Processing Unit 123 Decision Unit
[0810] 124 Communication Control Unit
[0811] 131 Sensor Unit
[0812] 132 storage units
[0813] 133 Input and Output Unit; 151 Position Information Sensor; 152 Camera Module; 153 LiDAR
[0814] 154 Radar
[0815] 155 sensor
[0816] 156 Driver monitoring sensor 201 Confirmation button 301 CPU
[0817] 302ROM
[0818] 303RAM
[0819] 304 bus
[0820] 305 Input and Output Interfaces 306 Input Units
[0821] 307 Output Unit
[0822] 308 memory cells
[0823] 309 Communication Unit
[0824] 310 drive
[0825] 311 Removable media
[0826] 321 sensor
[0827] 322 drive unit
Claims
1. An information processing device, comprising: Control unit, the control unit being configured to: Obtain the first level of autonomous driving processing performed by the mobile device, where The first level of autonomous driving corresponds to a first set of permitted activities associated with the driver of the mobile device; Based on the acquired first level of autonomous driving, control the output of a first notification to the driver, wherein the first notification indicates the first group of permitted activities; Detect the change from the first level of autonomous driving to the second level of autonomous driving, where The second level of autonomous driving corresponds to a second group of permitted activities that are different from the first group of permitted activities; Based on the change from the first level of autonomous driving to the second level of autonomous driving, the differences between the first group of permitted activities and the second group of permitted activities are determined; as well as Based on the aforementioned differences, the control outputs a second notification to the driver.
2. The information processing apparatus according to claim 1, wherein the control unit is further configured to control one of the following: The display unit displays the first group of permitted activities, or The voice output unit outputs voice related to the first group of permitted activities.
3. The information processing device according to claim 1, The control unit is further configured to obtain the first set of permitted activities based on the list of permitted activities corresponding to the autonomous driving level. The list of permitted activities corresponding to each autonomous driving level includes multiple autonomous driving levels and multiple permitted activities. The multiple levels of autonomous driving include the first level of autonomous driving and the second level of autonomous driving. The plurality of permitted activities includes the first group of permitted activities and the second group of permitted activities, and Each of the plurality of autonomous driving levels corresponds to a specific set of permitted activities among the plurality of permitted activities.
4. The information processing device according to claim 3, The control unit is further configured to: The first set of permitted activities is obtained from an external server or storage unit based on the list of permitted activities corresponding to the autonomous driving level. Based on the first set of permitted activities obtained, the control outputs the first notification to the driver.
5. The information processing device according to claim 1, wherein the control unit is further configured to: Based on the differences between the first group of permitted activities and the second group of permitted activities, determine the difference information; Generate display data including the aforementioned difference information, and Control the display unit to output the display data.
6. The information processing device according to claim 1, wherein the control unit is further configured to: Detects a decrease in automation level from the first level to the third level. Based on the reduction, specific allowed activities are removed from the first group of allowed activities; Generate an instruction to delete the display data of a specific allowed activity from the first group of allowed activities; and The control display unit outputs the display data.
7. The information processing device according to claim 1, wherein the control unit is further configured to: Based on the first notification, determine whether there is a confirmation response from the driver, and Based on the lack of confirmation from the driver, emergency response procedures were initiated.
8. The information processing device according to claim 7, wherein the control unit is further configured to stop the mobile device during the emergency response process.
9. The information processing device according to claim 1, wherein the control unit is further configured to: Determining driver activity based on sensor detection information. The determined driver's activity differs from the first group of permitted activities, and Emergency response procedures are executed based on the determination that the driver's activity differs from the first set of permitted activities.
10. The information processing device according to claim 9, wherein the control unit is further configured to stop the mobile device during the emergency response process.
11. The information processing apparatus according to claim 1, wherein the control unit is further configured to: Based on acquiring specific information from at least one of the mobile device's storage unit, sensor, external server, base station, or roadside communication unit; and The first level of autonomous driving is designated based on the specific information obtained.
12. The information processing device according to claim 1, wherein the control unit is further configured to: Obtain map information from the map information provider server; Based on the map information, the permissible autonomous driving level information for each of the multiple road segments is determined; and Based on the permitted autonomous driving level information, the first autonomous driving level is specified.
13. The information processing device according to claim 1, wherein the control unit is further configured to: Obtain a request from the driver specifying a particular activity; and The control outputs a third notification to the driver, wherein the third notification indicates one of the following: The specific activity corresponds to a permitted activity in the first group of permitted activities, or The specific activity is different from the first group of permitted activities.
14. The information processing apparatus according to claim 1, wherein the control unit is further configured to: Determine the duration for which the mobile device performs the first level of autonomous driving; Non-recommended activities are determined based on the specified duration; and The control outputs a third notification to the driver indicating the non-recommended activity.
15. The information processing apparatus according to claim 1, wherein the control unit is further configured to: Determine the duration for which the mobile device performs the first level of autonomous driving; Based on the determined duration, at least one of the recommended or non-recommended activities is identified; and The control outputs a third notification to the driver indicating at least one of the recommended or non-recommended activities.
16. An information processing system, comprising: The mobile device is configured as follows: Perform autonomous driving processing; as well as Obtain the first autonomous driving level of the autonomous driving process, and The control device is configured as follows: Provide the mobile device with a list of permitted activities corresponding to autonomous driving levels, wherein the list of permitted activities corresponding to autonomous driving levels includes multiple autonomous driving levels and multiple permitted activities; Obtain the first set of permitted activities from the plurality of permitted activities corresponding to the first level of autonomous driving of the mobile device, wherein The first group allows activities to be associated with the driver of the mobile device, and The plurality of autonomous driving levels includes the first autonomous driving level; and Based on the acquired first level of autonomous driving, the control outputs a first notification to the driver, wherein the first notification indicates the first group of permitted activities; Detect changes from the first level of autonomous driving to a second level of autonomous driving among the plurality of autonomous driving levels, wherein The second level of autonomous driving corresponds to the second set of permitted activities among the plurality of permitted activities, and The second group of permitted activities differs from the first group of permitted activities; Based on the change from the first level of autonomous driving to the second level of autonomous driving, the differences between the first group of permitted activities and the second group of permitted activities are determined; as well as Based on the aforementioned differences, the control outputs a second notification to the driver.
17. The information processing system according to claim 16, wherein the mobile device is further configured to obtain the first autonomous driving level from the control device.
18. An information processing method, comprising: In information processing equipment: Obtain the first level of autonomous driving processing performed by the mobile device, where The first level of autonomous driving corresponds to a first set of permitted activities associated with the driver of the mobile device; Based on the acquired first level of autonomous driving, the control outputs a first notification to the driver, wherein the first notification indicates the first group of permitted activities; Detect the change from the first level of autonomous driving to the second level of autonomous driving, where The second level of autonomous driving corresponds to a second group of permitted activities that are different from the first group of permitted activities; Based on the change from the first level of autonomous driving to the second level of autonomous driving, the differences between the first group of permitted activities and the second group of permitted activities are determined; and Based on the aforementioned differences, the control outputs a second notification to the driver.
Citation Information
Patent Citations
Driver assist design analysis system
JP2019018842A
Automatic driving control system
JP2019185246A
Travelling control system for vehicle
JP2020029238A
Driving authorization transfer determination device and driving authorization transfer determination method
CN109843689A
Head-up display device
CN209955763U
