Autonomous vehicle systems

Abstract

To solve human factor problems in a self-drive vehicle.SOLUTION: A system includes: map image data associated with a map coordinate; storage means constituted to store accompanying data that is associated with the map coordinate and unassociated with drive tasks; measurement means for determining a current position and status of a vehicle in the map coordinate; and information presentation means for presenting information to a driver of the vehicle, where the information presentation means is constituted to present the driver with the information in which the accompanying data relating to the current position of the vehicle in the map coordinate is superimposed on the map image data relating to the current position of the vehicle in the map coordinate, when driving of the vehicle is determined to be in a state which does not require operations of the driver based on the measured reliability.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a system for use in an autonomous vehicle. [Background technology]

[0002] The widespread use of car navigation systems, as exemplified by Patent Document 1, has brought about a major revolution in vehicle driving. Furthermore, there has been remarkable progress in autonomous driving technology in recent years, with the practical application of Level 3 autonomous driving on the horizon. Level 3 (conditional autonomous driving) refers to a system in which the system performs all driving tasks, but the driver must respond appropriately to requests for system intervention (see Non-Patent Document 1). Level 3 autonomous vehicles are now permitted to be driven in Japan under the revised Road Traffic Act, which came into effect on April 1, 2020.

[0003] However, at Level 3, the problem is that the human driver does not know when they will be required to intervene in the driving task. During Level 3 autonomous driving, the driver is essentially left with nothing to do, so to speak. Even if a situation that could lead to an accident occurs and the driver is required to take over immediately, it will take some time for the human driver's brain to keep up.

[0004] In other words, although a situation where the vehicle is forced to switch from automated driving to manual driving is generally considered an urgent matter (beyond the capabilities of the automated driving machine), there is an inherent problem in that it is not possible to shift the functions of the human brain from a task other than driving to the driving task (to take over driving) in an emergency.

[0005] These Level 3 issues are referred to in the technical field as "human factor" issues, and various solutions are being explored. For example, methods have been proposed such as having the autonomous driving system give a warning to the human driver before taking over, having the driver perform a visually stimulating subtask such as a number guessing game while autonomous driving is in progress, and blowing cool air onto the driver if the system determines that the driver is drowsy during autonomous driving (see Non-Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 01-263688 [Non-patent literature]

[0007] [Non-Patent Document 1] Regarding the classification of automated driving levels, Ministry of Land, Infrastructure, Transport and Tourism, March 19, 2018 http: / / www.mlit.go.jp / common / 001226541.pdf [Non-patent document 2] Presentation materials from the SIP Symposium on Human Factors Research and Development Results for Automated Driving Systems, held at Bellesalle Onarimon Tower 3F Hall on July 29, 2019, hosted by the National Institute of Advanced Industrial Science and Technology (AIST). https: / / seminar.tokyotobs-entry.jp / Symposium / https: / / web.archive.org / web / 20200818144741 / https: / / seminar.tokyotobs-entry.jp / Symposium / Summary of the Invention [Problem to be solved by the invention]

[0008] However, for the average human driver, being forced to intervene in automated driving using the conventional methods described above can be unpleasant. Being placed in the unnatural position of being bored in the driver's seat without having to drive, and then suddenly being asked to return to the driving task due to circumstances unrelated to their own actions, can be very stressful for a human driver.

[0009] Furthermore, situations such as being forced to perform subtasks like a number guessing game or being blown with cold air are purely for the convenience of the autonomous driving system and do not take into account the psychological discomfort felt by human drivers.

[0010] Although resolving the above-mentioned human factor issues is essential to popularizing Level 3 autonomous driving, the reality is that no effective measures have been found yet. [Means for solving the problem]

[0011] In view of the current state of the art, the present inventors have proposed a means for solving the human factor problem. That is, the following aspects can be provided in the embodiments of the present invention.

[0012] Aspect 1 1. A system for use in automated driving of a vehicle, comprising: a storage means configured to store map image data associated with map coordinates and associated data associated with the map coordinates and not related to a driving task; positioning means for determining the current vehicle position and status in said map coordinates; an information presentation means for presenting information to a driver of the vehicle; A means for measuring the reliability of the automatic driving of the vehicle; Including, The information presenting means is configured to present to the driver, when it is determined that operation of the vehicle does not require the driver's operation based on the measured reliability, information in which the map image data related to the current position of the vehicle in the map coordinates is superimposed with the accompanying data related to the current position of the vehicle in the map coordinates. A system characterized by:

[0013] Aspect 2 moreover an imaging means for capturing an image of the scenery outside the vehicle; editing means for creating the accompanying data based on the images and / or videos of the scenery photographed by the imaging means; 2. The system of embodiment 1, comprising:

[0014] Aspect 3 The system of aspect 1 or 2, wherein the accompanying data includes one or more types selected from the group consisting of natural topography and geological data, historical map image data, engineering technology data, data on historical figures or culture, literary and folktale data, tourist information data, data on indigenous flora and fauna, and disaster prevention data associated with the map coordinates.

[0015] Aspect 4 4. The system according to any one of aspects 1 to 3, wherein the associated data includes audio, and the information presentation means includes an audio transmission device.

[0016] Aspect 5 5. The system of claim 4, wherein the voice transmission device is a speaker mounted on the vehicle or a speaker carried by the driver.

[0017] Aspect 6 The system of any one of aspects 1 to 5 further includes a means for measuring the driver's readiness state, and when the driver's readiness state is determined to be below a given threshold, the information presentation means emphasizes the information when providing it.

[0018] Aspect 7 7. The system according to any one of aspects 1 to 6, wherein the map image data is displayed as a bird's-eye view map, a planar map, or a 3D map on the information presentation means. [Effects of the Invention]

[0019] The embodiments of the present invention have the significant effect of solving the human factor issues of Level 3 autonomous driving, and furthermore, making it possible to provide a new driving experience (user experience) that could not be realized with existing vehicles. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram illustrating the principle by which environmental information is obtained from unique ambient light. [Figure 2] FIG. 1 is a schematic diagram illustrating the principle by which the recognition of the structure of the environment is updated as the observer moves. [Figure 3A] 1 is a schematic diagram showing the interior of a vehicle to which the system of the present invention is applied; [Figure 3B] FIG. 10 is a schematic diagram showing the interior of another vehicle to which the system of the present invention is applied. [Figure 4A] 1 shows an overview of an example in which the windshield of a vehicle is used as part of an information presentation means (an example in which the windshield is used as a head-up display) used in a system according to an embodiment of the present invention. [Figure 4B] 1 is a diagram showing the configuration around the driver's seat of a standard passenger car incorporating a system according to an embodiment of the present invention. [Figure 4C] 1 is a diagram showing the interior configuration of a standard passenger car incorporating a system according to an embodiment of the present invention. [Figure 5] 1 is a diagram illustrating a schematic overview of a system according to an embodiment of the present invention. [Figure 6] 1 is a schematic diagram illustrating the principle of a high-order guide according to an embodiment of the present invention; [Figure 7]1 shows an example of car navigation that can be realized by a system according to an embodiment of the present invention. [Figure 8] 1 shows an overview of a planning module included in a system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to these.

[0022] (Basic principles and affordances of the present invention) The inventors have completed the present invention by focusing on the spatial interrelationship and cognitive relationship between a user of an autonomous vehicle and an object outside the vehicle.

[0023] Affordance is a concept proposed by ecological psychologist James J. Gibson, and is defined as the meaning / value that an animal (organism) that exists in an environment can acquire by exploring the environment in which it lives. When an animal, such as a human, moves through space, the scenery of the surrounding environment changes as they move. This may seem obvious, but this changing scenery is nothing other than the light stimuli that the animal's eyes perceive. So, let's consider how the light that provides information about this scenery reaches the animal's eyes.

[0024] Light emitted from a light source such as the sun is scattered when it hits tiny dust particles as it passes through the atmosphere or water, and is also scattered and reflected off the surface of objects. This scattered and reflected light fills the atmosphere in all directions. This state is called "illumination," and "everywhere" under this illumination is surrounded by light coming from "everywhere." Gibson called this fact "ambient light."

[0025] This ambient light carries information from the surrounding surface conditions (also known as texture). In other words, ambient light projects the conditions and layout of the surfaces that surround it in all directions. Therefore, no matter where you are on Earth, there is a unique light structure that exists there (see Figure 1). This is why, when an animal moves, its eyes see different landscapes that correspond to each location at each time. In other words, as an animal moves, the structure of the environment that the animal perceives is also updated (see Figure 2).

[0026] Furthermore, as the animal moves, it will continue to perceive changes in texture. As it moves, distant objects in the structure of perspective appear to come closer and enlarge, and the approaching texture passes around the sides and bottom of the animal and flows away behind it. During this process, numerous affordances are contained within the texture. There is a chain of affordances here. Within this chain, the affordances of the environment in the depth direction (ahead of the animal's direction of travel) are generally occluded by buildings or other objects in front and cannot be detected. However, as the animal continues to move, new textures appear one after another from behind the occlusions, and affordances become apparent.

[0027] If we apply this to the driver and other passengers of an autonomous vehicle, the passengers will sense their own movement by detecting the scenery (caused by ambient light) visible from the driver's seat window, etc., and will dynamically receive affordance signals that indicate the space in which the vehicle can safely drive.

[0028] Generally, an autonomous vehicle (or its occupants) moves or stands still in space, and recognizes objects in the surrounding coordinates from their coordinates. The following fundamental relationship exists between space and recognition.

[0029] First, the occupant's coordinates correspond to the geometric coordinate range of the spatial situation in which it is potentially possible to view and recognize an object (the broad affordance coordinates in which a specific optical image of the object exists).

[0030] Secondly, when the user uses their will to look at an object, the image of the object is transmitted to the retina, and the user is able to recognize the composition.

[0031] For example, if you are in a location that has the affordance of being able to see a particular mountain, you can view that particular mountain (in real space) in a particular composition. This can be understood by considering the location of Cézanne's sketch of Mont Sainte-Victoire.

[0032] Based on these fundamental relationships, the present invention can be realized by utilizing spatial coordinate information. For example, a design that adds value to the view from the garden paths of a historic strolling garden is a prime example of linking coordinates with cognition.

[0033] The same can be said for virtual spaces (spaces represented on car navigation systems). For example, when a passenger (e.g., a driver) of an autonomous vehicle in a certain space recognizes a specific object (such as a natural terrain outside the vehicle, e.g., mountains, hills, capes, inlets, or mountain passes, or a rural area, city, facility, building, or other landscape) on a map image on the navigation screen, the system according to the present invention can sequentially provide the driver with information (data) about that specific object. Furthermore, such objects are not limited to actual objects outside the vehicle, but can also be applied to information about historical spaces, literary works, and other objects. Examples of such information include, but are not limited to, data about old townscapes that are now lost, data about old maps, data about people and literary works associated with the area, tourist guides about the area, data about the geology, flora and fauna, and disaster prevention measures (hazard maps). The information may also include data on the history of the industry in the area shown on the map, the history of the people and companies involved in the industry, and the history of innovation.

[0034] By providing such data (also referred to herein as "associated data") to the driver of an autonomous vehicle according to the vehicle's driving situation, the driver can continuously obtain information about their current location (the vehicle's current location). In other words, by presenting the driver with associated data, which is information related to the location where the autonomous vehicle is traveling, the driver's brain is continuously exposed to information relevant to the driver's current location. In other words, the information represented by such associated data is not directly related in real time to the driving task itself, which the driver cannot participate in during autonomous driving, but is related to the affordance signals conveyed by the autonomous vehicle and recognized by the driver.

[0035] In summary, the accompanying data referred to in this specification is distinguished from information directly related to the driving task in real time, such as information provided by conventional car navigation systems, such as weather, road congestion, and the driving conditions of nearby vehicles (also referred to as "driving guide navigation" in this specification). In other words, accompanying data can be thought of as information that is not directly related to the driving task but is related to the location (map coordinates) where the autonomous vehicle is traveling. Navigation that uses such accompanying data is also referred to as "advanced guide navigation" in this specification.

[0036] With this configuration of the present invention, the driver's readiness state (a term used in the technical field to describe the degree of "readiness" of consciousness for automated driving) can be maintained at a certain level or higher, and the driver will not become bored or drowsy. Nevertheless, the driver can maintain the readiness state by being exposed to the information presented by this system, even if they do not necessarily have to keep their eyes fixed on the maneuvering space. In other words, even if the automated driving system requests them to take over driving, a driver who maintains such a high readiness state can smoothly transition to the driving task without feeling any discomfort.

[0037] Furthermore, the effects of the present invention are not limited to level 3, but can also be applied to passengers in autonomous driving at levels 4 and 5. That is, as described in Non-Patent Document 1 above, level 4 refers to fully autonomous driving under specific conditions, such as on expressways, and level 5 refers to fully autonomous driving without any specific conditions.

[0038] Even during such automatic driving, the high-level guide navigation function according to the present invention may be fully utilized to enable the enjoyment of a spatiotemporal theater effect in many areas according to one's hobbies and tastes.

[0039] For example, the present invention can be used in Level 4 autonomous driving limited to a specific area (i.e., under specific conditions). While an autonomous bus is traveling a short distance from a train station to a ferry terminal, as is already being planned for practical use, the advanced navigation function of the present invention can be used to provide the driver and passengers with timely information about the scenery and history of the surrounding area along the route. Similarly, the present invention can be used to provide drivers with a weekly service of brief local history guides on autonomous buses for daily life support in remote islands or mountainous areas. Furthermore, such autonomous buses can be used for tourism purposes, and the advanced navigation function of the present invention can be used to improve the quality of tourist information. In this way, providing drivers and passengers of autonomous vehicles (including large vehicles such as buses) with map information and accompanying data overlaid on each other can prevent boredom and allow them to learn about the area they are traveling in.

[0040] The period following this application will be a crucial one for the upcoming launch of Level 3 autonomous vehicles. By utilizing this invention during this valuable period and expanding the use of advanced navigation functions while avoiding driver distraction, many vehicles equipped with advanced navigation will be on the road. This will allow the spatial data detection function to accumulate operational data on autonomous vehicles in various locations, as well as data on the behavior of vehicle occupants. By utilizing this data for machine learning (e.g., deep learning), the development of even more advanced systems will be promoted, ultimately creating a scenario that demonstrates the appeal of autonomous driving services to the market. In this way, it is clear that this invention will be used more widely in preparation for the practical launch of Level 4 and even Level 5 systems, which are expected to be available in the coming years from this application.

[0041] The example shown in Figure 3A is a schematic diagram of the driver's seat of a passenger vehicle (this figure illustrates a standard passenger vehicle, but the invention can also be applied to large vehicles and other specialized vehicles). In this example, a large display is placed in the center of the front of the vehicle to enable efficient use of the advanced navigation system, making information easily viewable from the driver's seat to the passenger seat. This large display may be an integrated unit, or multiple displays may be arranged side by side. This configuration makes it easier for passengers to obtain information when the advanced navigation system's diverse service content is provided, promoting their understanding and increasing their enjoyment. It also makes it easier for the system of the present invention to recognize changes in the passenger's readiness state, and facilitates understanding the status of the automated driving device and handing over operation in accordance with the above-mentioned revised Road Traffic Act.

[0042] By utilizing this invention in this way, we can foster a closer cooperative relationship between people and vehicles, increase the enjoyment of driving, deepen knowledge, and eventually build a new mobile civilization. A superior self-driving vehicle equipped with the advanced guide navigation system of the present invention can travel over large terrain, expand its field of view, and expand the awareness of the human brain. Please understand this in conjunction with the explanations of Figures 5 and 6 below.

[0043] FIG. 3B shows a schematic diagram of the interior of a passenger car according to another embodiment of the present invention. In this example, a high-level guide navigation system (also referred to as a high-level planning project system) incorporating the "planning" module described later with reference to FIG. 8 and other figures is applied. In addition to the effects obtained in the example of FIG. 3A, this example can also display survey and photographic data from the location where the passenger car is traveling, and present plan simulations. The passenger car can also be used as a place for remote meetings. This has the effect of making space-time studies more efficient, so to speak.

[0044] 3A or 3B to a large vehicle, it will be appreciated that the information presented to the driver of that large vehicle can also be presented to passengers, either via their own individual displays or via a large display (continuous display) installed to partially or completely cover the ceiling, walls, or windows of the vehicle.

[0045] On the other hand, even if the driver is provided with only information related to the driving task itself from the car navigation system, as in the past, the driver is not / cannot be involved in the driving task during automated driving, so this is boring and has the opposite effect on maintaining a state of readiness. Generally, it is difficult for humans to maintain interest in things that they cannot be involved in.

[0046] Furthermore, as previously mentioned, even if the driver is provided with information (such as television programs or games) that is unrelated to the driving task or affordance signals, as in the past, the readiness state cannot be maintained.

[0047] The system according to the embodiment of the present invention overcomes the drawbacks of the conventional technology and also provides an excellent user experience. In other words, for the first time in the age of autonomous driving, the driver is freed from driving and can freely access the space outside the vehicle, enjoying and learning about the wonderful values ​​that fill that space. This user experience is unprecedented in the automotive field, and it can be said to create a new era of time-space theater in which the inside and outside of the vehicle are a common moving stage that transcends time and space.

[0048] (Basic configuration of an autonomous driving system) A system according to an embodiment of the present invention may include a storage means, a positioning means, an information presentation means, and a means for measuring the reliability of the autonomous driving of a vehicle.

[0049] The storage means may be a storage used by a computer, a car navigation system (car navigation system), or the like, and is configured to store map image data associated with map coordinates and accompanying data. As described above, the accompanying data relates to information associated with map coordinates but not related to the driving task.

[0050] The map image data can be used arbitrarily by a navigation system such as a car navigation system, and when processed by a processor, can be displayed as, for example, a bird's-eye view map, a planar map, or a 3D map. Note that 3D maps include, but are not limited to, 3D maps that can be displayed using means such as holograms, 3D displays, and 3D printers.

[0051] The accompanying data relates to information associated with the map coordinates, such as natural topography and geological data, historical map image data, engineering technology data, historical figures or cultural data, literature and folklore data, tourist information data, data on inhabiting flora and fauna, and disaster prevention data. Additionally, the accompanying data may include data on "innovation simulation" (described later).

[0052] Geological information data may include data on geology and geography, such as data on the existence of large and medium-sized landforms such as strata and volcanoes, rivers, lakes, etc. For example, the system can determine when an autonomous vehicle is approaching the western edge of the Fossa Magna (Itoigawa-Shizuoka Tectonic Line) and present information about the Fossa Magna to the driver.

[0053] Historical map image data may include not only so-called antique maps, but also data on maps that served as settings for literary or artistic works. For example, it may be data on a map tracing the route of "Tokaido Chu Hizakurige." The system can determine whether the location where the self-driving vehicle is traveling corresponds to a location that served as a setting for such a work, and present information about that to the driver.

[0054] The engineering data may include data related to engineering such as architecture, civil engineering, bridges, roads, etc. For example, the system may determine that an autonomous vehicle has entered the Seto Ohashi Bridge or that the autonomous vehicle is in a position where the driver can see the Seto Ohashi Bridge, and may present the driver with information about the history of the Seto Ohashi Bridge, its structure, related local governments, etc.

[0055] Data on historical figures or cultures can include data on people or cultures associated with a particular area. For example, the system may determine that an autonomous vehicle is approaching Sakuradamon Gate and present data on Ii Naosuke to the driver.

[0056] Literature and folklore data can include data on literature and folklore related to the area (or movies and manga related to them). For example, the system can determine that an autonomous vehicle is in a position where the driver can see Mount Hakkoda (an example of a medium terrain), and present information (such as bibliography and reviews) about Jiro Nitta's novel "Death March on Mount Hakkoda" to the driver.

[0057] The tourist information data may include data about tourism, travel, local products, etc. For example, the system may determine that an autonomous vehicle is approaching the coast where Izura Rokkakudo Temple is located in Ibaraki Prefecture, or that the autonomous vehicle is in a position where the driver can see Izura Rokkakudo Temple, and present tourist information (such as information about admission times and admission fees) to the driver. Alternatively, the system may determine that an autonomous vehicle is approaching a point where the Izura Rokkakudo Temple is not yet visible to the occupant, but is in the vicinity, and present information about Okakura Tenshin to the occupant.

[0058] Data on inhabiting flora and fauna may include not only wild fauna and flora, but also data on livestock and crops on ranches, farms, etc. For example, the system may determine that an autonomous vehicle is approaching a fishing port in Onomichi City, Hiroshima Prefecture, or that the autonomous vehicle is in a position within the driver's line of sight of the fishing port, and present the driver with information about scorpionfish and other fish that can be caught at the fishing port.

[0059] Disaster prevention data may include information about earthquakes, floods, rock falls, landslides, etc. related to the region. For example, if the system determines that the autonomous vehicle is in a position where the driver can see a monument in an area that has been hit by a historical disaster in the past, it can present information about the disaster and the monument to the driver.

[0060] The accompanying data is not limited to visual images (still images and videos) but may also include audio information. It may also include information related to the sense of smell, touch, or taste obtained through some kind of sensor or air blowing means.

[0061] The positioning means included in the system determines the current vehicle position in map coordinates and may be, but is not limited to, GPS or the Quasi-Zenith Satellite System (QZS) that can be used by car navigation systems.

[0062] The information presentation means included in this system may be any means for presenting information to the vehicle driver, such as a display (such as a car navigation system monitor, a television monitor, or a mobile device screen). Alternatively, it may be a projector that projects information onto the window or wall of an autonomous vehicle. When such a projector is used on a window, it is possible to obtain the effect of directly overlaying information onto the passing scenery seen outside the window. In this case, the scenery outside the window can be considered to be treated as data that reflects map image data (dominating as map image data). Alternatively, in such a case, this system may use both map image data and the scenery outside the window to overlay associated data.

[0063] FIG. 4A shows an example in which the windshield of a vehicle is used as part of an information presentation means (an example in which the windshield is used as a head-up display). The head-up display according to this embodiment may include a projector capable of projecting information onto the windshield, an occupant position sensor for detecting the driver's position, line of sight, and the like, and an imaging means for capturing an image of the scenery outside the windshield. The imaging means (such as a camera) captures an image of the scenery outside the vehicle, and computer means processes the image to correlate it with map image data as described above, thereby determining which coordinates on the map correspond to which topographical features (such as mountains) in the scenery (obtaining the scenery of the afford point of the scenery). Then, the mountains associated with the coordinates on the map can be associated with accompanying data (such as the name of the mountain) related to the mountains.

[0064] The occupant position sensor can calculate the line of sight of the driver or other occupant based on the occupant's posture and physical characteristics, and simulate the scenery as seen by the occupant's eyes. The projector can then project the accompanying data so as to overlay it on the scenery the occupant is supposed to see. Preferably, the projector's projection can be automatically corrected according to changes in the vehicle's operation and the occupant's posture.

[0065] Means for measuring the reliability of autonomous driving of a vehicle may include means known in the art, such as people / object sensors, radar, lidar, and V2X (vehicle to X). Furthermore, such means may also be integrated with the steering control system. These means allow the autonomous driving system to know when to switch between autonomous driving and manual driving. For example, the degree of reliability may be displayed on the information presentation means described above using graphs, numerical values, etc.

[0066] Based on the reliability thus measured, when the vehicle is in a state where it is determined that the vehicle's operation does not require driver operation (i.e., an automatic driving state), information in which map image data related to the current vehicle position in map coordinates is superimposed with accompanying data related to the current vehicle position in map coordinates is presented to the driver via the information presentation means.

[0067] In some embodiments, the system may further include an imaging means (such as a camera) for capturing images of the scenery outside the vehicle. The system may also include an editing means (such as computer software) for creating associated data based on images and / or videos of the scenery captured by the imaging means. The editing means can process and edit the information stored in the database to automatically generate associated data.

[0068] For example, landscape images obtained by an imaging means may be associated with the position and track of the corresponding autonomous vehicle, and classified through machine learning performed by a computer processor to generate accompanying data that can be overlaid on map image data. The acquisition of landscape images by such an imaging means is preferably continuous, but may also be intermittent or discontinuous.

[0069] In some embodiments, determining whether a driver of an automated vehicle can see a particular scene associated with associated data may be based on map image data, such as by including data on the elevation, shape, or other suitable geometry of the terrain or relief present in the map image data, and performing calculations to simulate the driver's field of view. Preferably, the driver's physical characteristics (e.g., height, eye position, etc.) may also be factored into the calculations.

[0070] In another embodiment, the determination of whether the driver of the autonomous vehicle can see the specific scene associated with the accompanying data may be performed by the imaging means described above. For example, such an imaging means may be installed near the driver's eyes to reproduce the scene from the driver's perspective. Alternatively, as another method, the scene from the driver's perspective may be reproduced by combining images acquired by imaging means (preferably multiple imaging means) taking into account the relative distance and angle between the imaging means and the driver's eyes.

[0071] In some embodiments, the system may include an audio output device such as a speaker. For example, (a portion of) the accompanying data including audio data may be output from the audio output device. The speaker may be mounted in the vehicle or may be a portable speaker carried by the driver.

[0072] In some embodiments, a smart speaker may be used as the voice transmission device, and may accept voice input from the driver or other passengers in addition to the voice output described above. Such voice input may be used to operate the system. For example, such a smart speaker may be used to search for and set navigation destinations for the autonomous vehicle, or to adjust the brightness and volume of displayed content. Furthermore, the system may be adjusted to suit the user's intelligence level, such as by changing the level of expertise of the guide.

[0073] In some embodiments, the system may include a means for measuring the driver's readiness state. Examples of such a means include a sensor (e.g., a camera) for measuring the driver's eye movement, a sensor for measuring the driver's posture, and a sensor for measuring the driver's brain waves. The driver's readiness state may be evaluated based on the results of such measurements and quantified based on some measure. The system may determine whether the quantified readiness state is below a threshold determined based on some standard (e.g., a standard that may be established by a government agency or an industry association). The readiness state may also be displayed on a display means (which may be the information presentation means described above or another display) included in the system. Examples of such a display include, but are not limited to, numerical values, bar graphs, progress bars, and the like.

[0074] In some embodiments, if the driver's readiness state is determined to be below a given threshold, the information presentation means may provide enhanced information, such as increased contrast or brightness when presenting the information visually, or increased volume when presenting the information audibly.

[0075] FIG. 4B is a diagram showing the configuration of the area around the driver's seat of a standard passenger car (standard car) incorporating a system according to an embodiment of the present invention. It is assumed that this standard car can be operated at any level of autonomous driving. (For example, in the case of level 5 autonomous driving, the steering wheel can be retracted, as indicated by the dotted line in the figure.)

[0076] In the example shown in Fig. 4B, information can be presented on information presentation means (such as displays) around the driver's seat (cockpit), for example, as shown in *2 and *3 in Fig. 5 and Fig. 6. In this example, multiple displays are arranged side by side across the space between the driver's seat and passenger seat of a right-hand drive vehicle, and various information can be presented to the driver (and, if necessary, the passenger in the passenger seat).

[0077] 4C is a diagram showing the interior configuration of a standard passenger car (standard car) incorporating a system according to an embodiment of the present invention. This standard car may be equipped with the driver's seat shown in FIG. 4B.

[0078] Information presentation means (displays) such as those shown in Figure 4A or 4B above are installed around the front seat (driver's seat) in the vehicle, allowing the driver to view information and, if desired, providing information to passengers (users) in the front or rear seats. A separate information work desk is also provided in the rear seats, where information presentation means (displays, mobile PCs, etc.) can also be installed. The information presentation means in the front seats and the information presentation means in the rear seats can operate in conjunction with each other via a network using an in-vehicle LAN or wireless communication lines, allowing each passenger, including the driver, to share information. Users outside the vehicle and planners from the "planning" section (described below) can also remotely connect to such a network to share information.

[0079] In the examples shown in Figures 4B and 4C, information about the geological features of river terraces is presented and shared with each passenger in the vehicle. In other words, as the passenger vehicle moves (from left to right in the figure), the advanced navigation system of the present invention installed in the passenger vehicle can provide navigation with nearby geological features as accompanying data (based on a topographical and geological database, for example). Such navigation can provide, for example, a bird's-eye view image, as shown in the lower part of Figure 4C, with the accompanying data superimposed on the passenger vehicle's current location. This allows passengers to be provided with information not only about the scenery they can see directly from the vehicle window, but also about geological features (such as the formation of river terraces) that are not visible to the naked eye or that they would not notice without prior knowledge, even if they were visible.

[0080] By enabling this type of information provision and sharing, the space-time theater effect can be applied to multiple passengers in the vehicle, making it possible to analyze, plan, and project the area while traveling. In other words, the interior of the vehicle itself can become a mobile remote office, and a teleworking base where passengers can enjoy the rich natural environment. This effect was not possible with conventional car navigation systems.

[0081] Furthermore, the system according to the present invention can also adjust how information is displayed on an information presentation means (display) with a certain display area. For example, in the image shown at the bottom of Fig. 4C, the standard-sized vehicle is moving from left to right on the screen, so by displaying the position of the vehicle on the front seat side displays to the left of the center, it becomes easier to recognize and grasp the overall driving space.

[0082] FIG. 5 is a diagram showing a schematic overview of a system according to an embodiment of the present invention. The maneuvering space (*4) shown in the lower left of the figure is what the driver would have had to focus on in the past. Autonomous driving eliminates this need for focus. With this system, the driver senses (for example, through the windshield) the entire space (*1) surrounding the vehicle. This perception is carried out by the driver's brain function (*6b) via the driver's visual system and other sensory inputs (*6a).

[0083] While the vehicle is being driven automatically by the automated driving system (*12), the reliability of that automated driving is continuously measured and evaluated (*11). During automated driving, the high-level guide navigation system shown in the upper left of the figure (*2) functions to maintain the driver's readiness.

[0084] The advanced guide navigation is displayed superimposed on the map image data. In this case, the display may be switched between the conventional maneuver guide navigation (*5) and the advanced guide navigation, or the maneuver guide navigation and the advanced guide navigation may be superimposed. The maneuver guide navigation may display a parameter (STATUS) that quantifies the readiness state as described above, or a parameter that quantifies the reliability of automated driving (*5).

[0085] Additionally, the scenery (landscape) outside the vehicle is continuously captured by an imaging device and is used to automatically generate data (*7) for use in advanced navigation guidance. All of this data may be automatically edited, or some or all of it may be saved and sent for offline editing (*8). The data thus created can be linked to the vehicle's current location (coordinates on map image data) (*9) and displayed as advanced navigation guidance.

[0086] Based on feedback from the system for autonomous driving and the system for measuring the reliability of autonomous driving (*13a-13d), the reliability of autonomous driving and the driver's readiness state can be measured (*12b). If it is determined that the reliability of autonomous driving has decreased (driver intervention is necessary), a request to take over driving is made to the driver (*11a), and if the request is accepted (*11b), the handover can be carried out (*11c). In addition, based on such feedback (*12c), the output of the maneuvering guide navigation system can be adjusted to highlight and guide changes in the maneuvering environment (*10).

[0087] This structure is based on the fact that the driver's brain has a rich set of high-level thought and behavioral programs built on spatial awareness. In other words, the driver's spatial awareness can easily transition to a dynamic steering program within a limited steering space.(*6c) Therefore, this steering program works in conjunction with the basal ganglia loop involved in steering, increasing the likelihood of guiding the vehicle into a safe travel zone.

[0088] Figure 6 is a schematic diagram illustrating the principle of a high-order guide according to an embodiment of the present invention. First, from the ambient light conditions (*1) based on the time, season, weather, etc., the imaging means of this system (such as a landscape camera) can obtain information on the texture of the object (affordance signal) (*2).

[0089] Based on this information, the processor in this system can classify and recognize (through machine learning, etc.) images of various objects (landforms, mountains, capes, inlets, etc.) that have been photographed(*6). By matching the recognized objects A, B, etc. with coordinate data(*5) obtained from a spatial coordinate system database (map database), it is possible to generate data(*3) that positions each object in geometric space.

[0090] Furthermore, by associating the recognized objects A, B, etc. with semantic data obtained from a map database (such as the fact that a certain mountain or cape is located at a certain point) (*4), it is possible to recognize what those objects are.

[0091] In this way, we can calculate and construct an affordable space system (i.e., data representing a spatial system in which semantic spaces are arranged in the landscape that the driver can see from the vehicle).(*9) Then, by linking the affordable space system with data showing the vehicle's position on a map (car navigation data)(*10), we can create a high-level guide that can be presented to the driver.(*8)

[0092] When the output from the Afford space system is passed to a high-level automatic guide generation system, the automatic generation system can continuously select guide targets (targets related to the information presented to the driver) according to the vehicle's operation.(*11a~11b) Guide targets can be extracted from a multi-layered spatiotemporal database (a database that overlays data related to nature, culture, literature, technology, topography, geology, history, etc. onto corresponding map image data)(*12).

[0093] The data objects A, B, etc. obtained in this way may be further organized into a hierarchy, and child objects corresponding to the existence level, knowledge level, emotion level, etc. Also, various contents such as images, icons, sounds, music, etc. may be received(*13) from an external server(*14) and used to edit such data objects.

[0094] Based on the data obtained in this way, or the output data obtained by passing that data to the offline editing system (*11c), the automatic playback system can control the timing of playback (presentation to the driver) of high-level guidance in accordance with changes in the scenery (the flow of scenery as the vehicle moves) (*15). In this way, the real scenery can be synchronized with the flow of consciousness of the driver and other passengers, thereby maintaining a high level of driver readiness. It also has the effect of satisfying the intellectual curiosity of passengers other than the driver (*15b). Information can also be presented via a head-up display (*16).

[0095] Communication between the vehicle occupants and the system may be an interactive interface (*7), which may include, for example, a speaker and a microphone (*17-19).

[0096] 7 shows an example of car navigation that can be realized by a system according to an embodiment of the present invention. In this example, Tokyo is the starting point and Niigata is the destination. The example route is shown by a dotted line.

[0097] The driver of the autonomous vehicle can set the destination and route just like a normal car navigation system. The system can also provide a (brief) introduction of the entire route. The following examples are mainly for large and medium terrains.

[0098] When the system determines that the vehicle has departed, passed through the Kanto Plain, and that the driver can see the Kanto Plain, it can output characteristics of the Kanto Plain (such as geological features) as a high-level guide, and can present information about the formation history of the alluvial lowlands, for example.

[0099] When the system determines that the vehicle has reached a point where the driver can see Mount Asama, Mount Haruna, or Mount Akagi, it can present information about the principles underlying the formation of the volcanic fronts of those mountains, for example.

[0100] When the system determines that the vehicle has reached a point where the Tone River terraces are visible to the driver, it can present information about the geological characteristics of those river terraces, for example.

[0101] If the system determines that a vehicle is entering / has entered a tunnel area (near Gunma Prefecture) that passes through the Tanigawa Mountains, it can present information about tunnel excavation technology and the construction history of the Kanetsu Tunnel, for example.

[0102] When the system determines that the vehicle has reached a point where the driver can see Echigo Yuzawa or Hakkaisan, it can present information about each of them (such as information about folk tales related to the area).

[0103] When the system determines that the vehicle has reached a point where the driver can see the Uonuma Formation and the river terraces, it can present information about each (such as historical and geological information).

[0104] When the system determines that the vehicle has reached a point where the driver can see the Niigata alluvial lowlands, it can present information such as the formation of raccoons.

[0105] A system according to another embodiment of the present invention may further include a module (subsystem) that provides "innovation simulation" information. In this specification, "innovation simulation" information refers to information about urban planning or industrial development in a certain region, and is information that can explain the origin and history of so-called innovations.

[0106] Figure 8 shows an overview of such a planning module. For example, such a module may be placed alongside the time-space database (*12) in Figure 6 so that it works in cooperation with it. As an example, Figure 8 shows a planning module that divides innovation propositions into three hierarchical levels: large-scale, medium-scale, and small-scale. While Figure 8 shows examples of success, it goes without saying that it is not limited to these and may also include unsuccessful examples.

[0107] Large-scale issues refer to advanced, innovative regional development and smart city development that affect the entire region.

[0108] Information on the proposition of advanced innovation-based regional development could be, for example, data on innovation in a certain region over time and space (e.g., information on the history of semiconductor and computer hardware / software innovation in Silicon Valley). In other words, the planning module can store information on how industry has developed based on a certain region's social and cultural foundations (such as what kind of people moved to the region and how they started businesses), economic foundations (such as what kind of people provided support for business startups and how), and topographical structure (such as how rivers and mountains are distributed and what facilities have been built where in relation to them) as "data on large-scale propositions."

[0109] Information related to the proposition of smart city development may include, for example, information on how the layout and control methods of supply networks such as electricity and hydrogen energy, information and communication networks, security systems, or automated mobility systems are related to the overall urban development of the area.

[0110] Medium-scale propositions refer to the development of teleworking districts and relocation plans for river flood areas, which are related to the composition of parts of a region (districts).

[0111] Information on the topic of developing teleworking districts could include, for example, information on where and how districts suitable for teleworking have been established. For example, it would be good to know that a certain company or local government has established such a district on a cool plateau so that teleworking can be done comfortably even in the summer.

[0112] Information related to the proposition of river flood area relocation planning could include, for example, information on the history or plans to relocate functions from areas judged to be at high risk of disaster based on hazard maps to other areas judged to be at low risk of disaster.

[0113] Small-scale propositions refer to disaster prevention measures, natural environment restoration plans, lifeline plans, etc., which are related to the composition of a small part (plot) of a region. Examples of such information include how water supply and sewerage systems are arranged or reforestation is carried out in a certain plateau or residential area.

[0114] The information described above can be stored in a storage means (server, storage, etc.) related to the "reference system" in Fig. 8. This information is not limited to information about a specific region, but may systematically include data about regions around the world.

[0115] Based on the data held by the "reference system" and the data held by the time-space database (*12) in Figure 6, a simulation can be provided of how development could proceed for a certain area that is the target of the advanced guide (note that this does not necessarily have to be the same area as the data stored in the "reference system"). This is referred to as a "planning proposal" in this specification. For example, by converting the layout of rivers and mountains in area A, the target of the advanced guide, into data and statistically comparing it with data for another area B held by the "reference system," it is possible to simulate how urban planning actually carried out in area B should be customized if it were to be realized in area A. This simulation can achieve a complex planning effect by moving around the site in all directions using this advanced guide navigation and adding evaluations.

[0116] (Examples of application to large, medium and semi-medium vehicles) The configurations shown in Figures 3A and 3B can also be applied to large vehicles such as buses, medium-sized vehicles such as microbuses, and medium-sized vehicles such as vans. Drivers of large vehicles such as buses are generally professionals, and even in the future, even with Level 3 autonomous driving, the driver's primary responsibility will be to monitor the vehicle's autonomous driving status, road conditions, and readiness status. Therefore, by using the system of the present invention, it is possible to efficiently prevent the driver from becoming distracted by managing and adjusting the overall status of the advanced navigation guidance service provided to passengers. In other words, although the situation is somewhat different from that in which a driver of a regular passenger vehicle maintains a readiness status while utilizing advanced navigation guidance during autonomous driving, it can be seen that the effects of the present invention can still be achieved.

[0117] For example, an integrated advanced guidance and navigation display (or a collection of multiple such displays) can be installed near the driver's seat of a bus and used to integrate and manage information. The management operations can be performed by the driver or remotely by another manager or navigator. Furthermore, the system of the present invention can be used to provide content to passengers. Individual advanced guidance and navigation displays and voice transmission devices can be installed near each passenger seat to output information personalized for each passenger. Such personalization can be achieved, for example, by analyzing the passenger's knowledge, interests, and preferences based on information input by the passenger, or by input from some database related to the passenger.

[0118] In addition, the ceiling, walls, and windows of large trains may be partially or entirely configured as large displays, allowing information to be provided to multiple passengers at once. This allows passengers to feel as if they are being shown information in a theater (also known as the "spatiotemporal theater effect").

[0119] In these large vehicles, each passenger seat may be a private (or semi-private) room. This allows passengers to gather together, helps protect their privacy, and helps prevent infection. Such private (or semi-private) rooms may be equipped with shields and air conditioning for disease prevention.

[0120] It will be understood that, according to the disclosure of this specification, various other operation routes and combinations of accompanying data are possible in addition to the examples described above.

[0121] In an embodiment of the present invention, a device (hardware) capable of implementing the system, a program, and a product (any medium, carrier wave, module, etc.) storing part or all of the program in a form executable by a user may also be provided. The system according to the embodiment of the present invention may be integrated or distributed. In the case of a distributed system, computational processing may be performed by communicating with a remote resource (such as a server) outside the system.

[0122] As described above, the present invention can utilize a variety of databases. A particularly advantageous feature of the present invention is its ability to utilize valuable databases that systematize information such as national land infrastructure map information, including topography, elevation, aerial photographs, and geological distribution. While such databases have traditionally been used on a desk, the present invention allows them to be used on-site, in a moving vehicle, or in a mobile office. The system of the present invention can provide experiences based on such databases, along with various representations of detected landscapes, along with spatial and temporal axes. Therefore, the present invention allows users to acquire a more complex knowledge system, which is believed to lead to new enjoyment, excitement, and creativity.

Claims

1. 1. A system for use in automated driving of a vehicle, comprising: a storage means configured to store map image data associated with map coordinates and associated data associated with the map coordinates and not related to a driving task; positioning means for determining the current vehicle position and status in said map coordinates; a first information presentation means for presenting information to a driver of the vehicle; a second information presentation means for presenting information to an occupant of the vehicle; a network means operatively connecting the first information presentation means and the second information presentation means; A means for measuring the reliability of the autonomous driving of the vehicle; an imaging means for photographing the scenery outside the vehicle; Including, the first information presentation means is configured to present to the driver, when it is determined that operation of the vehicle does not require the driver's operation based on the measured reliability, information in which the map image data related to the current position of the vehicle at the map coordinates is superimposed with the associated data related to the current position of the vehicle at the map coordinates; and the second information presentation means is configured to be operable in cooperation with the first information presentation means via the network means; and determining how one or more objects in a scene external to the vehicle correspond to the map coordinates based on the data obtained from the imaging means; A system characterized by:

2. The system according to claim 1 , configured to be able to associate a plurality of types of the accompanying data with the map coordinates and / or the object.

3. The system according to claim 2 , wherein the association of the plurality of types of accompanying data with the map coordinates and / or the object is performed by hierarchizing the data.

4. 4. The system according to claim 2 or 3, wherein the system is configured to determine which of a plurality of types of the associated data the information presented by the first information presentation means and / or the second information presentation means includes based on personalization for each of the driver and / or the passenger.

5. A system for use in automatic driving of a vehicle, comprising: a storage means configured to store map image data associated with map coordinates and associated data associated with the map coordinates and not related to a driving task; positioning means for determining the current vehicle position and status in said map coordinates; a first information presentation means for presenting information to a driver of the vehicle; a second information presentation means for presenting information to an occupant of the vehicle; a network means operatively connecting the first information presentation means and the second information presentation means; A means for measuring the reliability of the automatic driving of the vehicle; Including, the first information presentation means is configured to present to the driver, when it is determined that operation of the vehicle does not require the driver's operation based on the measured reliability, information in which the map image data related to the current position of the vehicle at the map coordinates is superimposed with the associated data related to the current position of the vehicle at the map coordinates; and the second information presentation means is configured to be operable in cooperation with the first information presentation means via the network means; and The system is configured to determine whether the driver can see the scenery associated with the accompanying data based on the map image data. A system characterized by:

6. The system of any one of claims 1 to 5, wherein the accompanying data includes one or more types selected from the group consisting of data on urban planning or industrial development of the area, natural topography and geological data, historical map image data, engineering technology data, data on historical figures or culture, literary and folktale data, tourist information data, data on inhabiting flora and fauna, and disaster prevention data, all of which are associated with the map coordinates.

7. The system according to any one of claims 1 to 6, configured to provide vehicle navigation information based on the associated data by the first information presentation means and / or the second information presentation means.

8. The system of claim 7 , wherein the navigation information is provided based on the accompanying data including natural terrain and geological data.

9. 9. The system of claim 8, wherein the navigation information is provided based on topographical, geological, or geographical information or aerial, planimetric, or 3D maps, or a combination thereof, of the terrain surrounding the vehicle.

10. The system according to any one of claims 1 to 9, wherein the network means has a function of communicating with the outside of the vehicle.

11. 11. The system according to claim 1, wherein the associated data includes audio, and the first information presentation means and / or the second information presentation means includes an audio transmission device.

12. 12. The system of claim 11, wherein the audio output device is a speaker mounted on the vehicle or a speaker carried by the driver.

13. The system according to any one of claims 1 to 12, further comprising means for measuring the driver's readiness state, wherein if the driver's readiness state is determined to be below a given threshold, the first information presentation means and / or the second information presentation means emphasize when providing information.

14. The system according to any one of claims 1 to 13, wherein the map image data is displayed as an overhead map, a planar map, or a three-dimensional map on the first information presentation means and / or the second information presentation means.

15. The system according to any one of claims 1 to 14, wherein the first information presentation means and / or the second information presentation means are configured as a part or the whole of the ceiling surface, wall surface, and window of the vehicle.

16. 5. The system according to claim 1, wherein the system is configured to determine whether the driver can see the scenery associated with the accompanying data based on the map image data.

17. 5. The system according to claim 1, wherein the imaging means is further configured to determine whether the driver can see the scenery associated with the associated data.

18. 18. The system according to claim 16, wherein the first information presentation means is configured to present information when it is determined that the driver can see the scenery associated with the accompanying data.

19. A vehicle comprising a system according to any one of claims 1 to 18.

Images