System

Abstract

A system is provided.SOLUTION: A system including means for a user to set a destination and a route, means for indicating an optimal route based on route information received from a server, means for acquiring and analyzing information on a current location and surroundings of the user in real time, means for assisting a field of view of the user using a camera and performing route guidance using arrows and text, and means for analyzing a voice command and updating an indication in accordance with a route change or a question.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a need for a safe and efficient way for cyclists to receive route guidance while traveling, without having to check a map or stop if they get lost. Receiving route guidance without obstructing the view is especially important on dark roads, in busy traffic areas, or when using a walking stick or carrying luggage. The present invention aims to provide a system that solves these problems and improves the safety and convenience of cyclists and pedestrians. [Means for solving the problem]

[0005] The present invention provides a system for providing route guidance by combining a plurality of means. The system includes the following components.

[0006] The system includes a means for users to set their destination and route, a means for indicating the optimal route based on route information received from a server, a means for acquiring and analyzing information about the user's current location and surroundings in real time, a means for assisting the user's field of vision using a camera and providing route guidance using arrows and text, and a means for analyzing voice commands and updating instructions in response to route changes or questions. This allows users to receive route guidance without obstructing their view, improving safety and efficiency during travel.

[0007] "User" refers to an individual or user who uses the system to receive directions.

[0008] "Destination" is location information indicating the place the user wants to reach.

[0009] "Route information" is data that indicates the route a user must take to reach a destination.

[0010] A "server" is a computer system that receives requests from users and performs route calculations and provides real-time data.

[0011] A "camera" is a photographic device used to obtain information about a user's current location and surroundings.

[0012] "Field of view" refers to the range of vision a user can see through the smart glasses.

[0013] An "arrow" is a visual symbol that indicates the direction of travel to the user.

[0014] "Text" is a string of characters used to provide directions or other information to the user.

[0015] A "voice command" is a voice instruction or question that the user issues through the microphone in the smart glasses.

[0016] "Analysis" is the means by which the system interprets voice commands and surrounding information and processes them appropriately.

[0017] "Route change" refers to an operation in which the user switches the current route to another route. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to exemplary embodiments.

[0040] System Overview

[0041] This system consists of a user, a device (a smartphone and smart glasses), and a server. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time.

[0042] Program processing

[0043] Server Processing

[0044] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0045] Terminal processing (smartphone)

[0046] The smartphone sends the destination information entered by the user to the server. Upon receiving route information from the server, the smartphone synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis to the server. Specifically, when a user says "I want to change the route" to the smart glasses, the voice is transferred to the smartphone, analyzed, and then sent to the server.

[0047] Terminal processing (smart glasses)

[0048] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[0049] Specific examples

[0050] 1. If you're heading to the cafe by bicycle

[0051] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bike. The smart glasses use a camera to determine their current location and show turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view.

[0052] 2. When walking to the station on a crowded street at night

[0053] The user inputs the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, they display warnings such as "Watch your left. A car is approaching."

[0054] As described above, the system of the present invention provides a multifunctional means for users to receive route guidance safely and efficiently. In particular, the unobstructed route guidance and voice command operation function allow users to enjoy greater safety and convenience while traveling.

[0055] The processing flow will be explained below.

[0056] Server Processing Steps

[0057] Step 1:

[0058] The server receives the destination information sent by the user via the smartphone.

[0059] Step 2:

[0060] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[0061] Step 3:

[0062] The server sends the calculated route information to the smartphone.

[0063] Step 4:

[0064] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[0065] Step 5:

[0066] The server resends the updated route information to the smartphone.

[0067] Smartphone processing steps

[0068] Step 1:

[0069] The user launches a dedicated app on their smartphone and inputs their destination.

[0070] Step 2:

[0071] The smartphone transmits the input destination information to the server.

[0072] Step 3:

[0073] The smartphone synchronizes the route information received from the server with the smart glasses.

[0074] Step 4:

[0075] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[0076] Step 5:

[0077] The smartphone transmits the analyzed voice data to the server as text.

[0078] Smart Glasses Processing Steps

[0079] Step 1:

[0080] The user puts on the smart glasses and starts moving.

[0081] Step 2:

[0082] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[0083] Step 3:

[0084] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[0085] Step 4:

[0086] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[0087] Step 5:

[0088] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[0089] User processing steps

[0090] Step 1:

[0091] The user inputs the destination into the smartphone and sets the route.

[0092] Step 2:

[0093] The user puts on the smart glasses and begins moving along the specified route.

[0094] Step 3:

[0095] Users can check their direction by following the arrows and text displayed through the smart glasses.

[0096] Step 4:

[0097] Users can use voice commands to change routes or ask questions as needed.

[0098] Step 5:

[0099] The user follows the new instructions from the smart glasses to reach their destination.

[0100] The above is the specific flow of each processing step of this system.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Conventional navigation systems do not adequately ensure safety and efficiency in the process of users reaching their destination, and often fail to provide appropriate guidance, especially on complex routes or in congested environments. They also lack the ability to provide route guidance that takes into account real-time changes in traffic conditions, weather, and other environmental factors, resulting in inconvenience for users. Furthermore, they lack an interface that utilizes voice commands, making hands-free operation difficult.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for using a camera to assist the user's vision and for providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing video data acquired by the camera to identify the user's current location and direction of travel, means for analyzing voice commands and transmitting them to the server, means for recalculating the route under specific conditions (e.g., traffic conditions or weather) and providing the latest route information to the terminal, and means for synchronizing data between terminals using wireless communication, thereby enabling a user to receive real-time route guidance to reach their destination safely and efficiently.

[0106] A "destination" is a location that a user ultimately wants to reach.

[0107] A "route" is information about a route to reach a destination.

[0108] A "server" is a computer device that processes and stores data on a network.

[0109] The term "terminal" refers to any device that is directly operated by a user, and in this invention includes smartphones, smart glasses, etc.

[0110] "Route information" refers to the optimal route calculated by the server to reach the destination.

[0111] "Current location" refers to the location where the user is currently located.

[0112] "Surrounding information" refers to environmental information around the user's current location, including traffic conditions and weather.

[0113] The term "photography device" refers to a device that captures images or videos, and includes a camera in this invention.

[0114] A "voice command" is an instruction that a user inputs by voice.

[0115] "Analysis" refers to the process of processing acquired data to extract useful information.

[0116] "Synchronization" refers to the act of matching information to unify data across multiple terminals.

[0117] MODE FOR CARRYING OUT THE INVENTION

[0118] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. The system is composed of a user, a terminal (a smartphone and smart glasses), and a server.

[0119] System configuration

[0120] The user inputs their destination using their smartphone, and the information is sent to the server. The server then calculates the optimal route based on the received destination information using external services such as Google Maps API. The calculated route information is sent to the smartphone, which then synchronizes it with the smart glasses.

[0121] Smart glasses use a camera to determine the user's current location and provide directions by displaying arrows and text in the user's field of view. Communication between the smart glasses and smartphones is via wireless communication such as Bluetooth or Wi-Fi. Smart glasses also accept voice commands and transmit the voice data to the smartphone. The smartphone analyzes the voice data and sends it to a server to provide new routes and additional information.

[0122] Specific examples

[0123] Example 1: Cycling to a cafe

[0124] The user enters the location of the cafe into their smartphone. The smartphone sends the information to a server, which then calls the Google Maps API to calculate the optimal cycling route. The route information calculated by the server is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts cycling. The smart glasses use a camera to determine the user's current location in real time and display arrows and text such as "Turn right in 100 meters."

[0125] Example 2: Walking to the station on a crowded street at night

[0126] The user inputs the "station location" into their smartphone. The smartphone sends this information to the server, which then calls the Google Maps API to calculate a safe walking route. The server sets the route, taking into account traffic conditions and nighttime safety. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins walking. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching."

[0127] Prompt Sentence Examples

[0128] Enter "I want to go to XX Cafe" into your smartphone as your destination and start navigation. Also, if you say "I want to change the route" using the voice command, the server will calculate a new route and new instructions will be displayed on the smart glasses.

[0129] This system allows users to enjoy greater safety and convenience while traveling. With many features, such as real-time traffic and weather updates and the ability to operate using voice commands, users can receive optimal guidance in a variety of environments and situations.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Processing step details

[0132] Step 1:

[0133] The user starts up the smartphone and opens the navigation app. The user inputs their destination using the keyboard or touch screen. For example, they input the destination name, such as "cafe." The input data is converted to JSON format within the smartphone. At this point, the JSON format destination information is generated as input data (destination information).

[0134] Step 2:

[0135] The smartphone sends the destination information entered by the user to the server. Specifically, it uses an HTTP POST request to send the destination information to the server. The smartphone attaches the data entered by the user to the HTTP request and sends it to the server. The output is that the server receives the destination information.

[0136] Step 3:

[0137] The server calculates the optimal route based on the received destination information. This is achieved by calling a third-party API (e.g., Google Maps API). The server sends the destination information in an appropriate format to the API and obtains the optimal route information. The input is the received destination information, and the output is the calculated route information (in JSON format).

[0138] Step 4:

[0139] The server sends the calculated route information in JSON format to the smartphone. The server creates an HTTP response, attaches the route information, and sends it to the smartphone. The smartphone receives the HTTP response and obtains the route information. The output is the JSON-formatted route information received by the smartphone.

[0140] Step 5:

[0141] The smartphone synchronizes the received route information with the smart glasses. The smartphone transmits the route information to the smart glasses using wireless communication such as Bluetooth or Wi-Fi. The input is the route information received from the server, and the output is the route information received by the smart glasses.

[0142] Step 6:

[0143] Smart glasses use a built-in camera to determine the user's current location and direction of travel. Specifically, they use computer vision technology to analyze the video data captured by the camera and determine the user's current location and direction of travel. The input is the video data captured by the camera, and the output is the current location information and direction of travel information as the analysis results.

[0144] Step 7:

[0145] The smart glasses provide route guidance using arrows and text overlays based on the analyzed current location and direction of travel. Specifically, they display route guidance such as "Turn right in 100 meters" in the user's field of view. The input is the current location and direction of travel as analyzed results, and the output is the arrows and text displayed in the user's field of view.

[0146] Step 8:

[0147] The user inputs a voice command, for example, "I want to change my route" or "Where is the next turn?" The input is the user's voice command, and the output is the voice data captured by the smart glasses' internal microphone.

[0148] Step 9:

[0149] The smart glasses transmit the captured audio data to a smartphone using Bluetooth communication. The input is the audio data captured by the smart glasses, and the output is the audio data received by the smartphone.

[0150] Step 10:

[0151] The smartphone analyzes the received voice data and sends it to the server. It converts the voice data into text using voice recognition software and sends the text information to the server as an HTTP request. The input is the voice data received from the smart glasses, and the output is the analyzed text information and a request to send it to the server.

[0152] Step 11:

[0153] The server calculates a new route based on the received text information and sends that information to the smartphone. It then calls the Google Maps API again to obtain the new route information and sends it back to the smartphone. The input is the parsed text information, and the output is the recalculated route information.

[0154] Step 12:

[0155] The smartphone synchronizes the new route information received from the server with the smart glasses, and the smart glasses display new instructions to the user based on that information. The input is the recalculated route information received from the server, and the output is the new route guidance information displayed on the smart glasses.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Conventional navigation systems have the problem of making it difficult for users to reach their destination safely and efficiently while traveling. Self-driving vehicles, in particular, are required to provide appropriate route guidance while grasping the surrounding situation in real time. Furthermore, systems are susceptible to external factors such as traffic conditions and weather changes, and may not update routes in a timely manner. To solve these problems, a system using more advanced technology is required.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's vision using a camera and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle, and means for updating the route based on real-time updates of traffic conditions and weather information, thereby enabling a user to reach their destination safely and efficiently in an autonomous vehicle.

[0161] The "means for the user to set a destination and a route" is an interface device for inputting destination information and requesting the server for the optimal route.

[0162] The "means for indicating the optimum route based on the route information received from the server" is a device that analyzes the route information sent from the server and provides guidance to the user on the optimum direction of travel.

[0163] "Means for acquiring and analyzing information about the user's current location and surroundings in real time" refers to a device that uses cameras and sensors to collect and analyze information about the user's location and surrounding environment.

[0164] "Means for using a camera to assist the user's field of vision and providing route guidance using arrows and text" refers to a device that guides the user by displaying arrows and text based on images captured by a camera.

[0165] The "means for analyzing voice commands and updating instructions in response to route changes or questions" refers to a device that recognizes a user's voice input and updates route information and navigation instructions based on the analysis results.

[0166] "Means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle" refers to a device that monitors the surrounding situation using a camera inside the vehicle and provides visual information to the user through a display.

[0167] The "means for updating the route based on real-time updates of traffic conditions and weather information" refers to a device that analyzes traffic and weather information obtained from outside and updates the route information accordingly.

[0168] System Overview

[0169] As an embodiment of the present invention, a navigation system for an autonomous vehicle is taken as an example. This system is composed of a user, a terminal (an in-vehicle display and a smartphone), and a server. The user can set a destination and receive route guidance. The terminal receives destination and route information from the server and updates it in real time. The terminal also uses a camera to obtain information on the user's current location and surrounding conditions, and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the terminal in real time.

[0170] Program processing

[0171] 1. Server Processing

[0172] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into the in-car display or smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. The calculated route information is then sent to the in-car display and smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0173] 2. Terminal processing (in-vehicle displays and smartphones)

[0174] The in-car display and smartphone send the destination information entered by the user to the server. When route information is received from the server, it is displayed on the in-car display and smartphone. The device also uses a camera to acquire the current location in real time and analyzes the direction of travel. The user receives visual instructions and warnings through the device. For example, instructions such as "Turn left 100 meters ahead" are displayed on the in-car display.

[0175] 3. Analyzing voice commands and updating instructions

[0176] The vehicle is equipped with a voice recognition microphone, and when the user enters a voice command such as "I want to change my route," the voice is sent as a request to the server. The server analyzes the voice command, generates a new route and instructions, and displays them on the in-vehicle display. If the user asks a question, the server also provides answers both audibly and visually.

[0177] Hardware and software used

[0178] Hardware

[0179] Cameras, microphones, in-vehicle displays, smartphones

[0180] software

[0181] Server: Route calculation and data update (Google Maps API, TensorFlow)

[0182] Smartphone app: Set destination, receive route (Android / iOS)

[0183] In-car display: Real-time information display (Android Auto, Apple CarPlay)

[0184] Speech analysis: Analysis and conversion of voice commands (Google Cloud Speech-to-Text, Amazon Transcribe)

[0185] Specific examples

[0186] Traveling within cities in autonomous vehicles

[0187] When a user sets a "station" as their destination on their smartphone, the in-car display shows the "optimal route to the station." If traffic conditions change along the way, the route is updated in real time, and instructions such as "Change the next left turn to a right turn" are displayed.

[0188] Example prompts for generative AI models

[0189] "How does the navigation system react when a user sets a train station as a destination on their smartphone? Also, please explain in detail how the system updates the route if traffic information changes along the way."

[0190] This system enables autonomous vehicles to provide safer and more efficient route guidance to users.

[0191] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0192] Step 1:

[0193] The user inputs the destination into a smartphone or on-board display. The user sets the destination using a touchscreen or voice input. This input is recorded in the device as the destination's coordinates and name (e.g., "station").

[0194] Step 2:

[0195] The device sends destination information to the server. The smartphone or in-car display sends the destination information entered by the user to the server as an HTTP request. The server receives the input and analyzes the destination coordinates.

[0196] Step 3:

[0197] The server calculates the optimal route and generates route information. The server calls the Google Maps API and calculates the optimal route based on the destination coordinates. In doing so, it also takes into account traffic conditions and weather data when selecting the route. Route information is generated as detailed information for each step (e.g., distance to the next right turn).

[0198] Step 4:

[0199] The server sends the generated route information to the device. The route information is sent as an HTTP response to the in-vehicle display and smartphone. The device analyzes the received information and prepares to display it visually to the user.

[0200] Step 5:

[0201] The device uses a camera to obtain the user's current location and direction of travel in real time. The onboard camera captures video data, and the device analyzes the video in real time. Image recognition technology (e.g., OpenCV) is used to identify the location and direction of travel.

[0202] Step 6:

[0203] The device displays visual instructions, such as arrows and text overlaid on the camera image, in real time in the user's field of view.

[0204] Step 7:

[0205] The server periodically updates traffic and weather information and recalculates route information as needed. The server calls external APIs (e.g., traffic information API, weather API) at regular intervals to obtain the latest information. When new information is obtained, it recalculates route information as needed and sends it to the device.

[0206] Step 8:

[0207] The user inputs a voice command. While driving, the user inputs a voice command such as "I want to change the route." This command is transmitted to the terminal through a microphone in the vehicle.

[0208] Step 9:

[0209] The device analyzes the voice command and sends a request to the server. The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then analyzed. After that, a new request is sent to the server based on the analysis results.

[0210] Step 10:

[0211] The server generates new route information and sends it to the device. The server receives the new route request and recalculates the route. The generated route information is sent to the device in real time and displayed to the user.

[0212] This program's processing allows users to reach their destination safely and efficiently based on the latest route information.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[0215] System Overview

[0216] This system consists of a user, a device (smartphone and smart glasses), a server, and an emotion engine. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time. The emotion engine has the ability to analyze the user's voice and facial expressions and recognize the user's emotional state.

[0217] Program processing

[0218] Server Processing

[0219] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0220] Terminal processing (smartphone)

[0221] The smartphone sends destination information entered by the user to the server. Upon receiving route information from the server, it synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis results to the server. It also receives emotional information from the emotion engine and requests the server to provide appropriate route guidance based on the user's emotional state.

[0222] Terminal processing (smart glasses)

[0223] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[0224] Emotion engine processing

[0225] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, the emotion engine acquires voice tone and facial expression data from the smart glasses' camera and microphone, and analyzes this data to determine whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotion information is sent to the smartphone and then forwarded to the server. Based on this information, the server provides route guidance and warnings appropriate to the user's emotional state.

[0226] Specific examples

[0227] 1. If you're heading to the cafe by bicycle

[0228] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and indicate turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view. At the same time, the emotion engine analyzes the user's tone of voice and, if it determines that the user is feeling anxious, an additional warning is given.

[0229] 2. When walking to the station on a crowded street at night

[0230] The user enters the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, warnings such as "Watch your left. A car is approaching" are displayed. If the emotion engine recognizes that the user's stress level is high, a voice prompt to calm them down is played.

[0231] As described above, the system of the present invention provides a multifunctional means for users to receive safe and efficient route guidance. In addition to unobstructed route guidance and voice command operation, the system's emotion engine allows for flexible guidance that responds to the user's emotions. This allows users to enjoy greater safety and convenience while traveling.

[0232] The processing flow will be explained below.

[0233] Server Processing Steps

[0234] Step 1:

[0235] The server receives the destination information sent by the user via the smartphone.

[0236] Step 2:

[0237] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[0238] Step 3:

[0239] The server sends the calculated route information to the smartphone.

[0240] Step 4:

[0241] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[0242] Step 5:

[0243] The server receives the user's emotion information transferred from the emotion engine and adjusts the route instructions as necessary.

[0244] Step 6:

[0245] The server resends the updated route information to the smartphone.

[0246] Smartphone processing steps

[0247] Step 1:

[0248] The user launches a dedicated app on their smartphone and inputs their destination.

[0249] Step 2:

[0250] The smartphone transmits the input destination information to the server.

[0251] Step 3:

[0252] The smartphone synchronizes the route information received from the server with the smart glasses.

[0253] Step 4:

[0254] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[0255] Step 5:

[0256] The smartphone transmits the analyzed voice data to the server as text.

[0257] Step 6:

[0258] The smartphone transmits the user's facial and voice data acquired from the smart glasses to the emotion engine.

[0259] Step 7:

[0260] The user's emotional state information received from the emotion engine is transmitted to a server.

[0261] Smart Glasses Processing Steps

[0262] Step 1:

[0263] The user puts on the smart glasses and starts moving.

[0264] Step 2:

[0265] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[0266] Step 3:

[0267] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[0268] Step 4:

[0269] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[0270] Step 5:

[0271] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[0272] Step 6:

[0273] The smart glasses' camera and microphone analyze the user's facial expressions and tone of voice in real time and send the data to the emotion engine.

[0274] Emotion Engine Processing Steps

[0275] Step 1:

[0276] The emotion engine analyzes the user's voice tone and facial expressions received from the smart glasses.

[0277] Step 2:

[0278] The emotion engine recognizes the user's emotional state based on the analysis results.

[0279] Step 3:

[0280] The emotion engine sends the recognized emotion information to the smartphone.

[0281] User processing steps

[0282] Step 1:

[0283] The user inputs the destination into the smartphone and sets the route.

[0284] Step 2:

[0285] The user puts on the smart glasses and begins moving along the specified route.

[0286] Step 3:

[0287] Users can check their direction by following the arrows and text displayed through the smart glasses.

[0288] Step 4:

[0289] Users can use voice commands to change routes or ask questions as needed.

[0290] Step 5:

[0291] The user follows the new instructions from the smart glasses to reach their destination.

[0292] The above is the specific flow of each processing step of this system.

[0293] Example 2

[0294] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0295] Conventional navigation systems can sometimes make it difficult for users to find the optimal route to their destination, and they also lack real-time route adjustments and guidance that reflects the user's emotional state. In particular, they are unable to provide appropriate guidance to users who are feeling anxious or stressed, making it difficult to ensure the safety and convenience of users.

[0296] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to set a destination and a route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's field of vision using an imaging device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing the user's emotional state, and means for flexibly adjusting the content of guidance based on the user's emotional state. This allows the user to receive optimal route guidance in real time and enables flexible guidance according to the user's emotional state, greatly improving safety and convenience.

[0297] "User" refers to an individual who uses the system to travel to a destination.

[0298] A "destination" refers to a specific location that a user wishes to reach.

[0299] A "route" refers to the path a user takes to reach a destination.

[0300] The term "server" refers to a computer system that receives information from a user, processes the data, generates route information, and transmits it to a terminal.

[0301] "Terminal" refers to a device used by a user, such as a smartphone or smart glasses.

[0302] "Route information" refers to data regarding the optimal route a user needs to take to reach a destination.

[0303] "Real-time" refers to instantly recognizing and reflecting the current situation or state.

[0304] The term "photography device" refers to a device such as a camera for acquiring information about the user's field of view and current location.

[0305] An "arrow" refers to a graphic that visually indicates the direction of travel to the user.

[0306] "Text" refers to a string of characters that provides information to the user.

[0307] "Voice command" refers to a means by which a user issues instructions to a system by voice.

[0308] "Emotional state" refers to the user's psychological state, and includes emotions such as anxiety, impatience, and joy.

[0309] "Analysis" refers to the detailed analysis of acquired data to clarify its meaning.

[0310] "Guidance content" refers to information provided to a user, including route information and other instructions.

[0311] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. This system is composed of a user, a terminal (a smartphone and smart glasses), a server, and an emotion engine. Below, an embodiment of the present invention will be described in detail.

[0312] System Overview

[0313] The user inputs their destination using a smartphone application. This information is sent from the smartphone to a server. The server uses the Google Maps API to calculate the optimal route and sends that information to the smartphone. The smartphone then synchronizes the received route information with the smart glasses. The smart glasses use a built-in camera to determine the user's current location and direction in real time, and provide directions by displaying arrows and text in the user's field of view.

[0314] Server Processing

[0315] The server calculates the optimal route based on the destination information sent by the user. The server calls the Google Maps API to calculate the optimal route based on the destination and current location, taking into account real-time information such as traffic conditions and weather data. The calculated route information is also sent to the smartphone using encryption technology.

[0316] Terminal processing (smartphone)

[0317] The smartphone sends destination information entered by the user to the server. It uses Bluetooth and Wi-Fi to synchronize route information received from the server with the smart glasses. When a voice command is sent from the smart glasses, the smartphone also sends that information to the server. It also receives the user's emotional state from the emotion engine and requests appropriate route guidance from the server.

[0318] Terminal processing (smart glasses)

[0319] Smart glasses use a camera to determine the user's current location and direction in real time. They analyze the captured video and GPS data to determine the user's current location and direction. As a result, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also send voice commands to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is sent from the smart glasses to a server via the smartphone, and new instructions are generated and displayed on the smart glasses.

[0320] Emotion engine processing

[0321] The emotion engine analyzes the user's emotional state based on data obtained from the smart glasses' camera and microphone. Voice tone and facial expression data are used for the analysis. It determines whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotional information is sent to the smartphone and then forwarded to the server. The server uses this information to provide flexible route guidance and warnings according to the user's emotional state.

[0322] Specific example explanation

[0323] 1. If you're heading to the cafe by bicycle

[0324] The user enters the location of the cafe into their smartphone. The smartphone sends this information to the server, which uses the Google Maps API to calculate the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and provide directions using text and arrows, such as "Turn right in 100 meters." At the same time, if the emotion engine detects the user's anxiety, additional warnings are given.

[0325] Prompt Sentence Examples

[0326] Explain the steps and specific actions to take to get from your home to a nearby cafe using a bicycle, smart glasses, and a smartphone.

[0327] 2. When walking to the station on a crowded street at night

[0328] The user enters the "station location" into their smartphone. The smartphone sends this information to the server. The server uses the Google Maps API to calculate the optimal route, taking into account traffic conditions and safety at night. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins moving. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching." If the emotion engine detects that the user is stressed, a voice prompt to calm them down is played.

[0329] Prompt Sentence Examples

[0330] Explain the specific steps to safely receive directions using smart glasses and a smartphone when walking down a crowded street at night to the station.

[0331] Through these examples, the system of the present invention provides a versatile means for users to receive route guidance safely and efficiently. By utilizing unobstructed route guidance and voice commands, users can enjoy safety and convenience while traveling. Furthermore, the emotion engine enables flexible guidance based on the user's emotions.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1:

[0334] The user operates the smartphone to input a destination. The user opens the smartphone's application and inputs the name and address of the destination. For example, the user inputs "the location of the cafe." This input information is saved on the smartphone as data about the location the user wants to go to.

[0335] Input: Destination information (cafe name and address)

[0336] Output: Destination information saved on your smartphone

[0337] Specific operation:

[0338] The user launches an application on the smartphone.

[0339] The user enters the "location of the cafe" in the text box.

[0340] Step 2:

[0341] The device (smartphone) sends destination information to the server. The smartphone sends destination information entered by the user to the server. Communication is usually via Wi-Fi or mobile data networks.

[0342] Input: Destination information saved on your smartphone

[0343] Output: Destination information sent to the server

[0344] Specific operation:

[0345] The smartphone sends destination information in packet format to the server.

[0346] Encryption technology is used to ensure the stability of data communication.

[0347] Step 3:

[0348] The server calculates the optimal route. Based on the destination information received, the server uses external services such as Google Maps API to calculate the optimal route, while taking into account real-time information such as traffic conditions and weather data.

[0349] Input: Destination information, current traffic conditions, weather data

[0350] Output: Optimal route information

[0351] Specific operation:

[0352] The server calls the Google Maps API and starts route calculation.

[0353] Route information is updated in real time based on traffic and weather data.

[0354] Step 4:

[0355] The server sends route information to the device (smartphone). The calculated route information is sent to the smartphone in encrypted packet format. The smartphone receives and stores this information.

[0356] Input: Optimal route information

[0357] Output: Route information sent to your smartphone

[0358] Specific operation:

[0359] The server encrypts the route information and sends it to your smartphone.

[0360] The smartphone decodes the received route information and stores it in its internal storage.

[0361] Step 5:

[0362] The device (smartphone) synchronizes the route information with the smart glasses. The smartphone then synchronizes the received route information with the smart glasses via Bluetooth or Wi-Fi.

[0363] Input: Route information saved on your smartphone

[0364] Output: Route information synchronized to smart glasses

[0365] Specific operation:

[0366] Your smartphone connects to the smart glasses using Bluetooth or Wi-Fi.

[0367] Transfer the route information to your smart glasses and check the synchronization.

[0368] Step 6:

[0369] Smart glasses acquire location information. Using a built-in camera and GPS, smart glasses determine the user's location in real time and determine the direction of travel.

[0370] Input: Camera image, GPS data

[0371] Output: User's current location

[0372] Specific operation:

[0373] The smart glasses use a camera to capture images of the surrounding area.

[0374] Use the built-in GPS sensor to determine your current location.

[0375] Step 7:

[0376] The smart glasses provide route guidance. Based on the acquired current location information, the smart glasses display arrows and text in the user's field of view to provide route guidance to the user. For example, specific instructions such as "Turn right in 100 meters" are displayed.

[0377] Input: User's current location information, route information

[0378] Output: Directions displayed in the field of view

[0379] Specific operation:

[0380] The smart glasses generate arrows and text based on route information.

[0381] Instructions such as "Turn right in 100 meters" are displayed in the user's field of vision.

[0382] Step 8:

[0383] The emotion engine analyzes the user's emotional state. The emotion engine analyzes voice tone and facial expressions based on data obtained from the smart glasses' camera and microphone to determine the user's emotional state.

[0384] Input: Voice data, facial expression data

[0385] Output: User's emotional state information

[0386] Specific operation:

[0387] The emotion engine analyzes the tone of the voice to determine whether the user is feeling anxious.

[0388] Identify emotional states based on facial expressions captured by a camera.

[0389] Step 9:

[0390] Providing guidance based on emotional information. The server generates appropriate guidance content based on the user's emotional information obtained from the emotion engine and sends it to the smart glasses. This provides flexible guidance according to the user's emotional state.

[0391] Input: User's emotional state information

[0392] Output: Guidance information according to emotions

[0393] Specific operation:

[0394] The server generates messages based on the emotional information to provide special attention or reassurance.

[0395] The smart glasses will display a message such as "You are almost at your destination. Proceed safely."

[0396] As described above, through these processing steps, the system provides the user with safe and efficient route guidance, and is also able to respond flexibly to the user's emotional state.

[0397] (Application example 2)

[0398] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0399] In self-driving vehicles, conventional navigation systems only provide route guidance, making it difficult to provide instructions that fully consider the emotional state and safety of passengers while driving. There is also a need for a system that can reduce the mental burden on drivers and enable them to reach their destination more safely. It is necessary to sense the anxiety and stress felt by users while driving in real time and respond appropriately.

[0400] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0401] In this invention, the server includes means for allowing the user to set a destination and route, means for indicating the optimal route based on route information received from the server, means for acquiring and analyzing the user's current location and scenery in real time, means for assisting the user's vision using a visual device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, and means for adjusting operation or issuing additional warnings based on the user's emotional state using emotion analysis means, thereby enabling safer and more secure automated driving that is responsive to the driver's emotional state.

[0402] A "user" is a vehicle occupant who uses the system to set a destination and route and receives navigation.

[0403] A "destination" is the final location or place that a user wishes to reach.

[0404] A "route" is the path or direction of travel to reach a destination.

[0405] A "means" is a method, device, or system for achieving a specific purpose.

[0406] A "server" is a central computer system that performs route calculations and real-time data processing.

[0407] "Real time" refers to the time frame for immediate processing and response to ongoing events.

[0408] "Analysis" is the process of examining and processing information or data in detail to extract meaning.

[0409] A "visual device" is a device that supports a user's vision and presents information in real time.

[0410] An "arrow" is an indicator displayed on a visual device to indicate the direction in which the user should move.

[0411] "Text" means written information displayed on a visual device to provide guidance or instructions to a user.

[0412] A "voice command" is an instruction or question given by a user through speech.

[0413] "Emotion analysis" is the process of recognizing and analyzing a user's emotional state based on their voice and facial expression data.

[0414] "Operation adjustment" refers to changing or optimizing the autonomous driving speed, route, settings, etc. based on the user's emotional state.

[0415] "Attention" refers to a visual or audio warning or instruction to alert the user.

[0416] The above are definitions of important words included in the claims.

[0417] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0418] System Overview

[0419] This system consists of a user, a terminal (the central control device of the autonomous vehicle), a server, and an emotion analysis means. The user sets a destination and route and gets into the autonomous vehicle. The terminal receives destination and route information from the server and updates it in real time. The terminal uses a camera and microphone to obtain the user's current location information and emotional state, and provides route guidance through a visual device. The emotion analysis means also has the function of analyzing the user's voice and facial expressions to recognize the user's emotional state.

[0420] Hardware and Software

[0421] Hardware

[0422] Autonomous vehicle: central control unit, camera, microphone, GPS

[0423] Server: Responsible for route calculation and real-time data acquisition

[0424] software

[0425] Google Maps API: Used for route calculation

[0426] Emotion analysis tools: AI models that analyze voice tone and facial expressions (e.g., developed using OpenCV and TensorFlow)

[0427] Real-time data processing: APIs used to obtain traffic and weather data

[0428] Program processing

[0429] Server Processing

[0430] The server calculates the optimal route based on the destination information set by the user. It receives destination information sent from the central control unit of the autonomous vehicle and calculates the optimal route using external services such as Google Maps API. It sends the calculated route information to the autonomous vehicle and updates the route based on real-time information. It also generates emotion-responsive driving instructions based on emotion data received from the emotion analysis means.

[0431] Terminal processing (central control unit of autonomous vehicles)

[0432] The central control unit of the autonomous vehicle controls autonomous driving based on route information received from the server. It also passes data acquired from the camera and microphone to an emotion analysis means to analyze the user's emotional state in real time. Based on the emotional information acquired from the emotion analysis means, it adjusts driving and issues warnings to improve safety. For example, if the user feels anxious, it may adjust the driving speed or play relaxing music.

[0433] Processing of sentiment analysis tools

[0434] The emotion analysis means analyzes the user's voice and facial expression data to recognize their emotional state. Specifically, it uses data acquired from the camera and microphone to determine whether the user is feeling anxiety, impatience, joy, or other emotions. This information is sent to the central control unit and server in real time and used to adjust operation and provide warnings.

[0435] Specific examples

[0436] 1. Dealing with stressed drivers

[0437] If the autonomous vehicle determines through emotion analysis that the driver is feeling stressed, the system will play relaxing music and suggest the best place to rest, for example, "There is a rest area nearby, so we recommend you take a 10-minute break."

[0438] 2. Examples of prompts

[0439] "Design a system that provides route guidance and warnings to help drivers relax. The server calculates the optimal route taking into account traffic conditions and weather, and the vehicle processes emotion data in real time and responds appropriately based on the emotion analysis results while driving."

[0440] The above is a specific embodiment for carrying out the invention. This system realizes safe and secure automated driving according to the emotional state of the user.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] The user sets the destination.

[0444] Input: The user inputs a destination into the central control unit of the autonomous vehicle.

[0445] Data processing and calculation: The central control unit receives the input destination information and compares it with GPS information.

[0446] Output: Sends the set destination information to the server.

[0447] Step 2:

[0448] The server calculates the optimal route.

[0449] Input: Destination information sent from the autonomous vehicle.

[0450] Data processing and calculation: Using the Google Maps API, the optimal route from the current location to the destination is calculated, taking into account traffic conditions and weather data, and optimization is performed in real time.

[0451] Output: Send the calculated optimal route information to the central control unit of the autonomous vehicle.

[0452] Step 3:

[0453] The self-driving vehicle will begin operating based on the route information.

[0454] Input: Optimal route information received from the server.

[0455] Data processing and calculation: The central control unit controls the operation based on the route information.

[0456] Output: Actual driving control of the vehicle.

[0457] Step 4:

[0458] The camera and microphone capture information about the surroundings and the user's emotions.

[0459] Input: Video data captured by the camera of the autonomous vehicle, audio data recorded by the microphone.

[0460] Data processing and calculation: The emotion analysis means analyzes the video data and audio data in real time to extract the user's emotional state.

[0461] Output: Sending the user's emotional information to the central control unit and server.

[0462] Step 5:

[0463] The server adjusts operations and issues warnings based on emotional information.

[0464] Input: User emotion information received from emotion analysis means.

[0465] Data processing and calculation: The server analyzes the emotional information and generates instructions and warning messages to adjust operations.

[0466] Output: Sends generated instructions and reminder messages to the autonomous vehicle.

[0467] Step 6:

[0468] The autonomous vehicle will carry out instructions and give warnings.

[0469] Input: Instructions and reminder messages sent by the server.

[0470] Data processing and calculation: The central control unit adjusts operation according to instructions, plays relaxing music, suggests resting spots, etc.

[0471] Output: Visual and audio guidance to the user and operational coordination.

[0472] The above is the processing flow of the program for the system for carrying out the invention. By performing appropriate processing and calculations based on the data input at each step and generating output, safe and efficient autonomous driving that responds to the user's emotional state becomes possible.

[0473] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0474] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0475] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0476] [Second embodiment]

[0477] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0478] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0479] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0480] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0481] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0482] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0483] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0484] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0485] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0486] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0487] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0488] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0489] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to exemplary embodiments.

[0490] System Overview

[0491] This system consists of a user, a device (a smartphone and smart glasses), and a server. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time.

[0492] Program processing

[0493] Server Processing

[0494] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0495] Terminal processing (smartphone)

[0496] The smartphone sends the destination information entered by the user to the server. Upon receiving route information from the server, the smartphone synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis to the server. Specifically, when a user says "I want to change the route" to the smart glasses, the voice is transferred to the smartphone, analyzed, and then sent to the server.

[0497] Terminal processing (smart glasses)

[0498] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[0499] Specific examples

[0500] 1. If you're heading to the cafe by bicycle

[0501] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bike. The smart glasses use a camera to determine their current location and show turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view.

[0502] 2. When walking to the station on a crowded street at night

[0503] The user inputs the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, they display warnings such as "Watch your left. A car is approaching."

[0504] As described above, the system of the present invention provides a multifunctional means for users to receive route guidance safely and efficiently. In particular, the unobstructed route guidance and voice command operation function allow users to enjoy greater safety and convenience while traveling.

[0505] The processing flow will be explained below.

[0506] Server Processing Steps

[0507] Step 1:

[0508] The server receives the destination information sent by the user via the smartphone.

[0509] Step 2:

[0510] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[0511] Step 3:

[0512] The server sends the calculated route information to the smartphone.

[0513] Step 4:

[0514] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[0515] Step 5:

[0516] The server resends the updated route information to the smartphone.

[0517] Smartphone processing steps

[0518] Step 1:

[0519] The user launches a dedicated app on their smartphone and inputs their destination.

[0520] Step 2:

[0521] The smartphone transmits the input destination information to the server.

[0522] Step 3:

[0523] The smartphone synchronizes the route information received from the server with the smart glasses.

[0524] Step 4:

[0525] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[0526] Step 5:

[0527] The smartphone transmits the analyzed voice data to the server as text.

[0528] Smart Glasses Processing Steps

[0529] Step 1:

[0530] The user puts on the smart glasses and starts moving.

[0531] Step 2:

[0532] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[0533] Step 3:

[0534] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[0535] Step 4:

[0536] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[0537] Step 5:

[0538] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[0539] User processing steps

[0540] Step 1:

[0541] The user inputs the destination into the smartphone and sets the route.

[0542] Step 2:

[0543] The user puts on the smart glasses and begins moving along the specified route.

[0544] Step 3:

[0545] Users can check their direction by following the arrows and text displayed through the smart glasses.

[0546] Step 4:

[0547] Users can use voice commands to change routes or ask questions as needed.

[0548] Step 5:

[0549] The user follows the new instructions from the smart glasses to reach their destination.

[0550] The above is the specific flow of each processing step of this system.

[0551] Example 1

[0552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Conventional navigation systems do not adequately ensure safety and efficiency in the process of users reaching their destination, and often fail to provide appropriate guidance, especially on complex routes or in congested environments. They also lack the ability to provide route guidance that takes into account real-time changes in traffic conditions, weather, and other environmental factors, resulting in inconvenience for users. Furthermore, they lack an interface that utilizes voice commands, making hands-free operation difficult.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0555] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for using a camera to assist the user's vision and for providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing video data acquired by the camera to identify the user's current location and direction of travel, means for analyzing voice commands and transmitting them to the server, means for recalculating the route under specific conditions (e.g., traffic conditions or weather) and providing the latest route information to the terminal, and means for synchronizing data between terminals using wireless communication, thereby enabling a user to receive real-time route guidance to reach their destination safely and efficiently.

[0556] A "destination" is a location that a user ultimately wants to reach.

[0557] A "route" is information about a route to reach a destination.

[0558] A "server" is a computer device that processes and stores data on a network.

[0559] The term "terminal" refers to any device that is directly operated by a user, and in this invention includes smartphones, smart glasses, etc.

[0560] "Route information" refers to the optimal route calculated by the server to reach the destination.

[0561] "Current location" refers to the location where the user is currently located.

[0562] "Surrounding information" refers to environmental information around the user's current location, including traffic conditions and weather.

[0563] The term "photography device" refers to a device that captures images or videos, and includes a camera in this invention.

[0564] A "voice command" is an instruction that a user inputs by voice.

[0565] "Analysis" refers to the process of processing acquired data to extract useful information.

[0566] "Synchronization" refers to the act of matching information to unify data across multiple terminals.

[0567] MODE FOR CARRYING OUT THE INVENTION

[0568] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. The system is composed of a user, a terminal (a smartphone and smart glasses), and a server.

[0569] System configuration

[0570] The user inputs their destination using their smartphone, and the information is sent to the server. The server then calculates the optimal route based on the received destination information using external services such as Google Maps API. The calculated route information is sent to the smartphone, which then synchronizes it with the smart glasses.

[0571] Smart glasses use a camera to determine the user's current location and provide directions by displaying arrows and text in the user's field of view. Communication between the smart glasses and smartphones is via wireless communication such as Bluetooth or Wi-Fi. Smart glasses also accept voice commands and transmit the voice data to the smartphone. The smartphone analyzes the voice data and sends it to a server to provide new routes and additional information.

[0572] Specific examples

[0573] Example 1: Cycling to a cafe

[0574] The user enters the location of the cafe into their smartphone. The smartphone sends the information to a server, which then calls the Google Maps API to calculate the optimal cycling route. The route information calculated by the server is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts cycling. The smart glasses use a camera to determine the user's current location in real time and display arrows and text such as "Turn right in 100 meters."

[0575] Example 2: Walking to the station on a crowded street at night

[0576] The user inputs the "station location" into their smartphone. The smartphone sends this information to the server, which then calls the Google Maps API to calculate a safe walking route. The server sets the route, taking into account traffic conditions and nighttime safety. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins walking. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching."

[0577] Prompt Sentence Examples

[0578] Enter "I want to go to XX Cafe" into your smartphone as your destination and start navigation. Also, if you say "I want to change the route" using the voice command, the server will calculate a new route and new instructions will be displayed on the smart glasses.

[0579] This system allows users to enjoy greater safety and convenience while traveling. With many features, such as real-time traffic and weather updates and the ability to operate using voice commands, users can receive optimal guidance in a variety of environments and situations.

[0580] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0581] Processing step details

[0582] Step 1:

[0583] The user starts up the smartphone and opens the navigation app. The user inputs their destination using the keyboard or touch screen. For example, they input the destination name, such as "cafe." The input data is converted to JSON format within the smartphone. At this point, the JSON format destination information is generated as input data (destination information).

[0584] Step 2:

[0585] The smartphone sends the destination information entered by the user to the server. Specifically, it uses an HTTP POST request to send the destination information to the server. The smartphone attaches the data entered by the user to the HTTP request and sends it to the server. The output is that the server receives the destination information.

[0586] Step 3:

[0587] The server calculates the optimal route based on the received destination information. This is achieved by calling a third-party API (e.g., Google Maps API). The server sends the destination information in an appropriate format to the API and obtains the optimal route information. The input is the received destination information, and the output is the calculated route information (in JSON format).

[0588] Step 4:

[0589] The server sends the calculated route information in JSON format to the smartphone. The server creates an HTTP response, attaches the route information, and sends it to the smartphone. The smartphone receives the HTTP response and obtains the route information. The output is the JSON-formatted route information received by the smartphone.

[0590] Step 5:

[0591] The smartphone synchronizes the received route information with the smart glasses. The smartphone transmits the route information to the smart glasses using wireless communication such as Bluetooth or Wi-Fi. The input is the route information received from the server, and the output is the route information received by the smart glasses.

[0592] Step 6:

[0593] Smart glasses use a built-in camera to determine the user's current location and direction of travel. Specifically, they use computer vision technology to analyze the video data captured by the camera and determine the user's current location and direction of travel. The input is the video data captured by the camera, and the output is the current location information and direction of travel information as the analysis results.

[0594] Step 7:

[0595] The smart glasses provide route guidance using arrows and text overlays based on the analyzed current location and direction of travel. Specifically, they display route guidance such as "Turn right in 100 meters" in the user's field of view. The input is the current location and direction of travel as analyzed results, and the output is the arrows and text displayed in the user's field of view.

[0596] Step 8:

[0597] The user inputs a voice command, for example, "I want to change my route" or "Where is the next turn?" The input is the user's voice command, and the output is the voice data captured by the smart glasses' internal microphone.

[0598] Step 9:

[0599] The smart glasses transmit the captured audio data to a smartphone using Bluetooth communication. The input is the audio data captured by the smart glasses, and the output is the audio data received by the smartphone.

[0600] Step 10:

[0601] The smartphone analyzes the received voice data and sends it to the server. It converts the voice data into text using voice recognition software and sends the text information to the server as an HTTP request. The input is the voice data received from the smart glasses, and the output is the analyzed text information and a request to send it to the server.

[0602] Step 11:

[0603] The server calculates a new route based on the received text information and sends that information to the smartphone. It then calls the Google Maps API again to obtain the new route information and sends it back to the smartphone. The input is the parsed text information, and the output is the recalculated route information.

[0604] Step 12:

[0605] The smartphone synchronizes the new route information received from the server with the smart glasses, and the smart glasses display new instructions to the user based on that information. The input is the recalculated route information received from the server, and the output is the new route guidance information displayed on the smart glasses.

[0606] (Application example 1)

[0607] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] Conventional navigation systems have the problem of making it difficult for users to reach their destination safely and efficiently while traveling. Self-driving vehicles, in particular, are required to provide appropriate route guidance while grasping the surrounding situation in real time. Furthermore, systems are susceptible to external factors such as traffic conditions and weather changes, and may not update routes in a timely manner. To solve these problems, a system using more advanced technology is required.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0610] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's vision using a camera and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle, and means for updating the route based on real-time updates of traffic conditions and weather information, thereby enabling a user to reach their destination safely and efficiently in an autonomous vehicle.

[0611] The "means for the user to set a destination and a route" is an interface device for inputting destination information and requesting the server for the optimal route.

[0612] The "means for indicating the optimum route based on the route information received from the server" is a device that analyzes the route information sent from the server and provides guidance to the user on the optimum direction of travel.

[0613] "Means for acquiring and analyzing information about the user's current location and surroundings in real time" refers to a device that uses cameras and sensors to collect and analyze information about the user's location and surrounding environment.

[0614] "Means for using a camera to assist the user's field of vision and providing route guidance using arrows and text" refers to a device that guides the user by displaying arrows and text based on images captured by a camera.

[0615] The "means for analyzing voice commands and updating instructions in response to route changes or questions" refers to a device that recognizes a user's voice input and updates route information and navigation instructions based on the analysis results.

[0616] "Means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle" refers to a device that monitors the surrounding situation using a camera inside the vehicle and provides visual information to the user through a display.

[0617] The "means for updating the route based on real-time updates of traffic conditions and weather information" refers to a device that analyzes traffic and weather information obtained from outside and updates the route information accordingly.

[0618] System Overview

[0619] As an embodiment of the present invention, a navigation system for an autonomous vehicle is taken as an example. This system is composed of a user, a terminal (an in-vehicle display and a smartphone), and a server. The user can set a destination and receive route guidance. The terminal receives destination and route information from the server and updates it in real time. The terminal also uses a camera to obtain information on the user's current location and surrounding conditions, and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the terminal in real time.

[0620] Program processing

[0621] 1. Server Processing

[0622] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into the in-car display or smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. The calculated route information is then sent to the in-car display and smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0623] 2. Terminal processing (in-vehicle displays and smartphones)

[0624] The in-car display and smartphone send the destination information entered by the user to the server. When route information is received from the server, it is displayed on the in-car display and smartphone. The device also uses a camera to acquire the current location in real time and analyzes the direction of travel. The user receives visual instructions and warnings through the device. For example, instructions such as "Turn left 100 meters ahead" are displayed on the in-car display.

[0625] 3. Analyzing voice commands and updating instructions

[0626] The vehicle is equipped with a voice recognition microphone, and when the user enters a voice command such as "I want to change my route," the voice is sent as a request to the server. The server analyzes the voice command, generates a new route and instructions, and displays them on the in-vehicle display. If the user asks a question, the server also provides answers both audibly and visually.

[0627] Hardware and software used

[0628] Hardware

[0629] Cameras, microphones, in-vehicle displays, smartphones

[0630] software

[0631] Server: Route calculation and data update (Google Maps API, TensorFlow)

[0632] Smartphone app: Set destination, receive route (Android / iOS)

[0633] In-car display: Real-time information display (Android Auto, Apple CarPlay)

[0634] Speech analysis: Analysis and conversion of voice commands (Google Cloud Speech-to-Text, Amazon Transcribe)

[0635] Specific examples

[0636] Traveling within cities in autonomous vehicles

[0637] When a user sets a "station" as their destination on their smartphone, the in-car display shows the "optimal route to the station." If traffic conditions change along the way, the route is updated in real time, and instructions such as "Change the next left turn to a right turn" are displayed.

[0638] Example prompts for generative AI models

[0639] "How does the navigation system react when a user sets a train station as a destination on their smartphone? Also, please explain in detail how the system updates the route if traffic information changes along the way."

[0640] This system enables autonomous vehicles to provide safer and more efficient route guidance to users.

[0641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0642] Step 1:

[0643] The user inputs the destination into a smartphone or on-board display. The user sets the destination using a touchscreen or voice input. This input is recorded in the device as the destination's coordinates and name (e.g., "station").

[0644] Step 2:

[0645] The device sends destination information to the server. The smartphone or in-car display sends the destination information entered by the user to the server as an HTTP request. The server receives the input and analyzes the destination coordinates.

[0646] Step 3:

[0647] The server calculates the optimal route and generates route information. The server calls the Google Maps API and calculates the optimal route based on the destination coordinates. In doing so, it also takes into account traffic conditions and weather data when selecting the route. Route information is generated as detailed information for each step (e.g., distance to the next right turn).

[0648] Step 4:

[0649] The server sends the generated route information to the device. The route information is sent as an HTTP response to the in-vehicle display and smartphone. The device analyzes the received information and prepares to display it visually to the user.

[0650] Step 5:

[0651] The device uses a camera to obtain the user's current location and direction of travel in real time. The onboard camera captures video data, and the device analyzes the video in real time. Image recognition technology (e.g., OpenCV) is used to identify the location and direction of travel.

[0652] Step 6:

[0653] The device displays visual instructions, such as arrows and text overlaid on the camera image, in real time in the user's field of view.

[0654] Step 7:

[0655] The server periodically updates traffic and weather information and recalculates route information as needed. The server calls external APIs (e.g., traffic information API, weather API) at regular intervals to obtain the latest information. When new information is obtained, it recalculates route information as needed and sends it to the device.

[0656] Step 8:

[0657] The user inputs a voice command. While driving, the user inputs a voice command such as "I want to change the route." This command is transmitted to the terminal through a microphone in the vehicle.

[0658] Step 9:

[0659] The device analyzes the voice command and sends a request to the server. The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then analyzed. After that, a new request is sent to the server based on the analysis results.

[0660] Step 10:

[0661] The server generates new route information and sends it to the device. The server receives the new route request and recalculates the route. The generated route information is sent to the device in real time and displayed to the user.

[0662] This program's processing allows users to reach their destination safely and efficiently based on the latest route information.

[0663] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0664] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[0665] System Overview

[0666] This system consists of a user, a device (smartphone and smart glasses), a server, and an emotion engine. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time. The emotion engine has the ability to analyze the user's voice and facial expressions and recognize the user's emotional state.

[0667] Program processing

[0668] Server Processing

[0669] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0670] Terminal processing (smartphone)

[0671] The smartphone sends destination information entered by the user to the server. Upon receiving route information from the server, it synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis results to the server. It also receives emotional information from the emotion engine and requests the server to provide appropriate route guidance based on the user's emotional state.

[0672] Terminal processing (smart glasses)

[0673] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[0674] Emotion engine processing

[0675] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, the emotion engine acquires voice tone and facial expression data from the smart glasses' camera and microphone, and analyzes this data to determine whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotion information is sent to the smartphone and then forwarded to the server. Based on this information, the server provides route guidance and warnings appropriate to the user's emotional state.

[0676] Specific examples

[0677] 1. If you're heading to the cafe by bicycle

[0678] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and indicate turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view. At the same time, the emotion engine analyzes the user's tone of voice and, if it determines that the user is feeling anxious, an additional warning is given.

[0679] 2. When walking to the station on a crowded street at night

[0680] The user enters the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, warnings such as "Watch your left. A car is approaching" are displayed. If the emotion engine recognizes that the user's stress level is high, a voice prompt to calm them down is played.

[0681] As described above, the system of the present invention provides a multifunctional means for users to receive safe and efficient route guidance. In addition to unobstructed route guidance and voice command operation, the system's emotion engine allows for flexible guidance that responds to the user's emotions. This allows users to enjoy greater safety and convenience while traveling.

[0682] The processing flow will be explained below.

[0683] Server Processing Steps

[0684] Step 1:

[0685] The server receives the destination information sent by the user via the smartphone.

[0686] Step 2:

[0687] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[0688] Step 3:

[0689] The server sends the calculated route information to the smartphone.

[0690] Step 4:

[0691] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[0692] Step 5:

[0693] The server receives the user's emotion information transferred from the emotion engine and adjusts the route instructions as necessary.

[0694] Step 6:

[0695] The server resends the updated route information to the smartphone.

[0696] Smartphone processing steps

[0697] Step 1:

[0698] The user launches a dedicated app on their smartphone and inputs their destination.

[0699] Step 2:

[0700] The smartphone transmits the input destination information to the server.

[0701] Step 3:

[0702] The smartphone synchronizes the route information received from the server with the smart glasses.

[0703] Step 4:

[0704] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[0705] Step 5:

[0706] The smartphone transmits the analyzed voice data to the server as text.

[0707] Step 6:

[0708] The smartphone transmits the user's facial and voice data acquired from the smart glasses to the emotion engine.

[0709] Step 7:

[0710] The user's emotional state information received from the emotion engine is transmitted to a server.

[0711] Smart Glasses Processing Steps

[0712] Step 1:

[0713] The user puts on the smart glasses and starts moving.

[0714] Step 2:

[0715] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[0716] Step 3:

[0717] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[0718] Step 4:

[0719] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[0720] Step 5:

[0721] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[0722] Step 6:

[0723] The smart glasses' camera and microphone analyze the user's facial expressions and tone of voice in real time and send the data to the emotion engine.

[0724] Emotion Engine Processing Steps

[0725] Step 1:

[0726] The emotion engine analyzes the user's voice tone and facial expressions received from the smart glasses.

[0727] Step 2:

[0728] The emotion engine recognizes the user's emotional state based on the analysis results.

[0729] Step 3:

[0730] The emotion engine sends the recognized emotion information to the smartphone.

[0731] User processing steps

[0732] Step 1:

[0733] The user inputs the destination into the smartphone and sets the route.

[0734] Step 2:

[0735] The user puts on the smart glasses and begins moving along the specified route.

[0736] Step 3:

[0737] Users can check their direction by following the arrows and text displayed through the smart glasses.

[0738] Step 4:

[0739] Users can use voice commands to change routes or ask questions as needed.

[0740] Step 5:

[0741] The user follows the new instructions from the smart glasses to reach their destination.

[0742] The above is the specific flow of each processing step of this system.

[0743] Example 2

[0744] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] Conventional navigation systems can sometimes make it difficult for users to find the optimal route to their destination, and they also lack real-time route adjustments and guidance that reflects the user's emotional state. In particular, they are unable to provide appropriate guidance to users who are feeling anxious or stressed, making it difficult to ensure the safety and convenience of users.

[0746] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to set a destination and a route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's field of vision using an imaging device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing the user's emotional state, and means for flexibly adjusting the content of guidance based on the user's emotional state. This allows the user to receive optimal route guidance in real time and enables flexible guidance according to the user's emotional state, greatly improving safety and convenience.

[0747] "User" refers to an individual who uses the system to travel to a destination.

[0748] A "destination" refers to a specific location that a user wishes to reach.

[0749] A "route" refers to the path a user takes to reach a destination.

[0750] The term "server" refers to a computer system that receives information from a user, processes the data, generates route information, and transmits it to a terminal.

[0751] "Terminal" refers to a device used by a user, such as a smartphone or smart glasses.

[0752] "Route information" refers to data regarding the optimal route a user needs to take to reach a destination.

[0753] "Real-time" refers to instantly recognizing and reflecting the current situation or state.

[0754] The term "photography device" refers to a device such as a camera for acquiring information about the user's field of view and current location.

[0755] An "arrow" refers to a graphic that visually indicates the direction of travel to the user.

[0756] "Text" refers to a string of characters that provides information to the user.

[0757] "Voice command" refers to a means by which a user issues instructions to a system by voice.

[0758] "Emotional state" refers to the user's psychological state, and includes emotions such as anxiety, impatience, and joy.

[0759] "Analysis" refers to the detailed analysis of acquired data to clarify its meaning.

[0760] "Guidance content" refers to information provided to a user, including route information and other instructions.

[0761] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. This system is composed of a user, a terminal (a smartphone and smart glasses), a server, and an emotion engine. Below, an embodiment of the present invention will be described in detail.

[0762] System Overview

[0763] The user inputs their destination using a smartphone application. This information is sent from the smartphone to a server. The server uses the Google Maps API to calculate the optimal route and sends that information to the smartphone. The smartphone then synchronizes the received route information with the smart glasses. The smart glasses use a built-in camera to determine the user's current location and direction in real time, and provide directions by displaying arrows and text in the user's field of view.

[0764] Server Processing

[0765] The server calculates the optimal route based on the destination information sent by the user. The server calls the Google Maps API to calculate the optimal route based on the destination and current location, taking into account real-time information such as traffic conditions and weather data. The calculated route information is also sent to the smartphone using encryption technology.

[0766] Terminal processing (smartphone)

[0767] The smartphone sends destination information entered by the user to the server. It uses Bluetooth and Wi-Fi to synchronize route information received from the server with the smart glasses. When a voice command is sent from the smart glasses, the smartphone also sends that information to the server. It also receives the user's emotional state from the emotion engine and requests appropriate route guidance from the server.

[0768] Terminal processing (smart glasses)

[0769] Smart glasses use a camera to determine the user's current location and direction in real time. They analyze the captured video and GPS data to determine the user's current location and direction. As a result, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also send voice commands to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is sent from the smart glasses to a server via the smartphone, and new instructions are generated and displayed on the smart glasses.

[0770] Emotion engine processing

[0771] The emotion engine analyzes the user's emotional state based on data obtained from the smart glasses' camera and microphone. Voice tone and facial expression data are used for the analysis. It determines whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotional information is sent to the smartphone and then forwarded to the server. The server uses this information to provide flexible route guidance and warnings according to the user's emotional state.

[0772] Specific example explanation

[0773] 1. If you're heading to the cafe by bicycle

[0774] The user enters the location of the cafe into their smartphone. The smartphone sends this information to the server, which uses the Google Maps API to calculate the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and provide directions using text and arrows, such as "Turn right in 100 meters." At the same time, if the emotion engine detects the user's anxiety, additional warnings are given.

[0775] Prompt Sentence Examples

[0776] Explain the steps and specific actions to take to get from your home to a nearby cafe using a bicycle, smart glasses, and a smartphone.

[0777] 2. When walking to the station on a crowded street at night

[0778] The user enters the "station location" into their smartphone. The smartphone sends this information to the server. The server uses the Google Maps API to calculate the optimal route, taking into account traffic conditions and safety at night. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins moving. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching." If the emotion engine detects that the user is stressed, a voice prompt to calm them down is played.

[0779] Prompt Sentence Examples

[0780] Explain the specific steps to safely receive directions using smart glasses and a smartphone when walking down a crowded street at night to the station.

[0781] Through these examples, the system of the present invention provides a versatile means for users to receive route guidance safely and efficiently. By utilizing unobstructed route guidance and voice commands, users can enjoy safety and convenience while traveling. Furthermore, the emotion engine enables flexible guidance based on the user's emotions.

[0782] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0783] Step 1:

[0784] The user operates the smartphone to input a destination. The user opens the smartphone's application and inputs the name and address of the destination. For example, the user inputs "the location of the cafe." This input information is saved on the smartphone as data about the location the user wants to go to.

[0785] Input: Destination information (cafe name and address)

[0786] Output: Destination information saved on your smartphone

[0787] Specific operation:

[0788] The user launches an application on the smartphone.

[0789] The user enters the "location of the cafe" in the text box.

[0790] Step 2:

[0791] The device (smartphone) sends destination information to the server. The smartphone sends destination information entered by the user to the server. Communication is usually via Wi-Fi or mobile data networks.

[0792] Input: Destination information saved on your smartphone

[0793] Output: Destination information sent to the server

[0794] Specific operation:

[0795] The smartphone sends destination information in packet format to the server.

[0796] Encryption technology is used to ensure the stability of data communication.

[0797] Step 3:

[0798] The server calculates the optimal route. Based on the destination information received, the server uses external services such as Google Maps API to calculate the optimal route, while taking into account real-time information such as traffic conditions and weather data.

[0799] Input: Destination information, current traffic conditions, weather data

[0800] Output: Optimal route information

[0801] Specific operation:

[0802] The server calls the Google Maps API and starts route calculation.

[0803] Route information is updated in real time based on traffic and weather data.

[0804] Step 4:

[0805] The server sends route information to the device (smartphone). The calculated route information is sent to the smartphone in encrypted packet format. The smartphone receives and stores this information.

[0806] Input: Optimal route information

[0807] Output: Route information sent to your smartphone

[0808] Specific operation:

[0809] The server encrypts the route information and sends it to your smartphone.

[0810] The smartphone decodes the received route information and stores it in its internal storage.

[0811] Step 5:

[0812] The device (smartphone) synchronizes the route information with the smart glasses. The smartphone then synchronizes the received route information with the smart glasses via Bluetooth or Wi-Fi.

[0813] Input: Route information saved on your smartphone

[0814] Output: Route information synchronized to smart glasses

[0815] Specific operation:

[0816] Your smartphone connects to the smart glasses using Bluetooth or Wi-Fi.

[0817] Transfer the route information to your smart glasses and check the synchronization.

[0818] Step 6:

[0819] Smart glasses acquire location information. Using a built-in camera and GPS, smart glasses determine the user's location in real time and determine the direction of travel.

[0820] Input: Camera image, GPS data

[0821] Output: User's current location

[0822] Specific operation:

[0823] The smart glasses use a camera to capture images of the surrounding area.

[0824] Use the built-in GPS sensor to determine your current location.

[0825] Step 7:

[0826] The smart glasses provide route guidance. Based on the acquired current location information, the smart glasses display arrows and text in the user's field of view to provide route guidance to the user. For example, specific instructions such as "Turn right in 100 meters" are displayed.

[0827] Input: User's current location information, route information

[0828] Output: Directions displayed in the field of view

[0829] Specific operation:

[0830] The smart glasses generate arrows and text based on route information.

[0831] Instructions such as "Turn right in 100 meters" are displayed in the user's field of vision.

[0832] Step 8:

[0833] The emotion engine analyzes the user's emotional state. The emotion engine analyzes voice tone and facial expressions based on data obtained from the smart glasses' camera and microphone to determine the user's emotional state.

[0834] Input: Voice data, facial expression data

[0835] Output: User's emotional state information

[0836] Specific operation:

[0837] The emotion engine analyzes the tone of the voice to determine whether the user is feeling anxious.

[0838] Identify emotional states based on facial expressions captured by a camera.

[0839] Step 9:

[0840] Providing guidance based on emotional information. The server generates appropriate guidance content based on the user's emotional information obtained from the emotion engine and sends it to the smart glasses. This provides flexible guidance according to the user's emotional state.

[0841] Input: User's emotional state information

[0842] Output: Guidance information according to emotions

[0843] Specific operation:

[0844] The server generates messages based on the emotional information to provide special attention or reassurance.

[0845] The smart glasses will display a message such as "You are almost at your destination. Proceed safely."

[0846] As described above, through these processing steps, the system provides the user with safe and efficient route guidance, and is also able to respond flexibly to the user's emotional state.

[0847] (Application example 2)

[0848] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0849] In self-driving vehicles, conventional navigation systems only provide route guidance, making it difficult to provide instructions that fully consider the emotional state and safety of passengers while driving. There is also a need for a system that can reduce the mental burden on drivers and enable them to reach their destination more safely. It is necessary to sense the anxiety and stress felt by users while driving in real time and respond appropriately.

[0850] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0851] In this invention, the server includes means for allowing the user to set a destination and route, means for indicating the optimal route based on route information received from the server, means for acquiring and analyzing the user's current location and scenery in real time, means for assisting the user's vision using a visual device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, and means for adjusting operation or issuing additional warnings based on the user's emotional state using emotion analysis means, thereby enabling safer and more secure automated driving that is responsive to the driver's emotional state.

[0852] A "user" is a vehicle occupant who uses the system to set a destination and route and receives navigation.

[0853] A "destination" is the final location or place that a user wishes to reach.

[0854] A "route" is the path or direction of travel to reach a destination.

[0855] A "means" is a method, device, or system for achieving a specific purpose.

[0856] A "server" is a central computer system that performs route calculations and real-time data processing.

[0857] "Real time" refers to the time frame for immediate processing and response to ongoing events.

[0858] "Analysis" is the process of examining and processing information or data in detail to extract meaning.

[0859] A "visual device" is a device that supports a user's vision and presents information in real time.

[0860] An "arrow" is an indicator displayed on a visual device to indicate the direction in which the user should move.

[0861] "Text" means written information displayed on a visual device to provide guidance or instructions to a user.

[0862] A "voice command" is an instruction or question given by a user through speech.

[0863] "Emotion analysis" is the process of recognizing and analyzing a user's emotional state based on their voice and facial expression data.

[0864] "Operation adjustment" refers to changing or optimizing the autonomous driving speed, route, settings, etc. based on the user's emotional state.

[0865] "Attention" refers to a visual or audio warning or instruction to alert the user.

[0866] The above are definitions of important words included in the claims.

[0867] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0868] System Overview

[0869] This system consists of a user, a terminal (the central control device of the autonomous vehicle), a server, and an emotion analysis means. The user sets a destination and route and gets into the autonomous vehicle. The terminal receives destination and route information from the server and updates it in real time. The terminal uses a camera and microphone to obtain the user's current location information and emotional state, and provides route guidance through a visual device. The emotion analysis means also has the function of analyzing the user's voice and facial expressions to recognize the user's emotional state.

[0870] Hardware and Software

[0871] Hardware

[0872] Autonomous vehicle: central control unit, camera, microphone, GPS

[0873] Server: Responsible for route calculation and real-time data acquisition

[0874] software

[0875] Google Maps API: Used for route calculation

[0876] Emotion analysis tools: AI models that analyze voice tone and facial expressions (e.g., developed using OpenCV and TensorFlow)

[0877] Real-time data processing: APIs used to obtain traffic and weather data

[0878] Program processing

[0879] Server Processing

[0880] The server calculates the optimal route based on the destination information set by the user. It receives destination information sent from the central control unit of the autonomous vehicle and calculates the optimal route using external services such as Google Maps API. It sends the calculated route information to the autonomous vehicle and updates the route based on real-time information. It also generates emotion-responsive driving instructions based on emotion data received from the emotion analysis means.

[0881] Terminal processing (central control unit of autonomous vehicles)

[0882] The central control unit of the autonomous vehicle controls autonomous driving based on route information received from the server. It also passes data acquired from the camera and microphone to an emotion analysis means to analyze the user's emotional state in real time. Based on the emotional information acquired from the emotion analysis means, it adjusts driving and issues warnings to improve safety. For example, if the user feels anxious, it may adjust the driving speed or play relaxing music.

[0883] Processing of sentiment analysis tools

[0884] The emotion analysis means analyzes the user's voice and facial expression data to recognize their emotional state. Specifically, it uses data acquired from the camera and microphone to determine whether the user is feeling anxiety, impatience, joy, or other emotions. This information is sent to the central control unit and server in real time and used to adjust operation and provide warnings.

[0885] Specific examples

[0886] 1. Dealing with stressed drivers

[0887] If the autonomous vehicle determines through emotion analysis that the driver is feeling stressed, the system will play relaxing music and suggest the best place to rest, for example, "There is a rest area nearby, so we recommend you take a 10-minute break."

[0888] 2. Examples of prompts

[0889] "Design a system that provides route guidance and warnings to help drivers relax. The server calculates the optimal route taking into account traffic conditions and weather, and the vehicle processes emotion data in real time and responds appropriately based on the emotion analysis results while driving."

[0890] The above is a specific embodiment for carrying out the invention. This system realizes safe and secure automated driving according to the emotional state of the user.

[0891] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0892] Step 1:

[0893] The user sets the destination.

[0894] Input: The user inputs a destination into the central control unit of the autonomous vehicle.

[0895] Data processing and calculation: The central control unit receives the input destination information and compares it with GPS information.

[0896] Output: Sends the set destination information to the server.

[0897] Step 2:

[0898] The server calculates the optimal route.

[0899] Input: Destination information sent from the autonomous vehicle.

[0900] Data processing and calculation: Using the Google Maps API, the optimal route from the current location to the destination is calculated, taking into account traffic conditions and weather data, and optimization is performed in real time.

[0901] Output: Send the calculated optimal route information to the central control unit of the autonomous vehicle.

[0902] Step 3:

[0903] The self-driving vehicle will begin operating based on the route information.

[0904] Input: Optimal route information received from the server.

[0905] Data processing and calculation: The central control unit controls the operation based on the route information.

[0906] Output: Actual driving control of the vehicle.

[0907] Step 4:

[0908] The camera and microphone capture information about the surroundings and the user's emotions.

[0909] Input: Video data captured by the camera of the autonomous vehicle, audio data recorded by the microphone.

[0910] Data processing and calculation: The emotion analysis means analyzes the video data and audio data in real time to extract the user's emotional state.

[0911] Output: Sending the user's emotional information to the central control unit and server.

[0912] Step 5:

[0913] The server adjusts operations and issues warnings based on emotional information.

[0914] Input: User emotion information received from emotion analysis means.

[0915] Data processing and calculation: The server analyzes the emotional information and generates instructions and warning messages to adjust operations.

[0916] Output: Sends generated instructions and reminder messages to the autonomous vehicle.

[0917] Step 6:

[0918] The autonomous vehicle will carry out instructions and give warnings.

[0919] Input: Instructions and reminder messages sent by the server.

[0920] Data processing and calculation: The central control unit adjusts operation according to instructions, plays relaxing music, suggests resting spots, etc.

[0921] Output: Visual and audio guidance to the user and operational coordination.

[0922] The above is the processing flow of the program for the system for carrying out the invention. By performing appropriate processing and calculations based on the data input at each step and generating output, safe and efficient autonomous driving that responds to the user's emotional state becomes possible.

[0923] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0924] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0925] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0926] [Third embodiment]

[0927] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0928] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0929] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0930] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0931] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0932] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0933] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0934] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0935] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0936] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0937] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0938] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0939] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to exemplary embodiments.

[0940] System Overview

[0941] This system consists of a user, a device (a smartphone and smart glasses), and a server. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time.

[0942] Program processing

[0943] Server Processing

[0944] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[0945] Terminal processing (smartphone)

[0946] The smartphone sends the destination information entered by the user to the server. Upon receiving route information from the server, the smartphone synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis to the server. Specifically, when a user says "I want to change the route" to the smart glasses, the voice is transferred to the smartphone, analyzed, and then sent to the server.

[0947] Terminal processing (smart glasses)

[0948] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[0949] Specific examples

[0950] 1. If you're heading to the cafe by bicycle

[0951] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bike. The smart glasses use a camera to determine their current location and show turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view.

[0952] 2. When walking to the station on a crowded street at night

[0953] The user inputs the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, they display warnings such as "Watch your left. A car is approaching."

[0954] As described above, the system of the present invention provides a multifunctional means for users to receive route guidance safely and efficiently. In particular, the unobstructed route guidance and voice command operation function allow users to enjoy greater safety and convenience while traveling.

[0955] The processing flow will be explained below.

[0956] Server Processing Steps

[0957] Step 1:

[0958] The server receives the destination information sent by the user via the smartphone.

[0959] Step 2:

[0960] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[0961] Step 3:

[0962] The server sends the calculated route information to the smartphone.

[0963] Step 4:

[0964] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[0965] Step 5:

[0966] The server resends the updated route information to the smartphone.

[0967] Smartphone processing steps

[0968] Step 1:

[0969] The user launches a dedicated app on their smartphone and inputs their destination.

[0970] Step 2:

[0971] The smartphone transmits the input destination information to the server.

[0972] Step 3:

[0973] The smartphone synchronizes the route information received from the server with the smart glasses.

[0974] Step 4:

[0975] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[0976] Step 5:

[0977] The smartphone transmits the analyzed voice data to the server as text.

[0978] Smart Glasses Processing Steps

[0979] Step 1:

[0980] The user puts on the smart glasses and starts moving.

[0981] Step 2:

[0982] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[0983] Step 3:

[0984] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[0985] Step 4:

[0986] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[0987] Step 5:

[0988] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[0989] User processing steps

[0990] Step 1:

[0991] The user inputs the destination into the smartphone and sets the route.

[0992] Step 2:

[0993] The user puts on the smart glasses and begins moving along the specified route.

[0994] Step 3:

[0995] Users can check their direction by following the arrows and text displayed through the smart glasses.

[0996] Step 4:

[0997] Users can use voice commands to change routes or ask questions as needed.

[0998] Step 5:

[0999] The user follows the new instructions from the smart glasses to reach their destination.

[1000] The above is the specific flow of each processing step of this system.

[1001] Example 1

[1002] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1003] Conventional navigation systems do not adequately ensure safety and efficiency in the process of users reaching their destination, and often fail to provide appropriate guidance, especially on complex routes or in congested environments. They also lack the ability to provide route guidance that takes into account real-time changes in traffic conditions, weather, and other environmental factors, resulting in inconvenience for users. Furthermore, they lack an interface that utilizes voice commands, making hands-free operation difficult.

[1004] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1005] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for using a camera to assist the user's vision and for providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing video data acquired by the camera to identify the user's current location and direction of travel, means for analyzing voice commands and transmitting them to the server, means for recalculating the route under specific conditions (e.g., traffic conditions or weather) and providing the latest route information to the terminal, and means for synchronizing data between terminals using wireless communication, thereby enabling a user to receive real-time route guidance to reach their destination safely and efficiently.

[1006] A "destination" is a location that a user ultimately wants to reach.

[1007] A "route" is information about a route to reach a destination.

[1008] A "server" is a computer device that processes and stores data on a network.

[1009] The term "terminal" refers to any device that is directly operated by a user, and in this invention includes smartphones, smart glasses, etc.

[1010] "Route information" refers to the optimal route calculated by the server to reach the destination.

[1011] "Current location" refers to the location where the user is currently located.

[1012] "Surrounding information" refers to environmental information around the user's current location, including traffic conditions and weather.

[1013] The term "photography device" refers to a device that captures images or videos, and includes a camera in this invention.

[1014] A "voice command" is an instruction that a user inputs by voice.

[1015] "Analysis" refers to the process of processing acquired data to extract useful information.

[1016] "Synchronization" refers to the act of matching information to unify data across multiple terminals.

[1017] MODE FOR CARRYING OUT THE INVENTION

[1018] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. The system is composed of a user, a terminal (a smartphone and smart glasses), and a server.

[1019] System configuration

[1020] The user inputs their destination using their smartphone, and the information is sent to the server. The server then calculates the optimal route based on the received destination information using external services such as Google Maps API. The calculated route information is sent to the smartphone, which then synchronizes it with the smart glasses.

[1021] Smart glasses use a camera to determine the user's current location and provide directions by displaying arrows and text in the user's field of view. Communication between the smart glasses and smartphones is via wireless communication such as Bluetooth or Wi-Fi. Smart glasses also accept voice commands and transmit the voice data to the smartphone. The smartphone analyzes the voice data and sends it to a server to provide new routes and additional information.

[1022] Specific examples

[1023] Example 1: Cycling to a cafe

[1024] The user enters the location of the cafe into their smartphone. The smartphone sends the information to a server, which then calls the Google Maps API to calculate the optimal cycling route. The route information calculated by the server is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts cycling. The smart glasses use a camera to determine the user's current location in real time and display arrows and text such as "Turn right in 100 meters."

[1025] Example 2: Walking to the station on a crowded street at night

[1026] The user inputs the "station location" into their smartphone. The smartphone sends this information to the server, which then calls the Google Maps API to calculate a safe walking route. The server sets the route, taking into account traffic conditions and nighttime safety. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins walking. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching."

[1027] Prompt Sentence Examples

[1028] Enter "I want to go to XX Cafe" into your smartphone as your destination and start navigation. Also, if you say "I want to change the route" using the voice command, the server will calculate a new route and new instructions will be displayed on the smart glasses.

[1029] This system allows users to enjoy greater safety and convenience while traveling. With many features, such as real-time traffic and weather updates and the ability to operate using voice commands, users can receive optimal guidance in a variety of environments and situations.

[1030] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1031] Processing step details

[1032] Step 1:

[1033] The user starts up the smartphone and opens the navigation app. The user inputs their destination using the keyboard or touch screen. For example, they input the destination name, such as "cafe." The input data is converted to JSON format within the smartphone. At this point, the JSON format destination information is generated as input data (destination information).

[1034] Step 2:

[1035] The smartphone sends the destination information entered by the user to the server. Specifically, it uses an HTTP POST request to send the destination information to the server. The smartphone attaches the data entered by the user to the HTTP request and sends it to the server. The output is that the server receives the destination information.

[1036] Step 3:

[1037] The server calculates the optimal route based on the received destination information. This is achieved by calling a third-party API (e.g., Google Maps API). The server sends the destination information in an appropriate format to the API and obtains the optimal route information. The input is the received destination information, and the output is the calculated route information (in JSON format).

[1038] Step 4:

[1039] The server sends the calculated route information in JSON format to the smartphone. The server creates an HTTP response, attaches the route information, and sends it to the smartphone. The smartphone receives the HTTP response and obtains the route information. The output is the JSON-formatted route information received by the smartphone.

[1040] Step 5:

[1041] The smartphone synchronizes the received route information with the smart glasses. The smartphone transmits the route information to the smart glasses using wireless communication such as Bluetooth or Wi-Fi. The input is the route information received from the server, and the output is the route information received by the smart glasses.

[1042] Step 6:

[1043] Smart glasses use a built-in camera to determine the user's current location and direction of travel. Specifically, they use computer vision technology to analyze the video data captured by the camera and determine the user's current location and direction of travel. The input is the video data captured by the camera, and the output is the current location information and direction of travel information as the analysis results.

[1044] Step 7:

[1045] The smart glasses provide route guidance using arrows and text overlays based on the analyzed current location and direction of travel. Specifically, they display route guidance such as "Turn right in 100 meters" in the user's field of view. The input is the current location and direction of travel as analyzed results, and the output is the arrows and text displayed in the user's field of view.

[1046] Step 8:

[1047] The user inputs a voice command, for example, "I want to change my route" or "Where is the next turn?" The input is the user's voice command, and the output is the voice data captured by the smart glasses' internal microphone.

[1048] Step 9:

[1049] The smart glasses transmit the captured audio data to a smartphone using Bluetooth communication. The input is the audio data captured by the smart glasses, and the output is the audio data received by the smartphone.

[1050] Step 10:

[1051] The smartphone analyzes the received voice data and sends it to the server. It converts the voice data into text using voice recognition software and sends the text information to the server as an HTTP request. The input is the voice data received from the smart glasses, and the output is the analyzed text information and a request to send it to the server.

[1052] Step 11:

[1053] The server calculates a new route based on the received text information and sends that information to the smartphone. It then calls the Google Maps API again to obtain the new route information and sends it back to the smartphone. The input is the parsed text information, and the output is the recalculated route information.

[1054] Step 12:

[1055] The smartphone synchronizes the new route information received from the server with the smart glasses, and the smart glasses display new instructions to the user based on that information. The input is the recalculated route information received from the server, and the output is the new route guidance information displayed on the smart glasses.

[1056] (Application example 1)

[1057] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1058] Conventional navigation systems have the problem of making it difficult for users to reach their destination safely and efficiently while traveling. Self-driving vehicles, in particular, are required to provide appropriate route guidance while grasping the surrounding situation in real time. Furthermore, systems are susceptible to external factors such as traffic conditions and weather changes, and may not update routes in a timely manner. To solve these problems, a system using more advanced technology is required.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1060] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's vision using a camera and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle, and means for updating the route based on real-time updates of traffic conditions and weather information, thereby enabling a user to reach their destination safely and efficiently in an autonomous vehicle.

[1061] The "means for the user to set a destination and a route" is an interface device for inputting destination information and requesting the server for the optimal route.

[1062] The "means for indicating the optimum route based on the route information received from the server" is a device that analyzes the route information sent from the server and provides guidance to the user on the optimum direction of travel.

[1063] "Means for acquiring and analyzing information about the user's current location and surroundings in real time" refers to a device that uses cameras and sensors to collect and analyze information about the user's location and surrounding environment.

[1064] "Means for using a camera to assist the user's field of vision and providing route guidance using arrows and text" refers to a device that guides the user by displaying arrows and text based on images captured by a camera.

[1065] The "means for analyzing voice commands and updating instructions in response to route changes or questions" refers to a device that recognizes a user's voice input and updates route information and navigation instructions based on the analysis results.

[1066] "Means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle" refers to a device that monitors the surrounding situation using a camera inside the vehicle and provides visual information to the user through a display.

[1067] The "means for updating the route based on real-time updates of traffic conditions and weather information" refers to a device that analyzes traffic and weather information obtained from outside and updates the route information accordingly.

[1068] System Overview

[1069] As an embodiment of the present invention, a navigation system for an autonomous vehicle is taken as an example. This system is composed of a user, a terminal (an in-vehicle display and a smartphone), and a server. The user can set a destination and receive route guidance. The terminal receives destination and route information from the server and updates it in real time. The terminal also uses a camera to obtain information on the user's current location and surrounding conditions, and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the terminal in real time.

[1070] Program processing

[1071] 1. Server Processing

[1072] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into the in-car display or smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. The calculated route information is then sent to the in-car display and smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[1073] 2. Terminal processing (in-vehicle displays and smartphones)

[1074] The in-car display and smartphone send the destination information entered by the user to the server. When route information is received from the server, it is displayed on the in-car display and smartphone. The device also uses a camera to acquire the current location in real time and analyzes the direction of travel. The user receives visual instructions and warnings through the device. For example, instructions such as "Turn left 100 meters ahead" are displayed on the in-car display.

[1075] 3. Analyzing voice commands and updating instructions

[1076] The vehicle is equipped with a voice recognition microphone, and when the user enters a voice command such as "I want to change my route," the voice is sent as a request to the server. The server analyzes the voice command, generates a new route and instructions, and displays them on the in-vehicle display. If the user asks a question, the server also provides answers both audibly and visually.

[1077] Hardware and software used

[1078] Hardware

[1079] Cameras, microphones, in-vehicle displays, smartphones

[1080] software

[1081] Server: Route calculation and data update (Google Maps API, TensorFlow)

[1082] Smartphone app: Set destination, receive route (Android / iOS)

[1083] In-car display: Real-time information display (Android Auto, Apple CarPlay)

[1084] Speech analysis: Analysis and conversion of voice commands (Google Cloud Speech-to-Text, Amazon Transcribe)

[1085] Specific examples

[1086] Traveling within cities in autonomous vehicles

[1087] When a user sets a "station" as their destination on their smartphone, the in-car display shows the "optimal route to the station." If traffic conditions change along the way, the route is updated in real time, and instructions such as "Change the next left turn to a right turn" are displayed.

[1088] Example prompts for generative AI models

[1089] "How does the navigation system react when a user sets a train station as a destination on their smartphone? Also, please explain in detail how the system updates the route if traffic information changes along the way."

[1090] This system enables autonomous vehicles to provide safer and more efficient route guidance to users.

[1091] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1092] Step 1:

[1093] The user inputs the destination into a smartphone or on-board display. The user sets the destination using a touchscreen or voice input. This input is recorded in the device as the destination's coordinates and name (e.g., "station").

[1094] Step 2:

[1095] The device sends destination information to the server. The smartphone or in-car display sends the destination information entered by the user to the server as an HTTP request. The server receives the input and analyzes the destination coordinates.

[1096] Step 3:

[1097] The server calculates the optimal route and generates route information. The server calls the Google Maps API and calculates the optimal route based on the destination coordinates. In doing so, it also takes into account traffic conditions and weather data when selecting the route. Route information is generated as detailed information for each step (e.g., distance to the next right turn).

[1098] Step 4:

[1099] The server sends the generated route information to the device. The route information is sent as an HTTP response to the in-vehicle display and smartphone. The device analyzes the received information and prepares to display it visually to the user.

[1100] Step 5:

[1101] The device uses a camera to obtain the user's current location and direction of travel in real time. The onboard camera captures video data, and the device analyzes the video in real time. Image recognition technology (e.g., OpenCV) is used to identify the location and direction of travel.

[1102] Step 6:

[1103] The device displays visual instructions, such as arrows and text overlaid on the camera image, in real time in the user's field of view.

[1104] Step 7:

[1105] The server periodically updates traffic and weather information and recalculates route information as needed. The server calls external APIs (e.g., traffic information API, weather API) at regular intervals to obtain the latest information. When new information is obtained, it recalculates route information as needed and sends it to the device.

[1106] Step 8:

[1107] The user inputs a voice command. While driving, the user inputs a voice command such as "I want to change the route." This command is transmitted to the terminal through a microphone in the vehicle.

[1108] Step 9:

[1109] The device analyzes the voice command and sends a request to the server. The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then analyzed. After that, a new request is sent to the server based on the analysis results.

[1110] Step 10:

[1111] The server generates new route information and sends it to the device. The server receives the new route request and recalculates the route. The generated route information is sent to the device in real time and displayed to the user.

[1112] This program's processing allows users to reach their destination safely and efficiently based on the latest route information.

[1113] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1114] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[1115] System Overview

[1116] This system consists of a user, a device (smartphone and smart glasses), a server, and an emotion engine. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time. The emotion engine has the ability to analyze the user's voice and facial expressions and recognize the user's emotional state.

[1117] Program processing

[1118] Server Processing

[1119] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[1120] Terminal processing (smartphone)

[1121] The smartphone sends destination information entered by the user to the server. Upon receiving route information from the server, it synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis results to the server. It also receives emotional information from the emotion engine and requests the server to provide appropriate route guidance based on the user's emotional state.

[1122] Terminal processing (smart glasses)

[1123] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[1124] Emotion engine processing

[1125] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, the emotion engine acquires voice tone and facial expression data from the smart glasses' camera and microphone, and analyzes this data to determine whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotion information is sent to the smartphone and then forwarded to the server. Based on this information, the server provides route guidance and warnings appropriate to the user's emotional state.

[1126] Specific examples

[1127] 1. If you're heading to the cafe by bicycle

[1128] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and indicate turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view. At the same time, the emotion engine analyzes the user's tone of voice and, if it determines that the user is feeling anxious, an additional warning is given.

[1129] 2. When walking to the station on a crowded street at night

[1130] The user enters the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, warnings such as "Watch your left. A car is approaching" are displayed. If the emotion engine recognizes that the user's stress level is high, a voice prompt to calm them down is played.

[1131] As described above, the system of the present invention provides a multifunctional means for users to receive safe and efficient route guidance. In addition to unobstructed route guidance and voice command operation, the system's emotion engine allows for flexible guidance that responds to the user's emotions. This allows users to enjoy greater safety and convenience while traveling.

[1132] The processing flow will be explained below.

[1133] Server Processing Steps

[1134] Step 1:

[1135] The server receives the destination information sent by the user via the smartphone.

[1136] Step 2:

[1137] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[1138] Step 3:

[1139] The server sends the calculated route information to the smartphone.

[1140] Step 4:

[1141] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[1142] Step 5:

[1143] The server receives the user's emotion information transferred from the emotion engine and adjusts the route instructions as necessary.

[1144] Step 6:

[1145] The server resends the updated route information to the smartphone.

[1146] Smartphone processing steps

[1147] Step 1:

[1148] The user launches a dedicated app on their smartphone and inputs their destination.

[1149] Step 2:

[1150] The smartphone transmits the input destination information to the server.

[1151] Step 3:

[1152] The smartphone synchronizes the route information received from the server with the smart glasses.

[1153] Step 4:

[1154] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[1155] Step 5:

[1156] The smartphone transmits the analyzed voice data to the server as text.

[1157] Step 6:

[1158] The smartphone transmits the user's facial and voice data acquired from the smart glasses to the emotion engine.

[1159] Step 7:

[1160] The user's emotional state information received from the emotion engine is transmitted to a server.

[1161] Smart Glasses Processing Steps

[1162] Step 1:

[1163] The user puts on the smart glasses and starts moving.

[1164] Step 2:

[1165] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[1166] Step 3:

[1167] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[1168] Step 4:

[1169] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[1170] Step 5:

[1171] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[1172] Step 6:

[1173] The smart glasses' camera and microphone analyze the user's facial expressions and tone of voice in real time and send the data to the emotion engine.

[1174] Emotion Engine Processing Steps

[1175] Step 1:

[1176] The emotion engine analyzes the user's voice tone and facial expressions received from the smart glasses.

[1177] Step 2:

[1178] The emotion engine recognizes the user's emotional state based on the analysis results.

[1179] Step 3:

[1180] The emotion engine sends the recognized emotion information to the smartphone.

[1181] User processing steps

[1182] Step 1:

[1183] The user inputs the destination into the smartphone and sets the route.

[1184] Step 2:

[1185] The user puts on the smart glasses and begins moving along the specified route.

[1186] Step 3:

[1187] Users can check their direction by following the arrows and text displayed through the smart glasses.

[1188] Step 4:

[1189] Users can use voice commands to change routes or ask questions as needed.

[1190] Step 5:

[1191] The user follows the new instructions from the smart glasses to reach their destination.

[1192] The above is the specific flow of each processing step of this system.

[1193] Example 2

[1194] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1195] Conventional navigation systems can sometimes make it difficult for users to find the optimal route to their destination, and they also lack real-time route adjustments and guidance that reflects the user's emotional state. In particular, they are unable to provide appropriate guidance to users who are feeling anxious or stressed, making it difficult to ensure the safety and convenience of users.

[1196] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to set a destination and a route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's field of vision using an imaging device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing the user's emotional state, and means for flexibly adjusting the content of guidance based on the user's emotional state. This allows the user to receive optimal route guidance in real time and enables flexible guidance according to the user's emotional state, greatly improving safety and convenience.

[1197] "User" refers to an individual who uses the system to travel to a destination.

[1198] A "destination" refers to a specific location that a user wishes to reach.

[1199] A "route" refers to the path a user takes to reach a destination.

[1200] The term "server" refers to a computer system that receives information from a user, processes the data, generates route information, and transmits it to a terminal.

[1201] "Terminal" refers to a device used by a user, such as a smartphone or smart glasses.

[1202] "Route information" refers to data regarding the optimal route a user needs to take to reach a destination.

[1203] "Real-time" refers to instantly recognizing and reflecting the current situation or state.

[1204] The term "photography device" refers to a device such as a camera for acquiring information about the user's field of view and current location.

[1205] An "arrow" refers to a graphic that visually indicates the direction of travel to the user.

[1206] "Text" refers to a string of characters that provides information to the user.

[1207] "Voice command" refers to a means by which a user issues instructions to a system by voice.

[1208] "Emotional state" refers to the user's psychological state, and includes emotions such as anxiety, impatience, and joy.

[1209] "Analysis" refers to the detailed analysis of acquired data to clarify its meaning.

[1210] "Guidance content" refers to information provided to a user, including route information and other instructions.

[1211] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. This system is composed of a user, a terminal (a smartphone and smart glasses), a server, and an emotion engine. Below, an embodiment of the present invention will be described in detail.

[1212] System Overview

[1213] The user inputs their destination using a smartphone application. This information is sent from the smartphone to a server. The server uses the Google Maps API to calculate the optimal route and sends that information to the smartphone. The smartphone then synchronizes the received route information with the smart glasses. The smart glasses use a built-in camera to determine the user's current location and direction in real time, and provide directions by displaying arrows and text in the user's field of view.

[1214] Server Processing

[1215] The server calculates the optimal route based on the destination information sent by the user. The server calls the Google Maps API to calculate the optimal route based on the destination and current location, taking into account real-time information such as traffic conditions and weather data. The calculated route information is also sent to the smartphone using encryption technology.

[1216] Terminal processing (smartphone)

[1217] The smartphone sends destination information entered by the user to the server. It uses Bluetooth and Wi-Fi to synchronize route information received from the server with the smart glasses. When a voice command is sent from the smart glasses, the smartphone also sends that information to the server. It also receives the user's emotional state from the emotion engine and requests appropriate route guidance from the server.

[1218] Terminal processing (smart glasses)

[1219] Smart glasses use a camera to determine the user's current location and direction in real time. They analyze the captured video and GPS data to determine the user's current location and direction. As a result, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also send voice commands to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is sent from the smart glasses to a server via the smartphone, and new instructions are generated and displayed on the smart glasses.

[1220] Emotion engine processing

[1221] The emotion engine analyzes the user's emotional state based on data obtained from the smart glasses' camera and microphone. Voice tone and facial expression data are used for the analysis. It determines whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotional information is sent to the smartphone and then forwarded to the server. The server uses this information to provide flexible route guidance and warnings according to the user's emotional state.

[1222] Specific example explanation

[1223] 1. If you're heading to the cafe by bicycle

[1224] The user enters the location of the cafe into their smartphone. The smartphone sends this information to the server, which uses the Google Maps API to calculate the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and provide directions using text and arrows, such as "Turn right in 100 meters." At the same time, if the emotion engine detects the user's anxiety, additional warnings are given.

[1225] Prompt Sentence Examples

[1226] Explain the steps and specific actions to take to get from your home to a nearby cafe using a bicycle, smart glasses, and a smartphone.

[1227] 2. When walking to the station on a crowded street at night

[1228] The user enters the "station location" into their smartphone. The smartphone sends this information to the server. The server uses the Google Maps API to calculate the optimal route, taking into account traffic conditions and safety at night. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins moving. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching." If the emotion engine detects that the user is stressed, a voice prompt to calm them down is played.

[1229] Prompt Sentence Examples

[1230] Explain the specific steps to safely receive directions using smart glasses and a smartphone when walking down a crowded street at night to the station.

[1231] Through these examples, the system of the present invention provides a versatile means for users to receive route guidance safely and efficiently. By utilizing unobstructed route guidance and voice commands, users can enjoy safety and convenience while traveling. Furthermore, the emotion engine enables flexible guidance based on the user's emotions.

[1232] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1233] Step 1:

[1234] The user operates the smartphone to input a destination. The user opens the smartphone's application and inputs the name and address of the destination. For example, the user inputs "the location of the cafe." This input information is saved on the smartphone as data about the location the user wants to go to.

[1235] Input: Destination information (cafe name and address)

[1236] Output: Destination information saved on your smartphone

[1237] Specific operation:

[1238] The user launches an application on the smartphone.

[1239] The user enters the "location of the cafe" in the text box.

[1240] Step 2:

[1241] The device (smartphone) sends destination information to the server. The smartphone sends destination information entered by the user to the server. Communication is usually via Wi-Fi or mobile data networks.

[1242] Input: Destination information saved on your smartphone

[1243] Output: Destination information sent to the server

[1244] Specific operation:

[1245] The smartphone sends destination information in packet format to the server.

[1246] Encryption technology is used to ensure the stability of data communication.

[1247] Step 3:

[1248] The server calculates the optimal route. Based on the destination information received, the server uses external services such as Google Maps API to calculate the optimal route, while taking into account real-time information such as traffic conditions and weather data.

[1249] Input: Destination information, current traffic conditions, weather data

[1250] Output: Optimal route information

[1251] Specific operation:

[1252] The server calls the Google Maps API and starts route calculation.

[1253] Route information is updated in real time based on traffic and weather data.

[1254] Step 4:

[1255] The server sends route information to the device (smartphone). The calculated route information is sent to the smartphone in encrypted packet format. The smartphone receives and stores this information.

[1256] Input: Optimal route information

[1257] Output: Route information sent to your smartphone

[1258] Specific operation:

[1259] The server encrypts the route information and sends it to your smartphone.

[1260] The smartphone decodes the received route information and stores it in its internal storage.

[1261] Step 5:

[1262] The device (smartphone) synchronizes the route information with the smart glasses. The smartphone then synchronizes the received route information with the smart glasses via Bluetooth or Wi-Fi.

[1263] Input: Route information saved on your smartphone

[1264] Output: Route information synchronized to smart glasses

[1265] Specific operation:

[1266] Your smartphone connects to the smart glasses using Bluetooth or Wi-Fi.

[1267] Transfer the route information to your smart glasses and check the synchronization.

[1268] Step 6:

[1269] Smart glasses acquire location information. Using a built-in camera and GPS, smart glasses determine the user's location in real time and determine the direction of travel.

[1270] Input: Camera image, GPS data

[1271] Output: User's current location

[1272] Specific operation:

[1273] The smart glasses use a camera to capture images of the surrounding area.

[1274] Use the built-in GPS sensor to determine your current location.

[1275] Step 7:

[1276] The smart glasses provide route guidance. Based on the acquired current location information, the smart glasses display arrows and text in the user's field of view to provide route guidance to the user. For example, specific instructions such as "Turn right in 100 meters" are displayed.

[1277] Input: User's current location information, route information

[1278] Output: Directions displayed in the field of view

[1279] Specific operation:

[1280] The smart glasses generate arrows and text based on route information.

[1281] Instructions such as "Turn right in 100 meters" are displayed in the user's field of vision.

[1282] Step 8:

[1283] The emotion engine analyzes the user's emotional state. The emotion engine analyzes voice tone and facial expressions based on data obtained from the smart glasses' camera and microphone to determine the user's emotional state.

[1284] Input: Voice data, facial expression data

[1285] Output: User's emotional state information

[1286] Specific operation:

[1287] The emotion engine analyzes the tone of the voice to determine whether the user is feeling anxious.

[1288] Identify emotional states based on facial expressions captured by a camera.

[1289] Step 9:

[1290] Providing guidance based on emotional information. The server generates appropriate guidance content based on the user's emotional information obtained from the emotion engine and sends it to the smart glasses. This provides flexible guidance according to the user's emotional state.

[1291] Input: User's emotional state information

[1292] Output: Guidance information according to emotions

[1293] Specific operation:

[1294] The server generates messages based on the emotional information to provide special attention or reassurance.

[1295] The smart glasses will display a message such as "You are almost at your destination. Proceed safely."

[1296] As described above, through these processing steps, the system provides the user with safe and efficient route guidance, and is also able to respond flexibly to the user's emotional state.

[1297] (Application example 2)

[1298] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1299] In self-driving vehicles, conventional navigation systems only provide route guidance, making it difficult to provide instructions that fully consider the emotional state and safety of passengers while driving. There is also a need for a system that can reduce the mental burden on drivers and enable them to reach their destination more safely. It is necessary to sense the anxiety and stress felt by users while driving in real time and respond appropriately.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1301] In this invention, the server includes means for allowing the user to set a destination and route, means for indicating the optimal route based on route information received from the server, means for acquiring and analyzing the user's current location and scenery in real time, means for assisting the user's vision using a visual device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, and means for adjusting operation or issuing additional warnings based on the user's emotional state using emotion analysis means, thereby enabling safer and more secure automated driving that is responsive to the driver's emotional state.

[1302] A "user" is a vehicle occupant who uses the system to set a destination and route and receives navigation.

[1303] A "destination" is the final location or place that a user wishes to reach.

[1304] A "route" is the path or direction of travel to reach a destination.

[1305] A "means" is a method, device, or system for achieving a specific purpose.

[1306] A "server" is a central computer system that performs route calculations and real-time data processing.

[1307] "Real time" refers to the time frame for immediate processing and response to ongoing events.

[1308] "Analysis" is the process of examining and processing information or data in detail to extract meaning.

[1309] A "visual device" is a device that supports a user's vision and presents information in real time.

[1310] An "arrow" is an indicator displayed on a visual device to indicate the direction in which the user should move.

[1311] "Text" means written information displayed on a visual device to provide guidance or instructions to a user.

[1312] A "voice command" is an instruction or question given by a user through speech.

[1313] "Emotion analysis" is the process of recognizing and analyzing a user's emotional state based on their voice and facial expression data.

[1314] "Operation adjustment" refers to changing or optimizing the autonomous driving speed, route, settings, etc. based on the user's emotional state.

[1315] "Attention" refers to a visual or audio warning or instruction to alert the user.

[1316] The above are definitions of important words included in the claims.

[1317] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1318] System Overview

[1319] This system consists of a user, a terminal (the central control device of the autonomous vehicle), a server, and an emotion analysis means. The user sets a destination and route and gets into the autonomous vehicle. The terminal receives destination and route information from the server and updates it in real time. The terminal uses a camera and microphone to obtain the user's current location information and emotional state, and provides route guidance through a visual device. The emotion analysis means also has the function of analyzing the user's voice and facial expressions to recognize the user's emotional state.

[1320] Hardware and Software

[1321] Hardware

[1322] Autonomous vehicle: central control unit, camera, microphone, GPS

[1323] Server: Responsible for route calculation and real-time data acquisition

[1324] software

[1325] Google Maps API: Used for route calculation

[1326] Emotion analysis tools: AI models that analyze voice tone and facial expressions (e.g., developed using OpenCV and TensorFlow)

[1327] Real-time data processing: APIs used to obtain traffic and weather data

[1328] Program processing

[1329] Server Processing

[1330] The server calculates the optimal route based on the destination information set by the user. It receives destination information sent from the central control unit of the autonomous vehicle and calculates the optimal route using external services such as Google Maps API. It sends the calculated route information to the autonomous vehicle and updates the route based on real-time information. It also generates emotion-responsive driving instructions based on emotion data received from the emotion analysis means.

[1331] Terminal processing (central control unit of autonomous vehicles)

[1332] The central control unit of the autonomous vehicle controls autonomous driving based on route information received from the server. It also passes data acquired from the camera and microphone to an emotion analysis means to analyze the user's emotional state in real time. Based on the emotional information acquired from the emotion analysis means, it adjusts driving and issues warnings to improve safety. For example, if the user feels anxious, it may adjust the driving speed or play relaxing music.

[1333] Processing of sentiment analysis tools

[1334] The emotion analysis means analyzes the user's voice and facial expression data to recognize their emotional state. Specifically, it uses data acquired from the camera and microphone to determine whether the user is feeling anxiety, impatience, joy, or other emotions. This information is sent to the central control unit and server in real time and used to adjust operation and provide warnings.

[1335] Specific examples

[1336] 1. Dealing with stressed drivers

[1337] If the autonomous vehicle determines through emotion analysis that the driver is feeling stressed, the system will play relaxing music and suggest the best place to rest, for example, "There is a rest area nearby, so we recommend you take a 10-minute break."

[1338] 2. Examples of prompts

[1339] "Design a system that provides route guidance and warnings to help drivers relax. The server calculates the optimal route taking into account traffic conditions and weather, and the vehicle processes emotion data in real time and responds appropriately based on the emotion analysis results while driving."

[1340] The above is a specific embodiment for carrying out the invention. This system realizes safe and secure automated driving according to the emotional state of the user.

[1341] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1342] Step 1:

[1343] The user sets the destination.

[1344] Input: The user inputs a destination into the central control unit of the autonomous vehicle.

[1345] Data processing and calculation: The central control unit receives the input destination information and compares it with GPS information.

[1346] Output: Sends the set destination information to the server.

[1347] Step 2:

[1348] The server calculates the optimal route.

[1349] Input: Destination information sent from the autonomous vehicle.

[1350] Data processing and calculation: Using the Google Maps API, the optimal route from the current location to the destination is calculated, taking into account traffic conditions and weather data, and optimization is performed in real time.

[1351] Output: Send the calculated optimal route information to the central control unit of the autonomous vehicle.

[1352] Step 3:

[1353] The self-driving vehicle will begin operating based on the route information.

[1354] Input: Optimal route information received from the server.

[1355] Data processing and calculation: The central control unit controls the operation based on the route information.

[1356] Output: Actual driving control of the vehicle.

[1357] Step 4:

[1358] The camera and microphone capture information about the surroundings and the user's emotions.

[1359] Input: Video data captured by the camera of the autonomous vehicle, audio data recorded by the microphone.

[1360] Data processing and calculation: The emotion analysis means analyzes the video data and audio data in real time to extract the user's emotional state.

[1361] Output: Sending the user's emotional information to the central control unit and server.

[1362] Step 5:

[1363] The server adjusts operations and issues warnings based on emotional information.

[1364] Input: User emotion information received from emotion analysis means.

[1365] Data processing and calculation: The server analyzes the emotional information and generates instructions and warning messages to adjust operations.

[1366] Output: Sends generated instructions and reminder messages to the autonomous vehicle.

[1367] Step 6:

[1368] The autonomous vehicle will carry out instructions and give warnings.

[1369] Input: Instructions and reminder messages sent by the server.

[1370] Data processing and calculation: The central control unit adjusts operation according to instructions, plays relaxing music, suggests resting spots, etc.

[1371] Output: Visual and audio guidance to the user and operational coordination.

[1372] The above is the processing flow of the program for the system for carrying out the invention. By performing appropriate processing and calculations based on the data input at each step and generating output, safe and efficient autonomous driving that responds to the user's emotional state becomes possible.

[1373] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1374] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1375] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1376] [Fourth embodiment]

[1377] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1378] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1379] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1380] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1381] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1382] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1383] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1384] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1385] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1386] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1387] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1388] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1389] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to exemplary embodiments.

[1391] System Overview

[1392] This system consists of a user, a device (a smartphone and smart glasses), and a server. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time.

[1393] Program processing

[1394] Server Processing

[1395] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[1396] Terminal processing (smartphone)

[1397] The smartphone sends the destination information entered by the user to the server. Upon receiving route information from the server, the smartphone synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis to the server. Specifically, when a user says "I want to change the route" to the smart glasses, the voice is transferred to the smartphone, analyzed, and then sent to the server.

[1398] Terminal processing (smart glasses)

[1399] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[1400] Specific examples

[1401] 1. If you're heading to the cafe by bicycle

[1402] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bike. The smart glasses use a camera to determine their current location and show turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view.

[1403] 2. When walking to the station on a crowded street at night

[1404] The user inputs the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, they display warnings such as "Watch your left. A car is approaching."

[1405] As described above, the system of the present invention provides a multifunctional means for users to receive route guidance safely and efficiently. In particular, the unobstructed route guidance and voice command operation function allow users to enjoy greater safety and convenience while traveling.

[1406] The processing flow will be explained below.

[1407] Server Processing Steps

[1408] Step 1:

[1409] The server receives the destination information sent by the user via the smartphone.

[1410] Step 2:

[1411] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[1412] Step 3:

[1413] The server sends the calculated route information to the smartphone.

[1414] Step 4:

[1415] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[1416] Step 5:

[1417] The server resends the updated route information to the smartphone.

[1418] Smartphone processing steps

[1419] Step 1:

[1420] The user launches a dedicated app on their smartphone and inputs their destination.

[1421] Step 2:

[1422] The smartphone transmits the input destination information to the server.

[1423] Step 3:

[1424] The smartphone synchronizes the route information received from the server with the smart glasses.

[1425] Step 4:

[1426] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[1427] Step 5:

[1428] The smartphone transmits the analyzed voice data to the server as text.

[1429] Smart Glasses Processing Steps

[1430] Step 1:

[1431] The user puts on the smart glasses and starts moving.

[1432] Step 2:

[1433] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[1434] Step 3:

[1435] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[1436] Step 4:

[1437] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[1438] Step 5:

[1439] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[1440] User processing steps

[1441] Step 1:

[1442] The user inputs the destination into the smartphone and sets the route.

[1443] Step 2:

[1444] The user puts on the smart glasses and begins moving along the specified route.

[1445] Step 3:

[1446] Users can check their direction by following the arrows and text displayed through the smart glasses.

[1447] Step 4:

[1448] Users can use voice commands to change routes or ask questions as needed.

[1449] Step 5:

[1450] The user follows the new instructions from the smart glasses to reach their destination.

[1451] The above is the specific flow of each processing step of this system.

[1452] Example 1

[1453] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1454] Conventional navigation systems do not adequately ensure safety and efficiency in the process of users reaching their destination, and often fail to provide appropriate guidance, especially on complex routes or in congested environments. They also lack the ability to provide route guidance that takes into account real-time changes in traffic conditions, weather, and other environmental factors, resulting in inconvenience for users. Furthermore, they lack an interface that utilizes voice commands, making hands-free operation difficult.

[1455] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1456] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for using a camera to assist the user's vision and for providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing video data acquired by the camera to identify the user's current location and direction of travel, means for analyzing voice commands and transmitting them to the server, means for recalculating the route under specific conditions (e.g., traffic conditions or weather) and providing the latest route information to the terminal, and means for synchronizing data between terminals using wireless communication, thereby enabling a user to receive real-time route guidance to reach their destination safely and efficiently.

[1457] A "destination" is a location that a user ultimately wants to reach.

[1458] A "route" is information about a route to reach a destination.

[1459] A "server" is a computer device that processes and stores data on a network.

[1460] The term "terminal" refers to any device that is directly operated by a user, and in this invention includes smartphones, smart glasses, etc.

[1461] "Route information" refers to the optimal route calculated by the server to reach the destination.

[1462] "Current location" refers to the location where the user is currently located.

[1463] "Surrounding information" refers to environmental information around the user's current location, including traffic conditions and weather.

[1464] The term "photography device" refers to a device that captures images or videos, and includes a camera in this invention.

[1465] A "voice command" is an instruction that a user inputs by voice.

[1466] "Analysis" refers to the process of processing acquired data to extract useful information.

[1467] "Synchronization" refers to the act of matching information to unify data across multiple terminals.

[1468] MODE FOR CARRYING OUT THE INVENTION

[1469] The present invention relates to a route guidance system that enables a user to reach a destination safely and efficiently. The system is composed of a user, a terminal (a smartphone and smart glasses), and a server.

[1470] System configuration

[1471] The user inputs their destination using their smartphone, and the information is sent to the server. The server then calculates the optimal route based on the received destination information using external services such as Google Maps API. The calculated route information is sent to the smartphone, which then synchronizes it with the smart glasses.

[1472] Smart glasses use a camera to determine the user's current location and provide directions by displaying arrows and text in the user's field of view. Communication between the smart glasses and smartphones is via wireless communication such as Bluetooth or Wi-Fi. Smart glasses also accept voice commands and transmit the voice data to the smartphone. The smartphone analyzes the voice data and sends it to a server to provide new routes and additional information.

[1473] Specific examples

[1474] Example 1: Cycling to a cafe

[1475] The user enters the location of the cafe into their smartphone. The smartphone sends the information to a server, which then calls the Google Maps API to calculate the optimal cycling route. The route information calculated by the server is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts cycling. The smart glasses use a camera to determine the user's current location in real time and display arrows and text such as "Turn right in 100 meters."

[1476] Example 2: Walking to the station on a crowded street at night

[1477] The user inputs the "station location" into their smartphone. The smartphone sends this information to the server, which then calls the Google Maps API to calculate a safe walking route. The server sets the route, taking into account traffic conditions and nighttime safety. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins walking. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching."

[1478] Prompt Sentence Examples

[1479] Enter "I want to go to XX Cafe" into your smartphone as your destination and start navigation. Also, if you say "I want to change the route" using the voice command, the server will calculate a new route and new instructions will be displayed on the smart glasses.

[1480] This system allows users to enjoy greater safety and convenience while traveling. With many features, such as real-time traffic and weather updates and the ability to operate using voice commands, users can receive optimal guidance in a variety of environments and situations.

[1481] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1482] Processing step details

[1483] Step 1:

[1484] The user starts up the smartphone and opens the navigation app. The user inputs their destination using the keyboard or touch screen. For example, they input the destination name, such as "cafe." The input data is converted to JSON format within the smartphone. At this point, the JSON format destination information is generated as input data (destination information).

[1485] Step 2:

[1486] The smartphone sends the destination information entered by the user to the server. Specifically, it uses an HTTP POST request to send the destination information to the server. The smartphone attaches the data entered by the user to the HTTP request and sends it to the server. The output is that the server receives the destination information.

[1487] Step 3:

[1488] The server calculates the optimal route based on the received destination information. This is achieved by calling a third-party API (e.g., Google Maps API). The server sends the destination information in an appropriate format to the API and obtains the optimal route information. The input is the received destination information, and the output is the calculated route information (in JSON format).

[1489] Step 4:

[1490] The server sends the calculated route information in JSON format to the smartphone. The server creates an HTTP response, attaches the route information, and sends it to the smartphone. The smartphone receives the HTTP response and obtains the route information. The output is the JSON-formatted route information received by the smartphone.

[1491] Step 5:

[1492] The smartphone synchronizes the received route information with the smart glasses. The smartphone transmits the route information to the smart glasses using wireless communication such as Bluetooth or Wi-Fi. The input is the route information received from the server, and the output is the route information received by the smart glasses.

[1493] Step 6:

[1494] Smart glasses use a built-in camera to determine the user's current location and direction of travel. Specifically, they use computer vision technology to analyze the video data captured by the camera and determine the user's current location and direction of travel. The input is the video data captured by the camera, and the output is the current location information and direction of travel information as the analysis results.

[1495] Step 7:

[1496] The smart glasses provide route guidance using arrows and text overlays based on the analyzed current location and direction of travel. Specifically, they display route guidance such as "Turn right in 100 meters" in the user's field of view. The input is the current location and direction of travel as analyzed results, and the output is the arrows and text displayed in the user's field of view.

[1497] Step 8:

[1498] The user inputs a voice command, for example, "I want to change my route" or "Where is the next turn?" The input is the user's voice command, and the output is the voice data captured by the smart glasses' internal microphone.

[1499] Step 9:

[1500] The smart glasses transmit the captured audio data to a smartphone using Bluetooth communication. The input is the audio data captured by the smart glasses, and the output is the audio data received by the smartphone.

[1501] Step 10:

[1502] The smartphone analyzes the received voice data and sends it to the server. It converts the voice data into text using voice recognition software and sends the text information to the server as an HTTP request. The input is the voice data received from the smart glasses, and the output is the analyzed text information and a request to send it to the server.

[1503] Step 11:

[1504] The server calculates a new route based on the received text information and sends that information to the smartphone. It then calls the Google Maps API again to obtain the new route information and sends it back to the smartphone. The input is the parsed text information, and the output is the recalculated route information.

[1505] Step 12:

[1506] The smartphone synchronizes the new route information received from the server with the smart glasses, and the smart glasses display new instructions to the user based on that information. The input is the recalculated route information received from the server, and the output is the new route guidance information displayed on the smart glasses.

[1507] (Application example 1)

[1508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] Conventional navigation systems have the problem of making it difficult for users to reach their destination safely and efficiently while traveling. Self-driving vehicles, in particular, are required to provide appropriate route guidance while grasping the surrounding situation in real time. Furthermore, systems are susceptible to external factors such as traffic conditions and weather changes, and may not update routes in a timely manner. To solve these problems, a system using more advanced technology is required.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1511] In this invention, the server includes means for allowing a user to set a destination and route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's vision using a camera and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle, and means for updating the route based on real-time updates of traffic conditions and weather information, thereby enabling a user to reach their destination safely and efficiently in an autonomous vehicle.

[1512] The "means for the user to set a destination and a route" is an interface device for inputting destination information and requesting the server for the optimal route.

[1513] The "means for indicating the optimum route based on the route information received from the server" is a device that analyzes the route information sent from the server and provides guidance to the user on the optimum direction of travel.

[1514] "Means for acquiring and analyzing information about the user's current location and surroundings in real time" refers to a device that uses cameras and sensors to collect and analyze information about the user's location and surrounding environment.

[1515] "Means for using a camera to assist the user's field of vision and providing route guidance using arrows and text" refers to a device that guides the user by displaying arrows and text based on images captured by a camera.

[1516] The "means for analyzing voice commands and updating instructions in response to route changes or questions" refers to a device that recognizes a user's voice input and updates route information and navigation instructions based on the analysis results.

[1517] "Means for providing visual instructions and warnings to the user using a camera and display mounted on the vehicle" refers to a device that monitors the surrounding situation using a camera inside the vehicle and provides visual information to the user through a display.

[1518] The "means for updating the route based on real-time updates of traffic conditions and weather information" refers to a device that analyzes traffic and weather information obtained from outside and updates the route information accordingly.

[1519] System Overview

[1520] As an embodiment of the present invention, a navigation system for an autonomous vehicle is taken as an example. This system is composed of a user, a terminal (an in-vehicle display and a smartphone), and a server. The user can set a destination and receive route guidance. The terminal receives destination and route information from the server and updates it in real time. The terminal also uses a camera to obtain information on the user's current location and surrounding conditions, and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the terminal in real time.

[1521] Program processing

[1522] 1. Server Processing

[1523] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into the in-car display or smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. The calculated route information is then sent to the in-car display and smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[1524] 2. Terminal processing (in-vehicle displays and smartphones)

[1525] The in-car display and smartphone send the destination information entered by the user to the server. When route information is received from the server, it is displayed on the in-car display and smartphone. The device also uses a camera to acquire the current location in real time and analyzes the direction of travel. The user receives visual instructions and warnings through the device. For example, instructions such as "Turn left 100 meters ahead" are displayed on the in-car display.

[1526] 3. Analyzing voice commands and updating instructions

[1527] The vehicle is equipped with a voice recognition microphone, and when the user enters a voice command such as "I want to change my route," the voice is sent as a request to the server. The server analyzes the voice command, generates a new route and instructions, and displays them on the in-vehicle display. If the user asks a question, the server also provides answers both audibly and visually.

[1528] Hardware and software used

[1529] Hardware

[1530] Cameras, microphones, in-vehicle displays, smartphones

[1531] software

[1532] Server: Route calculation and data update (Google Maps API, TensorFlow)

[1533] Smartphone app: Set destination, receive route (Android / iOS)

[1534] In-car display: Real-time information display (Android Auto, Apple CarPlay)

[1535] Speech analysis: Analysis and conversion of voice commands (Google Cloud Speech-to-Text, Amazon Transcribe)

[1536] Specific examples

[1537] Traveling within cities in autonomous vehicles

[1538] When a user sets a "station" as their destination on their smartphone, the in-car display shows the "optimal route to the station." If traffic conditions change along the way, the route is updated in real time, and instructions such as "Change the next left turn to a right turn" are displayed.

[1539] Example prompts for generative AI models

[1540] "How does the navigation system react when a user sets a train station as a destination on their smartphone? Also, please explain in detail how the system updates the route if traffic information changes along the way."

[1541] This system enables autonomous vehicles to provide safer and more efficient route guidance to users.

[1542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1543] Step 1:

[1544] The user inputs the destination into a smartphone or on-board display. The user sets the destination using a touchscreen or voice input. This input is recorded in the device as the destination's coordinates and name (e.g., "station").

[1545] Step 2:

[1546] The device sends destination information to the server. The smartphone or in-car display sends the destination information entered by the user to the server as an HTTP request. The server receives the input and analyzes the destination coordinates.

[1547] Step 3:

[1548] The server calculates the optimal route and generates route information. The server calls the Google Maps API and calculates the optimal route based on the destination coordinates. In doing so, it also takes into account traffic conditions and weather data when selecting the route. Route information is generated as detailed information for each step (e.g., distance to the next right turn).

[1549] Step 4:

[1550] The server sends the generated route information to the device. The route information is sent as an HTTP response to the in-vehicle display and smartphone. The device analyzes the received information and prepares to display it visually to the user.

[1551] Step 5:

[1552] The device uses a camera to obtain the user's current location and direction of travel in real time. The onboard camera captures video data, and the device analyzes the video in real time. Image recognition technology (e.g., OpenCV) is used to identify the location and direction of travel.

[1553] Step 6:

[1554] The device displays visual instructions, such as arrows and text overlaid on the camera image, in real time in the user's field of view.

[1555] Step 7:

[1556] The server periodically updates traffic and weather information and recalculates route information as needed. The server calls external APIs (e.g., traffic information API, weather API) at regular intervals to obtain the latest information. When new information is obtained, it recalculates route information as needed and sends it to the device.

[1557] Step 8:

[1558] The user inputs a voice command. While driving, the user inputs a voice command such as "I want to change the route." This command is transmitted to the terminal through a microphone in the vehicle.

[1559] Step 9:

[1560] The device analyzes the voice command and sends a request to the server. The voice data is converted into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then analyzed. After that, a new request is sent to the server based on the analysis results.

[1561] Step 10:

[1562] The server generates new route information and sends it to the device. The server receives the new route request and recalculates the route. The generated route information is sent to the device in real time and displayed to the user.

[1563] This program's processing allows users to reach their destination safely and efficiently based on the latest route information.

[1564] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1565] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the embodiments of the present invention.

[1566] System Overview

[1567] This system consists of a user, a device (smartphone and smart glasses), a server, and an emotion engine. The user can set a destination and receive route guidance. The device receives destination and route information from the server and updates it in real time. The device also uses a camera to obtain the user's current location and provides route guidance using arrows and text. The server calculates destination and route information and provides the latest information to the device in real time. The emotion engine has the ability to analyze the user's voice and facial expressions and recognize the user's emotional state.

[1568] Program processing

[1569] Server Processing

[1570] The server calculates the optimal route based on the destination information sent by the user. Specifically, when the user enters the destination into their smartphone, that information is sent to the server. The server receives that information and calculates the optimal route using external services such as Google Maps API. It then sends the calculated route information to the smartphone. The server also obtains real-time information such as traffic conditions and weather data and updates the route as needed.

[1571] Terminal processing (smartphone)

[1572] The smartphone sends destination information entered by the user to the server. Upon receiving route information from the server, it synchronizes the information with the smart glasses. The smartphone also analyzes microphone input (voice commands) sent from the smart glasses and sends the analysis results to the server. It also receives emotional information from the emotion engine and requests the server to provide appropriate route guidance based on the user's emotional state.

[1573] Terminal processing (smart glasses)

[1574] Smart glasses use a camera to determine the user's current location and direction of travel in real time. Specifically, they analyze the video captured by the camera to determine where the user is and which direction they are heading. Based on this information, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also have the ability to receive the user's voice commands and send them to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is transferred from the smart glasses to the smartphone, where it is analyzed by the server, and new instructions are generated and displayed again on the smart glasses.

[1575] Emotion engine processing

[1576] The emotion engine analyzes the user's voice and facial expressions to recognize their emotional state. For example, the emotion engine acquires voice tone and facial expression data from the smart glasses' camera and microphone, and analyzes this data to determine whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotion information is sent to the smartphone and then forwarded to the server. Based on this information, the server provides route guidance and warnings appropriate to the user's emotional state.

[1577] Specific examples

[1578] 1. If you're heading to the cafe by bicycle

[1579] The user enters the location of the cafe into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and indicate turning directions using arrows and text. For example, instructions such as "Turn right in 100 meters" are displayed in the smart glasses' field of view. At the same time, the emotion engine analyzes the user's tone of voice and, if it determines that the user is feeling anxious, an additional warning is given.

[1580] 2. When walking to the station on a crowded street at night

[1581] The user enters the station location into their smartphone. The smartphone sends this information to a server, which calculates the optimal route. The server also takes into account traffic conditions at night and selects a safe route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts walking. The smart glasses use a camera to obtain information about the surrounding area and provide directions using text and arrows. For example, warnings such as "Watch your left. A car is approaching" are displayed. If the emotion engine recognizes that the user's stress level is high, a voice prompt to calm them down is played.

[1582] As described above, the system of the present invention provides a multifunctional means for users to receive safe and efficient route guidance. In addition to unobstructed route guidance and voice command operation, the system's emotion engine allows for flexible guidance that responds to the user's emotions. This allows users to enjoy greater safety and convenience while traveling.

[1583] The processing flow will be explained below.

[1584] Server Processing Steps

[1585] Step 1:

[1586] The server receives the destination information sent by the user via the smartphone.

[1587] Step 2:

[1588] The server calls the Google Maps API based on the user's current location and destination information to calculate the optimal route.

[1589] Step 3:

[1590] The server sends the calculated route information to the smartphone.

[1591] Step 4:

[1592] The server periodically obtains real-time data such as traffic conditions and weather information and updates the route information.

[1593] Step 5:

[1594] The server receives the user's emotion information transferred from the emotion engine and adjusts the route instructions as necessary.

[1595] Step 6:

[1596] The server resends the updated route information to the smartphone.

[1597] Smartphone processing steps

[1598] Step 1:

[1599] The user launches a dedicated app on their smartphone and inputs their destination.

[1600] Step 2:

[1601] The smartphone transmits the input destination information to the server.

[1602] Step 3:

[1603] The smartphone synchronizes the route information received from the server with the smart glasses.

[1604] Step 4:

[1605] The smartphone receives microphone input (voice commands) from the smart glasses and analyzes the voice data.

[1606] Step 5:

[1607] The smartphone transmits the analyzed voice data to the server as text.

[1608] Step 6:

[1609] The smartphone transmits the user's facial and voice data acquired from the smart glasses to the emotion engine.

[1610] Step 7:

[1611] The user's emotional state information received from the emotion engine is transmitted to a server.

[1612] Smart Glasses Processing Steps

[1613] Step 1:

[1614] The user puts on the smart glasses and starts moving.

[1615] Step 2:

[1616] The smart glasses' camera captures the user's current location and surrounding environment in real time.

[1617] Step 3:

[1618] The smart glasses analyze the images captured by the camera and recognize the current location and direction of travel.

[1619] Step 4:

[1620] The smart glasses provide directions to users by displaying arrows and text in an unobstructed manner.

[1621] Step 5:

[1622] The microphone in the smart glasses receives voice commands from the user and transmits the voice data to the smartphone.

[1623] Step 6:

[1624] The smart glasses' camera and microphone analyze the user's facial expressions and tone of voice in real time and send the data to the emotion engine.

[1625] Emotion Engine Processing Steps

[1626] Step 1:

[1627] The emotion engine analyzes the user's voice tone and facial expressions received from the smart glasses.

[1628] Step 2:

[1629] The emotion engine recognizes the user's emotional state based on the analysis results.

[1630] Step 3:

[1631] The emotion engine sends the recognized emotion information to the smartphone.

[1632] User processing steps

[1633] Step 1:

[1634] The user inputs the destination into the smartphone and sets the route.

[1635] Step 2:

[1636] The user puts on the smart glasses and begins moving along the specified route.

[1637] Step 3:

[1638] Users can check their direction by following the arrows and text displayed through the smart glasses.

[1639] Step 4:

[1640] Users can use voice commands to change routes or ask questions as needed.

[1641] Step 5:

[1642] The user follows the new instructions from the smart glasses to reach their destination.

[1643] The above is the specific flow of each processing step of this system.

[1644] Example 2

[1645] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1646] Conventional navigation systems can sometimes make it difficult for users to find the optimal route to their destination, and they also lack real-time route adjustments and guidance that reflects the user's emotional state. In particular, they are unable to provide appropriate guidance to users who are feeling anxious or stressed, making it difficult to ensure the safety and convenience of users.

[1647] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to set a destination and a route, means for indicating an optimal route based on route information received from the server, means for acquiring and analyzing information about the user's current location and surroundings in real time, means for assisting the user's field of vision using an imaging device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, means for analyzing the user's emotional state, and means for flexibly adjusting the content of guidance based on the user's emotional state. This allows the user to receive optimal route guidance in real time and enables flexible guidance according to the user's emotional state, greatly improving safety and convenience.

[1648] "User" refers to an individual who uses the system to travel to a destination.

[1649] A "destination" refers to a specific location that a user wishes to reach.

[1650] A "route" refers to the path a user takes to reach a destination.

[1651] The term "server" refers to a computer system that receives information from a user, processes the data, generates route information, and transmits it to a terminal.

[1652] "Terminal" refers to a device used by a user, such as a smartphone or smart glasses.

[1653] "Route information" refers to data regarding the optimal route a user needs to take to reach a destination.

[1654] "Real-time" refers to instantly recognizing and reflecting the current situation or state.

[1655] The term "photography device" refers to a device such as a camera for acquiring information about the user's field of view and current location.

[1656] An "arrow" refers to a graphic that visually indicates the direction of travel to the user.

[1657] "Text" refers to a string of characters that provides information to the user.

[1658] "Voice command" refers to a means by which a user issues instructions to a system by voice.

[1659] "Emotional state" refers to the user's psychological state, and includes emotions such as anxiety, impatience, and joy.

[1660] "Analysis" refers to the detailed analysis of acquired data to clarify its meaning.

[1661] "Guidance content" refers to information provided to a user, including route information and other instructions.

[1662] The present invention combines an emotion engine with a system for allowing a user to receive route guidance safely and efficiently. This system is composed of a user, a terminal (a smartphone and smart glasses), a server, and an emotion engine. Below, an embodiment of the present invention will be described in detail.

[1663] System Overview

[1664] The user inputs their destination using a smartphone application. This information is sent from the smartphone to a server. The server uses the Google Maps API to calculate the optimal route and sends that information to the smartphone. The smartphone then synchronizes the received route information with the smart glasses. The smart glasses use a built-in camera to determine the user's current location and direction in real time, and provide directions by displaying arrows and text in the user's field of view.

[1665] Server Processing

[1666] The server calculates the optimal route based on the destination information sent by the user. The server calls the Google Maps API to calculate the optimal route based on the destination and current location, taking into account real-time information such as traffic conditions and weather data. The calculated route information is also sent to the smartphone using encryption technology.

[1667] Terminal processing (smartphone)

[1668] The smartphone sends destination information entered by the user to the server. It uses Bluetooth and Wi-Fi to synchronize route information received from the server with the smart glasses. When a voice command is sent from the smart glasses, the smartphone also sends that information to the server. It also receives the user's emotional state from the emotion engine and requests appropriate route guidance from the server.

[1669] Terminal processing (smart glasses)

[1670] Smart glasses use a camera to determine the user's current location and direction in real time. They analyze the captured video and GPS data to determine the user's current location and direction. As a result, they display arrows and text in the user's field of view to provide route guidance. Smart glasses also send voice commands to a smartphone. For example, if a user asks, "Where is the next turn?", the voice is sent from the smart glasses to a server via the smartphone, and new instructions are generated and displayed on the smart glasses.

[1671] Emotion engine processing

[1672] The emotion engine analyzes the user's emotional state based on data obtained from the smart glasses' camera and microphone. Voice tone and facial expression data are used for the analysis. It determines whether the user is feeling anxiety, impatience, joy, or other emotions. The recognized emotional information is sent to the smartphone and then forwarded to the server. The server uses this information to provide flexible route guidance and warnings according to the user's emotional state.

[1673] Specific example explanation

[1674] 1. If you're heading to the cafe by bicycle

[1675] The user enters the location of the cafe into their smartphone. The smartphone sends this information to the server, which uses the Google Maps API to calculate the optimal route. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and starts riding their bicycle. The smart glasses use a camera to determine their current location and provide directions using text and arrows, such as "Turn right in 100 meters." At the same time, if the emotion engine detects the user's anxiety, additional warnings are given.

[1676] Prompt Sentence Examples

[1677] Explain the steps and specific actions to take to get from your home to a nearby cafe using a bicycle, smart glasses, and a smartphone.

[1678] 2. When walking to the station on a crowded street at night

[1679] The user enters the "station location" into their smartphone. The smartphone sends this information to the server. The server uses the Google Maps API to calculate the optimal route, taking into account traffic conditions and safety at night. The calculated route information is sent to the smartphone and then synchronized with the smart glasses. The user puts on the smart glasses and begins moving. The smart glasses use a camera to obtain information about the surrounding area and display warnings such as "Watch your left hand. A car is approaching." If the emotion engine detects that the user is stressed, a voice prompt to calm them down is played.

[1680] Prompt Sentence Examples

[1681] Explain the specific steps to safely receive directions using smart glasses and a smartphone when walking down a crowded street at night to the station.

[1682] Through these examples, the system of the present invention provides a versatile means for users to receive route guidance safely and efficiently. By utilizing unobstructed route guidance and voice commands, users can enjoy safety and convenience while traveling. Furthermore, the emotion engine enables flexible guidance based on the user's emotions.

[1683] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1684] Step 1:

[1685] The user operates the smartphone to input a destination. The user opens the smartphone's application and inputs the name and address of the destination. For example, the user inputs "the location of the cafe." This input information is saved on the smartphone as data about the location the user wants to go to.

[1686] Input: Destination information (cafe name and address)

[1687] Output: Destination information saved on your smartphone

[1688] Specific operation:

[1689] The user launches an application on the smartphone.

[1690] The user enters the "location of the cafe" in the text box.

[1691] Step 2:

[1692] The device (smartphone) sends destination information to the server. The smartphone sends destination information entered by the user to the server. Communication is usually via Wi-Fi or mobile data networks.

[1693] Input: Destination information saved on your smartphone

[1694] Output: Destination information sent to the server

[1695] Specific operation:

[1696] The smartphone sends destination information in packet format to the server.

[1697] Encryption technology is used to ensure the stability of data communication.

[1698] Step 3:

[1699] The server calculates the optimal route. Based on the destination information received, the server uses external services such as Google Maps API to calculate the optimal route, while taking into account real-time information such as traffic conditions and weather data.

[1700] Input: Destination information, current traffic conditions, weather data

[1701] Output: Optimal route information

[1702] Specific operation:

[1703] The server calls the Google Maps API and starts route calculation.

[1704] Route information is updated in real time based on traffic and weather data.

[1705] Step 4:

[1706] The server sends route information to the device (smartphone). The calculated route information is sent to the smartphone in encrypted packet format. The smartphone receives and stores this information.

[1707] Input: Optimal route information

[1708] Output: Route information sent to your smartphone

[1709] Specific operation:

[1710] The server encrypts the route information and sends it to your smartphone.

[1711] The smartphone decodes the received route information and stores it in its internal storage.

[1712] Step 5:

[1713] The device (smartphone) synchronizes the route information with the smart glasses. The smartphone then synchronizes the received route information with the smart glasses via Bluetooth or Wi-Fi.

[1714] Input: Route information saved on your smartphone

[1715] Output: Route information synchronized to smart glasses

[1716] Specific operation:

[1717] Your smartphone connects to the smart glasses using Bluetooth or Wi-Fi.

[1718] Transfer the route information to your smart glasses and check the synchronization.

[1719] Step 6:

[1720] Smart glasses acquire location information. Using a built-in camera and GPS, smart glasses determine the user's location in real time and determine the direction of travel.

[1721] Input: Camera image, GPS data

[1722] Output: User's current location

[1723] Specific operation:

[1724] The smart glasses use a camera to capture images of the surrounding area.

[1725] Use the built-in GPS sensor to determine your current location.

[1726] Step 7:

[1727] The smart glasses provide route guidance. Based on the acquired current location information, the smart glasses display arrows and text in the user's field of view to provide route guidance to the user. For example, specific instructions such as "Turn right in 100 meters" are displayed.

[1728] Input: User's current location information, route information

[1729] Output: Directions displayed in the field of view

[1730] Specific operation:

[1731] The smart glasses generate arrows and text based on route information.

[1732] Instructions such as "Turn right in 100 meters" are displayed in the user's field of vision.

[1733] Step 8:

[1734] The emotion engine analyzes the user's emotional state. The emotion engine analyzes voice tone and facial expressions based on data obtained from the smart glasses' camera and microphone to determine the user's emotional state.

[1735] Input: Voice data, facial expression data

[1736] Output: User's emotional state information

[1737] Specific operation:

[1738] The emotion engine analyzes the tone of the voice to determine whether the user is feeling anxious.

[1739] Identify emotional states based on facial expressions captured by a camera.

[1740] Step 9:

[1741] Providing guidance based on emotional information. The server generates appropriate guidance content based on the user's emotional information obtained from the emotion engine and sends it to the smart glasses. This provides flexible guidance according to the user's emotional state.

[1742] Input: User's emotional state information

[1743] Output: Guidance information according to emotions

[1744] Specific operation:

[1745] The server generates messages based on the emotional information to provide special attention or reassurance.

[1746] The smart glasses will display a message such as "You are almost at your destination. Proceed safely."

[1747] As described above, through these processing steps, the system provides the user with safe and efficient route guidance, and is also able to respond flexibly to the user's emotional state.

[1748] (Application example 2)

[1749] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1750] In self-driving vehicles, conventional navigation systems only provide route guidance, making it difficult to provide instructions that fully consider the emotional state and safety of passengers while driving. There is also a need for a system that can reduce the mental burden on drivers and enable them to reach their destination more safely. It is necessary to sense the anxiety and stress felt by users while driving in real time and respond appropriately.

[1751] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1752] In this invention, the server includes means for allowing the user to set a destination and route, means for indicating the optimal route based on route information received from the server, means for acquiring and analyzing the user's current location and scenery in real time, means for assisting the user's vision using a visual device and providing route guidance using arrows and text, means for analyzing voice commands and updating instructions in response to route changes or questions, and means for adjusting operation or issuing additional warnings based on the user's emotional state using emotion analysis means, thereby enabling safer and more secure automated driving that is responsive to the driver's emotional state.

[1753] A "user" is a vehicle occupant who uses the system to set a destination and route and receives navigation.

[1754] A "destination" is the final location or place that a user wishes to reach.

[1755] A "route" is the path or direction of travel to reach a destination.

[1756] A "means" is a method, device, or system for achieving a specific purpose.

[1757] A "server" is a central computer system that performs route calculations and real-time data processing.

[1758] "Real time" refers to the time frame for immediate processing and response to ongoing events.

[1759] "Analysis" is the process of examining and processing information or data in detail to extract meaning.

[1760] A "visual device" is a device that supports a user's vision and presents information in real time.

[1761] An "arrow" is an indicator displayed on a visual device to indicate the direction in which the user should move.

[1762] "Text" means written information displayed on a visual device to provide guidance or instructions to a user.

[1763] A "voice command" is an instruction or question given by a user through speech.

[1764] "Emotion analysis" is the process of recognizing and analyzing a user's emotional state based on their voice and facial expression data.

[1765] "Operation adjustment" refers to changing or optimizing the autonomous driving speed, route, settings, etc. based on the user's emotional state.

[1766] "Attention" refers to a visual or audio warning or instruction to alert the user.

[1767] The above are definitions of important words included in the claims.

[1768] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1769] System Overview

[1770] This system consists of a user, a terminal (the central control device of the autonomous vehicle), a server, and an emotion analysis means. The user sets a destination and route and gets into the autonomous vehicle. The terminal receives destination and route information from the server and updates it in real time. The terminal uses a camera and microphone to obtain the user's current location information and emotional state, and provides route guidance through a visual device. The emotion analysis means also has the function of analyzing the user's voice and facial expressions to recognize the user's emotional state.

[1771] Hardware and Software

[1772] Hardware

[1773] Autonomous vehicle: central control unit, camera, microphone, GPS

[1774] Server: Responsible for route calculation and real-time data acquisition

[1775] software

[1776] Google Maps API: Used for route calculation

[1777] Emotion analysis tools: AI models that analyze voice tone and facial expressions (e.g., developed using OpenCV and TensorFlow)

[1778] Real-time data processing: APIs used to obtain traffic and weather data

[1779] Program processing

[1780] Server Processing

[1781] The server calculates the optimal route based on the destination information set by the user. It receives destination information sent from the central control unit of the autonomous vehicle and calculates the optimal route using external services such as Google Maps API. It sends the calculated route information to the autonomous vehicle and updates the route based on real-time information. It also generates emotion-responsive driving instructions based on emotion data received from the emotion analysis means.

[1782] Terminal processing (central control unit of autonomous vehicles)

[1783] The central control unit of the autonomous vehicle controls autonomous driving based on route information received from the server. It also passes data acquired from the camera and microphone to an emotion analysis means to analyze the user's emotional state in real time. Based on the emotional information acquired from the emotion analysis means, it adjusts driving and issues warnings to improve safety. For example, if the user feels anxious, it may adjust the driving speed or play relaxing music.

[1784] Processing of sentiment analysis tools

[1785] The emotion analysis means analyzes the user's voice and facial expression data to recognize their emotional state. Specifically, it uses data acquired from the camera and microphone to determine whether the user is feeling anxiety, impatience, joy, or other emotions. This information is sent to the central control unit and server in real time and used to adjust operation and provide warnings.

[1786] Specific examples

[1787] 1. Dealing with stressed drivers

[1788] If the autonomous vehicle determines through emotion analysis that the driver is feeling stressed, the system will play relaxing music and suggest the best place to rest, for example, "There is a rest area nearby, so we recommend you take a 10-minute break."

[1789] 2. Examples of prompts

[1790] "Design a system that provides route guidance and warnings to help drivers relax. The server calculates the optimal route taking into account traffic conditions and weather, and the vehicle processes emotion data in real time and responds appropriately based on the emotion analysis results while driving."

[1791] The above is a specific embodiment for carrying out the invention. This system realizes safe and secure automated driving according to the emotional state of the user.

[1792] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1793] Step 1:

[1794] The user sets the destination.

[1795] Input: The user inputs a destination into the central control unit of the autonomous vehicle.

[1796] Data processing and calculation: The central control unit receives the input destination information and compares it with GPS information.

[1797] Output: Sends the set destination information to the server.

[1798] Step 2:

[1799] The server calculates the optimal route.

[1800] Input: Destination information sent from the autonomous vehicle.

[1801] Data processing and calculation: Using the Google Maps API, the optimal route from the current location to the destination is calculated, taking into account traffic conditions and weather data, and optimization is performed in real time.

[1802] Output: Send the calculated optimal route information to the central control unit of the autonomous vehicle.

[1803] Step 3:

[1804] The self-driving vehicle will begin operating based on the route information.

[1805] Input: Optimal route information received from the server.

[1806] Data processing and calculation: The central control unit controls the operation based on the route information.

[1807] Output: Actual driving control of the vehicle.

[1808] Step 4:

[1809] The camera and microphone capture information about the surroundings and the user's emotions.

[1810] Input: Video data captured by the camera of the autonomous vehicle, audio data recorded by the microphone.

[1811] Data processing and calculation: The emotion analysis means analyzes the video data and audio data in real time to extract the user's emotional state.

[1812] Output: Sending the user's emotional information to the central control unit and server.

[1813] Step 5:

[1814] The server adjusts operations and issues warnings based on emotional information.

[1815] Input: User emotion information received from emotion analysis means.

[1816] Data processing and calculation: The server analyzes the emotional information and generates instructions and warning messages to adjust operations.

[1817] Output: Sends generated instructions and reminder messages to the autonomous vehicle.

[1818] Step 6:

[1819] The autonomous vehicle will carry out instructions and give warnings.

[1820] Input: Instructions and reminder messages sent by the server.

[1821] Data processing and calculation: The central control unit adjusts operation according to instructions, plays relaxing music, suggests resting spots, etc.

[1822] Output: Visual and audio guidance to the user and operational coordination.

[1823] The above is the processing flow of the program for the system for carrying out the invention. By performing appropriate processing and calculations based on the data input at each step and generating output, safe and efficient autonomous driving that responds to the user's emotional state becomes possible.

[1824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1825] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1827] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1828] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1829] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1830] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1831] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1832] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1833] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1834] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1835] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1836] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1837] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1838] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1839] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1840] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1841] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1842] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1843] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1844] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1845] The following is further disclosed regarding the above embodiment.

[1846] (Claim 1)

[1847] means for a user to set a destination and a route;

[1848] means for indicating an optimal route based on route information received from a server;

[1849] A means of acquiring and analyzing the user's current location and surrounding information in real time;

[1850] A means for using a camera to assist the user's vision and provide route guidance using arrows and text;

[1851] A means of interpreting voice commands and updating instructions in response to route changes or questions;

[1852] A system including:

[1853] (Claim 2)

[1854] 2. The system according to claim 1, wherein the camera recognizes the user's current location and direction of travel and visually guides the user along the route.

[1855] (Claim 3)

[1856] 2. The system according to claim 1, wherein instructions input by voice commands are analyzed and transmitted to a server.

[1857] "Example 1"

[1858] (Claim 1)

[1859] means for a user to set a destination and a route;

[1860] means for indicating an optimal route based on route information received from a server;

[1861] A means of acquiring and analyzing the user's current location and surrounding information in real time;

[1862] a means for assisting the user's vision using a photographing device and providing route guidance using arrows and text;

[1863] A means of interpreting voice commands and updating instructions in response to route changes or questions;

[1864] means for analyzing video data acquired by the imaging device and identifying the user's current location and traveling direction;

[1865] means for analyzing and transmitting the voice command to a server;

[1866] A means to recalculate the route under certain conditions (e.g., traffic or weather) and provide updated route information to the device;

[1867] means for synchronizing data between terminals using wireless communication;

[1868] A system including:

[1869] (Claim 2)

[1870] 2. The system according to claim 1, wherein the image capturing device recognizes the user's current location and direction of travel and visually guides the user along the travel route.

[1871] (Claim 3)

[1872] 2. The system according to claim 1, wherein instructions input by voice commands are analyzed and transmitted to a server.

[1873] "Application Example 1"

[1874] (Claim 1)

[1875] means for a user to set a destination and a route;

[1876] means for indicating an optimal route based on route information received from a server;

[1877] A means of acquiring and analyzing the user's current location and surrounding information in real time;

[1878] A means for using a camera to assist the user's vision and provide route guidance using arrows and text;

[1879] A means of interpreting voice commands and updating instructions in response to route changes or questions;

[1880] means for providing visual instructions and warnings to a user using a camera and display mounted on the vehicle;

[1881] a means of updating the route based on real-time updates of traffic and weather information;

[1882] A system including:

[1883] (Claim 2)

[1884] 2. The system according to claim 1, wherein the camera ...

Claims

1. means for a user to set a destination and a route; means for indicating an optimal route based on route information received from a server; A means for acquiring and analyzing information about the user's current location and surroundings in real time; A means for using a camera to assist the user's vision and provide route guidance using arrows and text; A means of interpreting voice commands and updating instructions in response to route changes or questions; A system including:

2. The system according to claim 1, wherein the camera recognizes the user's current location and direction of travel and visually guides the user along the travel route.

3. 2. The system according to claim 1, wherein instructions input by voice commands are analyzed and transmitted to a server.

Images