System

Abstract

To provide a system.SOLUTION: A navigation system for a visually impaired person comprises: means for enabling conversation with a user using a natural language processing technique; means for generating a prompt text instructing a destination of the user based on the conversation; means for acquiring information on at least one a route to the destination and surrounding environment information, using the generated prompt text and a generative AI model; means for recognizing an emotional state of the user from the conversation using an emotion engine; means for adjusting a tone of voice guidance according to the emotional state of the user; and means for providing the voice guidance of the information with the adjusted tone to the visually impaired person in an automatically operated vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When visually impaired people move around, it is difficult for them to understand the route to their destination and grasp their surroundings, especially the number of stairs, the narrowness of the town, the condition of the footing, and the characteristics and colors of the surrounding buildings, which are difficult to imagine visually. [Means for solving the problem]

[0005] This invention provides a means to provide directions to a destination, a means to enable conversations with users using natural language processing technology, and a means to convey information about the surrounding environment to visually impaired people. The natural language processing technology uses a generative pre-training transformer (generative AI), and the surrounding environment information is obtained in conjunction with a map information service. This provides a sense of security when traveling to visually impaired people and makes it easier for them to understand their surroundings. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15]FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The navigation system for visually impaired people of the present invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[0029] "Example 2"

[0030] The navigation system for visually impaired people of the present invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[0031] The processing flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The user activates the system using a portable device such as a smartphone or wearable device.

[0034] Step 2: The user sets a destination using voice or touch input.

[0035] Step 3: The system connects with the map information service to calculate the optimal route to the set destination.

[0036] Step 4: The system will provide voice guidance for directions.

[0037] Step 5: Once the user begins their journey, if they have any questions or requests about the surrounding environment along the way, the system will respond through natural conversation using a generative pre-trained transformer (generative AI).

[0038] Step 6: The system uses information obtained from the map information service to provide a voice message describing the characteristics and colors of surrounding buildings.

[0039] Example 1

[0040] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] To reach their destination safely and efficiently, visually impaired people need detailed route guidance and information about their surrounding environment. However, conventional navigation systems have difficulty providing sufficient information to visually impaired people, particularly in terms of conveying information about the surrounding environment and building features. Furthermore, they lack the ability to respond to real-time questions and requests through natural conversation with the user. This often leads to anxiety and difficulty for visually impaired people when trying to reach their destination.

[0042] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0043] In this invention, the server includes: means for a user to set a destination; means for a terminal to transmit destination information to the server; means for the server to calculate an optimal route using a map information service; means for the server to transmit the route information calculated by the server to the terminal; means for the terminal to generate audio guidance based on the route information; means for the user to receive the audio guidance; means for the user to ask a question or make a request; means for the terminal to respond to the user's question or request using a generative AI model; means for the server to acquire surrounding environmental information and transmit it to the terminal; and means for the terminal to convey surrounding information to the user by voice based on the environmental information. This enables visually impaired people to receive detailed route guidance and surrounding environmental information by voice when heading to their destination, and can also respond to questions and requests in real time through natural conversation.

[0044] "User" refers to any individual who uses the navigation system, including visually impaired individuals.

[0045] "Terminal" refers to a portable device such as a smartphone or wearable device.

[0046] "Server" refers to a computer system that processes data using map information services and generative AI models.

[0047] "Destination information" refers to data relating to a destination set by a user.

[0048] "Map Information Service" refers to an online service that provides geographic information.

[0049] An "optimal route" refers to the most efficient route from your current location to your destination.

[0050] "Audio Guidance" means audio directions generated using text-to-speech software.

[0051] "Generative AI model" refers to natural language processing technology that uses generative pre-training transformers.

[0052] "Surrounding environmental information" refers to information such as the characteristics and color of a building, whether or not it has stairs, etc., obtained from map information services.

[0053] "Natural language processing technology" refers to language understanding and generation technology that enables conversation with a user.

[0054] This invention relates to a navigation system for the visually impaired, implemented on a portable device such as a smartphone or wearable device, that provides detailed route guidance and surrounding environmental information by voice as the user travels to their destination, and can respond to questions and requests in real time through natural conversation.

[0055] The server uses a map information service (e.g., Google® Maps API) to calculate the optimal route to the destination set by the user. The calculated route information is sent to the device over the Internet. The device uses text-to-speech software (e.g., Google Text-to-Speech®) to generate audio guidance based on the received route information.

[0056] Users can check the route to their destination through voice guidance. If they have questions or additional requests along the way, they can use voice input to ask questions or make requests. The device uses a generative pre-training transformer (e.g., OpenAI® GPT-3®) to generate natural-sounding responses to the user's questions and requests.

[0057] The server obtains information about the surrounding environment from a map information service and sends it to the device. Based on the obtained information, the device uses text-to-speech software to communicate information about the surroundings to the user aloud. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead."

[0058] As a specific example, when a user asks "Please tell me the way to the station," the following processing is performed.

[0059] 1. User: "Can you tell me the way to the station?"

[0060] 2. The device converts the voice input into text and sends it to the server.

[0061] 3. The server uses a map information service to calculate the optimal route and sends the route information to the device.

[0062] 4. The device generates voice guidance based on the route information and provides it to the user.

[0063] 5. User: "How far is the next intersection?"

[0064] 6. Using the generative AI model, the device responds, "The next intersection is approximately 200 meters away."

[0065] 7. The server acquires information about the surrounding environment and sends it to the device.

[0066] 8. The device will tell the user by voice, "There is a red brick building on your right."

[0067] In this way, navigation systems for the visually impaired provide users with directions to their destination through voice guidance and natural conversation.

[0068] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0069] Step 1:

[0070] The user sets the destination.

[0071] The user launches an application on their smartphone or wearable device and sets their destination using voice or text input. For example, they might say, "Please tell me the way to the station." The voice input is converted into text data by the device.

[0072] Step 2:

[0073] The terminal transmits the destination information to the server.

[0074] The terminal transmits the destination information entered by the user to the server as text data. Specifically, the destination information is transmitted to the server via the Internet. The input is text data, and the output is a transmission completion notification to the server.

[0075] Step 3:

[0076] The server uses a map information service to calculate the optimal route.

[0077] The server uses a map information service (e.g., Google Maps API (registered trademark)) based on the received destination information to calculate the optimal route from the current location to the destination. The input is destination information, and the output is optimal route information.

[0078] Step 4:

[0079] The server sends the calculated route information to the terminal.

[0080] The server converts the calculated route information into a data format and sends it to the terminal. The input is the optimal route information, and the output is a transmission completion notification to the terminal.

[0081] Step 5:

[0082] The device generates voice guidance based on the route information.

[0083] The device analyzes the received route information and generates voice guidance using text-to-speech software (e.g., Google Text-to-Speech (registered trademark)). The input is the route information, and the output is the voice guidance.

[0084] Step 6:

[0085] The user receives the audio guidance.

[0086] The user confirms the route to the destination through voice guidance provided by the device. For example, the user may receive instructions such as, "Turn right and go 100 meters until you see a red brick building. Turn left at the next intersection." The input is the voice guidance, and the output is the user's actions.

[0087] Step 7:

[0088] The user asks a question or makes a request.

[0089] If the user has any questions or additional requests along the way, they can use voice input to ask questions or make requests. For example, they can ask, "How far is it to the next intersection?" The input is voice input, and the output is text data.

[0090] Step 8:

[0091] The device uses the generative AI model to respond to the user's questions and requests.

[0092] The device uses a generative pre-training transformer (e.g., OpenAI® GPT-3®) to generate natural-sounding responses to user questions and requests. The input is the text data of the user's question or request, and the output is the text data of the response.

[0093] Step 9:

[0094] The server acquires information about the surrounding environment and sends it to the terminal.

[0095] The server obtains information about the surrounding environment from the map information service and sends it to the terminal. For example, it obtains information about the characteristics and colors of surrounding buildings, whether there are stairs, etc. The input is the environmental information from the map information service, and the output is a transmission completion notification to the terminal.

[0096] Step 10:

[0097] The device communicates surrounding information to the user via voice based on environmental information.

[0098] Based on the received environmental information, the device uses text-to-speech software to communicate information about the surroundings to the user aloud. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead." The input is environmental information, and the output is voice guidance.

[0099] (Application example 1)

[0100] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] When visually impaired people navigate or search for products in a physical store, they face challenges such as difficulty grasping information about their surroundings, making it difficult for them to understand the route to their destination and the layout of products. Furthermore, there is a lack of ways for visually impaired people to obtain information through natural conversation while navigating the store, which limits their in-store experience.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0103] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying information about the surrounding environment to a visually impaired person, means for guiding a user to a destination using map information within a store, and means for conveying information about the features and layout of products within the store and facility information by voice, thereby enabling visually impaired people to move around a physical store and search for products smoothly.

[0104] "Means of providing directions to the destination" refers to a function that calculates the optimal route to the destination set by the user and provides instructions through voice guidance, etc.

[0105] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses natural language processing technology such as generative pre-training transformers (generative AI) to respond to questions and requests from users through natural conversation.

[0106] "Means of conveying information about the surrounding environment to visually impaired people" is a function that uses audio to convey the characteristics and colors of surrounding buildings, the location of facilities, etc., based on information obtained from map information services.

[0107] "A means of guiding users to their destination using in-store map information" is a function that calculates the optimal route to the destination set by the user based on in-store map information and provides instructions via voice guidance.

[0108] "Means for communicating information about the features and layout of products in a store and about facilities by voice" is a function that communicates information about the features and layout of products and facilities in a store to users by voice.

[0109] The system for implementing this invention is designed to assist visually impaired people in navigating and locating products in physical stores. The system is implemented as an application installed on a portable device such as a smartphone or smart glasses.

[0110] Hardware and Software

[0111] Hardware

[0112] Smartphone

[0113] Smart Glasses

[0114] software

[0115] OpenAI API (registered trademark)

[0116] Map Information Service API

[0117] Data processing and calculation

[0118] A means of providing directions to a destination

[0119] The server uses the map information service API to obtain route information between the user's current location and the destination, and provides voice guidance for each step based on the obtained route information.

[0120] A means of enabling conversations with users using natural language processing technology

[0121] The server uses a generative pre-training transformer (generative AI) to generate natural responses to user questions, allowing users to obtain information through natural conversation.

[0122] A means of conveying information about the surrounding environment to visually impaired people

[0123] Based on information obtained from the map information service, the server will provide audio information such as the characteristics and colors of surrounding buildings and the location of facilities, allowing the user to understand the surrounding environment.

[0124] A method of guiding users to their destination using in-store map information

[0125] The server calculates the optimal route to the destination set by the user based on the store's map information and provides voice guidance, allowing the user to move smoothly within the store.

[0126] A means of communicating the features and layout of products in stores and facility information through audio

[0127] The server will then provide users with voice information about the features and locations of products and facilities within the store, allowing them to easily find the products and facilities they are looking for.

[0128] Specific examples

[0129] For example, if a user asks, "What's in the fresh vegetable section?", the generative AI will respond, "The fresh vegetable section has fresh vegetables such as tomatoes, lettuce, and cucumbers."

[0130] Prompt Sentence Examples

[0131] User: What's in the fresh produce section?

[0132] Generative AI: The fresh vegetable section is stocked with fresh vegetables such as tomatoes, lettuce, and cucumbers.

[0133] In this way, a system will be created that helps visually impaired people navigate and search for products smoothly within a physical store.

[0134] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0135] Step 1:

[0136] A user turns on their smartphone or smart glasses and starts a navigation application.

[0137] Input: User's current location and destination information

[0138] Output: Application startup and initialization

[0139] What happens: The user taps the application to launch it and enters their current location and destination.

[0140] Step 2:

[0141] The server uses the map information service API to obtain route information between the user's current location and the destination.

[0142] Input: User's current location and destination information

[0143] Output: Route information

[0144] Specific operation: The server sends a request to the map information service API to calculate and obtain the optimal route.

[0145] Step 3:

[0146] Based on the route information acquired by the server, instructions for each step are provided to the user via voice guidance.

[0147] Input: Route information

[0148] Output: Voice guidance

[0149] Specific operation: The server analyzes the route information and provides instructions for each step to the user via voice.

[0150] Step 4:

[0151] The user enters a question, and the server generates a response using a generative pre-training transformer (generative AI).

[0152] Input: User question

[0153] Output: Generative AI response

[0154] How it works: The user enters a question via voice or text, and the server uses generative AI to generate a natural-sounding response.

[0155] Step 5:

[0156] Based on information obtained from the map information service, the server will provide audio information such as the characteristics and colors of surrounding buildings and the location of facilities.

[0157] Input: Environmental information from a map information service

[0158] Output: Provides environmental information through audio

[0159] Specific operation: The server obtains environmental information from the map information service and conveys it to the user via voice.

[0160] Step 6:

[0161] The server guides the user to their destination based on map information within the store.

[0162] Input: Store map information

[0163] Output: Guidance by voice guidance

[0164] Specific operation: The server analyzes the map information within the store and guides the user to the desired location by voice.

[0165] Step 7:

[0166] The server communicates information about the features and layout of products and facilities in the store to the user via voice.

[0167] Input: Store product and facility information

[0168] Output: Provides product features and location information by voice

[0169] Specific operation: The server obtains information about products and facilities in the store and conveys it to the user via voice.

[0170] In this way, a system will be created that helps visually impaired people navigate and search for products smoothly within a physical store.

[0171] Example 2

[0172] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] There is a demand for navigation systems that allow visually impaired people to reach their destinations safely and efficiently. Conventional navigation systems do not provide sufficient information to visually impaired people, making it difficult to properly convey directions to their destination and information about the surrounding environment. They also lack the technology to enable natural dialogue with the user. This has made it difficult for visually impaired people to travel safely.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0175] In this invention, the server includes a means for a user to set a destination, a means for the server to calculate an optimal route, and a means for the terminal to generate audio guidance. This allows visually impaired people to understand the route to their destination by voice and travel safely and efficiently. The server also includes a means for a user to ask questions or make requests, a means for the server to perform natural language processing, and a means for the terminal to transmit the generated conversation content by voice. This allows visually impaired people to have natural conversations with the system and obtain the necessary information. The server also includes a means for the server to acquire surrounding environmental information and a means for the terminal to transmit the environmental information by voice. This allows visually impaired people to understand environmental information such as the features and colors of surrounding buildings by voice and travel safely.

[0176] The "means for the user to set a destination" is a function that provides an interface for the user to specify a destination using voice input or text input.

[0177] The "means for the server to calculate the optimal route" is a function that uses a map information service to calculate the optimal route to the destination set by the user.

[0178] The "means for the terminal to generate voice guidance" is a function that uses a voice synthesis engine to create voice guidance that is easy for the user to understand, based on the calculated route information.

[0179] The "means for the user to receive audio guidance" is a function that enables the user to hear audio guidance provided by the terminal.

[0180] The "means for users to ask questions or make requests" is a function that provides an interface for users to ask questions or make requests to the system using voice input.

[0181] The "means for the server to perform natural language processing" refers to the function of using a generative pre-training transformer to generate natural conversations in response to questions and requests from users.

[0182] The "means for transmitting the conversation content generated by the terminal by voice" is a function that converts the conversation content received from the server into voice using a voice synthesis engine and transmits it to the user.

[0183] The "means for the server to acquire surrounding environmental information" is a function for acquiring information about surrounding buildings and features from a map information service.

[0184] "Means for the terminal to convey environmental information by voice" is a function that converts the acquired environmental information into voice using a voice synthesis engine and conveys it to the user.

[0185] The navigation system for visually impaired persons of the present invention is implemented in a portable device such as a smartphone, a wearable device, etc. Specific embodiments of this system will be described below.

[0186] Hardware and software used

[0187] Hardware: Smartphones, wearable devices

[0188] Software: Map information services (e.g., Google Maps API (registered trademark)), speech synthesis engines (e.g., Google Text-to-Speech (registered trademark)), generative pre-training transformers (e.g., GPT-4 (registered trademark))

[0189] Program processing

[0190] 1. The user sets a destination.

[0191] The user launches an application on their smartphone or wearable device and sets their destination by voice or text input, for example, by saying, "I want to go to a nearby cafe."

[0192] 2. The server calculates the optimal route.

[0193] The server accesses a map information service and calculates the optimal route to the destination set by the user. For example, it uses the Google Maps API to get the optimal route from the current location to the nearest cafe.

[0194] 3. The device generates voice guidance.

[0195] The device generates voice guidance based on the route information received from the server. It uses a speech synthesis engine to create voice guidance that is easy for the user to understand. For example, it generates voice guidance such as, "Turn right and go 50 meters, and you will find a cafe on your left."

[0196] 4. The user receives the audio prompt.

[0197] The user listens to the voice guidance provided by the terminal and understands the route to the destination.

[0198] 5. The user has a question or request.

[0199] If the user has any questions or requests along the way, they can ask the question or request by voice, for example, "Which way should I turn at the next intersection?"

[0200] 6. The server performs natural language processing.

[0201] The server uses a generative pre-training transformer to generate natural-sounding conversations in response to user questions and requests. For example, it uses GPT-4 (registered trademark) to generate natural-sounding conversations such as "Turn right at the next intersection."

[0202] 7. The device will then transmit the generated conversation content aloud.

[0203] The terminal converts the conversation content received from the server into voice using a speech synthesis engine and conveys it to the user.

[0204] 8. The server obtains information about the surrounding environment.

[0205] The server obtains information about surrounding buildings and features from a map information service, such as the color and shape of the building and whether it has stairs.

[0206] 9. The device will provide audio information about the environment.

[0207] The device converts the acquired environmental information into speech using a speech synthesis engine and conveys it to the user. For example, it conveys environmental information to the user in a voice form, such as "There is a red brick building on your right" or "There are five steps ahead."

[0208] Examples of concrete examples and prompts

[0209] Example: If a user says, "I want to go to a nearby cafe," the server uses a map information service to calculate the optimal route, and the device provides voice guidance such as, "Turn right and go 50 meters, and you will find the cafe on your left."

[0210] Example prompts: "If the user says they want to go to a nearby cafe, calculate the best route and provide voice directions."; "If the user asks about a nearby building, describe its features voice-over."

[0211] In this way, the navigation system for the visually impaired of the present invention can provide the user with voice guidance and natural conversation, and convey information about the surrounding environment.

[0212] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0213] Step 1:

[0214] The user sets the destination.

[0215] Input: The user launches an application on their smartphone or wearable device and sets a destination by voice or text input. For example, they might say, "I want to go to a nearby cafe."

[0216] Data processing: The device converts voice input into text data and extracts destination information.

[0217] Output: The destination information is sent to the server in text format.

[0218] Specific operation: When a user speaks, "I want to go to a nearby cafe," the device uses voice recognition technology to convert this into text and sends the destination information to the server.

[0219] Step 2:

[0220] The server calculates the optimal route.

[0221] Input: Destination information received by the server from the device.

[0222] Data processing: The server sends a request to a map information service (e.g., Google Maps API (registered trademark)) to calculate the optimal route from the current location to the destination.

[0223] Output: The optimal route information is sent from the server to the device.

[0224] Specific operation: The server accesses the Google Maps API (registered trademark), obtains the optimal route from the current location to the nearest cafe, and sends that information to the device.

[0225] Step 3:

[0226] The terminal generates audio guidance.

[0227] Input: The best route information received from the server.

[0228] Data processing: The device uses a speech synthesis engine (e.g., Google Text-to-Speech®) to convert route information into voice guidance.

[0229] Output: Audio guidance is provided to the user.

[0230] Specific operation: The device generates and provides voice guidance to the user, such as "Turn right and go 50 meters, and you will find the cafe on your left."

[0231] Step 4:

[0232] The user receives the audio guidance.

[0233] Input: Voice guidance provided by the device.

[0234] Data processing: The user listens to and understands the audio guidance.

[0235] Output: The user gets directions to their destination.

[0236] Specific operation: The user listens to the voice guidance from the device and walks according to the instructions.

[0237] Step 5:

[0238] The user asks a question or makes a request.

[0239] Input: The user speaks a question or request, for example, "Which way do I turn at the next intersection?"

[0240] Data processing: The device converts voice input into text data and extracts questions or requests.

[0241] Output: The question or request is sent to the server in text format.

[0242] Specific operation: When a user speaks a question such as "Which way should I turn at the next intersection?", the device uses voice recognition technology to convert the question into text and sends it to the server.

[0243] Step 6:

[0244] The server performs natural language processing.

[0245] Input: Questions or requests received from the device.

[0246] Data processing: The server uses a generative pre-training transformer (e.g., GPT-4 (registered trademark)) to generate natural-sounding conversations in response to questions and requests.

[0247] Output: The generated conversation content is sent from the server to the device.

[0248] Specific operation: The server uses GPT-4 (registered trademark) to generate natural conversation such as "Turn right at the next intersection" and sends that content to the terminal.

[0249] Step 7:

[0250] The device then transmits the generated conversation content by voice.

[0251] Input: Generated conversation received from the server.

[0252] Data processing: The device uses a speech synthesis engine to convert the conversation into audio.

[0253] Output: The conversation is communicated to the user via audio.

[0254] Specific operation: The device conveys the generated conversation content to the user by voice.

[0255] Step 8:

[0256] The server acquires information about the surrounding environment.

[0257] Input: A request for surrounding environment information from a mapping service.

[0258] Data processing: The server obtains information about surrounding buildings and features from a map information service.

[0259] Output: The acquired environmental information is sent from the server to the device.

[0260] Specific operation: The server obtains information about surrounding buildings and features from the Google Maps API (registered trademark) and sends that information to the device.

[0261] Step 9:

[0262] The device will provide audio information about the environment.

[0263] Input: Environment information received from the server.

[0264] Data processing: The device uses a speech synthesis engine to convert environmental information into speech.

[0265] Output: Environmental information is communicated to the user via audio.

[0266] Specific operation: The device will verbally communicate environmental information to the user, such as "There is a red brick building on your right" and "There are five steps up ahead."

[0267] (Application example 2)

[0268] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] When visually impaired people shop comfortably in a brick-and-mortar store, they face challenges such as difficulty understanding product locations and store layouts. They also face challenges such as having to rely on others to obtain product information, making it difficult for them to enjoy shopping independently. Furthermore, they are unable to properly grasp information about their surroundings, making it difficult for them to navigate safely.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0271] In this invention, the server includes a means for providing directions to a destination, a means for enabling conversation with a user using natural language processing technology, a means for conveying surrounding environmental information to a visually impaired person, a means for guiding the location of products in a store and the layout of the store, and a means for providing product information, thereby enabling visually impaired people to shop independently, safely, and comfortably in a physical store.

[0272] "Means of providing directions to the destination" refers to a function that calculates the optimal route to the destination set by the user and provides instructions using voice guidance, etc.

[0273] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses a generative pre-training transformer (generative AI) to respond to questions and requests from users through natural conversation.

[0274] "A means of conveying information about the surrounding environment to visually impaired people" is a function that uses audio to convey the characteristics and colors of surrounding buildings, the location of obstacles, etc., based on information obtained from map information services.

[0275] "Means of providing guidance on the location of products within the store and the store layout" refers to a function that obtains map information within the store, calculates the optimal route to the location of the product specified by the user, and provides voice guidance.

[0276] "Means for providing product information" refers to a function that provides product features, prices, stock status, etc. via voice.

[0277] A system for implementing this invention is designed to help visually impaired people shop independently, safely, and comfortably in brick-and-mortar stores. The system is implemented as an application installed on a portable device such as a smartphone or smart glasses.

[0278] Hardware and software used

[0279] Hardware: Smartphones, smart glasses

[0280] Software: Python (registered trademark), OpenAI (registered trademark) API, Google Maps (registered trademark) API, gTTS (Google Text-to-Speech (registered trademark))

[0281] System configuration

[0282] 1. Means of providing directions to a destination:

[0283] The server uses the Google Maps® API to calculate the optimal route from the user's current location to the destination.

[0284] The device provides the calculated route information to the user as voice guidance.

[0285] 2. Means of enabling conversation with users using natural language processing technology:

[0286] The server uses the OpenAI® API to generate responses to questions posed by the user.

[0287] The terminal plays the generated response aloud.

[0288] 3. Means of communicating information about the surrounding environment to the visually impaired:

[0289] The server obtains information such as the characteristics and colors of surrounding buildings and the location of obstacles based on information obtained from the map information service.

[0290] The device will then communicate the acquired information to the user via voice.

[0291] 4. Navigating product locations and store layout within the store:

[0292] The server obtains map information about the store and calculates the optimal route to the location of the product specified by the user.

[0293] The device provides the calculated route information to the user as voice guidance.

[0294] 5. Means of providing product information:

[0295] The server obtains information such as product features, prices, and stock status.

[0296] The terminal provides the acquired product information to the user by voice.

[0297] Specific examples

[0298] For example, if a user asks, "How much does this item cost?", the server uses the OpenAI (registered trademark) API to generate a response such as, "The price of this item is 1,000 yen." The device then plays this response aloud to the user.

[0299] Prompt Sentence Examples

[0300] User: How much does this item cost?

[0301] AI: The price of this item is 1,000 yen.

[0302] In this way, a navigation application can be realized that allows visually impaired people to enjoy shopping comfortably in physical stores.

[0303] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0304] Step 1:

[0305] The user starts up a smartphone or smart glasses and starts the navigation application. The user then inputs their destination and product information by voice. The input voice data is converted into text data by the device.

[0306] Step 2:

[0307] The device sends text data to the server, which uses the Google Maps (registered trademark) API to calculate the optimal route from the user's current location to the destination. The input is the user's current location and destination information, and the output is the optimal route information.

[0308] Step 3:

[0309] The server transmits the calculated route information to the terminal, which then provides the route information to the user as voice guidance by converting text to speech using gTTS (Google Text-to-Speech®) and playing it back to the user.

[0310] Step 4:

[0311] When a user wants to know product information, they input a question by voice. The input voice data is converted into text data by the terminal.

[0312] Step 5:

[0313] The device sends text data to the server, which uses the OpenAI® API to generate a response to the user's question. The input is the user's question text, and the output is the response text.

[0314] Step 6:

[0315] The server sends the generated response text to the terminal, which then plays the response text aloud and provides it to the user. Specifically, the terminal converts the text into speech using gTTS and plays it back to the user.

[0316] Step 7:

[0317] The server obtains environmental information such as the characteristics and colors of surrounding buildings and the location of obstacles based on information obtained from the map information service. The input is data from the map information service, and the output is environmental information.

[0318] Step 8:

[0319] The server sends the acquired environmental information to the device, which then communicates the environmental information to the user by voice. Specifically, the device converts text to voice using gTTS and plays it back to the user.

[0320] Step 9:

[0321] The server obtains map information about the store and calculates the optimal route to the location of the product specified by the user. The input is the map information about the store and the product information specified by the user, and the output is the optimal route information.

[0322] Step 10:

[0323] The server sends the calculated route information to the device, which then provides the route information to the user as voice guidance by converting text to speech using gTTS and playing it back to the user.

[0324] Step 11:

[0325] The server obtains information such as product features, prices, and stock status. The input is data from the product database, and the output is product information.

[0326] Step 12:

[0327] The server sends the acquired product information to the terminal, which then provides the product information to the user by voice. Specifically, the terminal converts text to voice using gTTS and plays it back to the user.

[0328] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0329] "Example 1"

[0330] In one embodiment of the present invention, an emotion engine is incorporated into a navigation system for the visually impaired. This emotion engine has the function of recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to the emotion. Specifically, if the user makes a sound that indicates joy or relief, the emotion engine recognizes this and the voice guidance issues instructions in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the emotion engine recognizes this and the voice guidance issues instructions in a clearer and more forceful tone. This makes it possible to provide appropriate guidance according to the user's emotional state.

[0331] "Example 2"

[0332] In one embodiment of the present invention, an emotion engine is incorporated into a navigation system for the visually impaired. This emotion engine has the function of recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to the emotion. Specifically, if the user makes a sound that indicates joy or relief, the emotion engine recognizes this and the voice guidance issues instructions in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the emotion engine recognizes this and the voice guidance issues instructions in a clearer and more forceful tone. This makes it possible to provide appropriate guidance according to the user's emotional state.

[0333] The processing flow of each embodiment will be described below.

[0334] "Example 1"

[0335] Step 1: The user speaks their destination, which is converted into text by the system and recognized as a destination.

[0336] Step 2: The system uses a map information service to calculate the optimal route from your current location to your destination.

[0337] Step 3: The system uses generative AI to generate voice guidance based on the calculated route.

[0338] Step 4: The user provides vocal feedback to the system, which is analyzed by the emotion engine to recognize the user's emotional state.

[0339] Step 5: The emotion engine adjusts the tone of the voice guidance based on the recognized emotion. For example, if the user vocalizes with a voice that indicates joy or relief, the voice guidance will give instructions in a gentler tone. Conversely, if the user vocalizes with a voice that indicates anxiety or fear, the voice guidance will give instructions in a clearer and more forceful tone.

[0340] Example 1

[0341] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0342] In order for visually impaired people to reach their destination safely and efficiently, they need not only directions but also appropriate guidance that takes into account the surrounding environmental information and the user's emotional state. However, conventional navigation systems have difficulty providing these elements in an integrated manner, and are therefore unable to provide sufficient support for visually impaired people. Therefore, a system that enables visually impaired people to travel safely and efficiently is needed.

[0343] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0344] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, and means for recognizing the user's emotions and adjusting the tone of the voice guidance. This enables a visually impaired person to not only be provided with directions but also receive appropriate guidance according to the surrounding environmental information and emotions when heading to a destination.

[0345] "Means for providing directions to a destination" is a function that calculates the optimal route to the destination set by the user and provides that information to the user.

[0346] "Means for enabling conversation with users using natural language processing technology" refers to a function that converts voice input from users into text data, generates an appropriate response using natural language processing technology such as generative pre-training transformers, and conveys it back to the user as voice.

[0347] "Means for conveying surrounding environmental information to visually impaired people" is a function that conveys information such as the characteristics and colors of surrounding buildings obtained from a map information service to the user via voice.

[0348] The "means for recognizing the user's emotions and adjusting the tone of the voice guidance" is a function for recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to those emotions.

[0349] A "generative pre-training transformer" is a natural language processing model that is pre-trained using large amounts of text data, and is a technology for generating appropriate responses to user input.

[0350] A "map information service" is an online service that provides geographical information and is used to obtain a user's current location, destination, surrounding environmental information, etc.

[0351] The navigation system for visually impaired persons of the present invention is implemented in a portable device such as a smartphone, a wearable device, etc. Specific embodiments of this system will be described below.

[0352] Hardware and Software Configuration

[0353] Hardware

[0354] Devices: Smartphones, wearable devices (e.g., smartwatches, earphones)

[0355] Server: Cloud server

[0356] software

[0357] Map information services: Online map services (e.g., Google Maps (registered trademark) API)

[0358] Generative pre-training transformers: Natural language processing models (e.g., GPT-4 (registered trademark))

[0359] Emotion Engine: Software for Recognizing Emotions from Speech

[0360] Data processing and calculation

[0361] 1. Setting a destination and calculating a route

[0362] Users open an application on their smartphone or wearable device and enter their destination by voice or text.

[0363] The terminal transmits the input destination information to the server.

[0364] The server uses a map information service to calculate the optimal route to the destination set by the user.

[0365] The server sends the calculation results to the terminal.

[0366] 2. Audio Guidance Generation

[0367] The device generates voice guidance based on the received route information.

[0368] The device provides voice guidance to the user through the smartphone's built-in speaker or the wearable device's earphones.

[0369] 3. Conversational features based on natural language processing

[0370] If the user has any questions or requests while navigating, they can input the questions or requests by voice.

[0371] The terminal converts the input voice into text data.

[0372] The device sends the converted text data to the generative AI model as a prompt.

[0373] The generative AI model generates an appropriate response based on the prompt it receives.

[0374] The generative AI model returns the generated response to the device.

[0375] The device converts the response received from the generative AI model into speech and conveys it to the user.

[0376] 4. Providing information about the surrounding environment

[0377] The server obtains environmental information such as the characteristics and colors of surrounding buildings from a map information service.

[0378] The server transmits the acquired environmental information to the terminal.

[0379] The device generates audio guidance based on the received environmental information.

[0380] The device will then provide the user with specific information via voice, such as "There is a red brick building on your right" or "There are five steps ahead."

[0381] 5. Adjusting the tone of voice guidance using an emotion engine

[0382] The device is equipped with an emotion engine that recognizes emotions from the user's voice.

[0383] If the user makes vocalizations that indicate joy or relief, the emotion engine will recognize this and the voice guidance will provide instructions in a gentler tone.

[0384] If the user makes vocalizations that indicate anxiety or fear, the emotion engine will recognize this and the voice guidance will provide instructions in a clearer and more forceful tone.

[0385] Examples of concrete examples and prompts

[0386] Specific examples

[0387] The user asks, "How do I turn right at the next intersection?"

[0388] The device converts this question into text data and sends it as a prompt to the generative AI model.

[0389] The generative AI model generates the response, "To turn right at the next intersection, go 50 meters and then turn right."

[0390] The device converts this response into speech and relays it to the user.

[0391] Prompt Sentence Examples

[0392] "How do I turn right at the next intersection?"

[0393] "Please tell me the nearest cafe."

[0394] "Are there any obstacles ahead?"

[0395] In this way, the navigation system for the visually impaired of the present invention can provide appropriate guidance and information to the user.

[0396] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0397] Step 1:

[0398] Users open an application on their smartphone or wearable device and enter their destination by voice or text.

[0399] Input: User's voice or text destination information

[0400] Output: Destination information is sent to the device

[0401] Specific operation: The user speaks, "Please tell me the way to the station."

[0402] Step 2:

[0403] The terminal transmits the input destination information to the server.

[0404] Input: Destination information from the user

[0405] Output: The destination information is sent to the server.

[0406] Specific operation: The device converts voice input into text data and sends it to the server.

[0407] Step 3:

[0408] The server uses a map information service to calculate the optimal route to the destination set by the user.

[0409] Input: Destination information

[0410] Output: Optimal route information

[0411] Specific operation: The server accesses the map information service and calculates the route from the current location to the destination.

[0412] Step 4:

[0413] The server sends the calculation results to the terminal.

[0414] Input: Optimal route information

[0415] Output: Route information is sent to the device.

[0416] Specific operation: The server sends the calculated route information to the terminal.

[0417] Step 5:

[0418] The device generates voice guidance based on the received route information.

[0419] Input: Route information

[0420] Output: Voice guidance

[0421] Specific operation: The device will give voice instructions such as "Go 50 meters and turn right."

[0422] Step 6:

[0423] If the user has any questions or requests while navigating, they can input the questions or requests by voice.

[0424] Input: The user's spoken question or request

[0425] Output: Questions and requests are sent to the terminal.

[0426] Specific Action: The user asks, "How do I turn right at the next intersection?"

[0427] Step 7:

[0428] The terminal converts the input voice into text data.

[0429] Input: User's voice data

[0430] Output: Text data

[0431] Specific operation: The device uses voice recognition technology to convert speech into text.

[0432] Step 8:

[0433] The device sends the converted text data to the generative AI model as a prompt.

[0434] Input: Text data

[0435] Output: prompts to the generative AI model

[0436] Specific operation: The device sends text data to the generative AI model.

[0437] Step 9:

[0438] The generative AI model generates an appropriate response based on the prompt it receives.

[0439] Input: prompt

[0440] Output: Response text

[0441] Specific behavior: The generative AI model generates the response, "To turn right at the next intersection, go 50 meters and then turn right."

[0442] Step 10:

[0443] The generative AI model returns the generated response to the device.

[0444] Input: Response text

[0445] Output: The response text is sent to the terminal

[0446] Specific operation: The response generated by the generative AI model is sent to the terminal.

[0447] Step 11:

[0448] The device converts the response received from the generative AI model into speech and conveys it to the user.

[0449] Input: Response text

[0450] Output: Voice guidance

[0451] Specific operation: The device will tell the user by voice, "To turn right at the next intersection, go 50 meters and then turn right."

[0452] Step 12:

[0453] The server obtains environmental information such as the characteristics and colors of surrounding buildings from a map information service.

[0454] Input: User's current location

[0455] Output: Surrounding environment information

[0456] Specific operation: The server accesses the map information service and obtains the characteristics and colors of surrounding buildings.

[0457] Step 13:

[0458] The server transmits the acquired environmental information to the terminal.

[0459] Input: Surrounding environment information

[0460] Output: Environment information is sent to the terminal

[0461] Specific operation: The server sends the acquired environmental information to the terminal.

[0462] Step 14:

[0463] The device generates audio guidance based on the received environmental information.

[0464] Input: Environment information

[0465] Output: Voice guidance

[0466] Specific action: The device will tell the user, "There is a red brick building on your right."

[0467] Step 15:

[0468] The device is equipped with an emotion engine that recognizes emotions from the user's voice.

[0469] Input: User's voice data

[0470] Output: Emotion recognition result

[0471] Specific operation: The device analyzes the user's voice and recognizes emotions.

[0472] Step 16:

[0473] If the user makes vocalizations that indicate joy or relief, the emotion engine will recognize this and the voice guidance will provide instructions in a gentler tone.

[0474] Input: Emotion recognition results

[0475] Output: Gentle voice guidance

[0476] Specific action: The device will give instructions in a gentle tone, such as "Turn right at the next intersection."

[0477] Step 17:

[0478] If the user makes vocalizations that indicate anxiety or fear, the emotion engine will recognize this and the voice guidance will provide instructions in a clearer and more forceful tone.

[0479] Input: Emotion recognition results

[0480] Output: Clear and powerful voice guidance

[0481] Specific action: The device will issue instructions in a clear, forceful tone, such as "Turn right at the next intersection."

[0482] (Application example 1)

[0483] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0484] When visually impaired people use self-driving vehicles, they face challenges in understanding the route to their destination, grasping information about their surroundings, and receiving appropriate guidance according to their emotional state. To solve these challenges, a comprehensive navigation system is needed to enable visually impaired people to use self-driving vehicles with peace of mind.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0486] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, means for recognizing the user's emotions and adjusting the tone of the voice guidance according to the emotions, and means installed in the self-driving vehicle, thereby enabling a visually impaired person using a self-driving vehicle to understand the route to the destination, grasp information about the surrounding environment, and receive appropriate guidance according to their emotional state.

[0487] "Means for providing directions to a destination" means a function that calculates the optimal route to a destination set by the user and provides instructions to the user by voice guidance or other means.

[0488] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses natural language processing technology such as generative pre-training transformers (generative AI) to respond to questions and requests from users through natural conversation.

[0489] "Means for communicating surrounding environmental information to visually impaired people" refers to a function that communicates environmental information such as the characteristics and colors of surrounding buildings and obstacles to visually impaired people via voice, based on information obtained from map information services.

[0490] "Means for recognizing the user's emotions and adjusting the tone of the voice guidance according to those emotions" refers to a function that recognizes emotions from the user's voice and adjusts the tone of the voice guidance according to those emotions. For example, if the user is feeling anxious, instructions will be given in a clearer and stronger tone.

[0491] "Means installed in an automated vehicle" refers to a navigation system for visually impaired people that is installed in an automated vehicle to provide appropriate navigation and guidance to visually impaired people while the vehicle is in operation.

[0492] The system for implementing the present invention is a navigation system for the visually impaired, and includes the following main functions:

[0493] 1. System Program

[0494] The system includes a means for providing directions to a destination, a means for enabling a conversation with a user using natural language processing technology, a means for conveying surrounding environmental information to visually impaired people, a means for recognizing a user's emotions and adjusting the tone of the voice guidance according to the emotions, and a means for being installed in an autonomous vehicle.

[0495] 2. Program processing explanation

[0496] Hardware and Software

[0497] Hardware: Smartphones, smart glasses, head-mounted displays, and embedded systems in autonomous vehicles

[0498] Software: OpenAI (registered trademark) API, Google Maps (registered trademark) API, gTTS (Google Text-to-Speech (registered trademark), playsound (registered trademark)

[0499] Data processing and calculation

[0500] Means for providing directions to the destination: The server uses Google Maps (registered trademark) API to calculate the optimal route to the destination set by the user. The calculated route information is provided to the user as voice guidance.

[0501] Means of enabling conversation with users using natural language processing technology: The server uses OpenAI®'s Generative Pre-Trained Transformer (generative AI) to respond to user questions and requests through natural conversation.

[0502] Means of communicating surrounding environmental information to visually impaired people: Based on information obtained from the Google Maps (registered trademark) API, the server communicates environmental information such as the characteristics and colors of surrounding buildings and obstacles to visually impaired people by voice.

[0503] A means to recognize the user's emotions and adjust the tone of the voice guidance according to those emotions: The server analyzes the emotions from the user's voice and adjusts the tone of the voice guidance according to those emotions. For example, if the user is feeling anxious, instructions will be given in a clearer and more forceful tone.

[0504] Inclusion in self-driving vehicles: The system will be integrated into self-driving vehicles to provide appropriate navigation and guidance to visually impaired people while the vehicle is in operation.

[0505] 3. Specific Examples

[0506] If the user says, "I want to go to Tokyo Tower," the system will provide voice guidance saying, "Your destination is Tokyo Tower. There is a red brick building on your right."

[0507] Prompt Sentence Examples

[0508] User: I want to go to Tokyo Tower.

[0509] System: Your destination is Tokyo Tower. On your right is a red brick building.

[0510] In this way, an automated vehicle navigation system for the visually impaired is realized.

[0511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0512] Step 1:

[0513] The user sets the destination.

[0514] The user specifies a destination through voice input. For example, they might say, "I want to go to Tokyo Tower." The device receives this voice input and converts it into text data. The input is the user's voice data, and the output is text data.

[0515] Step 2:

[0516] The server calculates the route to the destination.

[0517] The server uses the Google Maps (registered trademark) API to calculate the optimal route to the destination set by the user. The input is text data of the destination specified by the user, and the output is JSON data containing route information. The server parses this JSON data and generates text for voice guidance.

[0518] Step 3:

[0519] The server conducts the conversation with the user.

[0520] The server uses OpenAI's (registered trademark) generative pre-trained transformer (generative AI) to respond to user questions and requests through natural conversation. The input is the user's text data, and the output is a generated response text. The server converts this response text into speech and sends it to the device.

[0521] Step 4:

[0522] The server acquires information about the surrounding environment.

[0523] The server obtains environmental information such as the characteristics and colors of surrounding buildings and obstacles based on information obtained from the Google Maps (registered trademark) API. The input is the coordinate data of the current location, and the output is JSON data containing environmental information. The server analyzes this JSON data and generates text for voice guidance.

[0524] Step 5:

[0525] The server recognizes the user's emotions.

[0526] The server analyzes the user's voice to determine their emotions. The input is the user's voice data, and the output is data indicating their emotional state. The server adjusts the tone of the voice guidance based on this emotional data.

[0527] Step 6:

[0528] The server provides the audio guidance.

[0529] The server converts the generated text for the voice guidance into audio using gTTS (Google Text-to-Speech (registered trademark)) and sends it to the terminal. The input is text data for the voice guidance, and the output is audio data. The terminal plays back this audio data and provides it to the user.

[0530] Step 7:

[0531] Autonomous vehicles perform navigation.

[0532] The autonomous vehicle travels to its destination based on route information and voice guidance received from the server. The input is route information and voice guidance, and the output is the vehicle's operating status. The vehicle plays the voice guidance at the appropriate time and provides it to the user.

[0533] Example 2

[0534] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0535] In order for visually impaired people to reach their destination safely and efficiently, they need not only directions but also appropriate guidance that takes into account the surrounding environmental information and the user's emotional state. However, conventional navigation systems have difficulty providing these elements in an integrated manner, and have faced the challenge of not providing sufficient support for visually impaired people to travel safely.

[0536] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0537] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, and means for recognizing the user's emotions and adjusting the tone of the voice guidance, thereby enabling comprehensive support for visually impaired people to reach their destination safely and efficiently.

[0538] "Means for providing directions to a destination" is a function that calculates the optimal route to a destination set by the user and guides the user along that route.

[0539] "Means for enabling conversation with users using natural language processing technology" refers to technology that enables responses in natural language to questions or requests entered by users via voice or text.

[0540] "A means of conveying information about the surrounding environment to visually impaired people" is a function that conveys information about surrounding buildings, terrain features, colors, etc. obtained from map information services to users via voice.

[0541] "Means for recognizing the user's emotions and adjusting the tone of the voice guidance" refers to a function that analyzes the emotions from the user's voice and changes the tone of the voice guidance according to those emotions.

[0542] A "generative pre-training transformer" is a natural language processing model that is pre-trained based on large amounts of text data, and is a technology for generating appropriate responses to user input.

[0543] "Map information service" means an online service that provides geographic information and is used to obtain a user's current location, destination, and surrounding environmental information.

[0544] The present invention relates to a navigation system for visually impaired people, which is implemented in a portable device such as a smartphone or a wearable device. Specific embodiments of this system will be described below.

[0545] First, the user sets their destination using their smartphone or wearable device by voice or text input, for example, "I want to go to the nearest cafe."

[0546] The server accesses a map information service (e.g., Google Maps® API) and calculates the optimal route from the user's current location to the destination, taking into account traffic conditions and pedestrian route information.

[0547] The device generates voice guidance based on the route information received from the server, providing specific instructions to the user, such as "Turn right 100 meters ahead."

[0548] While traveling, the user may voice additional questions or requests, such as "What kind of place is this?" or "How far is the next intersection?" The device converts the user's voice input into text and inputs it as a prompt to a generative AI model (e.g., GPT-4). For example, it generates a prompt such as "What kind of place is this?"

[0549] The generative AI model generates an appropriate response based on the prompt sentence, such as "There are red brick buildings around your current location. There are five steps ahead." The device then converts this response into speech and relays it to the user, allowing the user to receive information about their surroundings and additional instructions via voice.

[0550] Furthermore, the device uses information obtained from the map information service to provide voice information about the characteristics and colors of surrounding buildings, providing specific information such as "There is a red brick building on your right" or "There are five steps ahead."

[0551] The device is equipped with an emotion engine that recognizes emotions from the user's voice. This emotion engine adjusts the tone of the voice guidance according to the user's emotions. For example, if the user makes a voice that indicates joy or relief, the device will provide voice guidance in a gentler tone. Conversely, if the user makes a voice that indicates anxiety or fear, the device will provide voice guidance in a clearer and more forceful tone.

[0552] For example, if a user says "I want to go to the nearest cafe," the following specific actions will be taken:

[0553] 1. The user speaks, "I want to go to the nearest cafe."

[0554] 2. The server accesses a map information service and calculates the optimal route from the current location to the nearest cafe.

[0555] 3. The device will give the user a voice command saying, "Turn right in 100 meters."

[0556] 4. The user asks, "What kind of place is this?"

[0557] 5. The device inputs the prompt sentence "What kind of place is this?" into the generative AI model.

[0558] 6. The generative AI model generates a response: "There are red brick buildings around your current location. There are five steps ahead."

[0559] 7. The device converts this response into speech and relays it to the user.

[0560] 8. The device provides information about the surrounding environment, such as "There is a red brick building on your right."

[0561] 9. If the user expresses anxiety, the device will provide voice guidance in a clearer and more forceful tone.

[0562] In this way, navigation systems for the visually impaired support users in their journeys and provide them with information about their surrounding environment.

[0563] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0564] Step 1:

[0565] The user sets the destination.

[0566] Users can use a smartphone or wearable device to set their destination by voice or text input. For example, they can say, "I want to go to the nearest cafe." The input speech is converted into text by the device's voice recognition function. The output is the destination information in text format.

[0567] Step 2:

[0568] The server calculates the optimal route.

[0569] The server accesses a map information service (e.g., Google Maps (registered trademark) API) and receives the user's current location and destination as input. The server calculates the optimal route based on this information. The calculation also takes into account traffic conditions and pedestrian route information. The output is the optimal route information.

[0570] Step 3:

[0571] The device provides voice guidance.

[0572] The device generates voice guidance based on the route information received from the server. For example, it gives specific instructions to the user by voice, such as "Turn right 100 meters ahead." The input is the route information from the server, and the output is the voice guidance.

[0573] Step 4:

[0574] The user asks a question or makes a request.

[0575] The user voices additional questions or requests as they move along, such as "What is the location like?" or "How far is the next intersection?" The input is the user's voice input, and the output is the question or request in text form.

[0576] Step 5:

[0577] The device inputs a prompt sentence into the generative AI model.

[0578] The device converts the user's voice input into text and inputs it as a prompt to a generative AI model (e.g., GPT-4 (registered trademark)). For example, it generates a prompt such as "What kind of place is this?" The input is the user's textual question or request, and the output is the prompt.

[0579] Step 6:

[0580] A generative AI model generates a response.

[0581] The generative AI model generates an appropriate response based on the prompt sentence, for example, "There are red brick buildings around your current location. There are five steps ahead." The input is the prompt sentence, and the output is a text response.

[0582] Step 7:

[0583] The device will then speak the response to the user.

[0584] The device converts the response from the generative AI model into speech and transmits it to the user, allowing the user to receive information about the surrounding environment and additional instructions by voice. The input is a text response from the generative AI model, and the output is a speech response.

[0585] Step 8:

[0586] The device provides information about the surrounding environment.

[0587] The device uses information obtained from the map information service to provide voice information about the characteristics and colors of surrounding buildings. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead." The input is environmental information from the map information service, and the output is environmental information in the form of voice.

[0588] Step 9:

[0589] The device recognizes the user's emotions and adjusts the tone of the voice guidance.

[0590] The device uses an emotion engine that recognizes emotions from the user's voice. If the user makes a sound that indicates joy or relief, the device will provide voice guidance in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the device will provide voice guidance in a clearer and more forceful tone. The input is the user's voice, and the output is voice guidance in the adjusted tone.

[0591] (Application example 2)

[0592] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0593] When visually impaired people use self-driving vehicles, it is difficult for them to properly understand the route to their destination and the surrounding environment.In addition, it is not possible to provide appropriate voice guidance according to the user's emotional state, making it difficult for them to travel safely.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0595] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, means for adjusting the tone of the voice guidance according to the emotional state of the user, and means for providing voice guidance to a visually impaired person in an autonomous vehicle. This enables a visually impaired person using an autonomous vehicle to properly understand directions to a destination and surrounding environmental information, and to receive appropriate voice guidance according to the emotional state of the user.

[0596] A "navigation system for the visually impaired" is a system that assists visually impaired users in traveling to their destination safely and efficiently.

[0597] A "means for providing directions to a destination" is a means that has the function of calculating the optimal route to a destination set by the user and communicating those directions to the user.

[0598] "Means for enabling conversation with a user using natural language processing technology" means means for realizing natural conversation with a user using technology that understands the user's voice input and generates an appropriate response.

[0599] "Means for conveying information about the surrounding environment to visually impaired people" refers to a means for conveying information about surrounding buildings and obstacles obtained from map information services, etc. to users via voice.

[0600] The "means for adjusting the tone of the voice guidance according to the emotional state of the user" refers to a means having a function of recognizing emotions from the user's voice and changing the tone of the voice guidance according to the emotions.

[0601] "Means for providing audio guidance to visually impaired persons in self-driving vehicles" refers to means for providing audio directions to a destination and information about the surrounding environment to visually impaired persons using self-driving vehicles.

[0602] A "map information service" is an online service that provides geographical information and is used for route calculations and obtaining information about the surrounding environment.

[0603] A "generative pre-trained transformer" (generative AI) is a natural language processing model that is pre-trained using large amounts of text data and is used to generate natural conversations with users.

[0604] A system for carrying out the present invention is implemented as a navigation system for the visually impaired in a portable device such as a smartphone or smart glasses. The system includes means for providing directions to a destination, means for enabling a conversation with a user using natural language processing technology, means for conveying surrounding environmental information to the visually impaired, means for adjusting the tone of the voice guidance depending on the emotional state of the user, and means for providing voice guidance to the visually impaired in an autonomous vehicle.

[0605] Hardware and software used

[0606] Hardware: Smartphones, smart glasses

[0607] Software: Python (registered trademark), Google Maps (registered trademark) API, Hugging Face Transformers (registered trademark), pyttsx3 (registered trademark)

[0608] System Operation

[0609] 1. Obtaining map information: The server uses the Google Maps (registered trademark) API to obtain route information from the user's current location to the destination, allowing it to calculate the optimal route to the destination set by the user.

[0610] 2. Voice Guidance: The server uses pyttsx3 (registered trademark) to provide voice guidance based on the acquired route information, allowing the user to confirm the route to the destination by voice.

[0611] 3. Emotion Recognition: The server uses the Hugging Face® emotion recognition model to recognize emotions from the user's voice, allowing it to adjust the tone of the voice guidance according to the user's emotional state.

[0612] 4. Conversation Processing: The server uses Hugging Face®'s generative AI model to enable natural conversation with the user, generating appropriate responses to the user's questions and requests.

[0613] Specific examples

[0614] For example, consider a scenario in which a user gets into an autonomous vehicle and sets a destination.

[0615] User: "Shinjuku Station, please."

[0616] System: "Calculating route to Shinjuku Station. There is a red brick building on your right. Turn left at the next intersection."

[0617] In this way, we can support visually impaired people in self-driving vehicles so that they can travel to their destination safely.

[0618] Prompt Sentence Examples

[0619] User: To Shinjuku Station, please.

[0620] System: Calculating route to Shinjuku Station. There is a red brick building on your right. Turn left at the next intersection.

[0621] This invention enables visually impaired people using self-driving vehicles to properly understand the route to their destination and information about the surrounding environment, and to be provided with appropriate voice guidance that matches the user's emotional state.

[0622] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0623] Step 1:

[0624] The user sets the destination.

[0625] Input: The user specifies a destination by voice (e.g., "Shinjuku Station, please.").

[0626] How it works: The device receives voice input and converts it to text using speech recognition technology.

[0627] Output: Destination information in text format (e.g. "Shinjuku Station").

[0628] Step 2:

[0629] The server obtains route information using a map information service.

[0630] Input: Textual destination information and user's current location information.

[0631] How it works: The server calls the Google Maps (registered trademark) API to calculate the optimal route from the current location to the destination.

[0632] Output: Route information (e.g. directions, distance, time required, etc.).

[0633] Step 3:

[0634] The server generates audio guidance based on the route information.

[0635] Input: Route information.

[0636] How it works: The server uses pyttsx3 (registered trademark) to convert route information into voice guidance.

[0637] Output: Voice guidance (e.g. "Turn left at the next intersection.").

[0638] Step 4:

[0639] The server recognizes the user's emotional state.

[0640] Input: User's voice input.

[0641] How it works: The server uses the Hugging Face® emotion recognition model to recognize emotions from the user's voice.

[0642] Output: Emotion information (e.g., "joy", "anxiety", etc.).

[0643] Step 5:

[0644] The server adjusts the tone of the voice guidance based on the emotion information.

[0645] Input: Emotional information and audio guidance.

[0646] Operation: The server adjusts the voice synthesis parameters (speed, pitch, etc.) of pyttsx3 (registered trademark) according to the emotion information.

[0647] Output: Adjusted voice guidance (e.g., "Turn left at the next intersection" (gentle tone)).

[0648] Step 6:

[0649] The server handles the conversation with the user.

[0650] Input: User's voice input.

[0651] How it works: The server uses Hugging Face®'s generative AI model to generate appropriate responses to the user's questions and requests.

[0652] Output: Response text (e.g., "Calculating route to Shinjuku Station.").

[0653] Step 7:

[0654] The server converts the response text into speech and provides it to the user.

[0655] Input: The response text.

[0656] How it works: The server uses pyttsx3® to convert the response text into speech.

[0657] Output: A voice response (e.g., "Calculating route to Shinjuku Station.").

[0658] In this way, when visually impaired people use self-driving vehicles, they will be able to properly understand the route to their destination and information about the surrounding environment, and will be able to receive appropriate voice guidance that matches the user's emotional state.

[0659] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0660] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0661] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0662] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0663] [Second embodiment]

[0664] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0665] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0666] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0667] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0668] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0669] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0670] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0671] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0672] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0673] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0674] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0675] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0676] "Example 1"

[0677] The navigation system for visually impaired people of this invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[0678] "Example 2"

[0679] The navigation system for visually impaired people of the present invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[0680] The processing flow of each embodiment will be described below.

[0681] "Example 1"

[0682] Step 1: The user activates the system using a portable device such as a smartphone or wearable device.

[0683] Step 2: The user sets a destination using voice or touch input.

[0684] Step 3: The system connects with the map information service to calculate the optimal route to the set destination.

[0685] Step 4: The system will provide voice guidance for directions.

[0686] Step 5: Once the user begins their journey, if they have any questions or requests about the surrounding environment along the way, the system will respond through natural conversation using a generative pre-trained transformer (generative AI).

[0687] Step 6: The system uses information obtained from the map information service to provide a voice message describing the characteristics and colors of surrounding buildings.

[0688] Example 1

[0689] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] To reach their destination safely and efficiently, visually impaired people need detailed route guidance and information about their surrounding environment. However, conventional navigation systems have difficulty providing sufficient information to visually impaired people, particularly in terms of conveying information about the surrounding environment and building features. Furthermore, they lack the ability to respond to real-time questions and requests through natural conversation with the user. This often leads to anxiety and difficulty for visually impaired people when trying to reach their destination.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0692] In this invention, the server includes: means for a user to set a destination; means for a terminal to transmit destination information to the server; means for the server to calculate an optimal route using a map information service; means for the server to transmit the route information calculated by the server to the terminal; means for the terminal to generate audio guidance based on the route information; means for the user to receive the audio guidance; means for the user to ask a question or make a request; means for the terminal to respond to the user's question or request using a generative AI model; means for the server to acquire surrounding environmental information and transmit it to the terminal; and means for the terminal to convey surrounding information to the user by voice based on the environmental information. This enables visually impaired people to receive detailed route guidance and surrounding environmental information by voice when heading to their destination, and can also respond to questions and requests in real time through natural conversation.

[0693] "User" refers to any individual who uses the navigation system, including visually impaired individuals.

[0694] "Terminal" refers to a portable device such as a smartphone or wearable device.

[0695] "Server" refers to a computer system that processes data using map information services and generative AI models.

[0696] "Destination information" refers to data relating to a destination set by a user.

[0697] "Map Information Service" refers to an online service that provides geographic information.

[0698] An "optimal route" refers to the most efficient route from your current location to your destination.

[0699] "Audio Guidance" means audio directions generated using text-to-speech software.

[0700] "Generative AI model" refers to natural language processing technology that uses generative pre-training transformers.

[0701] "Surrounding environmental information" refers to information such as the characteristics and color of a building, whether or not it has stairs, etc., obtained from map information services.

[0702] "Natural language processing technology" refers to language understanding and generation technology that enables conversation with a user.

[0703] This invention relates to a navigation system for the visually impaired, implemented on a portable device such as a smartphone or wearable device, that provides detailed route guidance and surrounding environmental information by voice as the user travels to their destination, and can respond to questions and requests in real time through natural conversation.

[0704] The server uses a map information service (e.g., Google Maps® API) to calculate the optimal route to the destination set by the user. The calculated route information is sent to the device over the Internet. The device uses text-to-speech software (e.g., Google Text-to-Speech®) to generate audio guidance based on the received route information.

[0705] Users can check the route to their destination through voice guidance. If they have questions or additional requests along the way, they can use voice input to ask questions or make requests. The device uses a generative pre-training transformer (e.g., OpenAI® GPT-3®) to generate natural-sounding responses to the user's questions and requests.

[0706] The server obtains information about the surrounding environment from a map information service and sends it to the device. Based on the obtained information, the device uses text-to-speech software to communicate information about the surroundings to the user aloud. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead."

[0707] As a specific example, when a user asks "Can you tell me the way to the station?", the following processing is performed.

[0708] 1. User: "Can you tell me the way to the station?"

[0709] 2. The device converts the voice input into text and sends it to the server.

[0710] 3. The server uses a map information service to calculate the optimal route and sends the route information to the device.

[0711] 4. The device generates voice guidance based on the route information and provides it to the user.

[0712] 5. User: "How far is the next intersection?"

[0713] 6. Using the generative AI model, the device responds, "The next intersection is approximately 200 meters away."

[0714] 7. The server acquires information about the surrounding environment and sends it to the device.

[0715] 8. The device will tell the user by voice, "There is a red brick building on your right."

[0716] In this way, navigation systems for the visually impaired provide users with directions to their destination through voice guidance and natural conversation.

[0717] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0718] Step 1:

[0719] The user sets the destination.

[0720] The user launches an application on their smartphone or wearable device and sets their destination using voice or text input. For example, they might say, "Please tell me the way to the station." The voice input is converted into text data by the device.

[0721] Step 2:

[0722] The terminal transmits the destination information to the server.

[0723] The terminal transmits the destination information entered by the user to the server as text data. Specifically, the destination information is transmitted to the server via the Internet. The input is text data, and the output is a transmission completion notification to the server.

[0724] Step 3:

[0725] The server uses a map information service to calculate the optimal route.

[0726] The server uses a map information service (e.g., Google Maps (registered trademark) API) based on the received destination information to calculate the optimal route from the current location to the destination. The input is destination information, and the output is optimal route information.

[0727] Step 4:

[0728] The server sends the calculated route information to the terminal.

[0729] The server converts the calculated route information into a data format and sends it to the terminal. The input is the optimal route information, and the output is a transmission completion notification to the terminal.

[0730] Step 5:

[0731] The device generates voice guidance based on the route information.

[0732] The device analyzes the received route information and generates voice guidance using text-to-speech software (e.g., Google Text-to-Speech (registered trademark)). The input is the route information, and the output is the voice guidance.

[0733] Step 6:

[0734] The user receives the audio guidance.

[0735] The user confirms the route to the destination through voice guidance provided by the device. For example, the user may receive instructions such as, "Turn right and go 100 meters until you see a red brick building. Turn left at the next intersection." The input is the voice guidance, and the output is the user's actions.

[0736] Step 7:

[0737] The user asks a question or makes a request.

[0738] If the user has any questions or additional requests along the way, they can use voice input to ask questions or make requests. For example, they can ask, "How far is it to the next intersection?" The input is voice input, and the output is text data.

[0739] Step 8:

[0740] The device uses the generative AI model to respond to the user's questions and requests.

[0741] The device uses a generative pre-training transformer (e.g., OpenAI® GPT-3®) to generate natural-sounding responses to user questions and requests. The input is the text data of the user's question or request, and the output is the text data of the response.

[0742] Step 9:

[0743] The server acquires information about the surrounding environment and sends it to the terminal.

[0744] The server obtains information about the surrounding environment from the map information service and sends it to the terminal. For example, it obtains information about the characteristics and colors of surrounding buildings, whether there are stairs, etc. The input is the environmental information from the map information service, and the output is a transmission completion notification to the terminal.

[0745] Step 10:

[0746] The device communicates surrounding information to the user via voice based on environmental information.

[0747] Based on the received environmental information, the device uses text-to-speech software to communicate information about the surroundings to the user aloud. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead." The input is environmental information, and the output is voice guidance.

[0748] (Application example 1)

[0749] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] When visually impaired people navigate or search for products in a physical store, they face challenges such as difficulty grasping information about their surroundings, making it difficult for them to understand the route to their destination and the layout of products. Furthermore, there is a lack of ways for visually impaired people to obtain information through natural conversation while navigating the store, which limits their in-store experience.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0752] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying information about the surrounding environment to a visually impaired person, means for guiding a user to a destination using map information within a store, and means for conveying information about the features and layout of products within the store and facility information by voice, thereby enabling visually impaired people to move around a physical store and search for products smoothly.

[0753] "Means of providing directions to the destination" refers to a function that calculates the optimal route to the destination set by the user and provides instructions through voice guidance, etc.

[0754] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses natural language processing technology such as generative pre-training transformers (generative AI) to respond to questions and requests from users through natural conversation.

[0755] "Means of conveying information about the surrounding environment to visually impaired people" is a function that uses audio to convey the characteristics and colors of surrounding buildings, the location of facilities, etc., based on information obtained from map information services.

[0756] "A means of guiding users to their destination using in-store map information" is a function that calculates the optimal route to the destination set by the user based on in-store map information and provides instructions via voice guidance.

[0757] "Means for communicating information about the features and layout of products in a store and about facilities by voice" is a function that communicates information about the features and layout of products and facilities in a store to users by voice.

[0758] The system for implementing this invention is designed to assist visually impaired people in navigating and locating products in physical stores. The system is implemented as an application installed on a portable device such as a smartphone or smart glasses.

[0759] Hardware and Software

[0760] Hardware

[0761] Smartphone

[0762] Smart Glasses

[0763] software

[0764] OpenAI(R) API

[0765] Map Information Service API

[0766] Data processing and calculation

[0767] A means of providing directions to a destination

[0768] The server uses the map information service API to obtain route information between the user's current location and the destination, and provides voice guidance for each step based on the obtained route information.

[0769] A means of enabling conversations with users using natural language processing technology

[0770] The server uses a generative pre-training transformer (generative AI) to generate natural responses to user questions, allowing users to obtain information through natural conversation.

[0771] A means of conveying information about the surrounding environment to visually impaired people

[0772] Based on information obtained from the map information service, the server will provide audio information such as the characteristics and colors of surrounding buildings and the location of facilities, allowing the user to understand the surrounding environment.

[0773] A method of guiding users to their destination using in-store map information

[0774] The server calculates the optimal route to the destination set by the user based on the store's map information and provides voice guidance, allowing the user to move smoothly within the store.

[0775] A means of communicating the features and layout of products in stores and facility information through audio

[0776] The server will then provide users with voice information about the features and locations of products and facilities within the store, allowing them to easily find the products and facilities they are looking for.

[0777] Specific examples

[0778] For example, if a user asks, "What's in the fresh vegetable section?", the generative AI will respond, "The fresh vegetable section has fresh vegetables such as tomatoes, lettuce, and cucumbers."

[0779] Prompt Sentence Examples

[0780] User: What's in the fresh produce section?

[0781] Generative AI: The fresh vegetable section is stocked with fresh vegetables such as tomatoes, lettuce, and cucumbers.

[0782] In this way, a system will be created that helps visually impaired people navigate and search for products smoothly within a physical store.

[0783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0784] Step 1:

[0785] A user turns on their smartphone or smart glasses and starts a navigation application.

[0786] Input: User's current location and destination information

[0787] Output: Application startup and initialization

[0788] What happens: The user taps the application to launch it and enters their current location and destination.

[0789] Step 2:

[0790] The server uses the map information service API to obtain route information between the user's current location and the destination.

[0791] Input: User's current location and destination information

[0792] Output: Route information

[0793] Specific operation: The server sends a request to the map information service API to calculate and obtain the optimal route.

[0794] Step 3:

[0795] Based on the route information acquired by the server, instructions for each step are provided to the user via voice guidance.

[0796] Input: Route information

[0797] Output: Voice guidance

[0798] Specific operation: The server analyzes the route information and provides instructions for each step to the user via voice.

[0799] Step 4:

[0800] The user enters a question, and the server generates a response using a generative pre-training transformer (generative AI).

[0801] Input: User question

[0802] Output: Generative AI response

[0803] How it works: The user enters a question via voice or text, and the server uses generative AI to generate a natural-sounding response.

[0804] Step 5:

[0805] Based on information obtained from the map information service, the server will provide audio information such as the characteristics and colors of surrounding buildings and the location of facilities.

[0806] Input: Environmental information from a map information service

[0807] Output: Provides environmental information through audio

[0808] Specific operation: The server obtains environmental information from the map information service and conveys it to the user via voice.

[0809] Step 6:

[0810] The server guides the user to their destination based on map information within the store.

[0811] Input: Store map information

[0812] Output: Guidance by voice guidance

[0813] Specific operation: The server analyzes the map information within the store and guides the user to the desired location by voice.

[0814] Step 7:

[0815] The server communicates information about the features and layout of products and facilities in the store to the user via voice.

[0816] Input: Store product and facility information

[0817] Output: Provides product features and location information by voice

[0818] Specific operation: The server obtains information about products and facilities in the store and conveys it to the user via voice.

[0819] In this way, a system will be created that helps visually impaired people navigate and search for products smoothly within a physical store.

[0820] Example 2

[0821] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0822] There is a demand for navigation systems that allow visually impaired people to reach their destinations safely and efficiently. Conventional navigation systems do not provide sufficient information to visually impaired people, making it difficult to properly convey directions to their destination and information about the surrounding environment. They also lack the technology to enable natural dialogue with the user. This has made it difficult for visually impaired people to travel safely.

[0823] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0824] In this invention, the server includes a means for a user to set a destination, a means for the server to calculate an optimal route, and a means for the terminal to generate audio guidance. This allows visually impaired people to understand the route to their destination by voice and travel safely and efficiently. The server also includes a means for a user to ask questions or make requests, a means for the server to perform natural language processing, and a means for the terminal to transmit the generated conversation content by voice. This allows visually impaired people to have natural conversations with the system and obtain the necessary information. The server also includes a means for the server to acquire surrounding environmental information and a means for the terminal to transmit the environmental information by voice. This allows visually impaired people to understand environmental information such as the features and colors of surrounding buildings by voice and travel safely.

[0825] The "means for the user to set a destination" is a function that provides an interface for the user to specify a destination using voice input or text input.

[0826] The "means for the server to calculate the optimal route" is a function that uses a map information service to calculate the optimal route to the destination set by the user.

[0827] The "means for the terminal to generate voice guidance" is a function that uses a voice synthesis engine to create voice guidance that is easy for the user to understand, based on the calculated route information.

[0828] The "means for the user to receive audio guidance" is a function that enables the user to hear audio guidance provided by the terminal.

[0829] The "means for users to ask questions or make requests" is a function that provides an interface for users to ask questions or make requests to the system using voice input.

[0830] The "means for the server to perform natural language processing" refers to the function of using a generative pre-training transformer to generate natural conversations in response to questions and requests from users.

[0831] The "means for transmitting the conversation content generated by the terminal by voice" is a function that converts the conversation content received from the server into voice using a voice synthesis engine and transmits it to the user.

[0832] The "means for the server to acquire surrounding environmental information" is a function for acquiring information about surrounding buildings and features from a map information service.

[0833] "Means for the terminal to convey environmental information by voice" is a function that converts the acquired environmental information into voice using a voice synthesis engine and conveys it to the user.

[0834] The navigation system for visually impaired persons of the present invention is implemented in a portable device such as a smartphone, a wearable device, etc. Specific embodiments of this system will be described below.

[0835] Hardware and software used

[0836] Hardware: Smartphones, wearable devices

[0837] Software: Map services (e.g., Google Maps® API), speech synthesis engines (e.g., Google Text-to-Speech®), generative pre-training transformers (e.g., GPT-4®)

[0838] Program processing

[0839] 1. The user sets a destination.

[0840] The user launches an application on their smartphone or wearable device and sets their destination by voice or text input, for example, by saying, "I want to go to a nearby cafe."

[0841] 2. The server calculates the optimal route.

[0842] The server accesses a map information service and calculates the optimal route to the destination set by the user. For example, it uses Google Maps (registered trademark) API to obtain the optimal route from the current location to the nearest cafe.

[0843] 3. The device generates voice guidance.

[0844] The device generates voice guidance based on the route information received from the server. It uses a speech synthesis engine to create voice guidance that is easy for the user to understand. For example, it generates voice guidance such as, "Turn right and go 50 meters, and you will find a cafe on your left."

[0845] 4. The user receives the audio prompt.

[0846] The user listens to the voice guidance provided by the terminal and understands the route to the destination.

[0847] 5. The user has a question or request.

[0848] If the user has any questions or requests along the way, they can ask the question or request by voice, for example, "Which way should I turn at the next intersection?"

[0849] 6. The server performs natural language processing.

[0850] The server uses a generative pre-training transformer to generate natural-sounding conversations in response to user questions and requests. For example, it uses GPT-4 (registered trademark) to generate natural-sounding conversations such as "Turn right at the next intersection."

[0851] 7. The device will then transmit the generated conversation content aloud.

[0852] The terminal converts the conversation content received from the server into voice using a speech synthesis engine and conveys it to the user.

[0853] 8. The server obtains information about the surrounding environment.

[0854] The server obtains information about surrounding buildings and features from a map information service, such as the color and shape of the building and whether it has stairs.

[0855] 9. The device will provide audio information about the environment.

[0856] The device converts the acquired environmental information into speech using a speech synthesis engine and conveys it to the user. For example, it conveys environmental information to the user in a voice form, such as "There is a red brick building on your right" or "There are five steps ahead."

[0857] Examples of specific examples and prompts

[0858] Example: If a user says, "I want to go to a nearby cafe," the server uses a map information service to calculate the optimal route, and the device provides voice guidance such as, "Turn right and go 50 meters, and you will find the cafe on your left."

[0859] Example prompts: "If the user says they want to go to a nearby cafe, calculate the best route and provide voice directions."; "If the user asks about a nearby building, describe its features voice-over."

[0860] In this way, the navigation system for the visually impaired of the present invention can provide the user with voice guidance and natural conversation, and convey information about the surrounding environment.

[0861] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0862] Step 1:

[0863] The user sets the destination.

[0864] Input: The user launches an application on their smartphone or wearable device and sets a destination by voice or text input. For example, they might say, "I want to go to a nearby cafe."

[0865] Data processing: The device converts voice input into text data and extracts destination information.

[0866] Output: The destination information is sent to the server in text format.

[0867] Specific operation: When a user speaks, "I want to go to a nearby cafe," the device uses voice recognition technology to convert this into text and sends the destination information to the server.

[0868] Step 2:

[0869] The server calculates the optimal route.

[0870] Input: Destination information received by the server from the device.

[0871] Data processing: The server sends a request to a map information service (e.g., Google Maps (registered trademark) API) to calculate the optimal route from the current location to the destination.

[0872] Output: The optimal route information is sent from the server to the device.

[0873] Specific operation: The server accesses the Google Maps (registered trademark) API, obtains the optimal route from the current location to the nearest cafe, and sends that information to the device.

[0874] Step 3:

[0875] The terminal generates audio guidance.

[0876] Input: The best route information received from the server.

[0877] Data processing: The device uses a speech synthesis engine (e.g., Google Text-to-Speech®) to convert route information into voice guidance.

[0878] Output: Audio guidance is provided to the user.

[0879] Specific operation: The device generates and provides voice guidance to the user, such as "Turn right and go 50 meters, and you will find the cafe on your left."

[0880] Step 4:

[0881] The user receives audio guidance

[0882] Input: Voice guidance provided by the device.

[0883] Data processing: The user listens to and understands the audio guidance.

[0884] Output: The user gets directions to their destination.

[0885] Specific operation: The user listens to the voice guidance from the device and walks according to the instructions.

[0886] Step 5:

[0887] The user asks a question or makes a request

[0888] Input: The user speaks a question or request, for example, "Which way do I turn at the next intersection?"

[0889] Data processing: The device converts voice input into text data and extracts questions or requests.

[0890] Output: The question or request is sent to the server in text format.

[0891] Specific operation: When a user speaks a question such as "Which way should I turn at the next intersection?", the device uses voice recognition technology to convert the question into text and sends it to the server.

[0892] Step 6:

[0893] The server performs natural language processing.

[0894] Input: Questions or requests received from the device.

[0895] Data processing: The server uses a generative pre-training transformer (e.g., GPT-4 (registered trademark)) to generate natural-sounding conversations in response to questions and requests.

[0896] Output: The generated conversation content is sent from the server to the device.

[0897] Specific operation: The server uses GPT-4 to generate natural conversation such as "Turn right at the next intersection" and sends that content to the device.

[0898] Step 7:

[0899] The device then transmits the generated conversation content by voice.

[0900] Input: Generated conversation received from the server.

[0901] Data processing: The device uses a speech synthesis engine to convert the conversation into audio.

[0902] Output: The conversation is communicated to the user via audio.

[0903] Specific operation: The device conveys the generated conversation content to the user by voice.

[0904] Step 8:

[0905] The server acquires information about the surrounding environment.

[0906] Input: A request for surrounding environment information from a mapping service.

[0907] Data processing: The server obtains information about surrounding buildings and features from a map information service.

[0908] Output: The acquired environmental information is sent from the server to the device.

[0909] What it does: The server retrieves information about surrounding buildings and features from the Google Maps API and sends that information to the device.

[0910] Step 9:

[0911] The device will provide audio information about the environment.

[0912] Input: Environment information received from the server.

[0913] Data processing: The device uses a speech synthesis engine to convert environmental information into speech.

[0914] Output: Environmental information is communicated to the user via audio.

[0915] Specific operation: The device will verbally communicate environmental information to the user, such as "There is a red brick building on your right" and "There are five steps up ahead."

[0916] (Application example 2)

[0917] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0918] When visually impaired people shop comfortably in a brick-and-mortar store, they face challenges such as difficulty understanding product locations and store layouts. They also face challenges such as having to rely on others to obtain product information, making it difficult for them to enjoy shopping independently. Furthermore, they are unable to properly grasp information about their surroundings, making it difficult for them to navigate safely.

[0919] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0920] In this invention, the server includes a means for providing directions to a destination, a means for enabling conversation with a user using natural language processing technology, a means for conveying surrounding environmental information to a visually impaired person, a means for guiding the location of products in a store and the layout of the store, and a means for providing product information, thereby enabling visually impaired people to shop independently, safely, and comfortably in a physical store.

[0921] "Means of providing directions to the destination" refers to a function that calculates the optimal route to the destination set by the user and provides instructions using voice guidance, etc.

[0922] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses a generative pre-training transformer (generative AI) to respond to questions and requests from users through natural conversation.

[0923] "A means of conveying information about the surrounding environment to visually impaired people" is a function that uses audio to convey the characteristics and colors of surrounding buildings, the location of obstacles, etc., based on information obtained from map information services.

[0924] "Means of providing guidance on the location of products within the store and the store layout" refers to a function that obtains map information within the store, calculates the optimal route to the location of the product specified by the user, and provides voice guidance.

[0925] "Means for providing product information" refers to a function that provides product features, prices, stock status, etc. via voice.

[0926] A system for implementing this invention is designed to help visually impaired people shop independently, safely, and comfortably in brick-and-mortar stores. The system is implemented as an application installed on a portable device such as a smartphone or smart glasses.

[0927] Hardware and software used

[0928] Hardware: Smartphones, smart glasses

[0929] Software: Python (registered trademark), OpenAI (registered trademark) API, Google Maps (registered trademark) API, gTTS (Google Text-to-Speech (registered trademark))

[0930] System configuration

[0931] 1. Means of providing directions to a destination:

[0932] The server uses the Google Maps® API to calculate the optimal route from the user's current location to the destination.

[0933] The device provides the calculated route information to the user as voice guidance.

[0934] 2. Means of enabling conversation with users using natural language processing technology:

[0935] The server uses the OpenAI® API to generate responses to questions posed by the user.

[0936] The terminal plays the generated response aloud.

[0937] 3. Means of communicating information about the surrounding environment to the visually impaired:

[0938] The server obtains information such as the characteristics and colors of surrounding buildings and the location of obstacles based on information obtained from the map information service.

[0939] The device will then communicate the acquired information to the user via voice.

[0940] 4. Navigating product locations and store layout within the store:

[0941] The server obtains map information about the store and calculates the optimal route to the location of the product specified by the user.

[0942] The device provides the calculated route information to the user as voice guidance.

[0943] 5. Means of providing product information:

[0944] The server obtains information such as product features, prices, and stock status.

[0945] The terminal provides the acquired product information to the user by voice.

[0946] Specific examples

[0947] For example, if a user asks, "How much does this item cost?", the server uses the OpenAI (registered trademark) API to generate a response such as, "The price of this item is 1,000 yen." The device then plays this response aloud to the user.

[0948] Prompt Sentence Examples

[0949] User: How much does this item cost?

[0950] AI: The price of this item is 1,000 yen.

[0951] In this way, a navigation application can be realized that allows visually impaired people to enjoy shopping comfortably in physical stores.

[0952] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0953] Step 1:

[0954] The user starts up a smartphone or smart glasses and starts the navigation application. The user then inputs their destination and product information by voice. The input voice data is converted into text data by the device.

[0955] Step 2:

[0956] The device sends text data to the server, which uses the Google Maps (registered trademark) API to calculate the optimal route from the user's current location to the destination. The input is the user's current location and destination information, and the output is the optimal route information.

[0957] Step 3:

[0958] The server transmits the calculated route information to the terminal, which then provides the route information to the user as voice guidance by converting text to speech using gTTS (Google Text-to-Speech®) and playing it back to the user.

[0959] Step 4:

[0960] When a user wants to know product information, they input a question by voice. The input voice data is converted into text data by the terminal.

[0961] Step 5:

[0962] The device sends text data to the server, which uses the OpenAI® API to generate a response to the user's question. The input is the user's question text, and the output is the response text.

[0963] Step 6:

[0964] The server sends the generated response text to the terminal, which then plays the response text aloud and provides it to the user. Specifically, the terminal converts the text into speech using gTTS and plays it back to the user.

[0965] Step 7:

[0966] The server obtains environmental information such as the characteristics and colors of surrounding buildings and the location of obstacles based on information obtained from the map information service. The input is data from the map information service, and the output is environmental information.

[0967] Step 8:

[0968] The server sends the acquired environmental information to the device, which then communicates the environmental information to the user by voice. Specifically, it uses gTTS to convert text to voice and plays it back to the user.

[0969] Step 9:

[0970] The server obtains map information about the store and calculates the optimal route to the location of the product specified by the user. The input is the map information about the store and the product information specified by the user, and the output is the optimal route information.

[0971] Step 10:

[0972] The server sends the calculated route information to the device, which then provides the route information to the user as voice guidance by converting text to speech using gTTS and playing it back to the user.

[0973] Step 11:

[0974] The server obtains information such as product features, prices, and stock status. The input is data from the product database, and the output is product information.

[0975] Step 12:

[0976] The server sends the acquired product information to the terminal, which then provides the product information to the user by voice. Specifically, the terminal converts text to voice using gTTS and plays it back to the user.

[0977] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0978] "Example 1"

[0979] In one embodiment of the present invention, an emotion engine is incorporated into a navigation system for the visually impaired. This emotion engine has the function of recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to the emotion. Specifically, if the user makes a sound that indicates joy or relief, the emotion engine recognizes this and the voice guidance issues instructions in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the emotion engine recognizes this and the voice guidance issues instructions in a clearer and more forceful tone. This makes it possible to provide appropriate guidance according to the user's emotional state.

[0980] "Example 2"

[0981] In one embodiment of the present invention, an emotion engine is incorporated into a navigation system for the visually impaired. This emotion engine has the function of recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to the emotion. Specifically, if the user makes a sound that indicates joy or relief, the emotion engine recognizes this and the voice guidance issues instructions in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the emotion engine recognizes this and the voice guidance issues instructions in a clearer and more forceful tone. This makes it possible to provide appropriate guidance according to the user's emotional state.

[0982] The processing flow of each embodiment will be described below.

[0983] "Example 1"

[0984] Step 1: The user speaks their destination, which is converted into text by the system and recognized as a destination.

[0985] Step 2: The system uses a map information service to calculate the optimal route from your current location to your destination.

[0986] Step 3: The system uses generative AI to generate voice guidance based on the calculated route.

[0987] Step 4: The user provides vocal feedback to the system, which is analyzed by the emotion engine to recognize the user's emotional state.

[0988] Step 5: The emotion engine adjusts the tone of the voice guidance based on the recognized emotion. For example, if the user vocalizes with a voice that indicates joy or relief, the voice guidance will give instructions in a gentler tone. Conversely, if the user vocalizes with a voice that indicates anxiety or fear, the voice guidance will give instructions in a clearer and more forceful tone.

[0989] Example 1

[0990] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0991] In order for visually impaired people to reach their destination safely and efficiently, they need not only directions but also appropriate guidance that takes into account the surrounding environmental information and the user's emotional state. However, conventional navigation systems have difficulty providing these elements in an integrated manner, and are therefore unable to provide sufficient support for visually impaired people. Therefore, a system that enables visually impaired people to travel safely and efficiently is needed.

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0993] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, and means for recognizing the user's emotions and adjusting the tone of the voice guidance. This enables a visually impaired person to not only be provided with directions but also receive appropriate guidance according to the surrounding environmental information and emotions when heading to a destination.

[0994] "Means for providing directions to a destination" is a function that calculates the optimal route to the destination set by the user and provides that information to the user.

[0995] "Means for enabling conversation with users using natural language processing technology" refers to a function that converts voice input from users into text data, generates an appropriate response using natural language processing technology such as generative pre-training transformers, and conveys it back to the user as voice.

[0996] "Means for conveying surrounding environmental information to visually impaired people" is a function that conveys information such as the characteristics and colors of surrounding buildings obtained from a map information service to the user via voice.

[0997] The "means for recognizing the user's emotions and adjusting the tone of the voice guidance" is a function for recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to those emotions.

[0998] A "generative pre-training transformer" is a natural language processing model that is pre-trained using large amounts of text data, and is a technology for generating appropriate responses to user input.

[0999] A "map information service" is an online service that provides geographical information and is used to obtain a user's current location, destination, surrounding environmental information, etc.

[1000] The navigation system for visually impaired persons of the present invention is implemented in a portable device such as a smartphone, a wearable device, etc. Specific embodiments of this system will be described below.

[1001] Hardware and Software Configuration

[1002] Hardware

[1003] Devices: Smartphones, wearable devices (e.g., smartwatches, earphones)

[1004] Server: Cloud server

[1005] software

[1006] Map information services: Online map services (e.g., Google Maps (registered trademark) API)

[1007] Generative pre-training transformers: Natural language processing models (e.g., GPT-4 (registered trademark))

[1008] Emotion Engine: Software for Recognizing Emotions from Speech

[1009] Data processing and calculation

[1010] 1. Setting a destination and calculating a route

[1011] Users open an application on their smartphone or wearable device and enter their destination by voice or text.

[1012] The terminal transmits the input destination information to the server.

[1013] The server uses a map information service to calculate the optimal route to the destination set by the user.

[1014] The server sends the calculation results to the terminal.

[1015] 2. Audio Guidance Generation

[1016] The device generates voice guidance based on the received route information.

[1017] The device provides voice guidance to the user through the smartphone's built-in speaker or the wearable device's earphones.

[1018] 3. Conversational features based on natural language processing

[1019] If the user has any questions or requests while navigating, they can input the questions or requests by voice.

[1020] The terminal converts the input voice into text data.

[1021] The device sends the converted text data to the generative AI model as a prompt.

[1022] The generative AI model generates an appropriate response based on the prompt it receives.

[1023] The generative AI model returns the generated response to the device.

[1024] The device converts the response received from the generative AI model into speech and conveys it to the user.

[1025] 4. Providing information about the surrounding environment

[1026] The server obtains environmental information such as the characteristics and colors of surrounding buildings from a map information service.

[1027] The server transmits the acquired environmental information to the terminal.

[1028] The device generates audio guidance based on the received environmental information.

[1029] The device will then provide the user with specific information via voice, such as "There is a red brick building on your right" or "There are five steps ahead."

[1030] 5. Adjusting the tone of voice guidance using an emotion engine

[1031] The device is equipped with an emotion engine that recognizes emotions from the user's voice.

[1032] If the user makes vocalizations that indicate joy or relief, the emotion engine will recognize this and the voice guidance will provide instructions in a gentler tone.

[1033] If the user makes vocalizations that indicate anxiety or fear, the emotion engine will recognize this and the voice guidance will provide instructions in a clearer and more forceful tone.

[1034] Examples of concrete examples and prompts

[1035] Specific examples

[1036] The user asks, "How do I turn right at the next intersection?"

[1037] The device converts this question into text data and sends it as a prompt to the generative AI model.

[1038] The generative AI model generates the response, "To turn right at the next intersection, go 50 meters and then turn right."

[1039] The device converts this response into speech and relays it to the user.

[1040] Prompt Sentence Examples

[1041] "How do I turn right at the next intersection?"

[1042] "Please tell me the nearest cafe."

[1043] "Are there any obstacles ahead?"

[1044] In this way, the navigation system for the visually impaired of the present invention can provide appropriate guidance and information to the user.

[1045] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1046] Step 1:

[1047] Users open an application on their smartphone or wearable device and enter their destination by voice or text.

[1048] Input: User's voice or text destination information

[1049] Output: Destination information is sent to the device

[1050] Specific operation: The user speaks, "Please tell me the way to the station."

[1051] Step 2:

[1052] The terminal transmits the input destination information to the server.

[1053] Input: Destination information from the user

[1054] Output: Destination information is sent to the server

[1055] Specific operation: The device converts voice input into text data and sends it to the server.

[1056] Step 3:

[1057] The server uses a map information service to calculate the optimal route to the destination set by the user.

[1058] Input: Destination information

[1059] Output: Optimal route information

[1060] Specific operation: The server accesses the map information service and calculates the route from the current location to the destination.

[1061] Step 4:

[1062] The server sends the calculation results to the terminal.

[1063] Input: Optimal route information

[1064] Output: Route information is sent to the device

[1065] Specific operation: The server sends the calculated route information to the terminal.

[1066] Step 5:

[1067] The device generates voice guidance based on the received route information.

[1068] Input: Route information

[1069] Output: Voice guidance

[1070] Specific operation: The device will give voice instructions such as "Go 50 meters and turn right."

[1071] Step 6:

[1072] If the user has any questions or requests while navigating, they can input the questions or requests by voice.

[1073] Input: The user's spoken question or request

[1074] Output: Questions and requests are sent to the terminal

[1075] Specific Action: The user asks, "How do I turn right at the next intersection?"

[1076] Step 7:

[1077] The terminal converts the input voice into text data.

[1078] Input: User's voice data

[1079] Output: Text data

[1080] Specific operation: The device uses voice recognition technology to convert speech into text.

[1081] Step 8:

[1082] The device sends the converted text data to the generative AI model as a prompt.

[1083] Input: Text data

[1084] Output: prompts to the generative AI model

[1085] Specific operation: The device sends text data to the generative AI model.

[1086] Step 9:

[1087] The generative AI model generates an appropriate response based on the prompt it receives.

[1088] Input: prompt

[1089] Output: Response text

[1090] Specific behavior: The generative AI model generates the response, "To turn right at the next intersection, go 50 meters and then turn right."

[1091] Step 10:

[1092] The generative AI model returns the generated response to the device.

[1093] Input: Response text

[1094] Output: The response text is sent to the terminal

[1095] Specific operation: The response generated by the generative AI model is sent to the terminal.

[1096] Step 11:

[1097] The device converts the response received from the generative AI model into speech and conveys it to the user.

[1098] Input: Response text

[1099] Output: Voice guidance

[1100] Specific operation: The device will tell the user by voice, "To turn right at the next intersection, go 50 meters and then turn right."

[1101] Step 12:

[1102] The server obtains environmental information such as the characteristics and colors of surrounding buildings from a map information service.

[1103] Input: User's current location

[1104] Output: Surrounding environment information

[1105] Specific operation: The server accesses the map information service and obtains the characteristics and colors of surrounding buildings.

[1106] Step 13:

[1107] The server transmits the acquired environmental information to the terminal.

[1108] Input: Surrounding environment information

[1109] Output: Environment information is sent to the terminal

[1110] Specific operation: The server sends the acquired environmental information to the terminal.

[1111] Step 14:

[1112] The device generates audio guidance based on the received environmental information.

[1113] Input: Environment information

[1114] Output: Voice guidance

[1115] Specific action: The device will tell the user, "There is a red brick building on your right."

[1116] Step 15:

[1117] The device is equipped with an emotion engine that recognizes emotions from the user's voice.

[1118] Input: User's voice data

[1119] Output: Emotion recognition result

[1120] Specific operation: The device analyzes the user's voice and recognizes emotions.

[1121] Step 16:

[1122] If the user makes vocalizations that indicate joy or relief, the emotion engine will recognize this and the voice guidance will provide instructions in a gentler tone.

[1123] Input: Emotion recognition results

[1124] Output: Gentle voice guidance

[1125] Specific action: The device will give instructions in a gentle tone, such as "Turn right at the next intersection."

[1126] Step 17:

[1127] If the user makes vocalizations that indicate anxiety or fear, the emotion engine will recognize this and the voice guidance will provide instructions in a clearer and more forceful tone.

[1128] Input: Emotion recognition results

[1129] Output: Clear and powerful voice guidance

[1130] Specific action: The device will issue instructions in a clear, forceful tone, such as "Turn right at the next intersection."

[1131] (Application example 1)

[1132] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1133] When visually impaired people use self-driving vehicles, they face challenges in understanding the route to their destination, grasping information about their surroundings, and receiving appropriate guidance according to their emotional state. To solve these challenges, a comprehensive navigation system is needed to enable visually impaired people to use self-driving vehicles with peace of mind.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1135] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, means for recognizing the user's emotions and adjusting the tone of the voice guidance according to the emotions, and means installed in the self-driving vehicle, thereby enabling a visually impaired person using a self-driving vehicle to understand the route to the destination, grasp information about the surrounding environment, and receive appropriate guidance according to their emotional state.

[1136] "Means for providing directions to a destination" means a function that calculates the optimal route to a destination set by the user and provides instructions to the user by voice guidance or other means.

[1137] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses natural language processing technology such as generative pre-training transformers (generative AI) to respond to questions and requests from users through natural conversation.

[1138] "Means for communicating surrounding environmental information to visually impaired people" refers to a function that communicates environmental information such as the characteristics and colors of surrounding buildings and obstacles to visually impaired people via voice, based on information obtained from map information services.

[1139] "Means for recognizing the user's emotions and adjusting the tone of the voice guidance according to those emotions" refers to a function that recognizes emotions from the user's voice and adjusts the tone of the voice guidance according to those emotions. For example, if the user is feeling anxious, instructions will be given in a clearer and stronger tone.

[1140] "Means installed in an automated vehicle" refers to a navigation system for visually impaired people that is installed in an automated vehicle to provide appropriate navigation and guidance to visually impaired people while the vehicle is in operation.

[1141] The system for implementing the present invention is a navigation system for the visually impaired, and includes the following main functions:

[1142] 1. System Program

[1143] The system includes a means for providing directions to a destination, a means for enabling a conversation with a user using natural language processing technology, a means for conveying surrounding environmental information to visually impaired people, a means for recognizing a user's emotions and adjusting the tone of the voice guidance according to the emotions, and a means for being installed in an autonomous vehicle.

[1144] 2. Program processing explanation

[1145] Hardware and Software

[1146] Hardware: Smartphones, smart glasses, head-mounted displays, and embedded systems in autonomous vehicles

[1147] Software: OpenAI (registered trademark) API, Google Maps (registered trademark) API, gTTS (Google Text-to-Speech (registered trademark), playsound (registered trademark)

[1148] Data processing and calculation

[1149] Means for providing directions to the destination: The server uses Google Maps (registered trademark) API to calculate the optimal route to the destination set by the user. The calculated route information is provided to the user as voice guidance.

[1150] Means of enabling conversation with users using natural language processing technology: The server uses OpenAI®'s Generative Pre-Trained Transformer (generative AI) to respond to user questions and requests through natural conversation.

[1151] Means of communicating surrounding environmental information to visually impaired people: Based on information obtained from the Google Maps (registered trademark) API, the server communicates environmental information such as the characteristics and colors of surrounding buildings and obstacles to visually impaired people by voice.

[1152] A means to recognize the user's emotions and adjust the tone of the voice guidance according to those emotions: The server analyzes the emotions from the user's voice and adjusts the tone of the voice guidance according to those emotions. For example, if the user is feeling anxious, instructions will be given in a clearer and more forceful tone.

[1153] Inclusion in self-driving vehicles: The system will be integrated into self-driving vehicles to provide appropriate navigation and guidance to visually impaired people while the vehicle is in operation.

[1154] 3. Specific Examples

[1155] If the user says, "I want to go to Tokyo Tower," the system will provide voice guidance saying, "Your destination is Tokyo Tower. There is a red brick building on your right."

[1156] Prompt Sentence Examples

[1157] User: I want to go to Tokyo Tower.

[1158] System: Your destination is Tokyo Tower. On your right is a red brick building.

[1159] In this way, an automated vehicle navigation system for the visually impaired is realized.

[1160] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1161] Step 1:

[1162] The user sets the destination.

[1163] The user specifies a destination through voice input. For example, they might say, "I want to go to Tokyo Tower." The device receives this voice input and converts it into text data. The input is the user's voice data, and the output is text data.

[1164] Step 2:

[1165] The server calculates the route to the destination.

[1166] The server uses the Google Maps (registered trademark) API to calculate the optimal route to the destination set by the user. The input is text data of the destination specified by the user, and the output is JSON data containing route information. The server parses this JSON data and generates text for voice guidance.

[1167] Step 3:

[1168] The server conducts the conversation with the user.

[1169] The server uses OpenAI's (registered trademark) generative pre-trained transformer (generative AI) to respond to user questions and requests through natural conversation. The input is the user's text data, and the output is a generated response text. The server converts this response text into speech and sends it to the device.

[1170] Step 4:

[1171] The server acquires information about the surrounding environment.

[1172] The server obtains environmental information such as the characteristics and colors of surrounding buildings and obstacles based on information obtained from the Google Maps (registered trademark) API. The input is the coordinate data of the current location, and the output is JSON data containing environmental information. The server analyzes this JSON data and generates text for voice guidance.

[1173] Step 5:

[1174] The server recognizes the user's emotions.

[1175] The server analyzes the user's voice to determine their emotions. The input is the user's voice data, and the output is data indicating their emotional state. The server adjusts the tone of the voice guidance based on this emotional data.

[1176] Step 6:

[1177] The server provides the audio guidance.

[1178] The server converts the generated text for the voice guidance into audio using gTTS (Google Text-to-Speech (registered trademark)) and sends it to the terminal. The input is text data for the voice guidance, and the output is audio data. The terminal plays back this audio data and provides it to the user.

[1179] Step 7:

[1180] Autonomous vehicles perform navigation.

[1181] The autonomous vehicle travels to its destination based on route information and voice guidance received from the server. The input is route information and voice guidance, and the output is the vehicle's operating status. The vehicle plays the voice guidance at the appropriate time and provides it to the user.

[1182] Example 2

[1183] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1184] In order for visually impaired people to reach their destination safely and efficiently, they need not only directions but also appropriate guidance that takes into account the surrounding environmental information and the user's emotional state. However, conventional navigation systems have difficulty providing these elements in an integrated manner, and have faced the challenge of not providing sufficient support for visually impaired people to travel safely.

[1185] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1186] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, and means for recognizing the user's emotions and adjusting the tone of the voice guidance, thereby enabling comprehensive support for visually impaired people to reach their destination safely and efficiently.

[1187] "Means for providing directions to a destination" is a function that calculates the optimal route to a destination set by the user and guides the user along that route.

[1188] "Means for enabling conversation with users using natural language processing technology" refers to technology that enables responses in natural language to questions or requests entered by users via voice or text.

[1189] "A means of conveying information about the surrounding environment to visually impaired people" is a function that conveys information about surrounding buildings, terrain features, colors, etc. obtained from map information services to users via voice.

[1190] "Means for recognizing the user's emotions and adjusting the tone of the voice guidance" refers to a function that analyzes the emotions from the user's voice and changes the tone of the voice guidance according to those emotions.

[1191] A "generative pre-training transformer" is a natural language processing model that is pre-trained based on large amounts of text data, and is a technology for generating appropriate responses to user input.

[1192] "Map information service" means an online service that provides geographic information and is used to obtain a user's current location, destination, and surrounding environmental information.

[1193] The present invention relates to a navigation system for visually impaired people, which is implemented in a portable device such as a smartphone or a wearable device. Specific embodiments of this system will be described below.

[1194] First, the user sets their destination using their smartphone or wearable device by voice or text input, for example, "I want to go to the nearest cafe."

[1195] The server accesses a map information service (e.g., Google Maps® API) and calculates the optimal route from the user's current location to the destination, taking into account traffic conditions and pedestrian route information.

[1196] The device generates voice guidance based on the route information received from the server, providing specific instructions to the user, such as "Turn right 100 meters ahead."

[1197] While traveling, the user may voice-input additional questions or requests, such as "What kind of place is this?" or "How far is the next intersection?" The device converts the user's voice input into text and inputs it as a prompt to a generative AI model (e.g., GPT-4 (registered trademark)). For example, it generates a prompt such as "What kind of place is this?"

[1198] The generative AI model generates an appropriate response based on the prompt sentence, such as "There are red brick buildings around your current location. There are five steps ahead." The device then converts this response into speech and relays it to the user, allowing the user to receive information about their surroundings and additional instructions via voice.

[1199] Furthermore, the device uses information obtained from the map information service to provide voice information about the characteristics and colors of surrounding buildings, providing specific information such as "There is a red brick building on your right" or "There are five steps ahead."

[1200] The device is equipped with an emotion engine that recognizes emotions from the user's voice. This emotion engine adjusts the tone of the voice guidance according to the user's emotions. For example, if the user makes a voice that indicates joy or relief, the device will provide voice guidance in a gentler tone. Conversely, if the user makes a voice that indicates anxiety or fear, the device will provide voice guidance in a clearer and more forceful tone.

[1201] For example, if a user says "I want to go to the nearest cafe," the following specific actions will be taken:

[1202] 1. The user speaks, "I want to go to the nearest cafe."

[1203] 2. The server accesses a map information service and calculates the optimal route from the current location to the nearest cafe.

[1204] 3. The device will give the user a voice command saying, "Turn right in 100 meters."

[1205] 4. The user asks, "What kind of place is this?"

[1206] 5. The device inputs the prompt sentence "What kind of place is this?" into the generative AI model.

[1207] 6. The generative AI model generates a response: "There are red brick buildings around your current location. There are five steps ahead."

[1208] 7. The device converts this response into speech and relays it to the user.

[1209] 8. The device provides information about the surrounding environment, such as "There is a red brick building on your right."

[1210] 9. If the user expresses anxiety, the device will provide voice guidance in a clearer and more forceful tone.

[1211] In this way, navigation systems for the visually impaired support users in their journeys and provide them with information about their surrounding environment.

[1212] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1213] Step 1:

[1214] The user sets the destination.

[1215] Users can use a smartphone or wearable device to set their destination by voice or text input. For example, they can say, "I want to go to the nearest cafe." The input speech is converted into text by the device's voice recognition function. The output is the destination information in text format.

[1216] Step 2:

[1217] The server calculates the optimal route.

[1218] The server accesses a map information service (e.g., Google Maps (registered trademark) API) and receives the user's current location and destination as input. The server calculates the optimal route based on this information. The calculation also takes into account traffic conditions and pedestrian route information. The output is the optimal route information.

[1219] Step 3:

[1220] The device provides voice guidance.

[1221] The device generates voice guidance based on the route information received from the server. For example, it gives specific instructions to the user by voice, such as "Turn right 100 meters ahead." The input is the route information from the server, and the output is the voice guidance.

[1222] Step 4:

[1223] The user asks a question or makes a request.

[1224] The user voices additional questions or requests as they move along, such as "What is the location like?" or "How far is the next intersection?" The input is the user's voice input, and the output is the question or request in text form.

[1225] Step 5:

[1226] The device inputs a prompt sentence into the generative AI model.

[1227] The device converts the user's voice input into text and inputs it as a prompt to a generative AI model (e.g., GPT-4 (registered trademark)). For example, the device generates a prompt such as "User: What kind of place is this?" The input is the user's textual question or request, and the output is the prompt.

[1228] Step 6:

[1229] A generative AI model generates a response.

[1230] The generative AI model generates an appropriate response based on the prompt sentence, for example, "There are red brick buildings around your current location. There are five steps ahead." The input is the prompt sentence, and the output is a text response.

[1231] Step 7:

[1232] The device will then speak the response to the user.

[1233] The device converts the response from the generative AI model into speech and transmits it to the user, allowing the user to receive information about the surrounding environment and additional instructions by voice. The input is a text response from the generative AI model, and the output is a speech response.

[1234] Step 8:

[1235] The device provides information about the surrounding environment.

[1236] The device uses information obtained from the map information service to provide voice information about the characteristics and colors of surrounding buildings. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead." The input is environmental information from the map information service, and the output is environmental information in the form of voice.

[1237] Step 9:

[1238] The device recognizes the user's emotions and adjusts the tone of the voice guidance.

[1239] The device uses an emotion engine that recognizes emotions from the user's voice. If the user makes a sound that indicates joy or relief, the device will provide voice guidance in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the device will provide voice guidance in a clearer and more forceful tone. The input is the user's voice, and the output is voice guidance in the adjusted tone.

[1240] (Application example 2)

[1241] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1242] When visually impaired people use self-driving vehicles, it is difficult for them to properly understand the route to their destination and the surrounding environment.In addition, it is not possible to provide appropriate voice guidance according to the user's emotional state, making it difficult for them to travel safely.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1244] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, means for adjusting the tone of the voice guidance according to the emotional state of the user, and means for providing voice guidance to a visually impaired person in an autonomous vehicle. This enables a visually impaired person using an autonomous vehicle to properly understand directions to a destination and surrounding environmental information, and to receive appropriate voice guidance according to the emotional state of the user.

[1245] A "navigation system for the visually impaired" is a system that assists visually impaired users in traveling to their destination safely and efficiently.

[1246] A "means for providing directions to a destination" is a means that has the function of calculating the optimal route to a destination set by the user and communicating those directions to the user.

[1247] "Means for enabling conversation with a user using natural language processing technology" means means for realizing natural conversation with a user using technology that understands the user's voice input and generates an appropriate response.

[1248] "Means for conveying information about the surrounding environment to visually impaired people" refers to a means for conveying information about surrounding buildings and obstacles obtained from map information services, etc. to users via voice.

[1249] The "means for adjusting the tone of the voice guidance according to the emotional state of the user" refers to a means having a function of recognizing emotions from the user's voice and changing the tone of the voice guidance according to the emotions.

[1250] "Means for providing audio guidance to visually impaired persons in self-driving vehicles" refers to means for providing audio directions to a destination and information about the surrounding environment to visually impaired persons using self-driving vehicles.

[1251] A "map information service" is an online service that provides geographical information and is used for route calculations and obtaining information about the surrounding environment.

[1252] A "generative pre-trained transformer" (generative AI) is a natural language processing model that is pre-trained using large amounts of text data and is used to generate natural conversations with users.

[1253] A system for carrying out the present invention is implemented as a navigation system for the visually impaired in a portable device such as a smartphone or smart glasses. The system includes means for providing directions to a destination, means for enabling a conversation with a user using natural language processing technology, means for conveying surrounding environmental information to the visually impaired, means for adjusting the tone of the voice guidance depending on the emotional state of the user, and means for providing voice guidance to the visually impaired in an autonomous vehicle.

[1254] Hardware and software used

[1255] Hardware: Smartphones, smart glasses

[1256] Software: Python (registered trademark), Google Maps (registered trademark) API, Hugging Face Transformers (registered trademark), pyttsx3 (registered trademark)

[1257] System Operation

[1258] 1. Obtaining map information: The server uses the Google Maps (registered trademark) API to obtain route information from the user's current location to the destination, allowing it to calculate the optimal route to the destination set by the user.

[1259] 2. Voice Guidance: The server uses pyttsx3 (registered trademark) to provide voice guidance based on the acquired route information, allowing the user to confirm the route to the destination by voice.

[1260] 3. Emotion Recognition: The server uses the Hugging Face® emotion recognition model to recognize emotions from the user's voice, allowing it to adjust the tone of the voice guidance according to the user's emotional state.

[1261] 4. Conversation Processing: The server uses Hugging Face®'s generative AI model to enable natural conversation with the user, generating appropriate responses to the user's questions and requests.

[1262] Specific examples

[1263] For example, consider a scenario in which a user gets into an autonomous vehicle and sets a destination.

[1264] User: "Shinjuku Station, please."

[1265] System: "Calculating route to Shinjuku Station. There is a red brick building on your right. Turn left at the next intersection."

[1266] In this way, we can support visually impaired people in self-driving vehicles so that they can travel to their destination safely.

[1267] Prompt Sentence Examples

[1268] User: To Shinjuku Station, please.

[1269] System: Calculating route to Shinjuku Station. There is a red brick building on your right. Turn left at the next intersection.

[1270] This invention enables visually impaired people using self-driving vehicles to properly understand the route to their destination and information about the surrounding environment, and to be provided with appropriate voice guidance that matches the user's emotional state.

[1271] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1272] Step 1:

[1273] The user sets the destination.

[1274] Input: The user specifies a destination by voice (e.g., "Shinjuku Station, please.").

[1275] How it works: The device receives voice input and converts it to text using speech recognition technology.

[1276] Output: Destination information in text format (e.g. "Shinjuku Station").

[1277] Step 2:

[1278] The server obtains route information using a map information service.

[1279] Input: Textual destination information and user's current location information.

[1280] How it works: The server calls the Google Maps (registered trademark) API to calculate the optimal route from the current location to the destination.

[1281] Output: Route information (e.g. directions, distance, time required, etc.).

[1282] Step 3:

[1283] The server generates audio guidance based on the route information.

[1284] Input: Route information.

[1285] How it works: The server uses pyttsx3 to convert route information into voice guidance.

[1286] Output: Voice guidance (e.g. "Turn left at the next intersection.").

[1287] Step 4:

[1288] The server recognizes the user's emotional state.

[1289] Input: User's voice input.

[1290] How it works: The server uses the Hugging Face® emotion recognition model to recognize emotions from the user's voice.

[1291] Output: Emotion information (e.g., "joy", "anxiety", etc.).

[1292] Step 5:

[1293] The server adjusts the tone of the voice guidance based on the emotion information.

[1294] Input: Emotional information and audio guidance.

[1295] Operation: The server adjusts the voice synthesis parameters (speed, pitch, etc.) of pyttsx3 (registered trademark) according to the emotion information.

[1296] Output: Adjusted voice guidance (e.g., "Turn left at the next intersection" (gentle tone)).

[1297] Step 6:

[1298] The server handles the conversation with the user.

[1299] Input: User's voice input.

[1300] How it works: The server uses Hugging Face®'s generative AI model to generate appropriate responses to the user's questions and requests.

[1301] Output: Response text (e.g., "Calculating route to Shinjuku Station.").

[1302] Step 7:

[1303] The server converts the response text into speech and provides it to the user.

[1304] Input: The response text.

[1305] How it works: The server uses pyttsx3® to convert the response text into speech.

[1306] Output: A voice response (e.g., "Calculating route to Shinjuku Station.").

[1307] In this way, when visually impaired people use self-driving vehicles, they will be able to properly understand the route to their destination and information about the surrounding environment, and will be able to receive appropriate voice guidance that matches the user's emotional state.

[1308] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1309] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1310] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1311] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1312] [Third embodiment]

[1313] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1314] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1315] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1316] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1317] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1318] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1319] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1320] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1321] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1322] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1323] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1324] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1325] "Example 1"

[1326] The navigation system for visually impaired people of the present invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[1327] "Example 2"

[1328] The navigation system for visually impaired people of this invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[1329] The processing flow of each embodiment will be described below.

[1330] "Example 1"

[1331] Step 1: The user activates the system using a portable device such as a smartphone or wearable device.

[1332] Step 2: The user sets a destination using voice or touch input.

[1333] Step 3: The system connects with the map information service to calculate the optimal route to the set destination.

[1334] Step 4: The system will provide voice guidance for directions.

[1335] Step 5: Once the user begins their journey, if they have any questions or requests about the surrounding environment along the way, the system will respond through natural conversation using a generative pre-trained transformer (generative AI).

[1336] Step 6: The system uses information obtained from the map information service to provide a voice message describing the characteristics and colors of surrounding buildings.

[1337] Example 1

[1338] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1339] To reach their destination safely and efficiently, visually impaired people need detailed route guidance and information about their surrounding environment. However, conventional navigation systems have difficulty providing sufficient information to visually impaired people, particularly in terms of conveying information about the surrounding environment and building features. Furthermore, they lack the ability to respond to real-time questions and requests through natural conversation with the user. This often leads to anxiety and difficulty for visually impaired people when trying to reach their destination.

[1340] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1341] In this invention, the server includes: means for a user to set a destination; means for a terminal to transmit destination information to the server; means for the server to calculate an optimal route using a map information service; means for the server to transmit the route information calculated by the server to the terminal; means for the terminal to generate audio guidance based on the route information; means for the user to receive the audio guidance; means for the user to ask a question or make a request; means for the terminal to respond to the user's question or request using a generative AI model; means for the server to acquire surrounding environmental information and transmit it to the terminal; and means for the terminal to convey surrounding information to the user by voice based on the environmental information. This enables visually impaired people to receive detailed route guidance and surrounding environmental information by voice when heading to their destination, and can also respond to questions and requests in real time through natural conversation.

[1342] "User" refers to any individual who uses the navigation system, including visually impaired individuals.

[1343] "Terminal" refers to a portable device such as a smartphone or wearable device.

[1344] "Server" refers to a computer system that processes data using map information services and generative AI models.

[1345] "Destination information" refers to data relating to a destination set by a user.

[1346] "Map Information Service" refers to an online service that provides geographic information.

[1347] An "optimal route" refers to the most efficient route from your current location to your destination.

[1348] "Audio Guidance" means audio directions generated using text-to-speech software.

[1349] "Generative AI model" refers to natural language processing technology that uses generative pre-training transformers.

[1350] "Surrounding environmental information" refers to information such as the characteristics and color of a building, whether or not it has stairs, etc., obtained from map information services.

[1351] "Natural language processing technology" refers to language understanding and generation technology that enables conversation with a user.

[1352] This invention relates to a navigation system for the visually impaired, implemented on a portable device such as a smartphone or wearable device, that provides detailed route guidance and surrounding environmental information by voice as the user travels to their destination, and can respond to questions and requests in real time through natural conversation.

[1353] The server uses a map information service (e.g., Google Maps® API) to calculate the optimal route to the destination set by the user. The calculated route information is sent to the device over the Internet. The device uses text-to-speech software (e.g., Google Text-to-Speech®) to generate audio guidance based on the received route information.

[1354] Users can check the route to their destination through voice guidance. If they have questions or additional requests along the way, they can use voice input to ask questions or make requests. The device uses a generative pre-training transformer (e.g., OpenAI® GPT-3®) to generate natural-sounding responses to the user's questions and requests.

[1355] The server obtains information about the surrounding environment from a map information service and sends it to the device. Based on the obtained information, the device uses text-to-speech software to communicate information about the surroundings to the user aloud. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead."

[1356] As a specific example, when a user asks "Can you tell me the way to the station?", the following processing is performed.

[1357] 1. User: "Can you tell me the way to the station?"

[1358] 2. The device converts the voice input into text and sends it to the server.

[1359] 3. The server uses a map information service to calculate the optimal route and sends the route information to the device.

[1360] 4. The device generates voice guidance based on the route information and provides it to the user.

[1361] 5. User: "How far is the next intersection?"

[1362] 6. Using the generative AI model, the device responds, "The next intersection is approximately 200 meters away."

[1363] 7. The server acquires information about the surrounding environment and sends it to the device.

[1364] 8. The device will tell the user by voice, "There is a red brick building on your right."

[1365] In this way, navigation systems for the visually impaired provide users with directions to their destination through voice guidance and natural conversation.

[1366] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1367] Step 1:

[1368] The user sets the destination.

[1369] The user launches an application on their smartphone or wearable device and sets their destination using voice or text input. For example, they might say, "Please tell me the way to the station." The voice input is converted into text data by the device.

[1370] Step 2:

[1371] The terminal transmits the destination information to the server.

[1372] The terminal transmits the destination information entered by the user to the server as text data. Specifically, the destination information is transmitted to the server via the Internet. The input is text data, and the output is a transmission completion notification to the server.

[1373] Step 3:

[1374] The server uses a map information service to calculate the optimal route.

[1375] The server uses a map information service (e.g., Google Maps (registered trademark) API) based on the received destination information to calculate the optimal route from the current location to the destination. The input is destination information, and the output is optimal route information.

[1376] Step 4:

[1377] The server sends the calculated route information to the terminal.

[1378] The server converts the calculated route information into a data format and sends it to the terminal. The input is the optimal route information, and the output is a transmission completion notification to the terminal.

[1379] Step 5:

[1380] The device generates voice guidance based on the route information.

[1381] The device analyzes the received route information and generates voice guidance using text-to-speech software (e.g., Google Text-to-Speech (registered trademark)). The input is the route information, and the output is the voice guidance.

[1382] Step 6:

[1383] The user receives the audio guidance.

[1384] The user confirms the route to the destination through voice guidance provided by the device. For example, the user may receive instructions such as, "Turn right and go 100 meters until you see a red brick building. Turn left at the next intersection." The input is the voice guidance, and the output is the user's actions.

[1385] Step 7:

[1386] The user asks a question or makes a request.

[1387] If the user has any questions or additional requests along the way, they can use voice input to ask questions or make requests. For example, they can ask, "How far is it to the next intersection?" The input is voice input, and the output is text data.

[1388] Step 8:

[1389] The device uses the generative AI model to respond to the user's questions and requests.

[1390] The device uses a generative pre-training transformer (e.g., OpenAI® GPT-3®) to generate natural-sounding responses to user questions and requests. The input is the text data of the user's question or request, and the output is the text data of the response.

[1391] Step 9:

[1392] The server acquires information about the surrounding environment and sends it to the terminal.

[1393] The server obtains information about the surrounding environment from the map information service and sends it to the terminal. For example, it obtains information about the characteristics and colors of surrounding buildings, the presence or absence of stairs, etc. The input is the environmental information from the map information service, and the output is a transmission completion notification to the terminal.

[1394] Step 10:

[1395] The device communicates surrounding information to the user via voice based on environmental information.

[1396] Based on the received environmental information, the device uses text-to-speech software to communicate information about the surroundings to the user aloud. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead." The input is environmental information, and the output is voice guidance.

[1397] (Application example 1)

[1398] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1399] When visually impaired people navigate or search for products in a physical store, they face challenges such as difficulty grasping information about their surroundings, making it difficult for them to understand the route to their destination and the layout of products. Furthermore, there is a lack of ways for visually impaired people to obtain information through natural conversation while navigating the store, which limits their in-store experience.

[1400] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1401] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying information about the surrounding environment to a visually impaired person, means for guiding a user to a destination using map information within a store, and means for conveying information about the features and layout of products within the store and facility information by voice, thereby enabling visually impaired people to move around a physical store and search for products smoothly.

[1402] "Means of providing directions to the destination" refers to a function that calculates the optimal route to the destination set by the user and provides instructions through voice guidance, etc.

[1403] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses natural language processing technology such as generative pre-training transformers (generative AI) to respond to questions and requests from users through natural conversation.

[1404] "Means of conveying information about the surrounding environment to visually impaired people" is a function that uses audio to convey the characteristics and colors of surrounding buildings, the location of facilities, etc., based on information obtained from map information services.

[1405] "A means of guiding users to their destination using in-store map information" is a function that calculates the optimal route to the destination set by the user based on in-store map information and provides instructions via voice guidance.

[1406] "Means for communicating information about the features and layout of products in a store and about facilities by voice" is a function that communicates information about the features and layout of products and facilities in a store to users by voice.

[1407] The system for implementing this invention is designed to assist visually impaired people in navigating and locating products in physical stores. The system is implemented as an application installed on a portable device such as a smartphone or smart glasses.

[1408] Hardware and Software

[1409] Hardware

[1410] Smartphone

[1411] Smart Glasses

[1412] software

[1413] OpenAI(R) API

[1414] Map Information Service API

[1415] Data processing and calculation

[1416] A means of providing directions to a destination

[1417] The server uses the map information service API to obtain route information between the user's current location and the destination, and provides voice guidance for each step based on the obtained route information.

[1418] A means of enabling conversations with users using natural language processing technology

[1419] The server uses a generative pre-training transformer (generative AI) to generate natural responses to user questions, allowing users to obtain information through natural conversation.

[1420] A means of conveying information about the surrounding environment to visually impaired people

[1421] Based on information obtained from the map information service, the server will provide audio information such as the characteristics and colors of surrounding buildings and the location of facilities, allowing the user to understand the surrounding environment.

[1422] A method of guiding users to their destination using in-store map information

[1423] The server calculates the optimal route to the destination set by the user based on the store's map information and provides voice guidance, allowing the user to move smoothly within the store.

[1424] A means of communicating the features and layout of products in stores and facility information through audio

[1425] The server will then provide users with voice information about the features and locations of products and facilities within the store, allowing them to easily find the products and facilities they are looking for.

[1426] Specific examples

[1427] For example, if a user asks, "What's in the fresh vegetable section?", the generative AI will respond, "The fresh vegetable section has fresh vegetables such as tomatoes, lettuce, and cucumbers."

[1428] Prompt Sentence Examples

[1429] User: What's in the fresh produce section?

[1430] Generative AI: The fresh vegetable section is stocked with fresh vegetables such as tomatoes, lettuce, and cucumbers.

[1431] In this way, a system will be created that helps visually impaired people navigate and search for products smoothly within a physical store.

[1432] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1433] Step 1:

[1434] A user turns on their smartphone or smart glasses and starts a navigation application.

[1435] Input: User's current location and destination information

[1436] Output: Application startup and initialization

[1437] What happens: The user taps the application to launch it and enters their current location and destination.

[1438] Step 2:

[1439] The server uses the map information service API to obtain route information between the user's current location and the destination.

[1440] Input: User's current location and destination information

[1441] Output: Route information

[1442] Specific operation: The server sends a request to the map information service API to calculate and obtain the optimal route.

[1443] Step 3:

[1444] Based on the route information acquired by the server, instructions for each step are provided to the user via voice guidance.

[1445] Input: Route information

[1446] Output: Voice guidance

[1447] Specific operation: The server analyzes the route information and provides instructions for each step to the user via voice.

[1448] Step 4:

[1449] The user enters a question, and the server generates a response using a generative pre-training transformer (generative AI).

[1450] Input: User question

[1451] Output: Generative AI response

[1452] How it works: The user enters a question via voice or text, and the server uses generative AI to generate a natural-sounding response.

[1453] Step 5:

[1454] Based on information obtained from the map information service, the server will provide audio information such as the characteristics and colors of surrounding buildings and the location of facilities.

[1455] Input: Environmental information from a map information service

[1456] Output: Provides environmental information through audio

[1457] Specific operation: The server obtains environmental information from the map information service and conveys it to the user via voice.

[1458] Step 6:

[1459] The server guides the user to their destination based on map information within the store.

[1460] Input: Store map information

[1461] Output: Guidance by voice guidance

[1462] Specific operation: The server analyzes the map information within the store and guides the user to the desired location by voice.

[1463] Step 7:

[1464] The server communicates information about the features and layout of products and facilities in the store to the user via voice.

[1465] Input: Store product and facility information

[1466] Output: Provides product features and location information by voice

[1467] Specific operation: The server obtains information about products and facilities in the store and conveys it to the user via voice.

[1468] In this way, a system will be created that helps visually impaired people navigate and search for products smoothly within a physical store.

[1469] Example 2

[1470] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1471] There is a demand for navigation systems that allow visually impaired people to reach their destinations safely and efficiently. Conventional navigation systems do not provide sufficient information to visually impaired people, making it difficult to properly convey directions to their destination and information about the surrounding environment. They also lack the technology to enable natural dialogue with the user. This has made it difficult for visually impaired people to travel safely.

[1472] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1473] In this invention, the server includes a means for a user to set a destination, a means for the server to calculate an optimal route, and a means for the terminal to generate audio guidance. This allows visually impaired people to understand the route to their destination by voice and travel safely and efficiently. The server also includes a means for a user to ask questions or make requests, a means for the server to perform natural language processing, and a means for the terminal to transmit the generated conversation content by voice. This allows visually impaired people to have natural conversations with the system and obtain the necessary information. The server also includes a means for the server to acquire surrounding environmental information and a means for the terminal to transmit the environmental information by voice. This allows visually impaired people to understand environmental information such as the features and colors of surrounding buildings by voice and travel safely.

[1474] The "means for the user to set a destination" is a function that provides an interface for the user to specify a destination using voice input or text input.

[1475] The "means for the server to calculate the optimal route" is a function that uses a map information service to calculate the optimal route to the destination set by the user.

[1476] The "means for the terminal to generate voice guidance" is a function that uses a voice synthesis engine to create voice guidance that is easy for the user to understand, based on the calculated route information.

[1477] The "means for the user to receive audio guidance" is a function that enables the user to hear audio guidance provided by the terminal.

[1478] The "means for users to ask questions or make requests" is a function that provides an interface for users to ask questions or make requests to the system using voice input.

[1479] The "means for the server to perform natural language processing" refers to the function of using a generative pre-training transformer to generate natural conversations in response to questions and requests from users.

[1480] The "means for transmitting the conversation content generated by the terminal by voice" is a function that converts the conversation content received from the server into voice using a voice synthesis engine and transmits it to the user.

[1481] The "means for the server to acquire surrounding environmental information" is a function for acquiring information about surrounding buildings and features from a map information service.

[1482] "Means for the terminal to convey environmental information by voice" is a function that converts the acquired environmental information into voice using a voice synthesis engine and conveys it to the user.

[1483] The navigation system for visually impaired persons of the present invention is implemented in a portable device such as a smartphone, a wearable device, etc. Specific embodiments of this system will be described below.

[1484] Hardware and software used

[1485] Hardware: Smartphones, wearable devices

[1486] Software: Map services (e.g., Google Maps® API), speech synthesis engines (e.g., Google Text-to-Speech®), generative pre-training transformers (e.g., GPT-4®)

[1487] Program processing

[1488] 1. The user sets a destination.

[1489] The user launches an application on their smartphone or wearable device and sets their destination by voice or text input, for example, by saying, "I want to go to a nearby cafe."

[1490] 2. The server calculates the optimal route.

[1491] The server accesses a map information service and calculates the optimal route to the destination set by the user. For example, it uses Google Maps (registered trademark) API to obtain the optimal route from the current location to the nearest cafe.

[1492] 3. The device generates voice guidance.

[1493] The device generates voice guidance based on the route information received from the server. It uses a speech synthesis engine to create voice guidance that is easy for the user to understand. For example, it generates voice guidance such as, "Turn right and go 50 meters, and you will find a cafe on your left."

[1494] 4. The user receives the audio prompt.

[1495] The user listens to the voice guidance provided by the terminal and understands the route to the destination.

[1496] 5. The user has a question or request.

[1497] If the user has any questions or requests along the way, they can ask the question or request by voice, for example, "Which way should I turn at the next intersection?"

[1498] 6. The server performs natural language processing.

[1499] The server uses a generative pre-training transformer to generate natural-sounding conversations in response to user questions and requests. For example, it uses GPT-4 (registered trademark) to generate natural-sounding conversations such as "Turn right at the next intersection."

[1500] 7. The device will then transmit the generated conversation content aloud.

[1501] The terminal converts the conversation content received from the server into voice using a speech synthesis engine and conveys it to the user.

[1502] 8. The server obtains information about the surrounding environment.

[1503] The server obtains information about surrounding buildings and features from a map information service, such as the color and shape of the building and whether it has stairs.

[1504] 9. The device will provide audio information about the environment.

[1505] The device converts the acquired environmental information into speech using a speech synthesis engine and conveys it to the user. For example, it conveys environmental information to the user in a voice form, such as "There is a red brick building on your right" or "There are five steps ahead."

[1506] Examples of specific examples and prompts

[1507] Example: If a user says, "I want to go to a nearby cafe," the server uses a map information service to calculate the optimal route, and the device provides voice guidance such as, "Turn right and go 50 meters, and you will find the cafe on your left."

[1508] Example prompts: "If the user says they want to go to a nearby cafe, calculate the best route and provide voice directions."; "If the user asks about a nearby building, describe its features voice-over."

[1509] In this way, the navigation system for the visually impaired of the present invention can provide the user with voice guidance and natural conversation, and convey information about the surrounding environment.

[1510] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1511] Step 1:

[1512] The user sets the destination.

[1513] Input: The user launches an application on their smartphone or wearable device and sets a destination by voice or text input. For example, they might say, "I want to go to a nearby cafe."

[1514] Data processing: The device converts voice input into text data and extracts destination information.

[1515] Output: The destination information is sent to the server in text format.

[1516] Specific operation: When a user speaks, "I want to go to a nearby cafe," the device uses voice recognition technology to convert this into text and sends the destination information to the server.

[1517] Step 2:

[1518] The server calculates the optimal route.

[1519] Input: Destination information received by the server from the device.

[1520] Data processing: The server sends a request to a map information service (e.g., Google Maps (registered trademark) API) to calculate the optimal route from the current location to the destination.

[1521] Output: The optimal route information is sent from the server to the device.

[1522] Specific operation: The server accesses the Google Maps (registered trademark) API, obtains the optimal route from the current location to the nearest cafe, and sends that information to the device.

[1523] Step 3:

[1524] The device generates voice guidance

[1525] Input: The best route information received from the server.

[1526] Data processing: The device uses a speech synthesis engine (e.g., Google Text-to-Speech®) to convert route information into voice guidance.

[1527] Output: Audio guidance is provided to the user.

[1528] Specific operation: The device generates and provides voice guidance to the user, such as "Turn right and go 50 meters, and you will find the cafe on your left."

[1529] Step 4:

[1530] The user receives the audio guidance.

[1531] Input: Voice guidance provided by the device.

[1532] Data processing: The user listens to and understands the audio guidance.

[1533] Output: The user gets directions to their destination.

[1534] Specific operation: The user listens to the voice guidance from the device and walks according to the instructions.

[1535] Step 5:

[1536] The user asks a question or makes a request.

[1537] Input: The user speaks a question or request, for example, "Which way do I turn at the next intersection?"

[1538] Data processing: The device converts voice input into text data and extracts questions or requests.

[1539] Output: The question or request is sent to the server in text format.

[1540] Specific operation: When a user speaks a question such as "Which way should I turn at the next intersection?", the device uses voice recognition technology to convert the question into text and sends it to the server.

[1541] Step 6:

[1542] The server performs natural language processing.

[1543] Input: Questions or requests received from the device.

[1544] Data processing: The server uses a generative pre-training transformer (e.g., GPT-4 (registered trademark)) to generate natural-sounding conversations in response to questions and requests.

[1545] Output: The generated conversation content is sent from the server to the device.

[1546] Specific operation: The server uses GPT-4 to generate natural conversation such as "Turn right at the next intersection" and sends that content to the device.

[1547] Step 7:

[1548] The device then transmits the generated conversation content by voice.

[1549] Input: Generated conversation received from the server.

[1550] Data processing: The device uses a speech synthesis engine to convert the conversation into audio.

[1551] Output: The conversation is communicated to the user via audio.

[1552] Specific operation: The device conveys the generated conversation content to the user by voice.

[1553] Step 8:

[1554] The server acquires information about the surrounding environment.

[1555] Input: A request for surrounding environment information from a mapping service.

[1556] Data processing: The server obtains information about surrounding buildings and features from a map information service.

[1557] Output: The acquired environmental information is sent from the server to the device.

[1558] Specific operation: The server obtains information about surrounding buildings and features from the Google Maps (registered trademark) API and sends that information to the device.

[1559] Step 9:

[1560] The device will provide audio information about the environment.

[1561] Input: Environment information received from the server.

[1562] Data processing: The device uses a speech synthesis engine to convert environmental information into speech.

[1563] Output: Environmental information is communicated to the user via audio.

[1564] Specific operation: The device will verbally communicate environmental information to the user, such as "There is a red brick building on your right" and "There are five steps up ahead."

[1565] (Application example 2)

[1566] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1567] When visually impaired people shop comfortably in a brick-and-mortar store, they face challenges such as difficulty understanding product locations and store layouts. They also face challenges such as having to rely on others to obtain product information, making it difficult for them to enjoy shopping independently. Furthermore, they are unable to properly grasp information about their surroundings, making it difficult for them to navigate safely.

[1568] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1569] In this invention, the server includes a means for providing directions to a destination, a means for enabling a conversation with a user using natural language processing technology, a means for conveying surrounding environmental information to a visually impaired person, a means for guiding the location of products in a store and the layout of the store, and a means for providing product information, thereby enabling visually impaired people to shop independently, safely, and comfortably in a physical store.

[1570] "Means of providing directions to the destination" refers to a function that calculates the optimal route to the destination set by the user and provides instructions using voice guidance, etc.

[1571] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses a generative pre-training transformer (generative AI) to respond to questions and requests from users through natural conversation.

[1572] "A means of conveying information about the surrounding environment to visually impaired people" is a function that uses audio to convey the characteristics and colors of surrounding buildings, the location of obstacles, etc., based on information obtained from map information services.

[1573] "Means of providing guidance on the location of products within the store and the store layout" refers to a function that obtains map information within the store, calculates the optimal route to the location of the product specified by the user, and provides voice guidance.

[1574] "Means for providing product information" refers to a function that provides product features, prices, stock status, etc. via voice.

[1575] A system for implementing this invention is designed to help visually impaired people shop independently, safely, and comfortably in brick-and-mortar stores. The system is implemented as an application installed on a portable device such as a smartphone or smart glasses.

[1576] Hardware and software used

[1577] Hardware: Smartphones, smart glasses

[1578] Software: Python, OpenAI API, Google Maps API, gTTS (Google Text-to-Speech)

[1579] System configuration

[1580] 1. Means of providing directions to a destination:

[1581] The server uses the Google Maps API to calculate the optimal route from the user's current location to the destination.

[1582] The device provides the calculated route information to the user as voice guidance.

[1583] 2. Means of enabling conversation with users using natural language processing technology:

[1584] The server uses the OpenAI API to generate responses to questions posed by the user.

[1585] The terminal plays the generated response aloud.

[1586] 3. Means of communicating information about the surrounding environment to the visually impaired:

[1587] The server obtains information such as the characteristics and colors of surrounding buildings and the location of obstacles based on information obtained from the map information service.

[1588] The device will then communicate the acquired information to the user via voice.

[1589] 4. Navigating product locations and store layout within the store:

[1590] The server obtains map information about the store and calculates the optimal route to the location of the product specified by the user.

[1591] The device provides the calculated route information to the user as voice guidance.

[1592] 5. Means of providing product information:

[1593] The server obtains information such as product features, prices, and stock status.

[1594] The terminal provides the acquired product information to the user by voice.

[1595] Specific examples

[1596] For example, if a user asks, "How much does this item cost?", the server uses the OpenAI (registered trademark) API to generate a response such as, "The price of this item is 1,000 yen." The device then plays this response aloud to the user.

[1597] Prompt Sentence Examples

[1598] User: How much does this item cost?

[1599] AI: The price of this item is 1,000 yen.

[1600] In this way, a navigation application can be realized that allows visually impaired people to enjoy shopping comfortably in physical stores.

[1601] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1602] Step 1:

[1603] The user starts up a smartphone or smart glasses and starts the navigation application. The user then inputs their destination and product information by voice. The input voice data is converted into text data by the device.

[1604] Step 2:

[1605] The device sends text data to the server, which uses the Google Maps (registered trademark) API to calculate the optimal route from the user's current location to the destination. The input is the user's current location and destination information, and the output is the optimal route information.

[1606] Step 3:

[1607] The server transmits the calculated route information to the terminal, which then provides the route information to the user as voice guidance by converting text to speech using gTTS (Google Text-to-Speech®) and playing it back to the user.

[1608] Step 4:

[1609] When a user wants to know product information, they input a question by voice. The input voice data is converted into text data by the terminal.

[1610] Step 5:

[1611] The device sends text data to the server, which uses the OpenAI® API to generate a response to the user's question. The input is the user's question text, and the output is the response text.

[1612] Step 6:

[1613] The server sends the generated response text to the terminal, which then plays the response text aloud and provides it to the user. Specifically, the terminal converts the text into speech using gTTS and plays it back to the user.

[1614] Step 7:

[1615] The server obtains environmental information such as the characteristics and colors of surrounding buildings and the location of obstacles based on information obtained from the map information service. The input is data from the map information service, and the output is environmental information.

[1616] Step 8:

[1617] The server sends the acquired environmental information to the device, which then communicates the environmental information to the user by voice. Specifically, the device converts text to voice using gTTS and plays it back to the user.

[1618] Step 9:

[1619] The server obtains map information about the store and calculates the optimal route to the location of the product specified by the user. The input is the map information about the store and the product information specified by the user, and the output is the optimal route information.

[1620] Step 10:

[1621] The server sends the calculated route information to the device, which then provides the route information to the user as voice guidance by converting text to speech using gTTS and playing it back to the user.

[1622] Step 11:

[1623] The server obtains information such as product features, prices, and stock status. The input is data from the product database, and the output is product information.

[1624] Step 12:

[1625] The server sends the acquired product information to the terminal, which then provides the product information to the user by voice. Specifically, the terminal converts text to voice using gTTS and plays it back to the user.

[1626] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1627] "Example 1"

[1628] In one embodiment of the present invention, an emotion engine is incorporated into a navigation system for the visually impaired. This emotion engine has the function of recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to the emotion. Specifically, if the user makes a sound that indicates joy or relief, the emotion engine recognizes this and the voice guidance issues instructions in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the emotion engine recognizes this and the voice guidance issues instructions in a clearer and more forceful tone. This makes it possible to provide appropriate guidance according to the user's emotional state.

[1629] "Example 2"

[1630] In one embodiment of the present invention, an emotion engine is incorporated into a navigation system for the visually impaired. This emotion engine has the function of recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to the emotion. Specifically, if the user makes a sound that indicates joy or relief, the emotion engine recognizes this and the voice guidance issues instructions in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the emotion engine recognizes this and the voice guidance issues instructions in a clearer and more forceful tone. This makes it possible to provide appropriate guidance according to the user's emotional state.

[1631] The processing flow of each embodiment will be described below.

[1632] "Example 1"

[1633] Step 1: The user speaks their destination, which is converted into text by the system and recognized as a destination.

[1634] Step 2: The system uses a map information service to calculate the optimal route from your current location to your destination.

[1635] Step 3: The system uses generative AI to generate voice guidance based on the calculated route.

[1636] Step 4: The user provides vocal feedback to the system, which is analyzed by the emotion engine to recognize the user's emotional state.

[1637] Step 5: The emotion engine adjusts the tone of the voice guidance based on the recognized emotion. For example, if the user vocalizes with a voice that indicates joy or relief, the voice guidance will give instructions in a gentler tone. Conversely, if the user vocalizes with a voice that indicates anxiety or fear, the voice guidance will give instructions in a clearer and more forceful tone.

[1638] Example 1

[1639] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1640] In order for visually impaired people to reach their destination safely and efficiently, they need not only directions but also appropriate guidance that takes into account the surrounding environmental information and the user's emotional state. However, conventional navigation systems have difficulty providing these elements in an integrated manner, and are therefore unable to provide sufficient support for visually impaired people. Therefore, a system that enables visually impaired people to travel safely and efficiently is needed.

[1641] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1642] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, and means for recognizing the user's emotions and adjusting the tone of the voice guidance. This enables a visually impaired person to not only be provided with directions but also receive appropriate guidance according to the surrounding environmental information and emotions when heading to a destination.

[1643] "Means for providing directions to a destination" is a function that calculates the optimal route to the destination set by the user and provides that information to the user.

[1644] "Means for enabling conversation with users using natural language processing technology" refers to a function that converts voice input from users into text data, generates an appropriate response using natural language processing technology such as generative pre-training transformers, and conveys it back to the user as voice.

[1645] "Means for conveying surrounding environmental information to visually impaired people" is a function that conveys information such as the characteristics and colors of surrounding buildings obtained from a map information service to the user via voice.

[1646] The "means for recognizing the user's emotions and adjusting the tone of the voice guidance" is a function for recognizing emotions from the user's voice and adjusting the tone of the voice guidance according to those emotions.

[1647] A "generative pre-training transformer" is a natural language processing model that is pre-trained using large amounts of text data, and is a technology for generating appropriate responses to user input.

[1648] A "map information service" is an online service that provides geographical information and is used to obtain a user's current location, destination, surrounding environmental information, etc.

[1649] The navigation system for visually impaired persons of the present invention is implemented in a portable device such as a smartphone, a wearable device, etc. Specific embodiments of this system will be described below.

[1650] Hardware and Software Configuration

[1651] Hardware

[1652] Devices: Smartphones, wearable devices (e.g., smartwatches, earphones)

[1653] Server: Cloud server

[1654] software

[1655] Map information services: Online map services (e.g., Google Maps (registered trademark) API)

[1656] Generative pre-training transformers: Natural language processing models (e.g., GPT-4 (registered trademark))

[1657] Emotion Engine: Software for Recognizing Emotions from Speech

[1658] Data processing and calculation

[1659] 1. Setting a destination and calculating a route

[1660] Users open an application on their smartphone or wearable device and enter their destination by voice or text.

[1661] The terminal transmits the input destination information to the server.

[1662] The server uses a map information service to calculate the optimal route to the destination set by the user.

[1663] The server sends the calculation results to the terminal.

[1664] 2. Audio Guidance Generation

[1665] The device generates voice guidance based on the received route information.

[1666] The device provides voice guidance to the user through the smartphone's built-in speaker or the wearable device's earphones.

[1667] 3. Conversational features based on natural language processing

[1668] If the user has any questions or requests while navigating, they can input the questions or requests by voice.

[1669] The terminal converts the input voice into text data.

[1670] The device sends the converted text data to the generative AI model as a prompt.

[1671] The generative AI model generates an appropriate response based on the prompt it receives.

[1672] The generative AI model returns the generated response to the device.

[1673] The device converts the response received from the generative AI model into speech and conveys it to the user.

[1674] 4. Providing information about the surrounding environment

[1675] The server obtains environmental information such as the characteristics and colors of surrounding buildings from a map information service.

[1676] The server transmits the acquired environmental information to the terminal.

[1677] The device generates audio guidance based on the received environmental information.

[1678] The device will then provide the user with specific information via voice, such as "There is a red brick building on your right" or "There are five steps ahead."

[1679] 5. Adjusting the tone of voice guidance using an emotion engine

[1680] The device is equipped with an emotion engine that recognizes emotions from the user's voice.

[1681] If the user makes vocalizations that indicate joy or relief, the emotion engine will recognize this and the voice guidance will provide instructions in a gentler tone.

[1682] If the user makes vocalizations that indicate anxiety or fear, the emotion engine will recognize this and the voice guidance will provide instructions in a clearer and more forceful tone.

[1683] Examples of concrete examples and prompts

[1684] Specific examples

[1685] The user asks, "How do I turn right at the next intersection?"

[1686] The device converts this question into text data and sends it as a prompt to the generative AI model.

[1687] The generative AI model generates the response, "To turn right at the next intersection, go 50 meters and then turn right."

[1688] The device converts this response into speech and relays it to the user.

[1689] Prompt Sentence Examples

[1690] "How do I turn right at the next intersection?"

[1691] "Please tell me the nearest cafe."

[1692] "Are there any obstacles ahead?"

[1693] In this way, the navigation system for the visually impaired of the present invention can provide appropriate guidance and information to the user.

[1694] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1695] Step 1:

[1696] Users open an application on their smartphone or wearable device and enter their destination by voice or text.

[1697] Input: User's voice or text destination information

[1698] Output: The destination information is sent to the terminal.

[1699] Specific operation: The user speaks, "Please tell me the way to the station."

[1700] Step 2:

[1701] The terminal transmits the input destination information to the server.

[1702] Input: Destination information from the user

[1703] Output: The destination information is sent to the server.

[1704] Specific operation: The device converts voice input into text data and sends it to the server.

[1705] Step 3:

[1706] The server uses a map information service to calculate the optimal route to the destination set by the user.

[1707] Input: Destination information

[1708] Output: Optimal route information

[1709] Specific operation: The server accesses the map information service and calculates the route from the current location to the destination.

[1710] Step 4:

[1711] The server sends the calculation results to the terminal.

[1712] Input: Optimal route information

[1713] Output: Route information is sent to the device.

[1714] Specific operation: The server sends the calculated route information to the terminal.

[1715] Step 5:

[1716] The device generates voice guidance based on the received route information.

[1717] Input: Route information

[1718] Output: Voice guidance

[1719] Specific operation: The device will give voice instructions such as "Go 50 meters and turn right."

[1720] Step 6:

[1721] If the user has any questions or requests while navigating, they can input the questions or requests by voice.

[1722] Input: The user's spoken question or request

[1723] Output: Questions and requests are sent to the terminal.

[1724] Specific Action: The user asks, "How do I turn right at the next intersection?"

[1725] Step 7:

[1726] The terminal converts the input voice into text data.

[1727] Input: User's voice data

[1728] Output: Text data

[1729] Specific operation: The device uses voice recognition technology to convert speech into text.

[1730] Step 8:

[1731] The device sends the converted text data to the generative AI model as a prompt.

[1732] Input: Text data

[1733] Output: prompts to the generative AI model

[1734] Specific operation: The device sends text data to the generative AI model.

[1735] Step 9:

[1736] The generative AI model generates an appropriate response based on the prompt it receives.

[1737] Input: prompt

[1738] Output: Response text

[1739] Specific behavior: The generative AI model generates the response, "To turn right at the next intersection, go 50 meters and then turn right."

[1740] Step 10:

[1741] The generative AI model returns the generated response to the device.

[1742] Input: Response text

[1743] Output: The response text is sent to the terminal

[1744] Specific operation: The response generated by the generative AI model is sent to the terminal.

[1745] Step 11:

[1746] The device converts the response received from the generative AI model into speech and conveys it to the user.

[1747] Input: Response text

[1748] Output: Voice guidance

[1749] Specific operation: The device will tell the user by voice, "To turn right at the next intersection, go 50 meters and then turn right."

[1750] Step 12:

[1751] The server obtains environmental information such as the characteristics and colors of surrounding buildings from a map information service.

[1752] Input: User's current location

[1753] Output: Surrounding environment information

[1754] Specific operation: The server accesses the map information service and obtains the characteristics and colors of surrounding buildings.

[1755] Step 13:

[1756] The server transmits the acquired environmental information to the terminal.

[1757] Input: Surrounding environment information

[1758] Output: Environmental information is sent to the terminal.

[1759] Specific operation: The server sends the acquired environmental information to the terminal.

[1760] Step 14:

[1761] The device generates audio guidance based on the received environmental information.

[1762] Input: Environment information

[1763] Output: Voice guidance

[1764] Specific action: The device will tell the user, "There is a red brick building on your right."

[1765] Step 15:

[1766] The device is equipped with an emotion engine that recognizes emotions from the user's voice.

[1767] Input: User's voice data

[1768] Output: Emotion recognition result

[1769] Specific operation: The device analyzes the user's voice and recognizes emotions.

[1770] Step 16:

[1771] If the user makes vocalizations that indicate joy or relief, the emotion engine will recognize this and the voice guidance will provide instructions in a gentler tone.

[1772] Input: Emotion recognition results

[1773] Output: Gentle voice guidance

[1774] Specific action: The device will give instructions in a gentle tone, such as "Turn right at the next intersection."

[1775] Step 17:

[1776] If the user makes vocalizations that indicate anxiety or fear, the emotion engine will recognize this and the voice guidance will provide instructions in a clearer and more forceful tone.

[1777] Input: Emotion recognition results

[1778] Output: Clear and powerful voice guidance

[1779] Specific action: The device will issue instructions in a clear, forceful tone, such as "Turn right at the next intersection."

[1780] (Application example 1)

[1781] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1782] When visually impaired people use self-driving vehicles, they face challenges in understanding the route to their destination, grasping information about their surroundings, and receiving appropriate guidance according to their emotional state. To solve these challenges, a comprehensive navigation system is needed to enable visually impaired people to use self-driving vehicles with peace of mind.

[1783] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1784] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, means for recognizing the user's emotions and adjusting the tone of the voice guidance according to the emotions, and means installed in the self-driving vehicle, thereby enabling a visually impaired person using a self-driving vehicle to understand the route to the destination, grasp information about the surrounding environment, and receive appropriate guidance according to their emotional state.

[1785] "Means for providing directions to a destination" means a function that calculates the optimal route to a destination set by the user and provides instructions to the user by voice guidance or other means.

[1786] "Means for enabling conversation with users using natural language processing technology" refers to a function that uses natural language processing technology such as generative pre-training transformers (generative AI) to respond to questions and requests from users through natural conversation.

[1787] "Means for communicating surrounding environmental information to visually impaired people" refers to a function that uses audio to communicate environmental information such as the characteristics and colors of surrounding buildings and obstacles to visually impaired people based on information obtained from map information services.

[1788] "Means for recognizing the user's emotions and adjusting the tone of the voice guidance according to those emotions" refers to a function that recognizes emotions from the user's voice and adjusts the tone of the voice guidance according to those emotions. For example, if the user is feeling anxious, instructions will be given in a clearer and stronger tone.

[1789] "Means installed in an automated vehicle" refers to a function in which a navigation system for visually impaired people is installed in an automated vehicle to provide appropriate navigation and guidance to visually impaired people while the vehicle is in operation.

[1790] The system for implementing the present invention is a navigation system for the visually impaired, and includes the following main functions:

[1791] 1. System Program

[1792] The system includes a means for providing directions to a destination, a means for enabling a conversation with a user using natural language processing technology, a means for conveying surrounding environmental information to visually impaired people, a means for recognizing a user's emotions and adjusting the tone of the voice guidance according to the emotions, and a means for being installed in an autonomous vehicle.

[1793] 2. Program processing explanation

[1794] Hardware and Software

[1795] Hardware: Smartphones, smart glasses, head-mounted displays, and embedded systems in autonomous vehicles

[1796] Software: OpenAI (registered trademark) API, Google Maps (registered trademark) API, gTTS (Google Text-to-Speech (registered trademark), playsound (registered trademark)

[1797] Data processing and calculation

[1798] Means for providing directions to the destination: The server uses Google Maps (registered trademark) API to calculate the optimal route to the destination set by the user. The calculated route information is provided to the user as voice guidance.

[1799] Means of enabling conversation with users using natural language processing technology: The server uses OpenAI®'s Generative Pre-Trained Transformer (generative AI) to respond to user questions and requests through natural conversation.

[1800] Means of communicating surrounding environmental information to visually impaired people: Based on information obtained from the Google Maps (registered trademark) API, the server communicates environmental information such as the characteristics and colors of surrounding buildings and obstacles to visually impaired people by voice.

[1801] A means to recognize the user's emotions and adjust the tone of the voice guidance according to those emotions: The server analyzes the emotions from the user's voice and adjusts the tone of the voice guidance according to those emotions. For example, if the user is feeling anxious, instructions will be given in a clearer and more forceful tone.

[1802] Inclusion in self-driving vehicles: The system will be integrated into self-driving vehicles to provide appropriate navigation and guidance to visually impaired people while the vehicle is in operation.

[1803] 3. Specific Examples

[1804] If the user says, "I want to go to Tokyo Tower," the system will provide voice guidance saying, "Your destination is Tokyo Tower. There is a red brick building on your right."

[1805] Prompt Sentence Examples

[1806] User: I want to go to Tokyo Tower.

[1807] System: Your destination is Tokyo Tower. On your right is a red brick building.

[1808] In this way, an automated vehicle navigation system for the visually impaired is realized.

[1809] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1810] Step 1:

[1811] The user sets the destination.

[1812] The user specifies a destination through voice input. For example, they might say, "I want to go to Tokyo Tower." The device receives this voice input and converts it into text data. The input is the user's voice data, and the output is text data.

[1813] Step 2:

[1814] The server calculates the route to the destination.

[1815] The server uses the Google Maps API to calculate the optimal route to the destination set by the user. The input is text data of the destination specified by the user, and the output is JSON data containing route information. The server parses this JSON data and generates text for voice guidance.

[1816] Step 3:

[1817] The server conducts the conversation with the user.

[1818] The server uses OpenAI's generative pre-trained transformer (generative AI) to respond to user questions and requests through natural conversation. The input is the user's text data, and the output is a generated response text. The server converts this response text into speech and sends it to the device.

[1819] Step 4:

[1820] The server acquires information about the surrounding environment.

[1821] The server obtains environmental information such as the characteristics and colors of surrounding buildings and obstacles based on information obtained from the Google Maps API. The input is the coordinate data of the current location, and the output is JSON data containing environmental information. The server parses this JSON data and generates text for voice guidance.

[1822] Step 5:

[1823] The server recognizes the user's emotions.

[1824] The server analyzes the user's voice to determine their emotions. The input is the user's voice data, and the output is data indicating their emotional state. The server adjusts the tone of the voice guidance based on this emotional data.

[1825] Step 6:

[1826] The server provides the audio guidance.

[1827] The server converts the generated text for the voice guidance into audio using gTTS (Google Text-to-Speech (registered trademark)) and sends it to the terminal. The input is text data for the voice guidance, and the output is audio data. The terminal plays back this audio data and provides it to the user.

[1828] Step 7:

[1829] Autonomous vehicles perform navigation.

[1830] The autonomous vehicle travels to its destination based on route information and voice guidance received from the server. The input is route information and voice guidance, and the output is the vehicle's operating status. The vehicle plays the voice guidance at the appropriate time and provides it to the user.

[1831] Example 2

[1832] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1833] In order for visually impaired people to reach their destination safely and efficiently, they need not only directions but also appropriate guidance that takes into account the surrounding environmental information and the user's emotional state. However, conventional navigation systems have difficulty providing these elements in an integrated manner, and have faced the challenge of not providing sufficient support for visually impaired people to travel safely.

[1834] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1835] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, and means for recognizing the user's emotions and adjusting the tone of the voice guidance, thereby enabling comprehensive support for visually impaired people to reach their destination safely and efficiently.

[1836] "Means for providing directions to a destination" is a function that calculates the optimal route to a destination set by the user and guides the user along that route.

[1837] "Means for enabling conversation with users using natural language processing technology" refers to technology that enables responses in natural language to questions or requests entered by users via voice or text.

[1838] "A means of conveying information about the surrounding environment to visually impaired people" is a function that conveys information about surrounding buildings, terrain features, colors, etc. obtained from map information services to users via voice.

[1839] "Means for recognizing the user's emotions and adjusting the tone of the voice guidance" refers to a function that analyzes the emotions from the user's voice and changes the tone of the voice guidance according to those emotions.

[1840] A "generative pre-training transformer" is a natural language processing model that is pre-trained based on large amounts of text data, and is a technology for generating appropriate responses to user input.

[1841] "Map information service" means an online service that provides geographic information and is used to obtain a user's current location, destination, and surrounding environmental information.

[1842] The present invention relates to a navigation system for visually impaired people, which is implemented in a portable device such as a smartphone or a wearable device. Specific embodiments of this system will be described below.

[1843] First, the user sets their destination using their smartphone or wearable device by voice or text input, for example, "I want to go to the nearest cafe."

[1844] The server accesses a map information service (e.g., Google Maps® API) and calculates the optimal route from the user's current location to the destination, taking into account traffic conditions and pedestrian route information.

[1845] The device generates voice guidance based on the route information received from the server, providing specific instructions to the user, such as "Turn right 100 meters ahead."

[1846] While traveling, the user may voice-input additional questions or requests, such as "What kind of place is this?" or "How far is the next intersection?" The device converts the user's voice input into text and inputs it as a prompt to a generative AI model (e.g., GPT-4 (registered trademark)). For example, it generates a prompt such as "What kind of place is this?"

[1847] The generative AI model generates an appropriate response based on the prompt sentence, such as "There are red brick buildings around your current location. There are five steps ahead." The device then converts this response into speech and relays it to the user, allowing the user to receive information about their surroundings and additional instructions via voice.

[1848] Furthermore, the device uses information obtained from the map information service to provide voice information about the characteristics and colors of surrounding buildings, providing specific information such as "There is a red brick building on your right" or "There are five steps ahead."

[1849] The device is equipped with an emotion engine that recognizes emotions from the user's voice. This emotion engine adjusts the tone of the voice guidance according to the user's emotions. For example, if the user makes a voice that indicates joy or relief, the device will provide voice guidance in a gentler tone. Conversely, if the user makes a voice that indicates anxiety or fear, the device will provide voice guidance in a clearer and more forceful tone.

[1850] For example, if a user says "I want to go to the nearest cafe," the following specific actions will be taken:

[1851] 1. The user speaks, "I want to go to the nearest cafe."

[1852] 2. The server accesses a map information service and calculates the optimal route from the current location to the nearest cafe.

[1853] 3. The device will give the user a voice command saying, "Turn right in 100 meters."

[1854] 4. The user asks, "What kind of place is this?"

[1855] 5. The device inputs the prompt sentence "What kind of place is this?" into the generative AI model.

[1856] 6. The generative AI model generates a response: "There are red brick buildings around your current location. There are five steps ahead."

[1857] 7. The device converts this response into speech and relays it to the user.

[1858] 8. The device provides information about the surrounding environment, such as "There is a red brick building on your right."

[1859] 9. If the user expresses anxiety, the device will provide voice guidance in a clearer and more forceful tone.

[1860] In this way, navigation systems for the visually impaired support users in their journeys and provide them with information about their surrounding environment.

[1861] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1862] Step 1:

[1863] The user sets the destination.

[1864] Users can use a smartphone or wearable device to set their destination by voice or text input. For example, they can say, "I want to go to the nearest cafe." The input speech is converted into text by the device's voice recognition function. The output is the destination information in text format.

[1865] Step 2:

[1866] The server calculates the optimal route.

[1867] The server accesses a map information service (e.g., Google Maps (registered trademark) API) and receives the user's current location and destination as input. The server calculates the optimal route based on this information. The calculation also takes into account traffic conditions and pedestrian route information. The output is the optimal route information.

[1868] Step 3:

[1869] The device provides voice guidance.

[1870] The device generates voice guidance based on the route information received from the server. For example, it gives specific instructions to the user by voice, such as "Turn right 100 meters ahead." The input is the route information from the server, and the output is the voice guidance.

[1871] Step 4:

[1872] The user asks a question or makes a request.

[1873] The user voices additional questions or requests as they move along, such as "What is the location like?" or "How far is the next intersection?" The input is the user's voice input, and the output is the question or request in text form.

[1874] Step 5:

[1875] The device inputs a prompt sentence into the generative AI model.

[1876] The device converts the user's voice input into text and inputs it as a prompt to a generative AI model (e.g., GPT-4 (registered trademark)). For example, it generates a prompt such as "What kind of place is this?" The input is the user's textual question or request, and the output is the prompt.

[1877] Step 6:

[1878] A generative AI model generates a response.

[1879] The generative AI model generates an appropriate response based on the prompt sentence, for example, "There are red brick buildings around your current location. There are five steps ahead." The input is the prompt sentence, and the output is a text response.

[1880] Step 7:

[1881] The device will then speak the response to the user.

[1882] The device converts the response from the generative AI model into speech and transmits it to the user, allowing the user to receive information about the surrounding environment and additional instructions by voice. The input is a text response from the generative AI model, and the output is a speech response.

[1883] Step 8:

[1884] The device provides information about the surrounding environment.

[1885] The device uses information obtained from the map information service to provide voice information about the characteristics and colors of surrounding buildings. For example, it provides specific information such as "There is a red brick building on your right" or "There are five steps ahead." The input is environmental information from the map information service, and the output is environmental information in the form of voice.

[1886] Step 9:

[1887] The device recognizes the user's emotions and adjusts the tone of the voice guidance.

[1888] The device uses an emotion engine that recognizes emotions from the user's voice. If the user makes a sound that indicates joy or relief, the device will provide voice guidance in a gentler tone. Conversely, if the user makes a sound that indicates anxiety or fear, the device will provide voice guidance in a clearer and more forceful tone. The input is the user's voice, and the output is voice guidance in the adjusted tone.

[1889] (Application example 2)

[1890] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1891] When visually impaired people use self-driving vehicles, it is difficult for them to properly understand the route to their destination and the surrounding environment.In addition, it is not possible to provide appropriate voice guidance according to the user's emotional state, making it difficult for them to travel safely.

[1892] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1893] In this invention, the server includes means for providing directions to a destination, means for enabling conversation with a user using natural language processing technology, means for conveying surrounding environmental information to a visually impaired person, means for adjusting the tone of the voice guidance according to the emotional state of the user, and means for providing voice guidance to a visually impaired person in an autonomous vehicle. This enables a visually impaired person using an autonomous vehicle to properly understand directions to a destination and surrounding environmental information, and to receive appropriate voice guidance according to the emotional state of the user.

[1894] A "navigation system for the visually impaired" is a system that assists visually impaired users in traveling to their destination safely and efficiently.

[1895] A "means for providing directions to a destination" is a means that has the function of calculating the optimal route to a destination set by the user and communicating those directions to the user.

[1896] "Means for enabling conversation with a user using natural language processing technology" means means for realizing natural conversation with a user using technology that understands the user's voice input and generates an appropriate response.

[1897] "Means for conveying information about the surrounding environment to visually impaired people" refers to a means for conveying information about surrounding buildings and obstacles obtained from map information services, etc. to users via voice.

[1898] The "means for adjusting the tone of the voice guidance according to the emotional state of the user" refers to a means having a function of recognizing emotions from the user's voice and changing the tone of the voice guidance according to the emotions.

[1899] "Means for providing audio guidance to visually impaired persons in self-driving vehicles" refers to means for providing audio directions to a destination and information about the surrounding environment to visually impaired persons using self-driving vehicles.

[1900] A "map information service" is an online service that provides geographical information and is used for route calculations and obtaining information about the surrounding environment.

[1901] A "generative pre-trained transformer" (generative AI) is a natural language processing model that is pre-trained using large amounts of text data and is used to generate natural conversations with users.

[1902] A system for carrying out the present invention is implemented as a navigation system for the visually impaired in a portable device such as a smartphone or smart glasses. The system includes means for providing directions to a destination, means for enabling a conversation with a user using natural language processing technology, means for conveying surrounding environmental information to the visually impaired, means for adjusting the tone of the voice guidance depending on the emotional state of the user, and means for providing voice guidance to the visually impaired in an autonomous vehicle.

[1903] Hardware and software used

[1904] Hardware: Smartphones, smart glasses

[1905] Software: Python (registered trademark), Google Maps (registered trademark) API, Hugging Face Transformers (registered trademark), pyttsx3 (registered trademark)

[1906] System Operation

[1907] 1. Obtaining map information: The server uses the Google Maps (registered trademark) API to obtain route information from the user's current location to the destination, allowing it to calculate the optimal route to the destination set by the user.

[1908] 2. Voice Guidance: The server uses pyttsx3 (registered trademark) to provide voice guidance based on the acquired route information, allowing the user to confirm the route to the destination by voice.

[1909] 3. Emotion Recognition: The server uses the Hugging Face® emotion recognition model to recognize emotions from the user's voice, allowing it to adjust the tone of the voice guidance according to the user's emotional state.

[1910] 4. Conversation Processing: The server uses Hugging Face®'s generative AI model to enable natural conversation with the user, generating appropriate responses to the user's questions and requests.

[1911] Specific examples

[1912] For example, consider a scenario in which a user gets into an autonomous vehicle and sets a destination.

[1913] User: "Shinjuku Station, please."

[1914] System: "Calculating route to Shinjuku Station. There is a red brick building on your right. Turn left at the next intersection."

[1915] In this way, we can support visually impaired people in self-driving vehicles so that they can travel to their destination safely.

[1916] Prompt Sentence Examples

[1917] User: To Shinjuku Station, please.

[1918] System: Calculating route to Shinjuku Station. There is a red brick building on your right. Turn left at the next intersection.

[1919] This invention enables visually impaired people using self-driving vehicles to properly understand the route to their destination and information about the surrounding environment, and to be provided with appropriate voice guidance that matches the user's emotional state.

[1920] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1921] Step 1:

[1922] The user sets the destination.

[1923] Input: The user specifies a destination by voice (e.g., "Shinjuku Station, please.").

[1924] How it works: The device receives voice input and converts it to text using speech recognition technology.

[1925] Output: Destination information in text format (e.g. "Shinjuku Station").

[1926] Step 2:

[1927] The server obtains route information using a map information service.

[1928] Input: Textual destination information and user's current location information.

[1929] How it works: The server calls the Google Maps (registered trademark) API to calculate the optimal route from the current location to the destination.

[1930] Output: Route information (e.g. directions, distance, time required, etc.).

[1931] Step 3:

[1932] The server generates audio guidance based on the route information.

[1933] Input: Route information.

[1934] How it works: The server uses pyttsx3 to convert route information into voice guidance.

[1935] Output: Voice guidance (e.g. "Turn left at the next intersection.").

[1936] Step 4:

[1937] The server recognizes the user's emotional state.

[1938] Input: User's voice input.

[1939] How it works: The server uses the Hugging Face emotion recognition model to recognize emotions from the user's voice.

[1940] Output: Emotion information (e.g., "joy", "anxiety", etc.).

[1941] Step 5:

[1942] The server adjusts the tone of the voice guidance based on the emotion information.

[1943] Input: Emotional information and audio guidance.

[1944] Operation: The server adjusts the voice synthesis parameters (speed, pitch, etc.) of pyttsx3 (registered trademark) according to the emotion information.

[1945] Output: Adjusted voice guidance (e.g., "Turn left at the next intersection" (gentle tone)).

[1946] Step 6:

[1947] The server handles the conversation with the user.

[1948] Input: User's voice input.

[1949] How it works: The server uses Hugging Face®'s generative AI model to generate appropriate responses to the user's questions and requests.

[1950] Output: Response text (e.g., "Calculating route to Shinjuku Station.").

[1951] Step 7:

[1952] The server converts the response text into speech and provides it to the user.

[1953] Input: The response text.

[1954] How it works: The server uses pyttsx3® to convert the response text into speech.

[1955] Output: A voice response (e.g., "Calculating route to Shinjuku Station.").

[1956] In this way, when visually impaired people use self-driving vehicles, they will be able to properly understand the route to their destination and information about the surrounding environment, and will be able to receive appropriate voice guidance that matches the user's emotional state.

[1957] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1958] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1959] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1960] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1961] [Fourth embodiment]

[1962] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1963] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1964] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1965] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1966] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1967] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1968] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1969] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1970] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1971] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1972] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1973] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1974] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1975] "Example 1"

[1976] The navigation system for visually impaired people of the present invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[1977] "Example 2"

[1978] The navigation system for visually impaired people of the present invention is implemented on portable devices such as smartphones and wearable devices. To provide directions to a destination, it works with a map information service to calculate the optimal route to the destination set by the user and provides instructions via voice guidance. To enable conversation with the user using natural language processing technology, it uses a generative pre-training transformer (generative AI). This enables responses to questions and requests from the user through natural conversation. To communicate information about the surrounding environment to visually impaired people, it uses voice to describe the characteristics and colors of surrounding buildings based on information obtained from the map information service. For example, it provides specific information such as, "There is a red brick building on your right," or "There are five steps ahead."

[1979] The processing flow of each embodiment will be described below.

[1980] "Example 1"

[1981] Step 1: The user activates the system using a portable device such as a smartphone or wearable device.

[1982] Step 2: The user sets a destination using voice or touch input.

[1983] Step 3: The system connects with the map information service to calculate the optimal route to the set destination.

[1984] Step 4: The system will provide voice guidance for directions.

[1985] Step 5: Once the user begins their journey, if they have any questions or requests about the surrounding environment along the way, the system will respond through natural conversation using a generative pre-trained transformer (generative AI).

[1986] Step 6: The system uses information obtained from the map information service to provide a voice message describing the characteristics and colors of surrounding buildings.

[1987] Example 1

[1988] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1989] To reach their destination safely and efficiently, visually impaired people need detailed route guidance and information about their surrounding environment. However, conventional navigation systems have difficulty providing sufficient information to visually impaired people, particularly in terms of conveying information about the surrounding environment and building features. Furthermore, they lack the ability to respond to real-time questions and requests through natural conversation with the user. This often leads to anxiety and difficulty for visually impaired people when...

Claims

1. A navigation system for visually impaired people, comprising: A means for enabling conversation with users using natural language processing technology; means for generating a prompt sentence indicating a destination of the user based on the conversation; A means for acquiring at least one of information on a route to the destination and information on a surrounding environment using the generated prompt sentence and a generative AI model; means for recognizing an emotional state of a user from said conversation using an emotion engine; means for adjusting the tone of the audio guidance in response to the emotional state of the user; A means for providing voice guidance of the information to a visually impaired person in the autonomous vehicle using the adjusted tone; A system including:

2. The system of claim 1 , wherein the natural language processing technique uses generative AI.

3. The system according to claim 1 , wherein the surrounding environmental information is obtained in cooperation with a map information service and includes at least one of the characteristics and colors of surrounding buildings and the layout of facilities based on information obtained from the map information service.

Images