system

Abstract

Provide a system. [Solution] A system in a barrier-free map creation application designed for the visually and hearing impaired, comprising: means for receiving a user's location information and destination information; means for calculating the optimal route based on the received location information and destination information by referring to public transport timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for tracking the user's current location in real time as the user moves, and detecting and notifying of changes in congestion or route deviations that occur along the route; and means for providing the user with location information of information centers and support facilities around the destination after the user arrives.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When people with visual and hearing impairments travel by public transportation, there is a problem of insufficient support to reach their destinations safely and efficiently. Furthermore, there is a need for comprehensive support that not only provides directions to their destination, but also information on peak times, the location of barrier-free facilities, and information centers along the way. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a barrier-free map creation application designed for the visually and hearing impaired by providing the following means: A system that receives a user's location information and destination information; A system that calculates an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; A system that provides the calculated optimal route and related congestion information and barrier-free facility information to the user's device; A system that tracks the user's current location in real time as the user moves and detects and notifies the user of changes in congestion and route deviations that occur along the route; A system that provides the user with location information of information centers and support facilities near the destination after the user arrives. Furthermore, the system provides comprehensive support for the visually and hearing impaired by providing guidance information by voice guidance, text display, and vibration depending on the user's disability, and by providing an interface for the user to specify intermediate stops and desired stops when planning a route.

[0006] "User location information" is data indicating the location where the user is currently located.

[0007] "Destination information" is data indicating a specific location where the user wants to arrive.

[0008] "Public transport" refers to public transport such as buses, trains, subways and ferries.

[0009] "Timetable data" refers to data related to the operation schedules of public transportation.

[0010] "Congestion forecast data" is data used to predict congestion on public transportation and roads based on past data and real-time information.

[0011] "Barrier-free route information" refers to information on routes that people with disabilities can travel safely and conveniently.

[0012] The "optimal route" refers to the travel route that best suits the user's needs, and is usually selected taking into consideration time, distance, barrier-free facilities, and congestion.

[0013] "Route calculation" is the process of calculating the optimal travel route based on the user's current location and destination.

[0014] "Guidance information" refers to route instructions and related information necessary for a user to reach a destination.

[0015] "Real-time tracking" means constantly checking the user's current location in an up-to-date manner.

[0016] "Route deviation" refers to a state in which the user deviates from the designated route.

[0017] An "information center" refers to a counter or facility located in a public place that provides information to users.

[0018] "Support facilities" refers to facilities designed to assist with mobility, such as barrier-free toilets, rest areas, and elevators.

[0019] "Audio guide" is a system that provides information to visually impaired people through audio.

[0020] "Text display" is a system for providing information to hearing impaired people by displaying text information.

[0021] "Vibration guidance information" is a technology that uses vibration to provide specific information or warnings to the user.

[0022] "Interface" refers to the means by which a user interacts with a system, and includes input screens, operation buttons, etc. [Brief explanation of the drawings]

[0023] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. Specific embodiments of the application will be described below.

[0045] System Configuration

[0046] This system consists of a user device, a central server, and a database. User devices include smartphones and tablets. The central server is a high-performance computer system for route calculations and data processing. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0047] Enter user information

[0048] Users start the app, log in, and enter their current location and destination information. Current location information can also be obtained automatically using GPS. In addition, an interface is provided for specifying intermediate destinations and places to stop.

[0049] Route calculation and data collection

[0050] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0051] Providing guidance information to users

[0052] The user's device receives the calculated route information and provides guidance in a format tailored to the visually and hearing impaired. The visually impaired receive audio guidance, while the hearing impaired receive text and vibration guidance. Detailed guidance is also provided on important points (e.g., elevator locations, arrow sign directions, etc.).

[0053] Route Tracking

[0054] As users move, the device tracks their location in real time and detects and notifies them of changes in congestion or deviations along the route, allowing them to quickly get back on track.

[0055] Post-arrival support

[0056] When the user arrives at their destination, the device will notify them of their arrival. It will also provide information about the locations of nearby information centers and support facilities. For example, if the user arrives at a hospital, they will receive detailed information about the location of the nearest information center and elevator.

[0057] Specific examples

[0058] Example 1: Visually impaired person A travels from the station to the library

[0059] 1. Enter user information

[0060] Person A launches the app and voice-inputs his current location (station) and destination (library).

[0061] Specify the station information desk as your stopover point.

[0062] 2. Route calculation and data collection

[0063] The server receives the current location and destination and calculates the optimal route.

[0064] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0065] 3. Providing guidance information to users

[0066] The device provides guidance information to Mr. A via audio guide.

[0067] It includes detailed instructions such as "Turn left and the elevator is 50 meters ahead."

[0068] 4. Route Tracking

[0069] While Mr. A is traveling, the device tracks his current location in real time and notifies him if he strays from his route or if there is a change in congestion.

[0070] 5. Post-arrival support

[0071] When you arrive at the library, the device will announce, "You have arrived at the library. The entrance is on your right."

[0072] It also provides information about the library's information desk and barrier-free facilities.

[0073] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[0077] Step 2:

[0078] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[0079] Step 3:

[0080] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[0081] Step 4:

[0082] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[0083] Step 5:

[0084] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[0085] Step 6:

[0086] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[0087] Step 7:

[0088] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[0089] Step 8:

[0090] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[0091] Step 9:

[0092] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[0093] Step 10:

[0094] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[0095] Step 11:

[0096] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] There is a lack of systems that guide people with visual and hearing impairments to the optimal route from their current location to their destination. In particular, there are issues with the lack of real-time congestion information, route deviation notifications, and information on information centers and support facilities near the destination. There is also a need for more flexible guidance methods according to the type of disability, as well as an interface that allows users to specify intermediate stops and stopovers when planning a route.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for receiving user location information and destination information, means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information, means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's information processing device, means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations that occur along the route, means for providing location information of information centers and support facilities around the destination after the user arrives, and a barrier-free map creation application designed for the visually and hearing impaired. This enables the visually and hearing impaired to travel safely and efficiently and to be provided with necessary guidance information in a timely manner.

[0102] "User location information" refers to geographical information about the user's current location, and is data obtained using technology such as GPS.

[0103] "Destination information" is geographical information about the final destination that the user is aiming for.

[0104] "Public transport timetable data" refers to data relating to the operation schedules of public transport such as buses and trains.

[0105] "Congestion prediction data" is data that predicts the congestion of people and vehicles at specific times and locations.

[0106] "Barrier-free route information" refers to information about routes that everyone, including those with visual or hearing impairments, can travel on safely and comfortably.

[0107] The "optimal route" is the most efficient travel route between the current location set by the user and the destination, taking into consideration time, types of obstacles, and the like.

[0108] A "calculated optimal route" is the best route to travel calculated using the user's current location and destination and related data.

[0109] "Real-time location tracking" means keeping track of the user's current location at all times.

[0110] "Route deviation detection" refers to detecting when a user deviates from a pre-planned route.

[0111] "Location information of information centers and support facilities" is geographical information about information centers and facilities that support users located in the vicinity of the destination.

[0112] A "barrier-free map creation application designed for the visually and hearing impaired" is map software designed to guide the visually and hearing impaired to navigate safely and efficiently.

[0113] MODE FOR CARRYING OUT THE INVENTION

[0114] The present invention is a barrier-free map creation application designed for the visually and hearing impaired, and specific embodiments thereof will be described below.

[0115] System Configuration

[0116] This system consists of user devices, a central server, and a database. User devices include mobile information terminals such as smartphones and tablets, through which the application runs. The central server is a high-performance computer system that calculates routes and processes data. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0117] Enter user information

[0118] Users must launch the app on their device and log in. The application provides an interface for users to input their current location and destination information. Current location information can also be obtained automatically using GPS. Users can also specify intermediate locations and places they want to stop at.

[0119] Route calculation and data collection

[0120] The server accesses a database to reference public transportation timetable data, congestion forecast data, and barrier-free route information based on the current location and destination information received from the user. Based on this data, the server calculates the optimal route for the user. The route calculation includes routes that are safe and accessible, especially for the visually and hearing impaired.

[0121] Providing guidance information to users

[0122] The calculated optimal route information is sent from the server to the user's device. Based on the received route information, the device provides guidance information in the form of voice guidance for the visually impaired and text display or vibration for the hearing impaired. For example, the device may provide a voice guide saying, "Turn left and there is an elevator 50 meters ahead."

[0123] Route Tracking

[0124] While the user is traveling along a route, the device tracks their location in real time. The device periodically checks its location using GPS and notifies the user if there is a deviation from the route or changes in traffic congestion, allowing the user to quickly get back on track.

[0125] Post-arrival support

[0126] When the user arrives at their destination, the device will notify them, "You have arrived at your destination." It will also provide detailed location information for information centers and support facilities around the destination. For example, if they arrive at a hospital, they will be told, "The elevator is on the right side of the building."

[0127] Specific examples

[0128] Example 1: Visually impaired person A travels from the station to the library

[0129] 1. The user launches the app and speaks, "I want to go to the library from the station."

[0130] 2. The server analyzes the received information and calculates the optimal route by referring to public transport timetable data, congestion forecast data, and barrier-free route information.

[0131] 3. The device will give voice guidance saying, "Turn left and get on the elevator 50 meters ahead."

[0132] 4. The device tracks your location in real time while you are moving, and if you stray from your route, it will notify you, "Go back 10 meters and turn right."

[0133] 5. When the device arrives at its destination, it will provide detailed information such as, "You have arrived at the library. The entrance is on the right. Additionally, the information desk is on the left."

[0134] Example prompts for generative AI models

[0135] I'd like to design an application that provides optimal routes and guidance for visually impaired people traveling from the station to the library. Please explain the specific process for this application: based on current location and destination information, this application calculates the optimal route by referencing public transport timetable data, congestion forecast data, and barrier-free route information, and then provides guidance using audio guides and text displays.

[0136] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and is a system that provides necessary guidance information in a timely manner, ensuring the safety and security of users.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1: Enter your information

[0139] Users launch the app on their smartphone or tablet and log in.

[0140] The user specifies the destination information by voice input or text input, and the application can automatically obtain the current location information using the GPS function.

[0141] Input: User's current location information, destination information, (optional) intermediate locations and places to stop

[0142] Output: User input data (current location, destination information)

[0143] As a specific example of operation, when the user is at a station, he or she inputs "I want to go to the library" by voice.

[0144] Step 2: Route calculation and data collection

[0145] The server accesses a database based on the current location information and destination information received from the user, and obtains public transportation timetable data, congestion forecast data, and barrier-free route information.

[0146] The server uses the acquired data to calculate the optimal route using an algorithm that takes into account accessibility and congestion.

[0147] Input: User's current location information, destination information, public transport timetable data in the database, congestion forecast data, barrier-free route information

[0148] Output: Calculated optimal route information

[0149] As a specific example of how it works, the server "identifies the nearest station based on GPS coordinates and calculates the optimal route from that station, taking into account barrier-free information."

[0150] Step 3: Providing guidance to users

[0151] The server transmits the calculated optimum route information to the user's terminal.

[0152] Based on the route information received, the device provides voice guidance for visually impaired people and guidance information via text display and vibration for hearing impaired people.

[0153] Input: Calculated optimal route information

[0154] Output: Guidance information for the user (audio guide, text display, vibration guide)

[0155] As a specific example of operation, the user's device plays back a voice instruction saying, "Turn left and use the elevator 50 meters ahead."

[0156] Step 4: Path tracing

[0157] The device periodically acquires GPS data to track the user's location in real time as they move.

[0158] The device compares the acquired current location information with the calculated route and notifies the user if the user deviates from the route or if congestion occurs.

[0159] Input: Real-time GPS data, calculated route information

[0160] Output: Notifications about route deviations and congestion

[0161] As a specific example of how it works, the device will issue a voice notification saying, "You are currently off route. Go back 10 meters and turn right."

[0162] Step 5: Support after arrival

[0163] When the terminal detects the user's arrival, it notifies the user that "you have arrived at your destination."

[0164] The device also provides the user with detailed location information for information centers and support facilities around the destination.

[0165] Input: User's current location (destination arrival), support facility information in the database

[0166] Output: Arrival notification at destination, location information of support facilities

[0167] As a specific example of how it works, the device will say, "You have arrived at the library. The elevator is on the right side of the building."

[0168] In this way, the system is designed to enable the visually and hearing impaired to travel to their destination safely and efficiently, and is equipped with functions for real-time route tracking and detailed guidance around the destination.

[0169] (Application example 1)

[0170] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0171] When using public transportation or self-driving vehicles, the visually and hearing impaired face difficulties in reaching their destinations safely and efficiently due to insufficient route guidance and barrier-free information. It is also difficult to respond immediately to congestion information or route deviations while traveling. Therefore, there is a need for a comprehensive navigation system that enables the visually and hearing impaired to travel safely.

[0172] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0173] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations along the route; means for providing location information of information centers and support facilities around the destination after the user arrives; means for navigating the route to the destination in cooperation with an autonomous vehicle; and means for providing real-time guidance information to the visually and hearing impaired using a smartphone or smart glasses, thereby enabling the visually and hearing impaired to reach their destination safely and efficiently via a barrier-free route.

[0174] "User location information" is data that indicates the geographical location of the user, obtained using a mobile device or GPS function.

[0175] "Destination information" is geographical location data relating to the location where the user ultimately wants to arrive.

[0176] "Public transport timetable data" refers to information about the operation schedules of public transport such as buses and trains.

[0177] "Congestion prediction data" is information that predicts the congestion situation of people along a specified route or at a destination.

[0178] "Barrier-free route information" refers to information about safe, barrier-free routes that are easy for wheelchair users and those with visual impairments to navigate.

[0179] The "optimal route" is the most suitable route for traveling from the user's location information to the destination information, taking into consideration time, convenience, and safety.

[0180] "Location tracking" refers to the process of continuously monitoring a user's current location to determine their location in real time.

[0181] "Route deviation" means that the user deviates from the calculated optimal route.

[0182] "Information centers and support facilities" are locations or service facilities that provide assistance and information during travel or upon arrival at a destination.

[0183] An "autonomous vehicle" is a vehicle that operates autonomously without the need for human operation.

[0184] A "smartphone" is a portable information terminal that has various functions in addition to telephone functions, such as internet connection.

[0185] "Smart glasses" are eyeglass-type devices with built-in computers that are capable of augmented reality and information display.

[0186] "Real-time guidance information" means up-to-date route and situation information provided immediately at the current time.

[0187] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. The system consists of a user terminal, a central server, and a database. Specific embodiments are described below.

[0188] System Configuration

[0189] Hardware Configuration

[0190] The system uses mobile devices such as smartphones and smart glasses, which have GPS capabilities to acquire user location information. Sensor modules in autonomous vehicles are also utilized, and the system connects to a central server to update route information in real time.

[0191] Software Configuration

[0192] The central server is a high-performance computer system that accesses the OpenStreetMap API and PostgreSQL database to calculate barrier-free routes based on location and destination information received from user devices. Voice guidance uses Google® Text-to-Speech and AWS® Polly.

[0193] Data processing and route calculation

[0194] The server receives the user's current location and destination information and calculates the optimal barrier-free route based on this information. The calculation uses public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is sent to the user's device, and route information is provided in real time.

[0195] User guidance and real-time tracking

[0196] Based on the route information received, the user's device provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired. It works in conjunction with the autonomous vehicle to navigate the route to the destination. If congestion changes or the user deviates from the route while traveling, the device will be notified immediately.

[0197] Post-arrival support

[0198] When the user arrives at their destination, the device will notify them of their arrival and provide location information for nearby information centers and support facilities, allowing the visually and hearing impaired to smoothly receive support at their destination.

[0199] Specific examples

[0200] Example 1: B, a visually impaired person, travels to the hospital using an autonomous vehicle

[0201] 1. Enter user information

[0202] Person B launches the smartphone app and voice-inputs his current location (home) and destination (hospital).

[0203] 2. Route calculation

[0204] The server calculates the optimal barrier-free route based on the current location and destination, and updates the route information for the autonomous vehicle.

[0205] 3. Real-time guidance

[0206] Directions are provided as audio guides for the visually impaired, for example: "Turn left in 100 metres."

[0207] 4. Real-time route tracking

[0208] It works in conjunction with autonomous vehicles to monitor changes in congestion and route deviations during travel in real time, and notifies drivers as necessary.

[0209] 5. Post-arrival support

[0210] When the user arrives at the hospital, the smartphone will notify them with a voice message saying, "You have arrived at the hospital. The entrance is on your right." It will also provide information about the hospital's information desk and barrier-free facilities.

[0211] Prompt Sentence Examples

[0212] "Development of an autonomous vehicle navigation system for the visually and hearing impaired. Provides real-time route guidance using the following information:

[0213] User's current location and destination information

[0214] Calculating barrier-free routes using the OpenStreetMap API

[0215] Voice guidance for the visually impaired using Google Text-to-Speech and Amazon Polly

[0216] Real-time route updates and notifications

[0217] These functions will be implemented in Python, and guidance information will be displayed on smartphones and smart glasses.

[0218] In this way, the present invention provides a system that assists the visually and hearing impaired to move around more safely and efficiently.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The user starts up the device (smartphone or smart glasses) and inputs information about their current location and destination. The device automatically acquires the location information using GPS and obtains the destination information through voice input or text input.

[0222] Input: User's current location information, destination information

[0223] Output: Current location information and destination information obtained

[0224] Specific behavior:

[0225] The user launches the app and speaks, "From home to hospital." The device uses GPS to obtain the current location, converts the speech into text, and obtains the destination information.

[0226] Step 2:

[0227] The device sends its current location and destination information to the server, which then accesses the OpenStreetMap API and a PostgreSQL database to calculate a barrier-free route.

[0228] Input: current location information, destination information

[0229] Output: Calculated optimal barrier-free route

[0230] Specific behavior:

[0231] To calculate the shortest route from the current location to the destination, the server refers to public transportation timetable data and congestion forecast data and selects a route that includes barrier-free facilities.

[0232] Step 3:

[0233] The server then sends the calculated optimal route information to the terminal, which receives the information and provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired.

[0234] Input: Calculated optimal route information

[0235] Output: Voice guide, text display, vibration guide

[0236] Specific behavior:

[0237] The device provides voice guidance such as "Turn left in 50 meters," and also uses text display and vibration functions to inform users of route information.

[0238] Step 4:

[0239] As the user moves, the device tracks their current location in real time, and the server monitors the route congestion and deviations in real time, recalculating the route and notifying the user if necessary.

[0240] Input: Real-time user location information, congestion prediction data

[0241] Output: Updated route information, recalculated optimal route

[0242] Specific behavior:

[0243] The server periodically checks the user's current location, and if there is a deviation from the route or a change in congestion, it calculates a new route and notifies the terminal.

[0244] Step 5:

[0245] When the user arrives at their destination, the device will notify them that they have arrived at the hospital and provide them with location information for information centers and barrier-free facilities near their destination.

[0246] Input: User's location indicating arrival

[0247] Output: Arrival notification, information on information desks and barrier-free facilities

[0248] Specific behavior:

[0249] The device uses GPS information to confirm that the user has arrived at their destination, and then provides a voice guide saying, "You have arrived at the hospital. The entrance is on your right." The screen also displays information about information centers and the nearest barrier-free facilities.

[0250] By following these steps and implementing the present invention, it becomes possible for visually and hearing impaired people to reach their destination safely and efficiently.

[0251] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0252] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described below.

[0253] System Configuration

[0254] The system consists of a user's device, a central server, a database, and an emotion engine. The devices include smartphones and tablets, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculations and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0255] Enter user information

[0256] After launching the app and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate stops and places to stop.

[0257] Route calculation and data collection

[0258] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0259] Providing guidance information to users

[0260] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[0261] Emotion recognition by emotion engine

[0262] The emotion engine installed in the user's device detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response, etc.), for example, determining whether the user is stressed or relaxed.

[0263] Emotion-based route adjustment

[0264] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[0265] Route Tracking

[0266] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[0267] Post-arrival support

[0268] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[0269] Specific examples

[0270] Example 1: Visually impaired person B travels from the station to the shopping mall

[0271] 1. Enter user information

[0272] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[0273] Specify the station information desk as your stopover point.

[0274] 2. Route calculation and data collection

[0275] The server receives the current location and destination and calculates the optimal route.

[0276] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0277] 3. Providing guidance information to users

[0278] The device provides guidance information to Mr. B via audio guide.

[0279] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[0280] 4. Emotion Recognition by Emotion Engine

[0281] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[0282] 5. Emotion-based route adjustment

[0283] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[0284] 6. Tracking and Notifications

[0285] The device tracks Mr. B's current location in real time and transmits it to the server.

[0286] Provide appropriate notification of deviations from route or changes in congestion.

[0287] 7. Post-arrival support

[0288] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[0289] It also provides information about the nearest information center and barrier-free facilities.

[0290] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[0291] The processing flow will be explained below.

[0292] Step 1:

[0293] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[0294] Step 2:

[0295] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[0296] Step 3:

[0297] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[0298] Step 4:

[0299] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[0300] Step 5:

[0301] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[0302] Step 6:

[0303] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[0304] Step 7:

[0305] The emotion engine uses facial recognition and biometric information to detect the user's emotional state, for example, assessing the user's stress level.

[0306] Step 8:

[0307] If the stress level is high, the server recalculates the route, choosing a route that avoids congestion or is more relaxing, and sends this new route information to the device.

[0308] Step 9:

[0309] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[0310] Step 10:

[0311] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[0312] Step 11:

[0313] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[0314] Step 12:

[0315] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[0316] Step 13:

[0317] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[0318] Example 2

[0319] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] In order for the visually and hearing impaired to reach their destinations safely, it is necessary to provide appropriate guidance information, and furthermore, flexible route adjustments are required to respond to changes in route and congestion during travel. However, conventional barrier-free map creation applications do not take the user's emotional state into consideration, and are therefore unable to respond to sudden stress or anxiety that may occur during travel. In addition, they do not provide sufficient real-time notifications of changes in congestion or deviations along the route. This makes it difficult for the visually and hearing impaired to travel safely and comfortably.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0322] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state using facial recognition and biometric information; means for recalculating the route based on the emotional state and providing adjusted route information; means for tracking the current location in real time and detecting and notifying changes in congestion or deviations that occur along the route; and means for providing location information of information centers and support facilities around the destination after arrival. This not only enables visually and hearing impaired people to travel safely and efficiently, but also reduces stress and anxiety during travel, supporting a more comfortable journey.

[0323] "User location information" is data indicating the user's current location.

[0324] "Destination information" is data indicating the location to which the user is heading.

[0325] "Public transportation timetable data" refers to data related to the operation schedules of public transportation such as buses and trains.

[0326] "Congestion forecast data" is data that predicts congestion on public transportation systems and routes.

[0327] "Barrier-free route information" refers to information about routes that are easy to use for people with visual and hearing impairments and people with physical disabilities.

[0328] An "optimal route" is the most efficient and safe travel route for a user's travel purpose.

[0329] "Emotional state" is information indicating the user's psychological state, such as stress level and relaxation level.

[0330] "Facial recognition" is a technology that analyzes the facial features of a user and identifies that face.

[0331] "Biometric information" refers to physiological data such as the user's heart rate and galvanic skin response.

[0332] "Route recalculation" refers to the process of correcting the initially calculated route and calculating a new optimal route.

[0333] "Tracking current location in real time" means continually updating the current location as the user moves.

[0334] A "route deviation" is a situation in which a user deviates from the planned route.

[0335] "Notification" refers to conveying information to the user by voice, text, vibration, etc.

[0336] "Post-arrival support" is additional assistance information provided to the user after they reach their destination.

[0337] An "information center" is a facility that provides information and support to users.

[0338] "Support facilities" are facilities equipped with barrier-free facilities to support users' movements and activities.

[0339] This invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments based on the user's emotional state. The system consists of a user's device, a central server, a database, and an emotion engine. The device can be a smartphone or tablet, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculation and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0340] Users launch the application using a device such as a smartphone or tablet and log in. They then provide their current location, which is automatically acquired using GPS, and their destination information, which they input via voice or text. This information is then sent from the device to a central server.

[0341] The server first uses the received location and destination information to refer to a database and calculate the optimal route. This involves using public transport timetable data, congestion forecast data, and barrier-free route information. Once the route is calculated, the results are sent to the device, which then provides the information to the user. Audio guidance is provided to the visually impaired, and text and vibration guidance is provided to the hearing impaired.

[0342] Next, the device's built-in emotion engine detects the user's emotional state. This is done by analyzing facial recognition using a camera and heart rate and skin galvanic response using biometric sensors. For example, if the emotion engine determines that the user is in a high stress state, that information is sent to the server, which then recalculates the route and suggests a new route that will allow the user to travel more relaxed.

[0343] The user's current location is tracked in real time by the device and sent to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[0344] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[0345] Specific examples

[0346] Example 1: Visually impaired person B travels from the station to the shopping mall

[0347] 1. Enter user information

[0348] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[0349] Specify the station information desk as your stopover point.

[0350] 2. Route calculation and data collection

[0351] The server receives the current location and destination and calculates the optimal route.

[0352] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0353] 3. Providing guidance information to users

[0354] The device provides guidance information to Mr. B via audio guide.

[0355] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[0356] 4. Emotion Recognition by Emotion Engine

[0357] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[0358] 5. Emotion-based route adjustment

[0359] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[0360] 6. Tracking and Notifications

[0361] The device tracks Mr. B's current location in real time and transmits it to the server.

[0362] Provide appropriate notification of deviations from route or changes in congestion.

[0363] 7. Post-arrival support

[0364] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[0365] It also provides information about the nearest information center and barrier-free facilities.

[0366] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[0367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0368] Step 1:

[0369] The user launches the app and logs in. The user inputs their current location and destination by voice or text. The device receives this information, displays it on the screen, and sends it to the server. The input is in the form of "current location is XX station, destination is △△ shopping mall." The output is sending the information to the server. In concrete terms, the user inputs their current location and destination by voice, and the device recognizes the content and converts it into text.

[0370] Step 2:

[0371] The device activates the GPS module and acquires the user's current location. The acquired current location information is sent from the device to the server. The input is GPS data, and the output is the transmission of current location information. Specifically, the GPS module measures the user's location information, analyzes the data, generates specific location coordinates, and sends them to the server.

[0372] Step 3:

[0373] The server refers to a database based on the location and destination information it receives and calculates the optimal route. Public transport timetable data, congestion forecast data, and barrier-free route information are obtained from the database. The inputs are location information, destination information, and database information, and the output is optimal route data. In concrete terms, the server searches the database using an SQL query and calculates the optimal route.

[0374] Step 4:

[0375] The server sends the calculated optimal route to the terminal. Based on the route information received by the terminal, voice guidance, text display, and vibration notification are performed. The input is the optimal route data, and the output is guidance information for the user. In concrete terms, the terminal uses a voice guidance engine (e.g., TTS) and text display function to provide guidance information to the user.

[0376] Step 5:

[0377] The emotion engine installed on the device uses the camera and biometric sensors to recognize the user's face and detect their biometric information. The detected emotion information is sent to the server. The input is a facial image and heart rate data, and the output is emotion information. Specifically, the camera captures the user's facial expression, the biometric sensors measure the heart rate, and these data are analyzed by the emotion analysis module.

[0378] Step 6:

[0379] The server analyzes the received emotional information and recalculates a new optimal route. Based on the emotional state, the server suggests a relaxing route for the user. The input is emotional information, and the output is adjusted route data. Specifically, the server searches the database again and calculates a new route that may reduce stress.

[0380] Step 7:

[0381] The device tracks the user's current location in real time and sends the information to the server. The server monitors the current location information and notifies the device if it detects deviation from the route or changes in congestion. The input is the current location data and the output is notification information. In concrete terms, the device updates the GPS data as needed, and the server analyzes the data to detect any anomalies.

[0382] Step 8:

[0383] When the device arrives at the destination, it notifies the user of the arrival. It also provides location information of information centers and support facilities around the destination. The input is current location data and arrival information, and the output is support information. Specifically, the device checks the current location information, and upon arrival, provides information about information centers and support facilities by voice or text.

[0384] (Application example 2)

[0385] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] It is extremely difficult for visually and hearing impaired people to travel safely and efficiently on public transportation or in crowded places. To address this issue, flexible responses that take into account the user's emotional state, as well as route guidance, are required. In particular, when visually and hearing impaired people move around a factory, real-time emotion analysis and route adjustment are required, but this has been difficult to achieve with conventional technology.

[0387] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referring to public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state and recalculating the route based on the emotional state; and means for analyzing the user's emotional state in real time using an emotion recognition engine. This enables visually and hearing-impaired people to move around factories safely and efficiently while receiving flexible support according to their emotional state.

[0388] "Location Information" means data that indicates a user's current location, including data obtained using GPS or other location measurement technologies.

[0389] "Destination information" is data on the target point the user wishes to reach. It is set by the user initially and serves as the basis for route calculation.

[0390] "Public transport timetable data" is data showing the operating schedules of each public transport system. Based on this data, the optimal timing for travel is calculated.

[0391] "Congestion forecast data" refers to data that predicts congestion on public transportation and travel routes. This includes predictions based on real-time and past data.

[0392] "Barrier-free route information" is data on routes that are safe for the visually and hearing impaired, including information on facilities such as elevators and ramps.

[0393] "Route calculation" is a process of calculating the optimal travel route based on the user's current location and destination information.

[0394] "Guidance information" refers to instructions and information necessary for a user to reach their destination, including audio guidance and text displays.

[0395] "Real-time tracking" is the process of continuously monitoring a user's location in real time, using GPS or other means.

[0396] A "route deviation" is when the user deviates from the calculated route, which triggers a recalculation and notification.

[0397] "Changes in congestion" refers to changes in travel routes and the congestion status of public transportation, and it is necessary to understand the situation in real time.

[0398] "Location information of information centers and support facilities" is data indicating the locations of information centers and registered support facilities that can be used by the user around the destination.

[0399] "Emotional state" refers to the user's current psychological state, including stress and relaxation.

[0400] An "emotion recognition engine" is software that uses cameras and sensors to analyze a user's emotional state from their facial expressions and biometric information.

[0401] A "robot guidance system" is a system that includes a robot for safely guiding visually and hearing impaired people within a factory or other designated area.

[0402] "Flexible response" refers to the process of adjusting the route and changing the guidance method as needed depending on the user's emotional state and travel situation.

[0403] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described in detail below.

[0404] System Configuration

[0405] This system consists of a user's device, a central server, a database, and an emotion engine. Devices include smartphones, tablets, and robots used in factories. The emotion engine is software that uses OpenCV and Python emotion recognition libraries to detect the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system such as Amazon Web Services (AWS) or Google Cloud Platform (GCP), and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0406] Enter user information

[0407] After launching the application and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate destinations and places to stop.

[0408] Route calculation and data collection

[0409] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0410] Providing guidance information to users

[0411] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[0412] Emotion recognition by emotion engine

[0413] The emotion engine detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response). For example, it determines whether the user is stressed or relaxed. The emotion engine uses OpenCV and Python emotion recognition libraries.

[0414] Emotion-based route adjustment

[0415] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[0416] Route Tracking

[0417] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[0418] Post-arrival support

[0419] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[0420] Specific examples

[0421] Example 1: When a visually and hearing impaired person, B, moves around the factory

[0422] 1. Enter user information

[0423] Person B launches the app and voice-inputs his current location (factory entrance) and destination (designated work area).

[0424] Designate a rest room as a stopover location.

[0425] 2. Route calculation and data collection

[0426] The server receives the current location and destination and calculates the optimal route.

[0427] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0428] 3. Providing guidance information to users

[0429] The device provides voice guidance to Mr. B.

[0430] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[0431] 4. Emotion Recognition by Emotion Engine

[0432] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[0433] 5. Emotion-based route adjustment

[0434] The server takes stress levels into account and recalculates a more relaxing route.

[0435] New guidance information is sent to the device.

[0436] 6. Tracking and Notifications

[0437] The device tracks Mr. B's current location in real time and transmits it to the server.

[0438] Provide appropriate notification of deviations from route or changes in congestion.

[0439] 7. Post-arrival support

[0440] When he arrives at the work area, the terminal notifies Mr. B of his arrival.

[0441] It also provides information on the nearest information center and barrier-free facilities.

[0442] Example prompt sentence:

[0443] If employees are feeling anxious, calculate a relaxing route through the factory.

[0444] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0445] Step 1:

[0446] The user launches the application and logs in.

[0447] Input: User login information (ID, password).

[0448] Output: The user is logged in.

[0449] Specific operation: The terminal sends the login information to the server and is authenticated. After successful authentication, the main screen is displayed.

[0450] Step 2:

[0451] The user inputs current location information and destination information.

[0452] Input: Current location information (GPS), destination information (manual input or voice input)

[0453] Output: Current location and destination information data.

[0454] How it works: The device uses GPS to obtain current location information, and the user inputs destination information manually or by voice. An interface is also available for specifying additional intermediate destinations and stops.

[0455] Step 3:

[0456] The server calculates the optimal route based on the current location information and destination information received.

[0457] Input: current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[0458] Output: Optimal route information.

[0459] Specific operation: The server refers to the database, executes the routing algorithm based on the necessary data, and calculates the optimal route. The calculated route is sent to the terminal.

[0460] Step 4:

[0461] The terminal presents the calculated route information to the user.

[0462] Input: Optimal route information.

[0463] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[0464] Specific operation: The device provides route guidance in the form of voice, text, vibration, etc. depending on the type of disability the user has.

[0465] Step 5:

[0466] An emotion engine detects the user's emotional state.

[0467] Input: User's facial recognition data, biometric information (heart rate, galvanic skin response).

[0468] Output: Emotional state (stress level and relaxation level).

[0469] Specific operation: The device's camera and sensors collect data on the user's face and physical state, which are then analyzed using an emotion recognition library (OpenCV, Python library). The analysis results are then sent to the server.

[0470] Step 6:

[0471] The server recalculates the route based on the emotional state.

[0472] Input: Emotional state, current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[0473] Output: New optimal route information based on emotional state.

[0474] Specific operation: The server takes into account the emotional state, re-references the database, recalculates relaxing routes and routes during quiet times, and sends the results to the terminal.

[0475] Step 7:

[0476] The terminal then presents the recalculated new route information to the user.

[0477] Input: New optimal route information.

[0478] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[0479] Specific operation: The terminal receives the recalculated route information from the database and provides the user with new directions in an appropriate format.

[0480] Step 8:

[0481] The device tracks your location in real time and detects deviations from your route or changes in congestion.

[0482] Input: Current location information (GPS data), congestion forecast data.

[0483] Output: Route deviation notification, congestion change notification.

[0484] Specific operation: The device periodically updates its current location using its GPS function and sends it to the server. The server monitors the current location in real time and sends a notification to the device if it detects deviations or changes in congestion.

[0485] Step 9:

[0486] When the user arrives at their destination, they are provided with information on information centers and support facilities.

[0487] Input: Destination arrival notification.

[0488] Output: Location information of information centers and support facilities around the destination.

[0489] Specific operation: When the device detects arrival at the destination, it automatically provides the user with the location information of information centers and support facilities. This is done by voice or text notification.

[0490] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0491] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0492] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0501] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0506] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. Specific embodiments of the application will be described below.

[0507] System Configuration

[0508] This system consists of a user device, a central server, and a database. User devices include smartphones and tablets. The central server is a high-performance computer system for route calculations and data processing. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0509] Enter user information

[0510] Users start the app, log in, and enter their current location and destination information. Current location information can also be obtained automatically using GPS. In addition, an interface is provided for specifying intermediate destinations and places to stop.

[0511] Route calculation and data collection

[0512] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0513] Providing guidance information to users

[0514] The user's device receives the calculated route information and provides guidance in a format tailored to the visually and hearing impaired. The visually impaired receive audio guidance, while the hearing impaired receive text and vibration guidance. Detailed guidance is also provided on important points (e.g., elevator locations, arrow sign directions, etc.).

[0515] Route Tracking

[0516] As users move, the device tracks their location in real time and detects and notifies them of changes in congestion or deviations along the route, allowing them to quickly get back on track.

[0517] Post-arrival support

[0518] When the user arrives at their destination, the device will notify them of their arrival. It will also provide information about the locations of nearby information centers and support facilities. For example, if the user arrives at a hospital, they will receive detailed information about the location of the nearest information center and elevator.

[0519] Specific examples

[0520] Example 1: Visually impaired person A travels from the station to the library

[0521] 1. Enter user information

[0522] Person A launches the app and voice-inputs his current location (station) and destination (library).

[0523] Specify the station information desk as your stopover point.

[0524] 2. Route calculation and data collection

[0525] The server receives the current location and destination and calculates the optimal route.

[0526] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0527] 3. Providing guidance information to users

[0528] The device provides guidance information to Mr. A via audio guide.

[0529] It includes detailed instructions such as "Turn left and the elevator is 50 meters ahead."

[0530] 4. Route Tracking

[0531] While Mr. A is traveling, the device tracks his current location in real time and notifies him if he strays from his route or if there is a change in congestion.

[0532] 5. Post-arrival support

[0533] When you arrive at the library, the device will announce, "You have arrived at the library. The entrance is on your right."

[0534] It also provides information about the library's information desk and barrier-free facilities.

[0535] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently.

[0536] The processing flow will be explained below.

[0537] Step 1:

[0538] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[0539] Step 2:

[0540] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[0541] Step 3:

[0542] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[0543] Step 4:

[0544] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[0545] Step 5:

[0546] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[0547] Step 6:

[0548] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[0549] Step 7:

[0550] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[0551] Step 8:

[0552] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[0553] Step 9:

[0554] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[0555] Step 10:

[0556] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[0557] Step 11:

[0558] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[0559] Example 1

[0560] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0561] There is a lack of systems that guide people with visual and hearing impairments to the optimal route from their current location to their destination. In particular, there are issues with the lack of real-time congestion information, route deviation notifications, and information on information centers and support facilities near the destination. There is also a need for more flexible guidance methods according to the type of disability, as well as an interface that allows users to specify intermediate stops and stopovers when planning a route.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0563] In this invention, the server includes means for receiving user location information and destination information, means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information, means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's information processing device, means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations that occur along the route, means for providing location information of information centers and support facilities around the destination after the user arrives, and a barrier-free map creation application designed for the visually and hearing impaired. This enables the visually and hearing impaired to travel safely and efficiently and to be provided with necessary guidance information in a timely manner.

[0564] "User location information" refers to geographical information about the user's current location, and is data obtained using technology such as GPS.

[0565] "Destination information" is geographical information about the final destination that the user is aiming for.

[0566] "Public transport timetable data" refers to data relating to the operation schedules of public transport such as buses and trains.

[0567] "Congestion prediction data" is data that predicts the congestion of people and vehicles at specific times and locations.

[0568] "Barrier-free route information" refers to information about routes that everyone, including those with visual or hearing impairments, can travel on safely and comfortably.

[0569] The "optimal route" is the most efficient travel route between the current location set by the user and the destination, taking into consideration time, types of obstacles, and the like.

[0570] A "calculated optimal route" is the best route to travel calculated using the user's current location and destination and related data.

[0571] "Real-time location tracking" means keeping track of the user's current location at all times.

[0572] "Route deviation detection" refers to detecting when a user deviates from a pre-planned route.

[0573] "Location information of information centers and support facilities" is geographical information about information centers and facilities that support users located in the vicinity of the destination.

[0574] A "barrier-free map creation application designed for the visually and hearing impaired" is map software designed to guide the visually and hearing impaired to navigate safely and efficiently.

[0575] MODE FOR CARRYING OUT THE INVENTION

[0576] The present invention is a barrier-free map creation application designed for the visually and hearing impaired, and specific embodiments thereof will be described below.

[0577] System Configuration

[0578] This system consists of user devices, a central server, and a database. User devices include mobile information terminals such as smartphones and tablets, through which the application runs. The central server is a high-performance computer system that calculates routes and processes data. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0579] Enter user information

[0580] Users must launch the app on their device and log in. The application provides an interface for users to input their current location and destination information. Current location information can also be obtained automatically using GPS. Users can also specify intermediate locations and places they want to stop at.

[0581] Route calculation and data collection

[0582] The server accesses a database to reference public transportation timetable data, congestion forecast data, and barrier-free route information based on the current location and destination information received from the user. Based on this data, the server calculates the optimal route for the user. The route calculation includes routes that are safe and accessible, especially for the visually and hearing impaired.

[0583] Providing guidance information to users

[0584] The calculated optimal route information is sent from the server to the user's device. Based on the received route information, the device provides guidance information in the form of voice guidance for the visually impaired and text display or vibration for the hearing impaired. For example, the device may provide a voice guide saying, "Turn left and there is an elevator 50 meters ahead."

[0585] Route Tracking

[0586] While the user is traveling along a route, the device tracks their location in real time. The device periodically checks its location using GPS and notifies the user if there is a deviation from the route or changes in traffic congestion, allowing the user to quickly get back on track.

[0587] Post-arrival support

[0588] When the user arrives at their destination, the device will notify them, "You have arrived at your destination." It will also provide detailed location information for information centers and support facilities around the destination. For example, if they arrive at a hospital, they will be told, "The elevator is on the right side of the building."

[0589] Specific examples

[0590] Example 1: Visually impaired person A travels from the station to the library

[0591] 1. The user launches the app and speaks, "I want to go to the library from the station."

[0592] 2. The server analyzes the received information and calculates the optimal route by referring to public transport timetable data, congestion forecast data, and barrier-free route information.

[0593] 3. The device will give voice guidance saying, "Turn left and get on the elevator 50 meters ahead."

[0594] 4. The device tracks your location in real time while you are moving, and if you stray from your route, it will notify you, "Go back 10 meters and turn right."

[0595] 5. When the device arrives at its destination, it will provide detailed information such as, "You have arrived at the library. The entrance is on the right. Additionally, the information desk is on the left."

[0596] Example prompts for generative AI models

[0597] I'd like to design an application that provides optimal routes and guidance for visually impaired people traveling from the station to the library. Please explain the specific process for this application: based on current location and destination information, this application calculates the optimal route by referencing public transport timetable data, congestion forecast data, and barrier-free route information, and then provides guidance using audio guides and text displays.

[0598] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and is a system that provides necessary guidance information in a timely manner, ensuring the safety and security of users.

[0599] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0600] Step 1: Enter your information

[0601] Users launch the app on their smartphone or tablet and log in.

[0602] The user specifies the destination information by voice input or text input, and the application can automatically obtain the current location information using the GPS function.

[0603] Input: User's current location information, destination information, (optional) intermediate locations and places to stop

[0604] Output: User input data (current location, destination information)

[0605] As a specific example of operation, when the user is at a station, he or she inputs "I want to go to the library" by voice.

[0606] Step 2: Route calculation and data collection

[0607] The server accesses a database based on the current location information and destination information received from the user, and obtains public transportation timetable data, congestion forecast data, and barrier-free route information.

[0608] The server uses the acquired data to calculate the optimal route using an algorithm that takes into account accessibility and congestion.

[0609] Input: User's current location information, destination information, public transport timetable data in the database, congestion forecast data, barrier-free route information

[0610] Output: Calculated optimal route information

[0611] As a specific example of how it works, the server "identifies the nearest station based on GPS coordinates and calculates the optimal route from that station, taking into account barrier-free information."

[0612] Step 3: Providing guidance to users

[0613] The server transmits the calculated optimum route information to the user's terminal.

[0614] Based on the route information received, the device provides voice guidance for visually impaired people and guidance information via text display and vibration for hearing impaired people.

[0615] Input: Calculated optimal route information

[0616] Output: Guidance information for the user (audio guide, text display, vibration guide)

[0617] As a specific example of operation, the user's device plays back a voice instruction saying, "Turn left and use the elevator 50 meters ahead."

[0618] Step 4: Path tracing

[0619] The device periodically acquires GPS data to track the user's location in real time as they move.

[0620] The device compares the acquired current location information with the calculated route and notifies the user if the user deviates from the route or if congestion occurs.

[0621] Input: Real-time GPS data, calculated route information

[0622] Output: Notifications about route deviations and congestion

[0623] As a specific example of how it works, the device will issue a voice notification saying, "You are currently off route. Go back 10 meters and turn right."

[0624] Step 5: Support after arrival

[0625] When the terminal detects the user's arrival, it notifies the user that "you have arrived at your destination."

[0626] The device also provides the user with detailed location information for information centers and support facilities around the destination.

[0627] Input: User's current location (destination arrival), support facility information in the database

[0628] Output: Arrival notification at destination, location information of support facilities

[0629] As a specific example of how it works, the device will say, "You have arrived at the library. The elevator is on the right side of the building."

[0630] In this way, the system is designed to enable the visually and hearing impaired to travel to their destination safely and efficiently, and is equipped with functions for real-time route tracking and detailed guidance around the destination.

[0631] (Application example 1)

[0632] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] When using public transportation or self-driving vehicles, the visually and hearing impaired face difficulties in reaching their destinations safely and efficiently due to insufficient route guidance and barrier-free information. It is also difficult to respond immediately to congestion information or route deviations while traveling. Therefore, there is a need for a comprehensive navigation system that enables the visually and hearing impaired to travel safely.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0635] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations along the route; means for providing location information of information centers and support facilities around the destination after the user arrives; means for navigating the route to the destination in cooperation with an autonomous vehicle; and means for providing real-time guidance information to the visually and hearing impaired using a smartphone or smart glasses, thereby enabling the visually and hearing impaired to reach their destination safely and efficiently via a barrier-free route.

[0636] "User location information" is data that indicates the geographical location of the user, obtained using a mobile device or GPS function.

[0637] "Destination information" is geographical location data relating to the location where the user ultimately wants to arrive.

[0638] "Public transport timetable data" refers to information about the operation schedules of public transport such as buses and trains.

[0639] "Congestion prediction data" is information that predicts the congestion situation of people along a specified route or at a destination.

[0640] "Barrier-free route information" refers to information about safe, barrier-free routes that are easy for wheelchair users and those with visual impairments to navigate.

[0641] The "optimal route" is the most suitable route for traveling from the user's location information to the destination information, taking into consideration time, convenience, and safety.

[0642] "Location tracking" refers to the process of continuously monitoring a user's current location to determine their location in real time.

[0643] "Route deviation" means that the user deviates from the calculated optimal route.

[0644] "Information centers and support facilities" are locations or service facilities that provide assistance and information during travel or upon arrival at a destination.

[0645] An "autonomous vehicle" is a vehicle that operates autonomously without the need for human operation.

[0646] A "smartphone" is a portable information terminal that has various functions in addition to telephone functions, such as internet connection.

[0647] "Smart glasses" are eyeglass-type devices with built-in computers that are capable of augmented reality and information display.

[0648] "Real-time guidance information" means up-to-date route and situation information provided immediately at the current time.

[0649] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. The system consists of a user terminal, a central server, and a database. Specific embodiments are described below.

[0650] System Configuration

[0651] Hardware Configuration

[0652] The system uses mobile devices such as smartphones and smart glasses, which have GPS capabilities to acquire user location information. Sensor modules in autonomous vehicles are also utilized, and the system connects to a central server to update route information in real time.

[0653] Software Configuration

[0654] The central server is a high-performance computer system that accesses the OpenStreetMap API and PostgreSQL database to calculate barrier-free routes based on location and destination information received from user devices. Voice guidance is provided using Google Text-to-Speech and AWS Polly.

[0655] Data processing and route calculation

[0656] The server receives the user's current location and destination information and calculates the optimal barrier-free route based on this information. The calculation uses public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is sent to the user's device, and route information is provided in real time.

[0657] User guidance and real-time tracking

[0658] Based on the route information received, the user's device provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired. It works in conjunction with the autonomous vehicle to navigate the route to the destination. If congestion changes or the user deviates from the route while traveling, the device will be notified immediately.

[0659] Post-arrival support

[0660] When the user arrives at their destination, the device will notify them of their arrival and provide location information for nearby information centers and support facilities, allowing the visually and hearing impaired to smoothly receive support at their destination.

[0661] Specific examples

[0662] Example 1: B, a visually impaired person, travels to the hospital using an autonomous vehicle

[0663] 1. Enter user information

[0664] Person B launches the smartphone app and voice-inputs his current location (home) and destination (hospital).

[0665] 2. Route calculation

[0666] The server calculates the optimal barrier-free route based on the current location and destination, and updates the route information for the autonomous vehicle.

[0667] 3. Real-time guidance

[0668] Directions are provided as audio guides for the visually impaired, for example: "Turn left in 100 metres."

[0669] 4. Real-time route tracking

[0670] It works in conjunction with autonomous vehicles to monitor changes in congestion and route deviations during travel in real time, and notifies drivers as necessary.

[0671] 5. Post-arrival support

[0672] When the user arrives at the hospital, the smartphone will notify them with a voice message saying, "You have arrived at the hospital. The entrance is on your right." It will also provide information about the hospital's information desk and barrier-free facilities.

[0673] Prompt Sentence Examples

[0674] "Development of an autonomous vehicle navigation system for the visually and hearing impaired. Provides real-time route guidance using the following information:

[0675] User's current location and destination information

[0676] Calculating barrier-free routes using the OpenStreetMap API

[0677] Voice guidance for the visually impaired using Google Text-to-Speech and Amazon Polly

[0678] Real-time route updates and notifications

[0679] These functions will be implemented in Python, and guidance information will be displayed on smartphones and smart glasses.

[0680] In this way, the present invention provides a system that assists the visually and hearing impaired to move around more safely and efficiently.

[0681] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0682] Step 1:

[0683] The user starts up the device (smartphone or smart glasses) and inputs information about their current location and destination. The device automatically acquires the location information using GPS and obtains the destination information through voice input or text input.

[0684] Input: User's current location information, destination information

[0685] Output: Current location information and destination information obtained

[0686] Specific behavior:

[0687] The user launches the app and speaks, "From home to hospital." The device uses GPS to obtain the current location, converts the speech into text, and obtains the destination information.

[0688] Step 2:

[0689] The device sends its current location and destination information to the server, which then accesses the OpenStreetMap API and a PostgreSQL database to calculate a barrier-free route.

[0690] Input: current location information, destination information

[0691] Output: Calculated optimal barrier-free route

[0692] Specific behavior:

[0693] To calculate the shortest route from the current location to the destination, the server refers to public transportation timetable data and congestion forecast data and selects a route that includes barrier-free facilities.

[0694] Step 3:

[0695] The server then sends the calculated optimal route information to the terminal, which receives the information and provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired.

[0696] Input: Calculated optimal route information

[0697] Output: Voice guide, text display, vibration guide

[0698] Specific behavior:

[0699] The device provides voice guidance such as "Turn left in 50 meters," and also uses text display and vibration functions to inform users of route information.

[0700] Step 4:

[0701] As the user moves, the device tracks their current location in real time, and the server monitors the route congestion and deviations in real time, recalculating the route and notifying the user if necessary.

[0702] Input: Real-time user location information, congestion prediction data

[0703] Output: Updated route information, recalculated optimal route

[0704] Specific behavior:

[0705] The server periodically checks the user's current location, and if there is a deviation from the route or a change in congestion, it calculates a new route and notifies the terminal.

[0706] Step 5:

[0707] When the user arrives at their destination, the device will notify them that they have arrived at the hospital and provide them with location information for information centers and barrier-free facilities near their destination.

[0708] Input: User's location indicating arrival

[0709] Output: Arrival notification, information on information desks and barrier-free facilities

[0710] Specific behavior:

[0711] The device uses GPS information to confirm that the user has arrived at their destination, and then provides a voice guide saying, "You have arrived at the hospital. The entrance is on your right." The screen also displays information about information centers and the nearest barrier-free facilities.

[0712] By following these steps and implementing the present invention, it becomes possible for visually and hearing impaired people to reach their destination safely and efficiently.

[0713] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0714] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described below.

[0715] System Configuration

[0716] The system consists of a user's device, a central server, a database, and an emotion engine. The devices include smartphones and tablets, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculations and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0717] Enter user information

[0718] After launching the app and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate stops and places to stop.

[0719] Route calculation and data collection

[0720] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0721] Providing guidance information to users

[0722] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[0723] Emotion recognition by emotion engine

[0724] The emotion engine installed in the user's device detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response, etc.), for example, determining whether the user is stressed or relaxed.

[0725] Emotion-based route adjustment

[0726] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[0727] Route Tracking

[0728] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[0729] Post-arrival support

[0730] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[0731] Specific examples

[0732] Example 1: Visually impaired person B travels from the station to the shopping mall

[0733] 1. Enter user information

[0734] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[0735] Specify the station information desk as your stopover point.

[0736] 2. Route calculation and data collection

[0737] The server receives the current location and destination and calculates the optimal route.

[0738] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0739] 3. Providing guidance information to users

[0740] The device provides guidance information to Mr. B via audio guide.

[0741] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[0742] 4. Emotion Recognition by Emotion Engine

[0743] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[0744] 5. Emotion-based route adjustment

[0745] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[0746] 6. Tracking and Notifications

[0747] The device tracks Mr. B's current location in real time and transmits it to the server.

[0748] Provide appropriate notification of deviations from route or changes in congestion.

[0749] 7. Post-arrival support

[0750] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[0751] It also provides information about the nearest information center and barrier-free facilities.

[0752] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[0753] The processing flow will be explained below.

[0754] Step 1:

[0755] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[0756] Step 2:

[0757] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[0758] Step 3:

[0759] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[0760] Step 4:

[0761] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[0762] Step 5:

[0763] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[0764] Step 6:

[0765] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[0766] Step 7:

[0767] The emotion engine uses facial recognition and biometric information to detect the user's emotional state, for example, assessing the user's stress level.

[0768] Step 8:

[0769] If the stress level is high, the server recalculates the route, choosing a route that avoids congestion or is more relaxing, and sends this new route information to the device.

[0770] Step 9:

[0771] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[0772] Step 10:

[0773] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[0774] Step 11:

[0775] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[0776] Step 12:

[0777] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[0778] Step 13:

[0779] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[0780] Example 2

[0781] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0782] In order for the visually and hearing impaired to reach their destinations safely, it is necessary to provide appropriate guidance information, and furthermore, flexible route adjustments are required to respond to changes in route and congestion during travel. However, conventional barrier-free map creation applications do not take the user's emotional state into consideration, and are therefore unable to respond to sudden stress or anxiety that may occur during travel. In addition, they do not provide sufficient real-time notifications of changes in congestion or deviations along the route. This makes it difficult for the visually and hearing impaired to travel safely and comfortably.

[0783] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0784] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state using facial recognition and biometric information; means for recalculating the route based on the emotional state and providing adjusted route information; means for tracking the current location in real time and detecting and notifying changes in congestion or deviations that occur along the route; and means for providing location information of information centers and support facilities around the destination after arrival. This not only enables visually and hearing impaired people to travel safely and efficiently, but also reduces stress and anxiety during travel, supporting a more comfortable journey.

[0785] "User location information" is data indicating the user's current location.

[0786] "Destination information" is data indicating the location to which the user is heading.

[0787] "Public transportation timetable data" refers to data related to the operation schedules of public transportation such as buses and trains.

[0788] "Congestion forecast data" is data that predicts congestion on public transportation systems and routes.

[0789] "Barrier-free route information" refers to information about routes that are easy to use for people with visual and hearing impairments and people with physical disabilities.

[0790] An "optimal route" is the most efficient and safe travel route for a user's travel purpose.

[0791] "Emotional state" is information indicating the user's psychological state, such as stress level and relaxation level.

[0792] "Facial recognition" is a technology that analyzes the facial features of a user and identifies that face.

[0793] "Biometric information" refers to physiological data such as the user's heart rate and galvanic skin response.

[0794] "Route recalculation" refers to the process of correcting the initially calculated route and calculating a new optimal route.

[0795] "Tracking current location in real time" means continually updating the current location as the user moves.

[0796] A "route deviation" is a situation in which a user deviates from the planned route.

[0797] "Notification" refers to conveying information to the user by voice, text, vibration, etc.

[0798] "Post-arrival support" is additional assistance information provided to the user after they reach their destination.

[0799] An "information center" is a facility that provides information and support to users.

[0800] "Support facilities" are facilities equipped with barrier-free facilities to support users' movements and activities.

[0801] This invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments based on the user's emotional state. The system consists of a user's device, a central server, a database, and an emotion engine. The device can be a smartphone or tablet, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculation and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0802] Users launch the application using a device such as a smartphone or tablet and log in. They then provide their current location, which is automatically acquired using GPS, and their destination information, which they input via voice or text. This information is then sent from the device to a central server.

[0803] The server first uses the received location and destination information to refer to a database and calculate the optimal route. This involves using public transport timetable data, congestion forecast data, and barrier-free route information. Once the route is calculated, the results are sent to the device, which then provides the information to the user. Audio guidance is provided to the visually impaired, and text and vibration guidance is provided to the hearing impaired.

[0804] Next, the device's built-in emotion engine detects the user's emotional state. This is done by analyzing facial recognition using a camera and heart rate and skin galvanic response using biometric sensors. For example, if the emotion engine determines that the user is in a high stress state, that information is sent to the server, which then recalculates the route and suggests a new route that will allow the user to travel more relaxed.

[0805] The user's current location is tracked in real time by the device and sent to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[0806] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[0807] Specific examples

[0808] Example 1: Visually impaired person B travels from the station to the shopping mall

[0809] 1. Enter user information

[0810] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[0811] Specify the station information desk as your stopover point.

[0812] 2. Route calculation and data collection

[0813] The server receives the current location and destination and calculates the optimal route.

[0814] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0815] 3. Providing guidance information to users

[0816] The device provides guidance information to Mr. B via audio guide.

[0817] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[0818] 4. Emotion Recognition by Emotion Engine

[0819] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[0820] 5. Emotion-based route adjustment

[0821] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[0822] 6. Tracking and Notifications

[0823] The device tracks Mr. B's current location in real time and transmits it to the server.

[0824] Provide appropriate notification of deviations from route or changes in congestion.

[0825] 7. Post-arrival support

[0826] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[0827] It also provides information about the nearest information center and barrier-free facilities.

[0828] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[0829] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0830] Step 1:

[0831] The user launches the app and logs in. The user inputs their current location and destination by voice or text. The device receives this information, displays it on the screen, and sends it to the server. The input is in the form of "current location is XX station, destination is △△ shopping mall." The output is sending the information to the server. In concrete terms, the user inputs their current location and destination by voice, and the device recognizes the content and converts it into text.

[0832] Step 2:

[0833] The device activates the GPS module and acquires the user's current location. The acquired current location information is sent from the device to the server. The input is GPS data, and the output is the transmission of current location information. Specifically, the GPS module measures the user's location information, analyzes the data, generates specific location coordinates, and sends them to the server.

[0834] Step 3:

[0835] The server refers to a database based on the location and destination information it receives and calculates the optimal route. Public transport timetable data, congestion forecast data, and barrier-free route information are obtained from the database. The inputs are location information, destination information, and database information, and the output is optimal route data. In concrete terms, the server searches the database using an SQL query and calculates the optimal route.

[0836] Step 4:

[0837] The server sends the calculated optimal route to the terminal. Based on the route information received by the terminal, voice guidance, text display, and vibration notification are performed. The input is the optimal route data, and the output is guidance information for the user. In concrete terms, the terminal uses a voice guidance engine (e.g., TTS) and text display function to provide guidance information to the user.

[0838] Step 5:

[0839] The emotion engine installed on the device uses the camera and biometric sensors to recognize the user's face and detect their biometric information. The detected emotion information is sent to the server. The input is a facial image and heart rate data, and the output is emotion information. Specifically, the camera captures the user's facial expression, the biometric sensors measure the heart rate, and these data are analyzed by the emotion analysis module.

[0840] Step 6:

[0841] The server analyzes the received emotional information and recalculates a new optimal route. Based on the emotional state, the server suggests a relaxing route for the user. The input is emotional information, and the output is adjusted route data. Specifically, the server searches the database again and calculates a new route that may reduce stress.

[0842] Step 7:

[0843] The device tracks the user's current location in real time and sends the information to the server. The server monitors the current location information and notifies the device if it detects deviation from the route or changes in congestion. The input is the current location data and the output is notification information. In concrete terms, the device updates the GPS data as needed, and the server analyzes the data to detect any anomalies.

[0844] Step 8:

[0845] When the device arrives at the destination, it notifies the user of the arrival. It also provides location information of information centers and support facilities around the destination. The input is current location data and arrival information, and the output is support information. Specifically, the device checks the current location information, and upon arrival, provides information about information centers and support facilities by voice or text.

[0846] (Application example 2)

[0847] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0848] It is extremely difficult for visually and hearing impaired people to travel safely and efficiently on public transportation or in crowded places. To address this issue, flexible responses that take into account the user's emotional state, as well as route guidance, are required. In particular, when visually and hearing impaired people move around a factory, real-time emotion analysis and route adjustment are required, but this has been difficult to achieve with conventional technology.

[0849] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referring to public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state and recalculating the route based on the emotional state; and means for analyzing the user's emotional state in real time using an emotion recognition engine. This enables visually and hearing-impaired people to move around factories safely and efficiently while receiving flexible support according to their emotional state.

[0850] "Location Information" means data that indicates a user's current location, including data obtained using GPS or other location measurement technologies.

[0851] "Destination information" is data on the target point the user wishes to reach. It is set by the user initially and serves as the basis for route calculation.

[0852] "Public transport timetable data" is data showing the operating schedules of each public transport system. Based on this data, the optimal timing for travel is calculated.

[0853] "Congestion forecast data" refers to data that predicts congestion on public transportation and travel routes. This includes predictions based on real-time and past data.

[0854] "Barrier-free route information" is data on routes that are safe for the visually and hearing impaired, including information on facilities such as elevators and ramps.

[0855] "Route calculation" is a process of calculating the optimal travel route based on the user's current location and destination information.

[0856] "Guidance information" refers to instructions and information necessary for a user to reach their destination, including audio guidance and text displays.

[0857] "Real-time tracking" is the process of continuously monitoring a user's location in real time, using GPS or other means.

[0858] A "route deviation" is when the user deviates from the calculated route, which triggers a recalculation and notification.

[0859] "Changes in congestion" refers to changes in travel routes and the congestion status of public transportation, and it is necessary to understand the situation in real time.

[0860] "Location information of information centers and support facilities" is data indicating the locations of information centers and registered support facilities that can be used by the user around the destination.

[0861] "Emotional state" refers to the user's current psychological state, including stress and relaxation.

[0862] An "emotion recognition engine" is software that uses cameras and sensors to analyze a user's emotional state from their facial expressions and biometric information.

[0863] A "robot guidance system" is a system that includes a robot for safely guiding visually and hearing impaired people within a factory or other designated area.

[0864] "Flexible response" refers to the process of adjusting the route and changing the guidance method as needed depending on the user's emotional state and travel situation.

[0865] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described in detail below.

[0866] System Configuration

[0867] This system consists of a user's device, a central server, a database, and an emotion engine. Devices include smartphones, tablets, and robots used in factories. The emotion engine is software that uses OpenCV and Python emotion recognition libraries to detect the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system such as Amazon Web Services (AWS) or Google Cloud Platform (GCP), and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0868] Enter user information

[0869] After launching the application and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate destinations and places to stop.

[0870] Route calculation and data collection

[0871] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0872] Providing guidance information to users

[0873] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[0874] Emotion recognition by emotion engine

[0875] The emotion engine detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response). For example, it determines whether the user is stressed or relaxed. The emotion engine uses OpenCV and Python emotion recognition libraries.

[0876] Emotion-based route adjustment

[0877] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[0878] Route Tracking

[0879] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[0880] Post-arrival support

[0881] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[0882] Specific examples

[0883] Example 1: When a visually and hearing impaired person, B, moves around the factory

[0884] 1. Enter user information

[0885] Person B launches the app and voice-inputs his current location (factory entrance) and destination (designated work area).

[0886] Designate a rest room as a stopover location.

[0887] 2. Route calculation and data collection

[0888] The server receives the current location and destination and calculates the optimal route.

[0889] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0890] 3. Providing guidance information to users

[0891] The device provides voice guidance to Mr. B.

[0892] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[0893] 4. Emotion Recognition by Emotion Engine

[0894] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[0895] 5. Emotion-based route adjustment

[0896] The server takes stress levels into account and recalculates a more relaxing route.

[0897] New guidance information is sent to the device.

[0898] 6. Tracking and Notifications

[0899] The device tracks Mr. B's current location in real time and transmits it to the server.

[0900] Provide appropriate notification of deviations from route or changes in congestion.

[0901] 7. Post-arrival support

[0902] When he arrives at the work area, the terminal notifies Mr. B of his arrival.

[0903] It also provides information on the nearest information center and barrier-free facilities.

[0904] Example prompt sentence:

[0905] If employees are feeling anxious, calculate a relaxing route through the factory.

[0906] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0907] Step 1:

[0908] The user launches the application and logs in.

[0909] Input: User login information (ID, password).

[0910] Output: The user is logged in.

[0911] Specific operation: The terminal sends the login information to the server and is authenticated. After successful authentication, the main screen is displayed.

[0912] Step 2:

[0913] The user inputs current location information and destination information.

[0914] Input: Current location information (GPS), destination information (manual input or voice input)

[0915] Output: Current location and destination information data.

[0916] How it works: The device uses GPS to obtain current location information, and the user inputs destination information manually or by voice. An interface is also available for specifying additional intermediate destinations and stops.

[0917] Step 3:

[0918] The server calculates the optimal route based on the current location information and destination information received.

[0919] Input: current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[0920] Output: Optimal route information.

[0921] Specific operation: The server refers to the database, executes the routing algorithm based on the necessary data, and calculates the optimal route. The calculated route is sent to the terminal.

[0922] Step 4:

[0923] The terminal presents the calculated route information to the user.

[0924] Input: Optimal route information.

[0925] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[0926] Specific operation: The device provides route guidance in the form of voice, text, vibration, etc. depending on the type of disability the user has.

[0927] Step 5:

[0928] An emotion engine detects the user's emotional state.

[0929] Input: User's facial recognition data, biometric information (heart rate, galvanic skin response).

[0930] Output: Emotional state (stress level and relaxation level).

[0931] Specific operation: The device's camera and sensors collect data on the user's face and physical state, which are then analyzed using an emotion recognition library (OpenCV, Python library). The analysis results are then sent to the server.

[0932] Step 6:

[0933] The server recalculates the route based on the emotional state.

[0934] Input: Emotional state, current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[0935] Output: New optimal route information based on emotional state.

[0936] Specific operation: The server takes into account the emotional state, re-references the database, recalculates relaxing routes and routes during quiet times, and sends the results to the terminal.

[0937] Step 7:

[0938] The terminal then presents the recalculated new route information to the user.

[0939] Input: New optimal route information.

[0940] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[0941] Specific operation: The terminal receives the recalculated route information from the database and provides the user with new directions in an appropriate format.

[0942] Step 8:

[0943] The device tracks your location in real time and detects deviations from your route or changes in congestion.

[0944] Input: Current location information (GPS data), congestion forecast data.

[0945] Output: Route deviation notification, congestion change notification.

[0946] Specific operation: The device periodically updates its current location using its GPS function and sends it to the server. The server monitors the current location in real time and sends a notification to the device if it detects deviations or changes in congestion.

[0947] Step 9:

[0948] When the user arrives at their destination, they are provided with information on information centers and support facilities.

[0949] Input: Destination arrival notification.

[0950] Output: Location information of information centers and support facilities around the destination.

[0951] Specific operation: When the device detects arrival at the destination, it automatically provides the user with the location information of information centers and support facilities. This is done by voice or text notification.

[0952] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0953] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0954] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0955] [Third embodiment]

[0956] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0957] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0958] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0959] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0960] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0961] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0962] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0963] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0964] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0965] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0966] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0967] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0968] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. Specific embodiments of the application will be described below.

[0969] System Configuration

[0970] This system consists of a user device, a central server, and a database. User devices include smartphones and tablets. The central server is a high-performance computer system for route calculations and data processing. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[0971] Enter user information

[0972] Users start the app, log in, and enter their current location and destination information. Current location information can also be obtained automatically using GPS. In addition, an interface is provided for specifying intermediate destinations and places to stop.

[0973] Route calculation and data collection

[0974] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[0975] Providing guidance information to users

[0976] The user's device receives the calculated route information and provides guidance in a format tailored to the visually and hearing impaired. The visually impaired receive audio guidance, while the hearing impaired receive text and vibration guidance. Detailed guidance is also provided on important points (e.g., elevator locations, arrow sign directions, etc.).

[0977] Route Tracking

[0978] As users move, the device tracks their location in real time and detects and notifies them of changes in congestion or deviations along the route, allowing them to quickly get back on track.

[0979] Post-arrival support

[0980] When the user arrives at their destination, the device will notify them of their arrival. It will also provide information about the locations of nearby information centers and support facilities. For example, if the user arrives at a hospital, they will receive detailed information about the location of the nearest information center and elevator.

[0981] Specific examples

[0982] Example 1: Visually impaired person A travels from the station to the library

[0983] 1. Enter user information

[0984] Person A launches the app and voice-inputs his current location (station) and destination (library).

[0985] Specify the station information desk as your stopover point.

[0986] 2. Route calculation and data collection

[0987] The server receives the current location and destination and calculates the optimal route.

[0988] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[0989] 3. Providing guidance information to users

[0990] The device provides guidance information to Mr. A via audio guide.

[0991] It includes detailed instructions such as "Turn left and the elevator is 50 meters ahead."

[0992] 4. Route Tracking

[0993] While Mr. A is traveling, the device tracks his current location in real time and notifies him if he strays from his route or if there is a change in congestion.

[0994] 5. Post-arrival support

[0995] When you arrive at the library, the device will announce, "You have arrived at the library. The entrance is on your right."

[0996] It also provides information about the library's information desk and barrier-free facilities.

[0997] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently.

[0998] The processing flow will be explained below.

[0999] Step 1:

[1000] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[1001] Step 2:

[1002] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[1003] Step 3:

[1004] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[1005] Step 4:

[1006] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[1007] Step 5:

[1008] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[1009] Step 6:

[1010] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[1011] Step 7:

[1012] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[1013] Step 8:

[1014] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[1015] Step 9:

[1016] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[1017] Step 10:

[1018] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[1019] Step 11:

[1020] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[1021] Example 1

[1022] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1023] There is a lack of systems that guide people with visual and hearing impairments to the optimal route from their current location to their destination. In particular, there are issues with the lack of real-time congestion information, route deviation notifications, and information on information centers and support facilities near the destination. There is also a need for more flexible guidance methods according to the type of disability, as well as an interface that allows users to specify intermediate stops and stopovers when planning a route.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1025] In this invention, the server includes means for receiving user location information and destination information, means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information, means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's information processing device, means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations that occur along the route, means for providing location information of information centers and support facilities around the destination after the user arrives, and a barrier-free map creation application designed for the visually and hearing impaired. This enables the visually and hearing impaired to travel safely and efficiently and to be provided with necessary guidance information in a timely manner.

[1026] "User location information" refers to geographical information about the user's current location, and is data obtained using technology such as GPS.

[1027] "Destination information" is geographical information about the final destination that the user is aiming for.

[1028] "Public transport timetable data" refers to data relating to the operation schedules of public transport such as buses and trains.

[1029] "Congestion prediction data" is data that predicts the congestion of people and vehicles at specific times and locations.

[1030] "Barrier-free route information" refers to information about routes that everyone, including those with visual or hearing impairments, can travel on safely and comfortably.

[1031] The "optimal route" is the most efficient travel route between the current location set by the user and the destination, taking into consideration time, types of obstacles, and the like.

[1032] A "calculated optimal route" is the best route to travel calculated using the user's current location and destination and related data.

[1033] "Real-time location tracking" means keeping track of the user's current location at all times.

[1034] "Route deviation detection" refers to detecting when a user deviates from a pre-planned route.

[1035] "Location information of information centers and support facilities" is geographical information about information centers and facilities that support users located in the vicinity of the destination.

[1036] A "barrier-free map creation application designed for the visually and hearing impaired" is map software designed to guide the visually and hearing impaired to navigate safely and efficiently.

[1037] MODE FOR CARRYING OUT THE INVENTION

[1038] The present invention is a barrier-free map creation application designed for the visually and hearing impaired, and specific embodiments thereof will be described below.

[1039] System Configuration

[1040] This system consists of user devices, a central server, and a database. User devices include mobile information terminals such as smartphones and tablets, through which the application runs. The central server is a high-performance computer system that calculates routes and processes data. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1041] Enter user information

[1042] Users must launch the app on their device and log in. The application provides an interface for users to input their current location and destination information. Current location information can also be obtained automatically using GPS. Users can also specify intermediate locations and places they want to stop at.

[1043] Route calculation and data collection

[1044] The server accesses a database to reference public transportation timetable data, congestion forecast data, and barrier-free route information based on the current location and destination information received from the user. Based on this data, the server calculates the optimal route for the user. The route calculation includes routes that are safe and accessible, especially for the visually and hearing impaired.

[1045] Providing guidance information to users

[1046] The calculated optimal route information is sent from the server to the user's device. Based on the received route information, the device provides guidance information in the form of voice guidance for the visually impaired and text display or vibration for the hearing impaired. For example, the device may provide a voice guide saying, "Turn left and there is an elevator 50 meters ahead."

[1047] Route Tracking

[1048] While the user is traveling along a route, the device tracks their location in real time. The device periodically checks its location using GPS and notifies the user if there is a deviation from the route or changes in traffic congestion, allowing the user to quickly get back on track.

[1049] Post-arrival support

[1050] When the user arrives at their destination, the device will notify them, "You have arrived at your destination." It will also provide detailed location information for information centers and support facilities around the destination. For example, if they arrive at a hospital, they will be told, "The elevator is on the right side of the building."

[1051] Specific examples

[1052] Example 1: Visually impaired person A travels from the station to the library

[1053] 1. The user launches the app and speaks, "I want to go to the library from the station."

[1054] 2. The server analyzes the received information and calculates the optimal route by referring to public transport timetable data, congestion forecast data, and barrier-free route information.

[1055] 3. The device will give voice guidance saying, "Turn left and get on the elevator 50 meters ahead."

[1056] 4. The device tracks your location in real time while you are moving, and if you stray from your route, it will notify you, "Go back 10 meters and turn right."

[1057] 5. When the device arrives at its destination, it will provide detailed information such as, "You have arrived at the library. The entrance is on the right. Additionally, the information desk is on the left."

[1058] Example prompts for generative AI models

[1059] I'd like to design an application that provides optimal routes and guidance for visually impaired people traveling from the station to the library. Please explain the specific process for this application: based on current location and destination information, this application calculates the optimal route by referencing public transport timetable data, congestion forecast data, and barrier-free route information, and then provides guidance using audio guides and text displays.

[1060] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and is a system that provides necessary guidance information in a timely manner, ensuring the safety and security of users.

[1061] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1062] Step 1: Enter your information

[1063] Users launch the app on their smartphone or tablet and log in.

[1064] The user specifies the destination information by voice input or text input, and the application can automatically obtain the current location information using the GPS function.

[1065] Input: User's current location information, destination information, (optional) intermediate locations and places to stop

[1066] Output: User input data (current location, destination information)

[1067] As a specific example of operation, when the user is at a station, he or she inputs "I want to go to the library" by voice.

[1068] Step 2: Route calculation and data collection

[1069] The server accesses a database based on the current location information and destination information received from the user, and obtains public transportation timetable data, congestion forecast data, and barrier-free route information.

[1070] The server uses the acquired data to calculate the optimal route using an algorithm that takes into account accessibility and congestion.

[1071] Input: User's current location information, destination information, public transport timetable data in the database, congestion forecast data, barrier-free route information

[1072] Output: Calculated optimal route information

[1073] As a specific example of how it works, the server "identifies the nearest station based on GPS coordinates and calculates the optimal route from that station, taking into account barrier-free information."

[1074] Step 3: Providing guidance to users

[1075] The server transmits the calculated optimum route information to the user's terminal.

[1076] Based on the route information received, the device provides voice guidance for visually impaired people and guidance information via text display and vibration for hearing impaired people.

[1077] Input: Calculated optimal route information

[1078] Output: Guidance information for the user (audio guide, text display, vibration guide)

[1079] As a specific example of operation, the user's device plays back a voice instruction saying, "Turn left and use the elevator 50 meters ahead."

[1080] Step 4: Path tracing

[1081] The device periodically acquires GPS data to track the user's location in real time as they move.

[1082] The device compares the acquired current location information with the calculated route and notifies the user if the user deviates from the route or if congestion occurs.

[1083] Input: Real-time GPS data, calculated route information

[1084] Output: Notifications about route deviations and congestion

[1085] As a specific example of how it works, the device will issue a voice notification saying, "You are currently off route. Go back 10 meters and turn right."

[1086] Step 5: Support after arrival

[1087] When the terminal detects the user's arrival, it notifies the user that "you have arrived at your destination."

[1088] The device also provides the user with detailed location information for information centers and support facilities around the destination.

[1089] Input: User's current location (destination arrival), support facility information in the database

[1090] Output: Arrival notification at destination, location information of support facilities

[1091] As a specific example of how it works, the device will say, "You have arrived at the library. The elevator is on the right side of the building."

[1092] In this way, the system is designed to enable the visually and hearing impaired to travel to their destination safely and efficiently, and is equipped with functions for real-time route tracking and detailed guidance around the destination.

[1093] (Application example 1)

[1094] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] When using public transportation or self-driving vehicles, the visually and hearing impaired face difficulties in reaching their destinations safely and efficiently due to insufficient route guidance and barrier-free information. It is also difficult to respond immediately to congestion information or route deviations while traveling. Therefore, there is a need for a comprehensive navigation system that enables the visually and hearing impaired to travel safely.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1097] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations along the route; means for providing location information of information centers and support facilities around the destination after the user arrives; means for navigating the route to the destination in cooperation with an autonomous vehicle; and means for providing real-time guidance information to the visually and hearing impaired using a smartphone or smart glasses, thereby enabling the visually and hearing impaired to reach their destination safely and efficiently via a barrier-free route.

[1098] "User location information" is data that indicates the geographical location of the user, obtained using a mobile device or GPS function.

[1099] "Destination information" is geographical location data relating to the location where the user ultimately wants to arrive.

[1100] "Public transport timetable data" refers to information about the operation schedules of public transport such as buses and trains.

[1101] "Congestion prediction data" is information that predicts the congestion situation of people along a specified route or at a destination.

[1102] "Barrier-free route information" refers to information about safe, barrier-free routes that are easy for wheelchair users and those with visual impairments to navigate.

[1103] The "optimal route" is the most suitable route for traveling from the user's location information to the destination information, taking into consideration time, convenience, and safety.

[1104] "Location tracking" refers to the process of continuously monitoring a user's current location to determine their location in real time.

[1105] "Route deviation" means that the user deviates from the calculated optimal route.

[1106] "Information centers and support facilities" are locations or service facilities that provide assistance and information during travel or upon arrival at a destination.

[1107] An "autonomous vehicle" is a vehicle that operates autonomously without the need for human operation.

[1108] A "smartphone" is a portable information terminal that has various functions in addition to telephone functions, such as internet connection.

[1109] "Smart glasses" are eyeglass-type devices with built-in computers that are capable of augmented reality and information display.

[1110] "Real-time guidance information" means up-to-date route and situation information provided immediately at the current time.

[1111] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. The system consists of a user terminal, a central server, and a database. Specific embodiments are described below.

[1112] System Configuration

[1113] Hardware Configuration

[1114] The system uses mobile devices such as smartphones and smart glasses, which have GPS capabilities to acquire user location information. Sensor modules in autonomous vehicles are also utilized, and the system connects to a central server to update route information in real time.

[1115] Software Configuration

[1116] The central server is a high-performance computer system that accesses the OpenStreetMap API and PostgreSQL database to calculate barrier-free routes based on location and destination information received from user devices. Voice guidance is provided using Google Text-to-Speech and AWS Polly.

[1117] Data processing and route calculation

[1118] The server receives the user's current location and destination information and calculates the optimal barrier-free route based on this information. The calculation uses public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is sent to the user's device, and route information is provided in real time.

[1119] User guidance and real-time tracking

[1120] Based on the route information received, the user's device provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired. It works in conjunction with the autonomous vehicle to navigate the route to the destination. If congestion changes or the user deviates from the route while traveling, the device will be notified immediately.

[1121] Post-arrival support

[1122] When the user arrives at their destination, the device will notify them of their arrival and provide location information for nearby information centers and support facilities, allowing the visually and hearing impaired to smoothly receive support at their destination.

[1123] Specific examples

[1124] Example 1: B, a visually impaired person, travels to the hospital using an autonomous vehicle

[1125] 1. Enter user information

[1126] Person B launches the smartphone app and voice-inputs his current location (home) and destination (hospital).

[1127] 2. Route calculation

[1128] The server calculates the optimal barrier-free route based on the current location and destination, and updates the route information for the autonomous vehicle.

[1129] 3. Real-time guidance

[1130] Directions are provided as audio guides for the visually impaired, for example: "Turn left in 100 metres."

[1131] 4. Real-time route tracking

[1132] It works in conjunction with autonomous vehicles to monitor changes in congestion and route deviations during travel in real time, and notifies drivers as necessary.

[1133] 5. Post-arrival support

[1134] When the user arrives at the hospital, the smartphone will notify them with a voice message saying, "You have arrived at the hospital. The entrance is on your right." It will also provide information about the hospital's information desk and barrier-free facilities.

[1135] Prompt Sentence Examples

[1136] "Development of an autonomous vehicle navigation system for the visually and hearing impaired. Provides real-time route guidance using the following information:

[1137] User's current location and destination information

[1138] Calculating barrier-free routes using the OpenStreetMap API

[1139] Voice guidance for the visually impaired using Google Text-to-Speech and Amazon Polly

[1140] Real-time route updates and notifications

[1141] These functions will be implemented in Python, and guidance information will be displayed on smartphones and smart glasses.

[1142] In this way, the present invention provides a system that assists the visually and hearing impaired to move around more safely and efficiently.

[1143] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1144] Step 1:

[1145] The user starts up the device (smartphone or smart glasses) and inputs information about their current location and destination. The device automatically acquires the location information using GPS and obtains the destination information through voice input or text input.

[1146] Input: User's current location information, destination information

[1147] Output: Current location information and destination information obtained

[1148] Specific behavior:

[1149] The user launches the app and speaks, "From home to hospital." The device uses GPS to obtain the current location, converts the speech into text, and obtains the destination information.

[1150] Step 2:

[1151] The device sends its current location and destination information to the server, which then accesses the OpenStreetMap API and a PostgreSQL database to calculate a barrier-free route.

[1152] Input: current location information, destination information

[1153] Output: Calculated optimal barrier-free route

[1154] Specific behavior:

[1155] To calculate the shortest route from the current location to the destination, the server refers to public transportation timetable data and congestion forecast data and selects a route that includes barrier-free facilities.

[1156] Step 3:

[1157] The server then sends the calculated optimal route information to the terminal, which receives the information and provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired.

[1158] Input: Calculated optimal route information

[1159] Output: Voice guide, text display, vibration guide

[1160] Specific behavior:

[1161] The device provides voice guidance such as "Turn left in 50 meters," and also uses text display and vibration functions to inform users of route information.

[1162] Step 4:

[1163] As the user moves, the device tracks their current location in real time, and the server monitors the route congestion and deviations in real time, recalculating the route and notifying the user if necessary.

[1164] Input: Real-time user location information, congestion prediction data

[1165] Output: Updated route information, recalculated optimal route

[1166] Specific behavior:

[1167] The server periodically checks the user's current location, and if there is a deviation from the route or a change in congestion, it calculates a new route and notifies the terminal.

[1168] Step 5:

[1169] When the user arrives at their destination, the device will notify them that they have arrived at the hospital and provide them with location information for information centers and barrier-free facilities near their destination.

[1170] Input: User's location indicating arrival

[1171] Output: Arrival notification, information on information desks and barrier-free facilities

[1172] Specific behavior:

[1173] The device uses GPS information to confirm that the user has arrived at their destination, and then provides a voice guide saying, "You have arrived at the hospital. The entrance is on your right." The screen also displays information about information centers and the nearest barrier-free facilities.

[1174] By following these steps and implementing the present invention, it becomes possible for visually and hearing impaired people to reach their destination safely and efficiently.

[1175] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1176] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described below.

[1177] System Configuration

[1178] The system consists of a user's device, a central server, a database, and an emotion engine. The devices include smartphones and tablets, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculations and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1179] Enter user information

[1180] After launching the app and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate stops and places to stop.

[1181] Route calculation and data collection

[1182] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[1183] Providing guidance information to users

[1184] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[1185] Emotion recognition by emotion engine

[1186] The emotion engine installed in the user's device detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response, etc.), for example, determining whether the user is stressed or relaxed.

[1187] Emotion-based route adjustment

[1188] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[1189] Route Tracking

[1190] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[1191] Post-arrival support

[1192] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[1193] Specific examples

[1194] Example 1: Visually impaired person B travels from the station to the shopping mall

[1195] 1. Enter user information

[1196] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[1197] Specify the station information desk as your stopover point.

[1198] 2. Route calculation and data collection

[1199] The server receives the current location and destination and calculates the optimal route.

[1200] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1201] 3. Providing guidance information to users

[1202] The device provides guidance information to Mr. B via audio guide.

[1203] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[1204] 4. Emotion Recognition by Emotion Engine

[1205] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[1206] 5. Emotion-based route adjustment

[1207] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[1208] 6. Tracking and Notifications

[1209] The device tracks Mr. B's current location in real time and transmits it to the server.

[1210] Provide appropriate notification of deviations from route or changes in congestion.

[1211] 7. Post-arrival support

[1212] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[1213] It also provides information about the nearest information center and barrier-free facilities.

[1214] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[1215] The processing flow will be explained below.

[1216] Step 1:

[1217] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[1218] Step 2:

[1219] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[1220] Step 3:

[1221] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[1222] Step 4:

[1223] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[1224] Step 5:

[1225] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[1226] Step 6:

[1227] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[1228] Step 7:

[1229] The emotion engine uses facial recognition and biometric information to detect the user's emotional state, for example, assessing the user's stress level.

[1230] Step 8:

[1231] If the stress level is high, the server recalculates the route, choosing a route that avoids congestion or is more relaxing, and sends this new route information to the device.

[1232] Step 9:

[1233] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[1234] Step 10:

[1235] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[1236] Step 11:

[1237] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[1238] Step 12:

[1239] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[1240] Step 13:

[1241] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[1242] Example 2

[1243] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1244] In order for the visually and hearing impaired to reach their destinations safely, it is necessary to provide appropriate guidance information, and furthermore, flexible route adjustments are required to respond to changes in route and congestion during travel. However, conventional barrier-free map creation applications do not take the user's emotional state into consideration, and are therefore unable to respond to sudden stress or anxiety that may occur during travel. In addition, they do not provide sufficient real-time notifications of changes in congestion or deviations along the route. This makes it difficult for the visually and hearing impaired to travel safely and comfortably.

[1245] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1246] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state using facial recognition and biometric information; means for recalculating the route based on the emotional state and providing adjusted route information; means for tracking the current location in real time and detecting and notifying changes in congestion or deviations that occur along the route; and means for providing location information of information centers and support facilities around the destination after arrival. This not only enables visually and hearing impaired people to travel safely and efficiently, but also reduces stress and anxiety during travel, supporting a more comfortable journey.

[1247] "User location information" is data indicating the user's current location.

[1248] "Destination information" is data indicating the location to which the user is heading.

[1249] "Public transportation timetable data" refers to data related to the operation schedules of public transportation such as buses and trains.

[1250] "Congestion forecast data" is data that predicts congestion on public transportation systems and routes.

[1251] "Barrier-free route information" refers to information about routes that are easy to use for people with visual and hearing impairments and people with physical disabilities.

[1252] An "optimal route" is the most efficient and safe travel route for a user's travel purpose.

[1253] "Emotional state" is information indicating the user's psychological state, such as stress level and relaxation level.

[1254] "Facial recognition" is a technology that analyzes the facial features of a user and identifies that face.

[1255] "Biometric information" refers to physiological data such as the user's heart rate and galvanic skin response.

[1256] "Route recalculation" refers to the process of correcting the initially calculated route and calculating a new optimal route.

[1257] "Tracking current location in real time" means continually updating the current location as the user moves.

[1258] A "route deviation" is a situation in which a user deviates from the planned route.

[1259] "Notification" refers to conveying information to the user by voice, text, vibration, etc.

[1260] "Post-arrival support" is additional assistance information provided to the user after they reach their destination.

[1261] An "information center" is a facility that provides information and support to users.

[1262] "Support facilities" are facilities equipped with barrier-free facilities to support users' movements and activities.

[1263] This invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments based on the user's emotional state. The system consists of a user's device, a central server, a database, and an emotion engine. The device can be a smartphone or tablet, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculation and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1264] Users launch the application using a device such as a smartphone or tablet and log in. They then provide their current location, which is automatically acquired using GPS, and their destination information, which they input via voice or text. This information is then sent from the device to a central server.

[1265] The server first uses the received location and destination information to refer to a database and calculate the optimal route. This involves using public transport timetable data, congestion forecast data, and barrier-free route information. Once the route is calculated, the results are sent to the device, which then provides the information to the user. Audio guidance is provided to the visually impaired, and text and vibration guidance is provided to the hearing impaired.

[1266] Next, the device's built-in emotion engine detects the user's emotional state. This is done by analyzing facial recognition using a camera and heart rate and skin galvanic response using biometric sensors. For example, if the emotion engine determines that the user is in a high stress state, that information is sent to the server, which then recalculates the route and suggests a new route that will allow the user to travel more relaxed.

[1267] The user's current location is tracked in real time by the device and sent to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[1268] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[1269] Specific examples

[1270] Example 1: Visually impaired person B travels from the station to the shopping mall

[1271] 1. Enter user information

[1272] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[1273] Specify the station information desk as your stopover point.

[1274] 2. Route calculation and data collection

[1275] The server receives the current location and destination and calculates the optimal route.

[1276] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1277] 3. Providing guidance information to users

[1278] The device provides guidance information to Mr. B via audio guide.

[1279] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[1280] 4. Emotion Recognition by Emotion Engine

[1281] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[1282] 5. Emotion-based route adjustment

[1283] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[1284] 6. Tracking and Notifications

[1285] The device tracks Mr. B's current location in real time and transmits it to the server.

[1286] Provide appropriate notification of deviations from route or changes in congestion.

[1287] 7. Post-arrival support

[1288] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[1289] It also provides information about the nearest information center and barrier-free facilities.

[1290] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[1291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] The user launches the app and logs in. The user inputs their current location and destination by voice or text. The device receives this information, displays it on the screen, and sends it to the server. The input is in the form of "current location is XX station, destination is △△ shopping mall." The output is sending the information to the server. In concrete terms, the user inputs their current location and destination by voice, and the device recognizes the content and converts it into text.

[1294] Step 2:

[1295] The device activates the GPS module and acquires the user's current location. The acquired current location information is sent from the device to the server. The input is GPS data, and the output is the transmission of current location information. Specifically, the GPS module measures the user's location information, analyzes the data, generates specific location coordinates, and sends them to the server.

[1296] Step 3:

[1297] The server refers to a database based on the location and destination information it receives and calculates the optimal route. Public transport timetable data, congestion forecast data, and barrier-free route information are obtained from the database. The inputs are location information, destination information, and database information, and the output is optimal route data. In concrete terms, the server searches the database using an SQL query and calculates the optimal route.

[1298] Step 4:

[1299] The server sends the calculated optimal route to the terminal. Based on the route information received by the terminal, voice guidance, text display, and vibration notification are performed. The input is the optimal route data, and the output is guidance information for the user. In concrete terms, the terminal uses a voice guidance engine (e.g., TTS) and text display function to provide guidance information to the user.

[1300] Step 5:

[1301] The emotion engine installed on the device uses the camera and biometric sensors to recognize the user's face and detect their biometric information. The detected emotion information is sent to the server. The input is a facial image and heart rate data, and the output is emotion information. Specifically, the camera captures the user's facial expression, the biometric sensors measure the heart rate, and these data are analyzed by the emotion analysis module.

[1302] Step 6:

[1303] The server analyzes the received emotional information and recalculates a new optimal route. Based on the emotional state, the server suggests a relaxing route for the user. The input is emotional information, and the output is adjusted route data. Specifically, the server searches the database again and calculates a new route that may reduce stress.

[1304] Step 7:

[1305] The device tracks the user's current location in real time and sends the information to the server. The server monitors the current location information and notifies the device if it detects deviation from the route or changes in congestion. The input is the current location data and the output is notification information. In concrete terms, the device updates the GPS data as needed, and the server analyzes the data to detect any anomalies.

[1306] Step 8:

[1307] When the device arrives at the destination, it notifies the user of the arrival. It also provides location information of information centers and support facilities around the destination. The input is current location data and arrival information, and the output is support information. Specifically, the device checks the current location information, and upon arrival, provides information about information centers and support facilities by voice or text.

[1308] (Application example 2)

[1309] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1310] It is extremely difficult for visually and hearing impaired people to travel safely and efficiently on public transportation or in crowded places. To address this issue, flexible responses that take into account the user's emotional state, as well as route guidance, are required. In particular, when visually and hearing impaired people move around a factory, real-time emotion analysis and route adjustment are required, but this has been difficult to achieve with conventional technology.

[1311] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referring to public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state and recalculating the route based on the emotional state; and means for analyzing the user's emotional state in real time using an emotion recognition engine. This enables visually and hearing-impaired people to move around factories safely and efficiently while receiving flexible support according to their emotional state.

[1312] "Location Information" means data that indicates a user's current location, including data obtained using GPS or other location measurement technologies.

[1313] "Destination information" is data on the target point the user wishes to reach. It is set by the user initially and serves as the basis for route calculation.

[1314] "Public transport timetable data" is data showing the operating schedules of each public transport system. Based on this data, the optimal timing for travel is calculated.

[1315] "Congestion forecast data" refers to data that predicts congestion on public transportation and travel routes. This includes predictions based on real-time and past data.

[1316] "Barrier-free route information" is data on routes that are safe for the visually and hearing impaired, including information on facilities such as elevators and ramps.

[1317] "Route calculation" is a process of calculating the optimal travel route based on the user's current location and destination information.

[1318] "Guidance information" refers to instructions and information necessary for a user to reach their destination, including audio guidance and text displays.

[1319] "Real-time tracking" is the process of continuously monitoring a user's location in real time, using GPS or other means.

[1320] A "route deviation" is when the user deviates from the calculated route, which triggers a recalculation and notification.

[1321] "Changes in congestion" refers to changes in travel routes and the congestion status of public transportation, and it is necessary to understand the situation in real time.

[1322] "Location information of information centers and support facilities" is data indicating the locations of information centers and registered support facilities that can be used by the user around the destination.

[1323] "Emotional state" refers to the user's current psychological state, including stress and relaxation.

[1324] An "emotion recognition engine" is software that uses cameras and sensors to analyze a user's emotional state from their facial expressions and biometric information.

[1325] A "robot guidance system" is a system that includes a robot for safely guiding visually and hearing impaired people within a factory or other designated area.

[1326] "Flexible response" refers to the process of adjusting the route and changing the guidance method as needed depending on the user's emotional state and travel situation.

[1327] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described in detail below.

[1328] System Configuration

[1329] This system consists of a user's device, a central server, a database, and an emotion engine. Devices include smartphones, tablets, and robots used in factories. The emotion engine is software that uses OpenCV and Python emotion recognition libraries to detect the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system such as Amazon Web Services (AWS) or Google Cloud Platform (GCP), and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1330] Enter user information

[1331] After launching the application and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate destinations and places to stop.

[1332] Route calculation and data collection

[1333] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[1334] Providing guidance information to users

[1335] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[1336] Emotion recognition by emotion engine

[1337] The emotion engine detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response). For example, it determines whether the user is stressed or relaxed. The emotion engine uses OpenCV and Python emotion recognition libraries.

[1338] Emotion-based route adjustment

[1339] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[1340] Route Tracking

[1341] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[1342] Post-arrival support

[1343] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[1344] Specific examples

[1345] Example 1: When a visually and hearing impaired person, B, moves around the factory

[1346] 1. Enter user information

[1347] Person B launches the app and voice-inputs his current location (factory entrance) and destination (designated work area).

[1348] Designate a rest room as a stopover location.

[1349] 2. Route calculation and data collection

[1350] The server receives the current location and destination and calculates the optimal route.

[1351] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1352] 3. Providing guidance information to users

[1353] The device provides voice guidance to Mr. B.

[1354] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[1355] 4. Emotion Recognition by Emotion Engine

[1356] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[1357] 5. Emotion-based route adjustment

[1358] The server takes stress levels into account and recalculates a more relaxing route.

[1359] New guidance information is sent to the device.

[1360] 6. Tracking and Notifications

[1361] The device tracks Mr. B's current location in real time and transmits it to the server.

[1362] Provide appropriate notification of deviations from route or changes in congestion.

[1363] 7. Post-arrival support

[1364] When he arrives at the work area, the terminal notifies Mr. B of his arrival.

[1365] It also provides information on the nearest information center and barrier-free facilities.

[1366] Example prompt sentence:

[1367] If employees are feeling anxious, calculate a relaxing route through the factory.

[1368] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1369] Step 1:

[1370] The user launches the application and logs in.

[1371] Input: User login information (ID, password).

[1372] Output: The user is logged in.

[1373] Specific operation: The terminal sends the login information to the server and is authenticated. After successful authentication, the main screen is displayed.

[1374] Step 2:

[1375] The user inputs current location information and destination information.

[1376] Input: Current location information (GPS), destination information (manual input or voice input)

[1377] Output: Current location and destination information data.

[1378] How it works: The device uses GPS to obtain current location information, and the user inputs destination information manually or by voice. An interface is also available for specifying additional intermediate destinations and stops.

[1379] Step 3:

[1380] The server calculates the optimal route based on the current location information and destination information received.

[1381] Input: current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[1382] Output: Optimal route information.

[1383] Specific operation: The server refers to the database, executes the routing algorithm based on the necessary data, and calculates the optimal route. The calculated route is sent to the terminal.

[1384] Step 4:

[1385] The terminal presents the calculated route information to the user.

[1386] Input: Optimal route information.

[1387] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[1388] Specific operation: The device provides route guidance in the form of voice, text, vibration, etc. depending on the type of disability the user has.

[1389] Step 5:

[1390] An emotion engine detects the user's emotional state.

[1391] Input: User's facial recognition data, biometric information (heart rate, galvanic skin response).

[1392] Output: Emotional state (stress level and relaxation level).

[1393] Specific operation: The device's camera and sensors collect data on the user's face and physical state, which are then analyzed using an emotion recognition library (OpenCV, Python library). The analysis results are then sent to the server.

[1394] Step 6:

[1395] The server recalculates the route based on the emotional state.

[1396] Input: Emotional state, current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[1397] Output: New optimal route information based on emotional state.

[1398] Specific operation: The server takes into account the emotional state, re-references the database, recalculates relaxing routes and routes during quiet times, and sends the results to the terminal.

[1399] Step 7:

[1400] The terminal then presents the recalculated new route information to the user.

[1401] Input: New optimal route information.

[1402] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[1403] Specific operation: The terminal receives the recalculated route information from the database and provides the user with new directions in an appropriate format.

[1404] Step 8:

[1405] The device tracks your location in real time and detects deviations from your route or changes in congestion.

[1406] Input: Current location information (GPS data), congestion forecast data.

[1407] Output: Route deviation notification, congestion change notification.

[1408] Specific operation: The device periodically updates its current location using its GPS function and sends it to the server. The server monitors the current location in real time and sends a notification to the device if it detects deviations or changes in congestion.

[1409] Step 9:

[1410] When the user arrives at their destination, they are provided with information on information centers and support facilities.

[1411] Input: Destination arrival notification.

[1412] Output: Location information of information centers and support facilities around the destination.

[1413] Specific operation: When the device detects arrival at the destination, it automatically provides the user with the location information of information centers and support facilities. This is done by voice or text notification.

[1414] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1415] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1416] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1417] [Fourth embodiment]

[1418] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1419] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1420] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1421] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1422] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1423] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1424] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1425] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1426] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1429] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1430] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. Specific embodiments of the application will be described below.

[1432] System Configuration

[1433] This system consists of a user device, a central server, and a database. User devices include smartphones and tablets. The central server is a high-performance computer system for route calculations and data processing. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1434] Enter user information

[1435] Users start the app, log in, and enter their current location and destination information. Current location information can also be obtained automatically using GPS. In addition, an interface is provided for specifying intermediate destinations and places to stop.

[1436] Route calculation and data collection

[1437] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[1438] Providing guidance information to users

[1439] The user's device receives the calculated route information and provides guidance in a format tailored to the visually and hearing impaired. The visually impaired receive audio guidance, while the hearing impaired receive text and vibration guidance. Detailed guidance is also provided on important points (e.g., elevator locations, arrow sign directions, etc.).

[1440] Route Tracking

[1441] As users move, the device tracks their location in real time and detects and notifies them of changes in congestion or deviations along the route, allowing them to quickly get back on track.

[1442] Post-arrival support

[1443] When the user arrives at their destination, the device will notify them of their arrival. It will also provide information about the locations of nearby information centers and support facilities. For example, if the user arrives at a hospital, they will receive detailed information about the location of the nearest information center and elevator.

[1444] Specific examples

[1445] Example 1: Visually impaired person A travels from the station to the library

[1446] 1. Enter user information

[1447] Person A launches the app and voice-inputs his current location (station) and destination (library).

[1448] Specify the station information desk as your stopover point.

[1449] 2. Route calculation and data collection

[1450] The server receives the current location and destination and calculates the optimal route.

[1451] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1452] 3. Providing guidance information to users

[1453] The device provides guidance information to Mr. A via audio guide.

[1454] It includes detailed instructions such as "Turn left and the elevator is 50 meters ahead."

[1455] 4. Route Tracking

[1456] While Mr. A is traveling, the device tracks his current location in real time and notifies him if he strays from his route or if there is a change in congestion.

[1457] 5. Post-arrival support

[1458] When you arrive at the library, the device will announce, "You have arrived at the library. The entrance is on your right."

[1459] It also provides information about the library's information desk and barrier-free facilities.

[1460] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently.

[1461] The processing flow will be explained below.

[1462] Step 1:

[1463] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[1464] Step 2:

[1465] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[1466] Step 3:

[1467] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[1468] Step 4:

[1469] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[1470] Step 5:

[1471] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[1472] Step 6:

[1473] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[1474] Step 7:

[1475] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[1476] Step 8:

[1477] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[1478] Step 9:

[1479] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[1480] Step 10:

[1481] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[1482] Step 11:

[1483] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[1484] Example 1

[1485] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] There is a lack of systems that guide people with visual and hearing impairments to the optimal route from their current location to their destination. In particular, there are issues with the lack of real-time congestion information, route deviation notifications, and information on information centers and support facilities near the destination. There is also a need for more flexible guidance methods according to the type of disability, as well as an interface that allows users to specify intermediate stops and stopovers when planning a route.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1488] In this invention, the server includes means for receiving user location information and destination information, means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information, means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's information processing device, means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations that occur along the route, means for providing location information of information centers and support facilities around the destination after the user arrives, and a barrier-free map creation application designed for the visually and hearing impaired. This enables the visually and hearing impaired to travel safely and efficiently and to be provided with necessary guidance information in a timely manner.

[1489] "User location information" refers to geographical information about the user's current location, and is data obtained using technology such as GPS.

[1490] "Destination information" is geographical information about the final destination that the user is aiming for.

[1491] "Public transport timetable data" refers to data relating to the operation schedules of public transport such as buses and trains.

[1492] "Congestion prediction data" is data that predicts the congestion of people and vehicles at specific times and locations.

[1493] "Barrier-free route information" refers to information about routes that everyone, including those with visual or hearing impairments, can travel on safely and comfortably.

[1494] The "optimal route" is the most efficient travel route between the current location set by the user and the destination, taking into consideration time, types of obstacles, and the like.

[1495] A "calculated optimal route" is the best route to travel calculated using the user's current location and destination and related data.

[1496] "Real-time location tracking" means keeping track of the user's current location at all times.

[1497] "Route deviation detection" refers to detecting when a user deviates from a pre-planned route.

[1498] "Location information of information centers and support facilities" is geographical information about information centers and facilities that support users located in the vicinity of the destination.

[1499] A "barrier-free map creation application designed for the visually and hearing impaired" is map software designed to guide the visually and hearing impaired to navigate safely and efficiently.

[1500] MODE FOR CARRYING OUT THE INVENTION

[1501] The present invention is a barrier-free map creation application designed for the visually and hearing impaired, and specific embodiments thereof will be described below.

[1502] System Configuration

[1503] This system consists of user devices, a central server, and a database. User devices include mobile information terminals such as smartphones and tablets, through which the application runs. The central server is a high-performance computer system that calculates routes and processes data. The database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1504] Enter user information

[1505] Users must launch the app on their device and log in. The application provides an interface for users to input their current location and destination information. Current location information can also be obtained automatically using GPS. Users can also specify intermediate locations and places they want to stop at.

[1506] Route calculation and data collection

[1507] The server accesses a database to reference public transportation timetable data, congestion forecast data, and barrier-free route information based on the current location and destination information received from the user. Based on this data, the server calculates the optimal route for the user. The route calculation includes routes that are safe and accessible, especially for the visually and hearing impaired.

[1508] Providing guidance information to users

[1509] The calculated optimal route information is sent from the server to the user's device. Based on the received route information, the device provides guidance information in the form of voice guidance for the visually impaired and text display or vibration for the hearing impaired. For example, the device may provide a voice guide saying, "Turn left and there is an elevator 50 meters ahead."

[1510] Route Tracking

[1511] While the user is traveling along a route, the device tracks their location in real time. The device periodically checks its location using GPS and notifies the user if there is a deviation from the route or changes in traffic congestion, allowing the user to quickly get back on track.

[1512] Post-arrival support

[1513] When the user arrives at their destination, the device will notify them, "You have arrived at your destination." It will also provide detailed location information for information centers and support facilities around the destination. For example, if they arrive at a hospital, they will be told, "The elevator is on the right side of the building."

[1514] Specific examples

[1515] Example 1: Visually impaired person A travels from the station to the library

[1516] 1. The user launches the app and speaks, "I want to go to the library from the station."

[1517] 2. The server analyzes the received information and calculates the optimal route by referring to public transport timetable data, congestion forecast data, and barrier-free route information.

[1518] 3. The device will give voice guidance saying, "Turn left and get on the elevator 50 meters ahead."

[1519] 4. The device tracks your location in real time while you are moving, and if you stray from your route, it will notify you, "Go back 10 meters and turn right."

[1520] 5. When the device arrives at its destination, it will provide detailed information such as, "You have arrived at the library. The entrance is on the right. Additionally, the information desk is on the left."

[1521] Example prompts for generative AI models

[1522] I'd like to design an application that provides optimal routes and guidance for visually impaired people traveling from the station to the library. Please explain the specific process for this application: based on current location and destination information, this application calculates the optimal route by referencing public transport timetable data, congestion forecast data, and barrier-free route information, and then provides guidance using audio guides and text displays.

[1523] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and is a system that provides necessary guidance information in a timely manner, ensuring the safety and security of users.

[1524] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1525] Step 1: Enter your information

[1526] Users launch the app on their smartphone or tablet and log in.

[1527] The user specifies the destination information by voice input or text input, and the application can automatically obtain the current location information using the GPS function.

[1528] Input: User's current location information, destination information, (optional) intermediate locations and places to stop

[1529] Output: User input data (current location, destination information)

[1530] As a specific example of operation, when the user is at a station, he or she inputs "I want to go to the library" by voice.

[1531] Step 2: Route calculation and data collection

[1532] The server accesses a database based on the current location information and destination information received from the user, and obtains public transportation timetable data, congestion forecast data, and barrier-free route information.

[1533] The server uses the acquired data to calculate the optimal route using an algorithm that takes into account accessibility and congestion.

[1534] Input: User's current location information, destination information, public transport timetable data in the database, congestion forecast data, barrier-free route information

[1535] Output: Calculated optimal route information

[1536] As a specific example of how it works, the server "identifies the nearest station based on GPS coordinates and calculates the optimal route from that station, taking into account barrier-free information."

[1537] Step 3: Providing guidance to users

[1538] The server transmits the calculated optimum route information to the user's terminal.

[1539] Based on the route information received, the device provides voice guidance for visually impaired people and guidance information via text display and vibration for hearing impaired people.

[1540] Input: Calculated optimal route information

[1541] Output: Guidance information for the user (audio guide, text display, vibration guide)

[1542] As a specific example of operation, the user's device plays back a voice instruction saying, "Turn left and use the elevator 50 meters ahead."

[1543] Step 4: Path tracing

[1544] The device periodically acquires GPS data to track the user's location in real time as they move.

[1545] The device compares the acquired current location information with the calculated route and notifies the user if the user deviates from the route or if congestion occurs.

[1546] Input: Real-time GPS data, calculated route information

[1547] Output: Notifications about route deviations and congestion

[1548] As a specific example of how it works, the device will issue a voice notification saying, "You are currently off route. Go back 10 meters and turn right."

[1549] Step 5: Support after arrival

[1550] When the terminal detects the user's arrival, it notifies the user that "you have arrived at your destination."

[1551] The device also provides the user with detailed location information for information centers and support facilities around the destination.

[1552] Input: User's current location (destination arrival), support facility information in the database

[1553] Output: Arrival notification at destination, location information of support facilities

[1554] As a specific example of how it works, the device will say, "You have arrived at the library. The elevator is on the right side of the building."

[1555] In this way, the system is designed to enable the visually and hearing impaired to travel to their destination safely and efficiently, and is equipped with functions for real-time route tracking and detailed guidance around the destination.

[1556] (Application example 1)

[1557] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1558] When using public transportation or self-driving vehicles, the visually and hearing impaired face difficulties in reaching their destinations safely and efficiently due to insufficient route guidance and barrier-free information. It is also difficult to respond immediately to congestion information or route deviations while traveling. Therefore, there is a need for a comprehensive navigation system that enables the visually and hearing impaired to travel safely.

[1559] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1560] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for tracking the user's current location in real time as the user moves and detecting and notifying changes in congestion or deviations along the route; means for providing location information of information centers and support facilities around the destination after the user arrives; means for navigating the route to the destination in cooperation with an autonomous vehicle; and means for providing real-time guidance information to the visually and hearing impaired using a smartphone or smart glasses, thereby enabling the visually and hearing impaired to reach their destination safely and efficiently via a barrier-free route.

[1561] "User location information" is data that indicates the geographical location of the user, obtained using a mobile device or GPS function.

[1562] "Destination information" is geographical location data relating to the location where the user ultimately wants to arrive.

[1563] "Public transport timetable data" refers to information about the operation schedules of public transport such as buses and trains.

[1564] "Congestion prediction data" is information that predicts the congestion situation of people along a specified route or at a destination.

[1565] "Barrier-free route information" refers to information about safe, barrier-free routes that are easy for wheelchair users and those with visual impairments to navigate.

[1566] The "optimal route" is the most suitable route for traveling from the user's location information to the destination information, taking into consideration time, convenience, and safety.

[1567] "Location tracking" refers to the process of continuously monitoring a user's current location to determine their location in real time.

[1568] "Route deviation" means that the user deviates from the calculated optimal route.

[1569] "Information centers and support facilities" are locations or service facilities that provide assistance and information during travel or upon arrival at a destination.

[1570] An "autonomous vehicle" is a vehicle that operates autonomously without the need for human operation.

[1571] A "smartphone" is a portable information terminal that has various functions in addition to telephone functions, such as internet connection.

[1572] "Smart glasses" are eyeglass-type devices with built-in computers that are capable of augmented reality and information display.

[1573] "Real-time guidance information" means up-to-date route and situation information provided immediately at the current time.

[1574] The present invention is a barrier-free map creation application designed for the visually and hearing impaired. The system consists of a user terminal, a central server, and a database. Specific embodiments are described below.

[1575] System Configuration

[1576] Hardware Configuration

[1577] The system uses mobile devices such as smartphones and smart glasses, which have GPS capabilities to acquire user location information. Sensor modules in autonomous vehicles are also utilized, and the system connects to a central server to update route information in real time.

[1578] Software Configuration

[1579] The central server is a high-performance computer system that accesses the OpenStreetMap API and PostgreSQL database to calculate barrier-free routes based on location and destination information received from user devices. Voice guidance is provided using Google Text-to-Speech and AWS Polly.

[1580] Data processing and route calculation

[1581] The server receives the user's current location and destination information and calculates the optimal barrier-free route based on this information. The calculation uses public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is sent to the user's device, and route information is provided in real time.

[1582] User guidance and real-time tracking

[1583] Based on the route information received, the user's device provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired. It works in conjunction with the autonomous vehicle to navigate the route to the destination. If congestion changes or the user deviates from the route while traveling, the device will be notified immediately.

[1584] Post-arrival support

[1585] When the user arrives at their destination, the device will notify them of their arrival and provide location information for nearby information centers and support facilities, allowing the visually and hearing impaired to smoothly receive support at their destination.

[1586] Specific examples

[1587] Example 1: B, a visually impaired person, travels to the hospital using an autonomous vehicle

[1588] 1. Enter user information

[1589] Person B launches the smartphone app and voice-inputs his current location (home) and destination (hospital).

[1590] 2. Route calculation

[1591] The server calculates the optimal barrier-free route based on the current location and destination, and updates the route information for the autonomous vehicle.

[1592] 3. Real-time guidance

[1593] Directions are provided as audio guides for the visually impaired, for example: "Turn left in 100 metres."

[1594] 4. Real-time route tracking

[1595] It works in conjunction with autonomous vehicles to monitor changes in congestion and route deviations during travel in real time, and notifies drivers as necessary.

[1596] 5. Post-arrival support

[1597] When the user arrives at the hospital, the smartphone will notify them with a voice message saying, "You have arrived at the hospital. The entrance is on your right." It will also provide information about the hospital's information desk and barrier-free facilities.

[1598] Prompt Sentence Examples

[1599] "Development of an autonomous vehicle navigation system for the visually and hearing impaired. Provides real-time route guidance using the following information:

[1600] User's current location and destination information

[1601] Calculating barrier-free routes using the OpenStreetMap API

[1602] Voice guidance for the visually impaired using Google Text-to-Speech and Amazon Polly

[1603] Real-time route updates and notifications

[1604] These functions will be implemented in Python, and guidance information will be displayed on smartphones and smart glasses.

[1605] In this way, the present invention provides a system that assists the visually and hearing impaired to move around more safely and efficiently.

[1606] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1607] Step 1:

[1608] The user starts up the device (smartphone or smart glasses) and inputs information about their current location and destination. The device automatically acquires the location information using GPS and obtains the destination information through voice input or text input.

[1609] Input: User's current location information, destination information

[1610] Output: Current location information and destination information obtained

[1611] Specific behavior:

[1612] The user launches the app and speaks, "From home to hospital." The device uses GPS to obtain the current location, converts the speech into text, and obtains the destination information.

[1613] Step 2:

[1614] The device sends its current location and destination information to the server, which then accesses the OpenStreetMap API and a PostgreSQL database to calculate a barrier-free route.

[1615] Input: current location information, destination information

[1616] Output: Calculated optimal barrier-free route

[1617] Specific behavior:

[1618] To calculate the shortest route from the current location to the destination, the server refers to public transportation timetable data and congestion forecast data and selects a route that includes barrier-free facilities.

[1619] Step 3:

[1620] The server then sends the calculated optimal route information to the terminal, which receives the information and provides audio guidance for the visually impaired and text and vibration guidance for the hearing impaired.

[1621] Input: Calculated optimal route information

[1622] Output: Voice guide, text display, vibration guide

[1623] Specific behavior:

[1624] The device provides voice guidance such as "Turn left in 50 meters," and also uses text display and vibration functions to inform users of route information.

[1625] Step 4:

[1626] As the user moves, the device tracks their current location in real time, and the server monitors the route congestion and deviations in real time, recalculating the route and notifying the user if necessary.

[1627] Input: Real-time user location information, congestion prediction data

[1628] Output: Updated route information, recalculated optimal route

[1629] Specific behavior:

[1630] The server periodically checks the user's current location, and if there is a deviation from the route or a change in congestion, it calculates a new route and notifies the terminal.

[1631] Step 5:

[1632] When the user arrives at their destination, the device will notify them that they have arrived at the hospital and provide them with location information for information centers and barrier-free facilities near their destination.

[1633] Input: User's location indicating arrival

[1634] Output: Arrival notification, information on information desks and barrier-free facilities

[1635] Specific behavior:

[1636] The device uses GPS information to confirm that the user has arrived at their destination, and then provides a voice guide saying, "You have arrived at the hospital. The entrance is on your right." The screen also displays information about information centers and the nearest barrier-free facilities.

[1637] By following these steps and implementing the present invention, it becomes possible for visually and hearing impaired people to reach their destination safely and efficiently.

[1638] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1639] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described below.

[1640] System Configuration

[1641] The system consists of a user's device, a central server, a database, and an emotion engine. The devices include smartphones and tablets, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculations and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1642] Enter user information

[1643] After launching the app and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate stops and places to stop.

[1644] Route calculation and data collection

[1645] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[1646] Providing guidance information to users

[1647] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[1648] Emotion recognition by emotion engine

[1649] The emotion engine installed in the user's device detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response, etc.), for example, determining whether the user is stressed or relaxed.

[1650] Emotion-based route adjustment

[1651] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[1652] Route Tracking

[1653] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[1654] Post-arrival support

[1655] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[1656] Specific examples

[1657] Example 1: Visually impaired person B travels from the station to the shopping mall

[1658] 1. Enter user information

[1659] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[1660] Specify the station information desk as your stopover point.

[1661] 2. Route calculation and data collection

[1662] The server receives the current location and destination and calculates the optimal route.

[1663] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1664] 3. Providing guidance information to users

[1665] The device provides guidance information to Mr. B via audio guide.

[1666] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[1667] 4. Emotion Recognition by Emotion Engine

[1668] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[1669] 5. Emotion-based route adjustment

[1670] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[1671] 6. Tracking and Notifications

[1672] The device tracks Mr. B's current location in real time and transmits it to the server.

[1673] Provide appropriate notification of deviations from route or changes in congestion.

[1674] 7. Post-arrival support

[1675] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[1676] It also provides information about the nearest information center and barrier-free facilities.

[1677] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[1678] The processing flow will be explained below.

[1679] Step 1:

[1680] The user launches the app and logs in. The user's current location information is automatically acquired or manually entered, and the destination is entered. If necessary, the user can specify intermediate destinations or places they wish to stop at.

[1681] Step 2:

[1682] The device sends the entered current location information and destination information to the server, along with information about intermediate stops and places you want to stop at.

[1683] Step 3:

[1684] The server retrieves public transport timetable data from a database based on the received current location and destination information, as well as congestion forecast data and barrier-free route information.

[1685] Step 4:

[1686] The server analyzes the acquired data and calculates the optimal route for the user, taking into account time, distance, availability of barrier-free facilities, and traffic congestion.

[1687] Step 5:

[1688] The server transmits the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal.

[1689] Step 6:

[1690] The terminal receives the calculated route information and provides guidance information to the user in the form of voice guidance for the visually impaired and text display or vibration guidance for the hearing impaired.

[1691] Step 7:

[1692] The emotion engine uses facial recognition and biometric information to detect the user's emotional state, for example, assessing the user's stress level.

[1693] Step 8:

[1694] If the stress level is high, the server recalculates the route, choosing a route that avoids congestion or is more relaxing, and sends this new route information to the device.

[1695] Step 9:

[1696] The user begins to follow the route, and the device tracks the user's current location in real time and transmits it to the server.

[1697] Step 10:

[1698] The server monitors the tracked location information in real time and notifies the device if there is a deviation from the route or a change in congestion.

[1699] Step 11:

[1700] If there is a deviation from the route or a change in congestion, the device will notify the user so that they can adjust their guidance, providing new instructions via voice guidance, text display, or vibration.

[1701] Step 12:

[1702] When the user arrives at their destination, the device will confirm their arrival and notify them, and also provide them with location information for information centers and support facilities around the destination.

[1703] Step 13:

[1704] If the user is inside a building, the device will also display information about barrier-free routes and facilities within the building to support movement.

[1705] Example 2

[1706] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1707] In order for the visually and hearing impaired to reach their destinations safely, it is necessary to provide appropriate guidance information, and furthermore, flexible route adjustments are required to respond to changes in route and congestion during travel. However, conventional barrier-free map creation applications do not take the user's emotional state into consideration, and are therefore unable to respond to sudden stress or anxiety that may occur during travel. In addition, they do not provide sufficient real-time notifications of changes in congestion or deviations along the route. This makes it difficult for the visually and hearing impaired to travel safely and comfortably.

[1708] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1709] In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route by referencing public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state using facial recognition and biometric information; means for recalculating the route based on the emotional state and providing adjusted route information; means for tracking the current location in real time and detecting and notifying changes in congestion or deviations that occur along the route; and means for providing location information of information centers and support facilities around the destination after arrival. This not only enables visually and hearing impaired people to travel safely and efficiently, but also reduces stress and anxiety during travel, supporting a more comfortable journey.

[1710] "User location information" is data indicating the user's current location.

[1711] "Destination information" is data indicating the location to which the user is heading.

[1712] "Public transportation timetable data" refers to data related to the operation schedules of public transportation such as buses and trains.

[1713] "Congestion forecast data" is data that predicts congestion on public transportation systems and routes.

[1714] "Barrier-free route information" refers to information about routes that are easy to use for people with visual and hearing impairments and people with physical disabilities.

[1715] An "optimal route" is the most efficient and safe travel route for a user's travel purpose.

[1716] "Emotional state" is information indicating the user's psychological state, such as stress level and relaxation level.

[1717] "Facial recognition" is a technology that analyzes the facial features of a user and identifies that face.

[1718] "Biometric information" refers to physiological data such as the user's heart rate and galvanic skin response.

[1719] "Route recalculation" refers to the process of correcting the initially calculated route and calculating a new optimal route.

[1720] "Tracking current location in real time" means continually updating the current location as the user moves.

[1721] A "route deviation" is a situation in which a user deviates from the planned route.

[1722] "Notification" refers to conveying information to the user by voice, text, vibration, etc.

[1723] "Post-arrival support" is additional assistance information provided to the user after they reach their destination.

[1724] An "information center" is a facility that provides information and support to users.

[1725] "Support facilities" are facilities equipped with barrier-free facilities to support users' movements and activities.

[1726] This invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments based on the user's emotional state. The system consists of a user's device, a central server, a database, and an emotion engine. The device can be a smartphone or tablet, and the emotion engine is software that detects the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system for route calculation and data processing, and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1727] Users launch the application using a device such as a smartphone or tablet and log in. They then provide their current location, which is automatically acquired using GPS, and their destination information, which they input via voice or text. This information is then sent from the device to a central server.

[1728] The server first uses the received location and destination information to refer to a database and calculate the optimal route. This involves using public transport timetable data, congestion forecast data, and barrier-free route information. Once the route is calculated, the results are sent to the device, which then provides the information to the user. Audio guidance is provided to the visually impaired, and text and vibration guidance is provided to the hearing impaired.

[1729] Next, the device's built-in emotion engine detects the user's emotional state. This is done by analyzing facial recognition using a camera and heart rate and skin galvanic response using biometric sensors. For example, if the emotion engine determines that the user is in a high stress state, that information is sent to the server, which then recalculates the route and suggests a new route that will allow the user to travel more relaxed.

[1730] The user's current location is tracked in real time by the device and sent to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[1731] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[1732] Specific examples

[1733] Example 1: Visually impaired person B travels from the station to the shopping mall

[1734] 1. Enter user information

[1735] Person B launches the app and voice-inputs his current location (station) and destination (shopping mall).

[1736] Specify the station information desk as your stopover point.

[1737] 2. Route calculation and data collection

[1738] The server receives the current location and destination and calculates the optimal route.

[1739] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1740] 3. Providing guidance information to users

[1741] The device provides guidance information to Mr. B via audio guide.

[1742] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[1743] 4. Emotion Recognition by Emotion Engine

[1744] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[1745] 5. Emotion-based route adjustment

[1746] The server takes stress levels into account, recalculates a more relaxing route, and sends new guidance information to the device.

[1747] 6. Tracking and Notifications

[1748] The device tracks Mr. B's current location in real time and transmits it to the server.

[1749] Provide appropriate notification of deviations from route or changes in congestion.

[1750] 7. Post-arrival support

[1751] When he arrives at the shopping mall, the device notifies Mr. B of his arrival.

[1752] It also provides information about the nearest information center and barrier-free facilities.

[1753] In this way, the barrier-free map creation application of the present invention helps the visually and hearing impaired to move around safely and efficiently, and by combining it with an emotion engine, it provides support that is tailored to the user's emotional state.

[1754] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1755] Step 1:

[1756] The user launches the app and logs in. The user inputs their current location and destination by voice or text. The device receives this information, displays it on the screen, and sends it to the server. The input is in the form of "current location is XX station, destination is △△ shopping mall." The output is sending the information to the server. In concrete terms, the user inputs their current location and destination by voice, and the device recognizes the content and converts it into text.

[1757] Step 2:

[1758] The device activates the GPS module and acquires the user's current location. The acquired current location information is sent from the device to the server. The input is GPS data, and the output is the transmission of current location information. Specifically, the GPS module measures the user's location information, analyzes the data, generates specific location coordinates, and sends them to the server.

[1759] Step 3:

[1760] The server refers to a database based on the location and destination information it receives and calculates the optimal route. Public transport timetable data, congestion forecast data, and barrier-free route information are obtained from the database. The inputs are location information, destination information, and database information, and the output is optimal route data. In concrete terms, the server searches the database using an SQL query and calculates the optimal route.

[1761] Step 4:

[1762] The server sends the calculated optimal route to the terminal. Based on the route information received by the terminal, voice guidance, text display, and vibration notification are performed. The input is the optimal route data, and the output is guidance information for the user. In concrete terms, the terminal uses a voice guidance engine (e.g., TTS) and text display function to provide guidance information to the user.

[1763] Step 5:

[1764] The emotion engine installed on the device uses the camera and biometric sensors to recognize the user's face and detect their biometric information. The detected emotion information is sent to the server. The input is a facial image and heart rate data, and the output is emotion information. Specifically, the camera captures the user's facial expression, the biometric sensors measure the heart rate, and these data are analyzed by the emotion analysis module.

[1765] Step 6:

[1766] The server analyzes the received emotional information and recalculates a new optimal route. Based on the emotional state, the server suggests a relaxing route for the user. The input is emotional information, and the output is adjusted route data. Specifically, the server searches the database again and calculates a new route that may reduce stress.

[1767] Step 7:

[1768] The device tracks the user's current location in real time and sends the information to the server. The server monitors the current location information and notifies the device if it detects deviation from the route or changes in congestion. The input is the current location data and the output is notification information. In concrete terms, the device updates the GPS data as needed, and the server analyzes the data to detect any anomalies.

[1769] Step 8:

[1770] When the device arrives at the destination, it notifies the user of the arrival. It also provides location information of information centers and support facilities around the destination. The input is current location data and arrival information, and the output is support information. Specifically, the device checks the current location information, and upon arrival, provides information about information centers and support facilities by voice or text.

[1771] (Application example 2)

[1772] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1773] It is extremely difficult for visually and hearing impaired people to travel safely and efficiently on public transportation or in crowded places. To address this issue, flexible responses that take into account the user's emotional state, as well as route guidance, are required. In particular, when visually and hearing impaired people move around a factory, real-time emotion analysis and route adjustment are required, but this has been difficult to achieve with conventional technology.

[1774] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving user location information and destination information; means for calculating an optimal route based on the received location information and destination information by referring to public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimal route and related congestion information and barrier-free facility information to the user's terminal; means for detecting the user's emotional state and recalculating the route based on the emotional state; and means for analyzing the user's emotional state in real time using an emotion recognition engine. This enables visually and hearing-impaired people to move around factories safely and efficiently while receiving flexible support according to their emotional state.

[1775] "Location Information" means data that indicates a user's current location, including data obtained using GPS or other location measurement technologies.

[1776] "Destination information" is data on the target point the user wishes to reach. It is set by the user initially and serves as the basis for route calculation.

[1777] "Public transport timetable data" is data showing the operating schedules of each public transport system. Based on this data, the optimal timing for travel is calculated.

[1778] "Congestion forecast data" refers to data that predicts congestion on public transportation and travel routes. This includes predictions based on real-time and past data.

[1779] "Barrier-free route information" is data on routes that are safe for the visually and hearing impaired, including information on facilities such as elevators and ramps.

[1780] "Route calculation" is a process of calculating the optimal travel route based on the user's current location and destination information.

[1781] "Guidance information" refers to instructions and information necessary for a user to reach their destination, including audio guidance and text displays.

[1782] "Real-time tracking" is the process of continuously monitoring a user's location in real time, using GPS or other means.

[1783] A "route deviation" is when the user deviates from the calculated route, which triggers a recalculation and notification.

[1784] "Changes in congestion" refers to changes in travel routes and the congestion status of public transportation, and it is necessary to understand the situation in real time.

[1785] "Location information of information centers and support facilities" is data indicating the locations of information centers and registered support facilities that can be used by the user around the destination.

[1786] "Emotional state" refers to the user's current psychological state, including stress and relaxation.

[1787] An "emotion recognition engine" is software that uses cameras and sensors to analyze a user's emotional state from their facial expressions and biometric information.

[1788] A "robot guidance system" is a system that includes a robot for safely guiding visually and hearing impaired people within a factory or other designated area.

[1789] "Flexible response" refers to the process of adjusting the route and changing the guidance method as needed depending on the user's emotional state and travel situation.

[1790] The present invention combines an emotion engine with a barrier-free map creation application designed for the visually and hearing impaired to provide appropriate guidance information and route adjustments according to the user's emotional state. Specific embodiments of the present invention are described in detail below.

[1791] System Configuration

[1792] This system consists of a user's device, a central server, a database, and an emotion engine. Devices include smartphones, tablets, and robots used in factories. The emotion engine is software that uses OpenCV and Python emotion recognition libraries to detect the user's emotional state based on facial recognition and biometric information. The central server is a high-performance computer system such as Amazon Web Services (AWS) or Google Cloud Platform (GCP), and the database stores public transportation timetable data, congestion forecast data, and barrier-free route information.

[1793] Enter user information

[1794] After launching the application and logging in, users enter their current location and destination information. Current location information can also be obtained automatically using GPS. An interface is also provided for specifying intermediate destinations and places to stop.

[1795] Route calculation and data collection

[1796] The server calculates the optimal route based on the received current location and destination information, referring to public transport timetable data, congestion forecast data, and barrier-free route information. The calculated optimal route is then sent to the user's device.

[1797] Providing guidance information to users

[1798] The device receives the calculated route information and provides voice guidance for the visually impaired, text display and vibration guidance for the hearing impaired, as well as detailed guidance of important points (e.g., elevator locations, arrow sign directions, etc.).

[1799] Emotion recognition by emotion engine

[1800] The emotion engine detects the user's emotional state based on facial recognition and biometric information (e.g., heart rate, galvanic skin response). For example, it determines whether the user is stressed or relaxed. The emotion engine uses OpenCV and Python emotion recognition libraries.

[1801] Emotion-based route adjustment

[1802] If the emotion engine detects the user's stress level, the server will recalculate the route and adjust it to a more relaxing route for the user or a less crowded time. The emotion engine can also automatically switch to escort mode based on the user's emotional state, allowing the user to travel with peace of mind.

[1803] Route Tracking

[1804] As the user moves, the device tracks their current location in real time and sends it to the server. The server monitors the real-time location information and notifies the device if there is a deviation from the route or a change in congestion.

[1805] Post-arrival support

[1806] Upon arrival at the destination, the device will confirm the arrival and notify the user. It will also provide the location of information centers and support facilities around the destination. For example, if the user arrives at a hospital, detailed directions will be provided to the nearest information center and elevator.

[1807] Specific examples

[1808] Example 1: When a visually and hearing impaired person, B, moves around the factory

[1809] 1. Enter user information

[1810] Person B launches the app and voice-inputs his current location (factory entrance) and destination (designated work area).

[1811] Designate a rest room as a stopover location.

[1812] 2. Route calculation and data collection

[1813] The server receives the current location and destination and calculates the optimal route.

[1814] Access public transport timetable data, barrier-free route information, and congestion forecast data.

[1815] 3. Providing guidance information to users

[1816] The device provides voice guidance to Mr. B.

[1817] Provide detailed directions such as "Turn right and you will find the elevator 50 meters ahead."

[1818] 4. Emotion Recognition by Emotion Engine

[1819] While traveling, the emotion engine detects Mr. B's stress level and sends it to the server.

[1820] 5. Emotion-based route adjustment

[1821] The server takes stress levels into account and recalculates a more relaxing route.

[1822] New guidance information is sent to the device.

[1823] 6. Tracking and Notifications

[1824] The device tracks Mr. B's current location in real time and transmits it to the server.

[1825] Provide appropriate notification of deviations from route or changes in congestion.

[1826] 7. Post-arrival support

[1827] When he arrives at the work area, the terminal notifies Mr. B of his arrival.

[1828] It also provides information on the nearest information center and barrier-free facilities.

[1829] Example prompt sentence:

[1830] If employees are feeling anxious, calculate a relaxing route through the factory.

[1831] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1832] Step 1:

[1833] The user launches the application and logs in.

[1834] Input: User login information (ID, password).

[1835] Output: The user is logged in.

[1836] Specific operation: The terminal sends the login information to the server and is authenticated. After successful authentication, the main screen is displayed.

[1837] Step 2:

[1838] The user inputs current location information and destination information.

[1839] Input: Current location information (GPS), destination information (manual input or voice input)

[1840] Output: Current location and destination information data.

[1841] How it works: The device uses GPS to obtain current location information, and the user inputs destination information manually or by voice. An interface is also available for specifying additional intermediate destinations and stops.

[1842] Step 3:

[1843] The server calculates the optimal route based on the current location information and destination information received.

[1844] Input: current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[1845] Output: Optimal route information.

[1846] Specific operation: The server refers to the database, executes the routing algorithm based on the necessary data, and calculates the optimal route. The calculated route is sent to the terminal.

[1847] Step 4:

[1848] The terminal presents the calculated route information to the user.

[1849] Input: Optimal route information.

[1850] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[1851] Specific operation: The device provides route guidance in the form of voice, text, vibration, etc. depending on the type of disability the user has.

[1852] Step 5:

[1853] An emotion engine detects the user's emotional state.

[1854] Input: User's facial recognition data, biometric information (heart rate, galvanic skin response).

[1855] Output: Emotional state (stress level and relaxation level).

[1856] Specific operation: The device's camera and sensors collect data on the user's face and physical state, which are then analyzed using an emotion recognition library (OpenCV, Python library). The analysis results are then sent to the server.

[1857] Step 6:

[1858] The server recalculates the route based on the emotional state.

[1859] Input: Emotional state, current location information, destination information, public transport timetable data, congestion forecast data, barrier-free route information.

[1860] Output: New optimal route information based on emotional state.

[1861] Specific operation: The server takes into account the emotional state, re-references the database, recalculates relaxing routes and routes during quiet times, and sends the results to the terminal.

[1862] Step 7:

[1863] The terminal then presents the recalculated new route information to the user.

[1864] Input: New optimal route information.

[1865] Output: Audio guide for the visually impaired, text display and vibration guidance information for the hearing impaired.

[1866] Specific operation: The terminal receives the recalculated route information from the database and provides the user with new directions in an appropriate format.

[1867] Step 8:

[1868] The device tracks your location in real time and detects deviations from your route or changes in congestion.

[1869] Input: Current location information (GPS data), congestion forecast data.

[1870] Output: Route deviation notification, congestion change notification.

[1871] Specific operation: The device periodically updates its current location using its GPS function and sends it to the server. The server monitors the current location in real time and sends a notification to the device if it detects deviations or changes in congestion.

[1872] Step 9:

[1873] When the user arrives at their destination, they are provided with information on information centers and support facilities.

[1874] Input: Destination arrival notification.

[1875] Output: Location information of information centers and support facilities around the destination.

[1876] Specific operation: When the device detects arrival at the destination, it automatically provides the user with the location information of information centers and support facilities. This is done by voice or text notification.

[1877] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1878] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1879] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1880] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1881] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions...

Claims

1. In a barrier-free map creation application designed for the visually and hearing impaired, means for receiving user location information and destination information; A means for calculating an optimal route based on the received location information and destination information by referring to public transportation timetable data, congestion forecast data, and barrier-free route information; means for providing the calculated optimum route and related congestion information and barrier-free facility information to a user's terminal; A means for tracking the user's current location in real time as the user moves, and detecting and notifying changes in congestion or deviations from the route that occur along the route; a means for providing location information of information centers and support facilities around the destination after the user arrives; A system including:

2. The system according to claim 1 , further comprising means for providing guidance information by voice guidance or text display and vibration depending on the type of disability of the user.

3. The system according to claim 1, further comprising means for providing an interface for a user to specify intermediate points or desired stops when setting a route.

Images