System

Abstract

To provide a system.SOLUTION: A system includes: means that analyzes a space with generative AI on the basis of image information acquired with a wearable camera, position information of a position information system, and map information of a map information system; means that generates information for recognizing the surrounding environment on the basis of the information acquire by the means that analyzes a space; and means that feeds back the information for recognizing the surrounding environment to a user by voice in a natural language by utilizing 5G communication.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult for visually impaired people to navigate to their destination or to grasp risks in the surrounding environment in a timely manner without the help of a sighted person. [Means for solving the problem]

[0005] The system will analyze space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system, and provide feedback to users in natural language using 5G communications with low latency and excellent real-time performance. This will enable visually impaired people to be guided to their destination and to be able to grasp risks in the surrounding environment in a timely manner without the need for sighted people. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a wearable camera for visually impaired people is used to capture images of the surrounding environment. This wearable camera is part of a head-mounted display that can be worn on the head of a visually impaired person.

[0029] "Example 2"

[0030] The captured image information is then sent to a generative AI system along with precise location information obtained from a location information system and highly accurate map information obtained from a map information system. Based on this information, the generative AI system analyzes the space and generates the optimal route for the visually impaired person to travel.

[0031] "Example 3"

[0032] The generated route information is fed back to the visually impaired person in real time using 5G communications. The feedback is provided in natural language and in a format that is easy for the visually impaired to understand. For example, instructions might be given such as, "There is an obstacle five meters ahead. Move two meters to the right."

[0033] "Example 4"

[0034] This system will enable visually impaired people to receive guidance to their destination and to be aware of risks in their surroundings in a timely manner without the need for sighted people, greatly improving the freedom and safety of movement for visually impaired people.

[0035] The processing flow of each embodiment will be described below.

[0036] "Example 1"

[0037] Step 1: A visually impaired person wears a head-mounted display that incorporates a wearable camera to capture images of the surrounding environment from the visually impaired person's perspective.

[0038] Step 2: Image information from the wearable camera is analyzed in conjunction with a location information system and a map information system. The location information system uses GPS and other technologies to provide the visually impaired person with accurate location information, and the map information system provides highly accurate map information corresponding to that location. Step 3: The generative AI analyzes the space based on this information and generates the optimal route for the visually impaired person to travel.

[0039] "Example 2"

[0040] Step 1: The generated route information is fed back to the visually impaired person in real time using 5G communications.

[0041] Step 2: Feedback is given in natural language and in a form that is easy for a visually impaired person to understand, such as "There is an obstacle 5 meters ahead. Move 2 meters to the right."

[0042] Step 3: The visually impaired person can navigate according to this feedback and reach their destination safely.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] For visually impaired people to travel safely and efficiently, it is important to grasp information about their surroundings in real time and receive appropriate feedback. However, conventional systems do not provide enough information for visually impaired people to accurately recognize their surroundings, and they often require assistance from sighted people. In addition, due to the lack of real-time and low latency, it is difficult to grasp risks in a timely manner.

[0046] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communication; means for preprocessing video data and performing object recognition and text reading using a generative AI model; means for generating the analysis results as audio data and converting it into audio using speech synthesis software; means for transmitting the generated audio data to the visually impaired person's head-mounted display to notify them by audio; and means for providing guidance to the visually impaired person and timely identification of risks in the surrounding environment without the intervention of a sighted person. This enables the visually impaired person to accurately recognize their surrounding environment in real time and move safely and efficiently.

[0047] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0048] A "wearable camera" is a camera that can be worn by the user and is used as part of a head-mounted display worn on the head of a visually impaired person.

[0049] A "location information system" refers to a system that uses technology such as GPS to accurately identify a user's current location.

[0050] A "map information system" refers to a system that provides highly accurate map data and provides surrounding geographic information in conjunction with the user's location information.

[0051] "Generative AI" refers to a system that uses artificial intelligence technology to generate useful information from input data.

[0052] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0053] "Feedback in natural language" refers to conveying analysis results and information to users in natural language that is easy for humans to understand.

[0054] "Preprocessing" refers to processing that is performed before data analysis, and includes processes such as noise removal and resolution adjustment.

[0055] "Object recognition" refers to a technology that detects specific objects from image data and identifies their type and location.

[0056] "Text reading" refers to the technology of extracting character information from image data and recognizing it as text data.

[0057] "Speech synthesis software" refers to software for converting text data into voice data.

[0058] A "head-mounted display" is a display device worn by the user on the head, and has an audio output function for visually impaired people.

[0059] "Timely risk understanding" refers to analyzing surrounding environmental information in real time and providing users with immediate risk information.

[0060] This invention is a system that enables visually impaired people to recognize their surroundings in real time and move safely and efficiently. This system is implemented using hardware and software such as a wearable camera, a location information system, a map information system, a generative AI model, 5G communications, and speech synthesis software.

[0061] The server receives video data transmitted from a wearable camera worn on the head of a visually impaired person. This camera is used to capture the surrounding environment. The server uses an image processing library to preprocess the received video data. Preprocessing includes noise reduction and resolution adjustment.

[0062] The server then inputs the preprocessed video data into a generative AI model, which has capabilities such as object recognition and text reading. For example, an object detection model is used to recognize objects in the video. The generative AI model generates information that helps visually impaired people understand their surroundings.

[0063] The server uses speech synthesis software to generate voice data based on the analysis results obtained from the generative AI model. The analysis results are converted into voice in a natural language that is easy for visually impaired people to understand. For example, voice data such as "There is a pedestrian ahead" or "The traffic light is red" is generated.

[0064] The generated voice data is transmitted in real time with low latency via 5G communications to the visually impaired person's head-mounted display. The visually impaired person can then understand the analysis results through a built-in speaker in the head-mounted display. This enables the visually impaired person to receive guidance to their destination and be aware of risks in their surrounding environment in a timely manner, without the need for a sighted person.

[0065] As a concrete example, consider a scene in which a visually impaired person is walking around town. The video captured by the wearable camera includes traffic lights, pedestrians, and vehicles. The server receives this video data and uses a generative AI model to analyze the color of the traffic lights, the location of pedestrians, and the movement of vehicles. The analysis results are generated as audio data, and the visually impaired person is provided with information such as "The traffic light is red," "There is a pedestrian ahead," and "A car is approaching from the right."

[0066] Example prompt sentence:

[0067] "Imagine a scene where a visually impaired person is walking around town. Explain a system that analyzes video data captured by a wearable camera and provides audio notifications about traffic light colors, pedestrian locations, and vehicle movements."

[0068] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0069] Step 1:

[0070] The device uses a wearable camera worn on the head of the visually impaired person to capture images of the surrounding environment in real time. The input is the video data captured by the camera, and the output is the captured video data. Specifically, the camera captures video at 30 frames per second.

[0071] Step 2:

[0072] The terminal transmits the captured video data to the server. The input is the captured video data, and the output is the video data transmitted to the server. In concrete terms, the terminal compresses the video data and transmits it to the server.

[0073] Step 3:

[0074] The server preprocesses the received video data. The input is the video data sent from the device, and the output is the preprocessed video data. Specifically, the server uses an image processing library such as OpenCV to remove noise and adjust the resolution. For example, it adjusts the resolution of the video data to 640x480 pixels and applies a noise filter.

[0075] Step 4:

[0076] The server inputs the preprocessed video data into the generative AI model. The input is the preprocessed video data, and the output is the analysis results by the generative AI model. Specifically, the server uses the object detection model to recognize objects in the video. For example, it analyzes the color of traffic lights, the position of pedestrians, and the movement of vehicles.

[0077] Step 5:

[0078] The server generates the analysis results obtained from the generative AI model as voice data. The input is the analysis results from the generative AI model, and the output is voice data. Specifically, it uses voice synthesis software to convert the analysis results into voice. For example, it generates voice data such as "There is a pedestrian ahead" or "The traffic light is red."

[0079] Step 6:

[0080] The server transmits the generated voice data to the terminal. The input is the generated voice data, and the output is the voice data transmitted to the terminal. In concrete terms, the server compresses the voice data and transmits it to the terminal.

[0081] Step 7:

[0082] The device plays the received audio data. The input is the audio data sent from the server, and the output is the audio heard by the visually impaired person. Specifically, the device decodes the received audio data and plays it back from the speaker built into the head-mounted display. This allows the visually impaired person to recognize information about their surrounding environment through audio.

[0083] (Application example 1)

[0084] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] For visually impaired people to move around safely, it is important to accurately grasp their surroundings and receive appropriate feedback. However, conventional technologies have made it difficult for visually impaired people to recognize surrounding dangers in real time and take appropriate action. In addition, there is a lack of systems that enable visually impaired people to move around independently without the assistance of sighted people, which has hindered their independence.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0087] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; and means for detecting dangerous situations and obstacles using a generative AI model and notifying the user through audio and haptic feedback, thereby enabling visually impaired people to recognize surrounding dangers in real time and move safely.

[0088] "Visually impaired persons" refer to people who have visual impairments and are unable to obtain sufficient visual information.

[0089] A "wearable camera" is a camera that can be worn by the user, and is primarily a device attached to a head-mounted display.

[0090] A "location information system" refers to a system that uses technology such as GPS to accurately identify a user's current location.

[0091] A "map information system" is a system that provides geographical information and includes highly accurate map data.

[0092] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate new information and predictions.

[0093] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0094] "Natural language feedback" refers to providing information in natural language that is easy for users to understand.

[0095] "Sighted person" refers to a person who has no visual impairment and can normally obtain visual information.

[0096] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task.

[0097] "Dangerous Situation" means any environment or situation that may potentially pose a hazard to the User.

[0098] "Obstacle" means any object or structure that impedes the user's movement.

[0099] "Audio feedback" refers to a means of providing information to a user using audio.

[0100] "Haptic feedback" refers to a means of providing information to a user through the sense of touch.

[0101] A system for implementing this invention includes a wearable camera worn by a visually impaired person, a location information system, a map information system, a generative AI model, 5G communication, audio feedback, and haptic feedback.

[0102] System configuration

[0103] 1. Wearable camera: Built into a head-mounted display worn by visually impaired people, it captures the surrounding environment in real time.

[0104] 2. Location systems: Using technologies such as GPS to pinpoint a user's exact location.

[0105] 3. Map information system: Provides highly accurate map data and links it to the user's location information.

[0106] 4. Generative AI model: Analyzes image information acquired by a wearable camera and detects dangerous situations and obstacles.

[0107] 5. 5G communications: Realizing high-speed, low-latency communications, processing and transmitting information in real time.

[0108] 6. Audio feedback: Notify the user of detected information via audio.

[0109] 7. Haptic feedback: Providing information to the user through touch when needed.

[0110] Program processing

[0111] The server receives image information from the wearable camera and analyzes it using a generative AI model. This analysis detects dangerous situations and obstacles. The detection results are sent to the user's device in real time via 5G communication. The user's device then provides information to the user through audio and haptic feedback.

[0112] Hardware and software used

[0113] Hardware: Head-mounted display, smartphone, GPS module

[0114] Software: OpenCV (image processing library), TENSORFLOW (registered trademark) (generative AI model framework), pyttsx3 (voice synthesis library)

[0115] Specific examples

[0116] When a visually impaired person is walking down a sidewalk and sees an obstacle ahead, a wearable camera captures the image. A generative AI model analyzes the image and issues a voice notification saying, "There is an obstacle ahead." This notification is sent in real time via 5G communication, allowing the user to respond immediately.

[0117] Prompt Sentence Examples

[0118] "Detect if there are any obstacles ahead."

[0119] This prompt is used to instruct the generative AI model on a specific task.

[0120] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0121] Step 1:

[0122] The user puts on the wearable camera and starts the system.

[0123] Input: System startup via user interaction

[0124] Output: The camera begins capturing the surrounding environment in real time.

[0125] Specific operation: A user puts on the head-mounted display and turns on the system. The camera starts up and starts capturing video.

[0126] Step 2:

[0127] The video data acquired by the camera is sent to the server.

[0128] Input: Real-time video captured by a camera

[0129] Output: Video data sent to the server

[0130] Specific operation: The video data captured by the camera is sent to the server via 5G communication.

[0131] Step 3:

[0132] The video data received by the server is input into the generative AI model for analysis.

[0133] Input: Video data sent to the server

[0134] Output: Analysis results (detection of dangerous situations and obstacles)

[0135] How it works: The server preprocesses the video data it receives using OpenCV and then inputs it into a generative AI model trained with TensorFlow. The model analyzes the video and detects dangerous situations and obstacles.

[0136] Step 4:

[0137] The server sends the analysis results to the user's device via 5G communication.

[0138] Input: Analysis results from generative AI model

[0139] Output: Analysis results sent to the user's device

[0140] Specific operation: The server sends the analysis results to the user's smartphone in real time.

[0141] Step 5:

[0142] The user's device receives the analysis results and generates audio and haptic feedback.

[0143] Input: Analysis results sent from the server

[0144] Output: Audio and haptic feedback

[0145] Specific operation: The user's smartphone receives the analysis results and generates audio feedback using pyttsx3, and also generates haptic feedback if necessary.

[0146] Step 6:

[0147] The user receives audio and haptic feedback and takes appropriate action.

[0148] Input: Audio and haptic feedback

[0149] Output: User action (e.g., avoiding obstacles)

[0150] Specific actions: The user receives audio and haptic feedback and takes appropriate action, such as avoiding an obstacle ahead.

[0151] In this way, visually impaired people can recognize dangers around them in real time and move around safely.

[0152] Example 2

[0153] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] For visually impaired people to travel safely and efficiently, they need a system that can grasp information about their surroundings and their location in real time and guide them to the appropriate route. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there is insufficient support for visually impaired people to travel independently, and they often need the help of sighted people. This limits the freedom of movement of visually impaired people.

[0155] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image information captured by a user, means for acquiring accurate location information from a location information system, means for acquiring highly accurate map information from a map information system, means for transmitting the acquired image information, location information, and map information to a generative AI, means for the generative AI to analyze space based on this information and generate an optimal route for the visually impaired person to travel, means for transmitting the generated optimal route information to the user's terminal, and means for the user's terminal to display the received route information to the user. This enables the visually impaired person to grasp information about their surrounding environment in real time and travel safely and efficiently.

[0156] "User" refers to a visually impaired person who uses the system to take images and receive route information.

[0157] "Image information" refers to visual data captured by a user using a terminal.

[0158] A "location information system" refers to a system that uses technology such as GPS to determine a person's current location.

[0159] "Precise location information" refers to latitude and longitude data obtained from a location information system.

[0160] "Map information system" refers to a system that provides geographical information.

[0161] "High-precision map information" refers to detailed geographic data obtained from a map information system.

[0162] "Generative AI" refers to artificial intelligence that analyzes space based on the data it receives and generates the optimal route.

[0163] "Analyzing space" refers to the process of understanding the surrounding environment based on image information, location information, and map information.

[0164] An "optimal route" refers to a route that allows visually impaired people to travel safely and efficiently.

[0165] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[0166] "Route information" refers to data regarding travel routes generated by generative AI.

[0167] "Displaying" refers to the terminal providing the route information to the user visually or audibly.

[0168] This invention is a system for enabling visually impaired people to travel safely and efficiently, in which a generative AI generates an optimal route based on image information taken by the user, accurate location information obtained from a location information system, and highly accurate map information obtained from a map information system.

[0169] Hardware and software used

[0170] Location information system: GPS module

[0171] Map Information System: Geographical Information API

[0172] Generative AI: Artificial intelligence models (e.g., OpenAI (registered trademark) GPT-4 (registered trademark))

[0173] System configuration

[0174] (server)

[0175] The server receives image information taken by the user on the device. Specifically, the image data is sent to the server via an HTTP POST request. Next, the server obtains the current location information from the location information system. This is done by sending an API request to the GPS module and receiving latitude and longitude data. The server then obtains highly accurate map information from the map information system. This is done by sending a request to the geographic information provision API and receiving map data based on the specified location information.

[0176] The server sends the acquired image information, location information, and map information to the generative AI. This data is converted into JSON format and a POST request is sent to the generative AI's API endpoint. The generative AI analyzes the space based on the received information and generates the optimal route for the visually impaired person to travel. The generated optimal route information is sent from the server to the user's device.

[0177] (Terminal)

[0178] The device sends the image information taken by the user to the server. Specifically, the device's camera app captures the image and sends it to the server via an HTTP POST request. The device then displays the optimal route information received from the server to the user. The device application analyzes the route information and provides visual or audio guidance to the user.

[0179] (User)

[0180] The user takes an image using the device. Specifically, they open the camera app on their smartphone and capture the desired scene. Next, the user moves based on the optimal route information displayed on the device. They safely travel to their destination while checking the device's screen and audio guidance.

[0181] (Example)

[0182] The user takes an image with their smartphone. The smartphone (device) sends the captured image to the server via an HTTP POST request. The server obtains the current location information from the GPS module and obtains highly accurate map information using a geographic information provision API. The server sends the image information, location information, and map information to the generative AI. The generative AI analyzes the space based on this information and generates the optimal route for the visually impaired person to travel. The server then sends the generated optimal route information to the smartphone. The smartphone displays the optimal route information to the user and provides visual and audio guidance.

[0183] Prompt Sentence Examples

[0184] "Generate the optimal route for visually impaired people based on image information taken by the user, location information obtained from GPS, and map information obtained from the map information providing API."

[0185] In this way, the system helps visually impaired people navigate safely and efficiently.

[0186] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0187] Step 1:

[0188] The server receives image information taken by the user on the device. Specifically, the user takes an image using the camera app on their smartphone, and the image data is sent to the server via an HTTP POST request. The input is the image information taken by the user, and the output is the image data stored on the server.

[0189] Step 2:

[0190] The server obtains precise location information from the location information system. Specifically, the server sends an API request to the GPS module and receives the current latitude and longitude data. The input is the location request from the GPS module, and the output is the latitude and longitude data stored on the server.

[0191] Step 3:

[0192] The server obtains highly accurate map information from the map information system. Specifically, the server sends a request to the geographic information API and receives map data based on the specified location information. The input is the request to the geographic information API, and the output is highly accurate map information stored on the server.

[0193] Step 4:

[0194] The server sends the acquired image information, location information, and map information to the generative AI. Specifically, it converts this data into JSON format and sends a POST request to the generative AI's API endpoint. The input is the image information, location information, and map information, and the output is the data sent to the generative AI.

[0195] Step 5:

[0196] Generative AI analyzes the space based on the information it receives and generates the optimal route for visually impaired people to travel. Specifically, generative AI combines image analysis algorithms with map data to calculate a route that takes into account obstacles, road width, etc. The inputs are image information, location information, and map information, and the output is information about the optimal route.

[0197] Step 6:

[0198] The server sends the generated optimal route information to the user's device. Specifically, it sends the route information received from the generative AI to the device in JSON format. The input is the optimal route information from the generative AI, and the output is the route information sent to the device.

[0199] Step 7:

[0200] The terminal displays the optimal route information received from the server to the user. Specifically, the terminal application analyzes the route information and guides the user visually or audibly. The input is the optimal route information from the server, and the output is the route information displayed to the user.

[0201] Step 8:

[0202] The user travels based on the optimal route information displayed on the device. Specifically, the user travels safely to their destination while checking the device's screen and voice guidance. The input is the route information displayed on the device, and the output is the user's travel behavior.

[0203] (Application example 2)

[0204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] It is extremely difficult for visually impaired people to navigate safely and efficiently within a physical store. With conventional technology, visually impaired people require the assistance of sighted people to understand their surroundings and find the optimal route to their destination. It is also difficult to provide real-time information, and risk assessment during travel is insufficient. This limits the independent mobility of visually impaired people.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0207] In this invention, the server includes: means for analyzing space using a generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communication; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; means for capturing images of the store interior using a camera mounted on smart glasses and for the generative AI to provide an optimal route in real time based on the location information and map information; and means for the generative AI to convert the captured image information into a list format, acquire the location information and map information in JSON format, and calculate the optimal route. This enables visually impaired people to move safely and efficiently within physical stores.

[0208] "Visually impaired people" refers to people who have visual impairments and have difficulty using visual information.

[0209] A "wearable camera" refers to a camera device that can be worn on the body.

[0210] "Image information" refers to visual data captured by a photographing device such as a camera.

[0211] A "location information system" refers to a system that uses technology such as GPS to determine the location of a specific location, such as its latitude and longitude.

[0212] A "map information system" is a system that provides geographical information and includes highly accurate map data.

[0213] "Generative AI" refers to an artificial intelligence system that analyzes and generates information based on acquired data.

[0214] "Analyzing space" refers to understanding the surrounding environment and the structure of space based on acquired image information and location information.

[0215] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0216] "Real-time" refers to the property of data acquisition and processing occurring immediately and without delay.

[0217] "Providing feedback in natural language" means providing information in language that is easy for users to understand.

[0218] "Sighted people" refers to people who are not visually impaired and can use visual information.

[0219] "Guidance to a destination" refers to providing a route or method for a user to reach a destination.

[0220] "Timely risk assessment of the surrounding environment" means immediately recognizing potential dangers in the surrounding environment and informing users.

[0221] "Smart glasses" are eyeglass-type wearable devices equipped with functions such as a camera and display.

[0222] "List format" refers to a format in which data is represented as a list (array).

[0223] "JSON format" is an abbreviation for JavaScript (registered trademark) Object Notation, and refers to a lightweight data exchange format that expresses data in text format.

[0224] "Calculating the optimal route" refers to calculating the most efficient and safe route for a user to reach their destination.

[0225] The system for implementing this invention helps visually impaired people navigate safely and efficiently within a physical store. The system uses hardware and software such as wearable cameras, location information systems, map information systems, generative AI, 5G communications, and smart glasses.

[0226] First, the user puts on the smart glasses. The smart glasses are equipped with a camera that captures images of the store in real time. The captured image information is sent to the generative AI along with accurate location information obtained from a location information system and highly accurate map information obtained from a map information system.

[0227] Based on this information, the generative AI analyzes the space and generates the optimal route for the visually impaired. The generated route information is fed back to the user in real time with low latency using 5G communications. The feedback is provided in natural language in a format that is easy for the user to understand.

[0228] Specifically, the generative AI converts captured image information into a list format and obtains location and map information in JSON format. This allows it to calculate the optimal route and provide it to the user. For example, when a user moves through a supermarket, the generative AI analyzes the location of shelves, the width of aisles, etc., and suggests the safest and most efficient route.

[0229] An example of a prompt is, "Generate the optimal route for a visually impaired person to move safely within a supermarket. Your current location is latitude 35.6895, longitude 139.6917. Use generative AI to provide the optimal route based on image information within the store and highly accurate map information."

[0230] This system allows visually impaired people to be guided to their destination and to be aware of risks in their surroundings in a timely manner without relying on sighted people, thereby enabling them to move around independently.

[0231] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0232] Step 1:

[0233] A user wears smart glasses and moves around the store. The camera on the smart glasses captures images in real time.

[0234] Input: Real-time images of the store

[0235] Output: Captured image information

[0236] How it works: The camera in the smart glasses continuously captures images of the in-store environment and generates image data.

[0237] Step 2:

[0238] The device obtains precise location information from the location information system.

[0239] Input: Current location request

[0240] Output: Latitude and longitude location

[0241] What happens: The device uses a location information system such as GPS to determine the user's current location and obtains latitude and longitude data.

[0242] Step 3:

[0243] The terminal obtains highly accurate map information from the map information system.

[0244] Input: Current location

[0245] Output: High-precision map information

[0246] Specific operation: The device sends its current location information to the map information system and obtains detailed map data corresponding to that location.

[0247] Step 4:

[0248] The server sends the captured image information, location information, and map information to the generative AI.

[0249] Input: Image information, location information, map information

[0250] Output: Sending data to generative AI

[0251] Specific operation: The device sends the image information, location information, and map information it has acquired to the server, and the server then sends this data to the generative AI.

[0252] Step 5:

[0253] The generative AI analyzes the space based on the data it receives and generates the optimal route for visually impaired people to move around.

[0254] Input: Image information, location information, map information

[0255] Output: Optimal route information

[0256] Specific operation: The generative AI converts image information into a list format, analyzes location information and map information in JSON format, and calculates the optimal route.

[0257] Step 6:

[0258] The server uses 5G communications to provide the optimal route information generated to the user in real time.

[0259] Input: Optimal route information

[0260] Output: Natural language feedback

[0261] Specific operation: The server converts the optimal route information received from the generative AI into natural language and sends it to the user's smart glasses via 5G communication.

[0262] Step 7:

[0263] Users receive optimal route information provided in natural language through smart glasses, allowing them to move safely and efficiently within the store.

[0264] Input: Natural language feedback

[0265] Output: User movement

[0266] Specific operation: The user moves around the store following the generated optimal route while listening to audio guidance from the smart glasses.

[0267] Example 3

[0268] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] For visually impaired people to move safely and efficiently, they need to be able to grasp information about their surroundings and the location of obstacles in real time, and receive appropriate instructions. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there are limited means for visually impaired people to move independently without the assistance of sighted people. This has restricted the freedom and safety of their movement.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0271] In this invention, the server includes: means for analyzing space using a generative AI model based on environmental information acquired by a sensor device, accurate location information from a location information system, and highly accurate map information from a map information system; means for providing feedback to a user in natural language as information with low latency and excellent real-time performance using a communication network; means for providing guidance to a destination and timely risk assessment of the surrounding environment without the intervention of a sighted person; means for generating instructions in natural language by inputting prompt sentences into the generative AI model; and means for communicating the generated instructions to a user using a voice output function. This enables visually impaired people to grasp information about their surrounding environment in real time and move safely and efficiently.

[0272] A "sensor device" is a device for acquiring environmental information and is used to detect the situation around a visually impaired person in real time.

[0273] A "location information system" is a system that accurately identifies the current location of visually impaired people and provides location data using technologies such as GPS.

[0274] A "map information system" is a system that provides highly accurate map information and is used to calculate travel routes for visually impaired people.

[0275] A "generative AI model" is an artificial intelligence model that analyzes space based on collected data and generates instructions in natural language that are easy for visually impaired people to understand.

[0276] A "communications network" is a network for transmitting information in real time with low latency, and includes high-speed communication technologies such as 5G.

[0277] "Means for providing feedback in natural language" refers to means for providing instructions generated by a generative AI model in a form that is easy for visually impaired people to understand, and includes voice output functions, etc.

[0278] A "prompt sentence" is a sentence that is input into a generative AI model and serves as the basis for generating specific instructions for visually impaired people.

[0279] The "voice output function" is a function for conveying generated natural language instructions as voice to visually impaired people.

[0280] This invention is a system for enabling visually impaired people to move around safely and efficiently, and is implemented using sensor devices, a location information system, a map information system, a generative AI model, a communication network, and a voice output function.

[0281] The server first obtains environmental information from a sensor device. The sensor device is used to detect the visually impaired person's surroundings in real time. The server then uses a location information system to determine the visually impaired person's current location. Location information systems use technologies such as GPS. The server also obtains highly accurate map information using a map information system and calculates the visually impaired person's travel route.

[0282] The server uses this data to analyze the space using a generative AI model. The generative AI model generates instructions in natural language that are easy for visually impaired people to understand based on the collected data. At this time, a prompt sentence is input to the generative AI model. For example, the following prompt sentence can be used:

[0283] "The user is currently at the entrance to a park. There is a bench 5 meters ahead. Instruct the user to move 2 meters to the right."

[0284] Based on this prompt, the generative AI model generates the instruction, "There is a bench 5 meters ahead. Please move 2 meters to the right."

[0285] The server then transmits the generated natural language instructions to the visually impaired person's device via a communications network using high-speed communications technology such as 5G. The device then communicates the received instructions to the visually impaired person using its voice output function. For example, the device may play back a voice instruction such as, "There is a bench five meters ahead. Please move two meters to the right."

[0286] The user can move safely by following the voice instructions. This system enables the visually impaired person to be guided to their destination and to be aware of risks in the surrounding environment in a timely manner without the intervention of a sighted person. This significantly improves the freedom and safety of movement for the visually impaired person. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0287] Step 1: Data collection

[0288] The server acquires environmental information from the sensor devices. The input is real-time data from the sensor devices. Specifically, the server receives the data sent from the sensor devices and detects the situation around the visually impaired person. The output is environmental information data.

[0289] Step 2: Obtaining location information

[0290] The server uses a location information system to identify the current location of the visually impaired person. GPS data is used as input. Specifically, the server analyzes the GPS data to identify the exact location of the visually impaired person. The output is the current location data.

[0291] Step 3: Get map information

[0292] The server uses a map information system to obtain highly accurate map information. The input is a map database. Specifically, the server obtains the necessary map information from the map database and obtains basic data for calculating the travel route of a visually impaired person. The output is map information data.

[0293] Step 4: Data analysis

[0294] The server analyzes the space using a generative AI model based on the collected environmental information, current location data, and map information data. The inputs are environmental information data, current location data, and map information data. Specifically, the server inputs this data into the generative AI model and performs analysis to generate instructions in natural language that are easy for visually impaired people to understand. The output is analysis result data.

[0295] Step 5: Natural Language Generation

[0296] The server inputs a prompt sentence into the generative AI model to generate natural language instructions. The inputs include analysis result data and the prompt sentence. In concrete terms, the server inputs the following prompt sentence into the generative AI model: "The user's current location is the entrance to the park. There is a bench 5 meters ahead. Please instruct the user to move 2 meters to the right." The generative AI model generates natural language instructions based on this prompt sentence. The output is natural language instruction data.

[0297] Step 6: Feedback

[0298] The server sends the generated natural language instructions to the visually impaired person's device via a communication network. The input is natural language instruction data. Specifically, the server sends the instruction data to the device using 5G communication. The output is the instruction data sent to the device.

[0299] Step 7: Audio Output

[0300] The device uses its voice output function to communicate the received instruction data to the visually impaired person. The input is the instruction data sent to the device. Specifically, the device converts the instruction data into voice using voice synthesis technology and plays it back to the visually impaired person. The output is the voice instruction that was communicated to the visually impaired person.

[0301] Step 8: User Behavior

[0302] The user moves safely according to voice instructions. The input is the voice instructions. The specific action is for the user to follow the voice instructions and move in the specified direction. The output is the user's safe movement.

[0303] (Application example 3)

[0304] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0305] When visually impaired people use self-driving vehicles, they face the challenge of having difficulty grasping route information to their destination and risk information about the surrounding environment in real time. Furthermore, in order for visually impaired people to travel safely, they often need the assistance of sighted people, which limits their freedom of movement. There is a need to solve these challenges and enable visually impaired people to use self-driving vehicles safely and freely.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0307] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted persons; means for being installed in autonomous vehicles and providing real-time route information and risk information of the surrounding environment to enable visually impaired persons to use autonomous vehicles safely; and means including a speech synthesis engine for providing voice feedback. This enables visually impaired persons to understand route information and risk information of the surrounding environment in real time when using autonomous vehicles and to travel safely and freely.

[0308] "Visually impaired people" refers to people who have visual impairments and have difficulty using visual information.

[0309] A "wearable camera" refers to a camera device that can be worn on the body.

[0310] A "location information system" refers to a system that provides accurate location information using GPS or other technologies.

[0311] "Map information system" refers to a system that provides highly accurate map information.

[0312] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate information for specific purposes.

[0313] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0314] "Means for providing feedback in natural language" refers to means for providing information to users using natural language.

[0315] "Sighted people" refers to people who are not visually impaired and can use visual information.

[0316] An "autonomous vehicle" refers to a vehicle that can drive itself using artificial intelligence and sensor technology.

[0317] "Route information" refers to information about the route and direction to a destination.

[0318] "Risk information about the surrounding environment" refers to information about obstacles and dangerous situations that may be encountered during travel.

[0319] A "speech synthesis engine" refers to technology that converts text information into speech.

[0320] This invention is a system that enables visually impaired people to travel safely and freely when using self-driving vehicles by understanding route information and risk information about the surrounding environment in real time.

[0321] System configuration

[0322] The system consists of the following main components:

[0323] 1. Wearable camera: Worn by visually impaired people, it captures image information of the surrounding area.

[0324] 2. Location information system: Provides accurate location information using GPS etc.

[0325] 3. Map information system: Provides highly accurate map information.

[0326] 4. Generative AI: Analyzes space based on image information, location information, and map information obtained from a wearable camera, and generates optimal route information.

[0327] 5. 5G communications: Provides low-latency, real-time communications.

[0328] 6. Speech synthesis engine: Provides generated route and risk information as voice feedback in natural language.

[0329] Program processing

[0330] The server analyzes image information acquired from the wearable camera and integrates data obtained from the location information system and map information system. Based on this data, the generative AI generates optimal route information for the visually impaired. The generated route information is fed back to the visually impaired in real time using 5G communications. A speech synthesis engine is used for voice feedback, which is provided in natural language that is easy for the visually impaired to understand.

[0331] Hardware and software used

[0332] Hardware: Smartphones, wearable cameras, GPS devices

[0333] Software: Python, requests library, pyttsx3 library

[0334] Specific examples

[0335] For example, when a visually impaired person is using a self-driving vehicle, they may be given feedback such as:

[0336] "Turn left at the next intersection."

[0337] "There are pedestrians ahead, please be careful."

[0338] "We're 500 meters away from our destination."

[0339] Prompt Sentence Examples

[0340] An example of a prompt to be input to the generative AI model is as follows:

[0341] Develop an application that provides real-time voice feedback on route information and risk information about the surrounding environment to visually impaired people using self-driving vehicles. Route information is obtained from an API using 5G communications, and the pyttsx3 library is used for voice feedback. Specific examples of feedback include "Turn left at the next intersection," and "There are pedestrians ahead, so please be careful."

[0342] In this way, applications can be realized that allow visually impaired people to safely use autonomous vehicles.

[0343] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0344] Step 1:

[0345] The server receives image information from the wearable camera in real time. The input is image data from the wearable camera, and the output is image data for analysis. This image data captures the environment around the visually impaired person in real time.

[0346] Step 2:

[0347] The server obtains precise location information from a location information system. The input is location data from a GPS device, and the output is the current precise location information. This location information is used to determine the current location of the visually impaired person.

[0348] Step 3:

[0349] The server obtains high-precision map information from the map information system. The input is map data from the map information system, and the output is map information for analysis. This map information is used to calculate routes for visually impaired people.

[0350] Step 4:

[0351] The server uses generative AI to integrate the acquired image information, location information, and map information and analyze the space. The input is image data, location information, and map information, and the output is optimal route information. This analysis generates the optimal route for visually impaired people to travel.

[0352] Step 5:

[0353] The server uses 5G communications to send the generated route information to the visually impaired person's device in real time. The input is the optimal route information, and the output is the route information sent to the visually impaired person's device. 5G communications enable low-latency, real-time communications.

[0354] Step 6:

[0355] The device uses a speech synthesis engine to provide natural language voice feedback of the received route information. The input is route information and the output is voice feedback. The speech synthesis engine provides information in a format that is easy for visually impaired people to understand.

[0356] Step 7:

[0357] The user receives voice feedback and follows instructions to operate the autonomous vehicle. The input is voice feedback and the output is the user's actions. This allows the visually impaired to move around safely and freely.

[0358] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0359] "Example 1"

[0360] This invention is a system for safe mobility for the visually impaired. This system is composed of a wearable camera, a location information system, a map information system, generative AI, 5G communication, a natural language feedback system, and an emotion engine.

[0361] "Example 2"

[0362] The wearable camera captures the surrounding environment from the perspective of the visually impaired person. The location information system uses GPS and other technologies to provide the visually impaired person with accurate location information, and the map information system provides highly accurate map information corresponding to that location.

[0363] "Example 3"

[0364] The generative AI analyzes the space based on image information from the wearable camera and information from the location information system and map information system. It also incorporates the user's emotional information obtained from the emotion engine into the analysis. For example, if the user is feeling anxious, the generative AI will

[0365] Generate detailed route information.

[0366] "Example 4"

[0367] The generated route information is fed back to the visually impaired in real time using 5G communications, and the feedback is provided in natural language in a format that is easy for the visually impaired to understand.

[0368] "Example 5"

[0369] The emotion engine recognizes the emotions of the visually impaired person and adjusts the feedback accordingly. For example, if the visually impaired person is feeling anxious, the emotion engine conveys that information to the generative AI, which then generates more detailed route information.

[0370] The processing flow of each embodiment will be described below.

[0371] "Example 1"

[0372] Step 1: A visually impaired person wears a wearable camera.

[0373] Step 2: The wearable camera captures the surrounding environment and sends the image information to the generative AI.

[0374] Step 3: The location information system acquires the location information of the visually impaired person and sends that information to the generative AI.

[0375] Step 4: The map information system obtains map information based on the location information and sends that information to the generative AI.

[0376] Step 5: The emotion engine acquires the emotion information of the visually impaired person and sends that information to the generative AI.

[0377] Step 6: The generative AI analyzes the space based on the received information and generates optimal route information.

[0378] Step 7: The generated route information is fed back to the visually impaired person in real time via 5G communication. Feedback is provided in natural language.

[0379] "Example 2"

[0380] Step 1: A visually impaired person puts on the wearable camera and starts moving.

[0381] Step 2: The wearable camera captures the surrounding environment and sends the image information to the generative AI.

[0382] Step 3: The location information system acquires the location information of the visually impaired person and sends that information to the generative AI.

[0383] Step 4: The map information system obtains map information based on the location information and sends that information to the generative AI.

[0384] Step 5: The emotion engine acquires the emotion information of the visually impaired person and sends that information to the generative AI.

[0385] Step 6: The generative AI analyzes the space based on the received information and generates optimal route information.

[0386] Step 7: The generated route information is fed back to the visually impaired person in real time via 5G communication. The feedback is provided in natural language and in a format that is easy for the visually impaired to understand.

[0387] Step 8: The visually impaired person continues to move according to the feedback. During this time, the emotion engine monitors the visually impaired person's emotions in real time and sends this information to the generative AI.

[0388] Step 9: The generative AI takes emotional information into account, updates the route information as needed, and provides feedback to the visually impaired user.

[0389] Example 1

[0390] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] For visually impaired people to travel safely, they need to accurately understand their surroundings and select an appropriate route. However, with conventional technology, it was difficult for visually impaired people to understand their surroundings and select an appropriate route on their own. In addition, there were times when real-time information provision was delayed, making it difficult to ensure the safety of visually impaired people. Furthermore, feedback based on the emotional state of visually impaired people was not provided, which could cause stress and anxiety.

[0392] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0393] In this invention, the server includes: a means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; a means for providing feedback to users in natural language using 5G communications for low latency and excellent real-time information; and a means for analyzing the user's emotional state using an emotion engine and adjusting the content and tone of the feedback. This enables visually impaired people to accurately grasp their surroundings and provide information in real time to move safely. Furthermore, providing feedback according to the user's emotional state reduces stress and anxiety, allowing them to move with peace of mind.

[0394] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining information visually.

[0395] A "wearable camera" is a camera that can be worn by a user, typically on the head, and refers to a device that captures images of the surrounding environment.

[0396] A "location information system" refers to a system that uses GPS or other devices to determine a user's current location.

[0397] "Map information system" refers to a system that provides map data and acquires information about the user's current location and surrounding environment.

[0398] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate useful information for users.

[0399] "5G communication" refers to the fifth-generation mobile communication system, a technology that transmits data at high speeds and with low latency.

[0400] "Natural language feedback" refers to the means of converting analysis results into natural language and providing information to users in voice or text.

[0401] An "emotion engine" refers to a system that analyzes a user's emotional state and provides feedback according to that state.

[0402] "Means for analyzing space" refers to technology that analyzes the surrounding environment based on image information, location information, and map information obtained by a wearable camera.

[0403] "Information with excellent real-time characteristics" refers to information that is provided immediately with little delay.

[0404] "Means for adjusting the content and tone of feedback" refers to technology that changes the content of the information provided and the tone of the voice depending on the user's emotional state.

[0405] This invention is a support system for the safe movement of visually impaired people. The system is composed of a wearable camera, a location information system, a map information system, generative AI, 5G communication, a natural language feedback system, and an emotion engine.

[0406] The user wears a head-mounted display on their head, and a wearable camera captures high-resolution images of the surrounding environment. The camera continues to capture video data in real time. The device receives the video data from the wearable camera and sends it to a server using 5G communication. Using 5G communication enables high-speed data transfer with low latency.

[0407] The server receives the GPS data sent from the device and identifies the user's current location. The server then calls the map information system to obtain map information around the user's current location. The map information includes details of roads, buildings, obstacles, etc.

[0408] The server uses generative AI to analyze video data and map information. The server generates information about obstacles around the user and route guidance. For example, the server generates information such as, "There are stairs ahead. Please be careful."

[0409] The server uses a natural language feedback system to convert the analysis results into a voice message. The voice message is generated in a format that is easy for the user to understand. The server uses an emotion engine to analyze the user's stress level and emotional state and adjust the content and tone of the feedback. For example, if the user is nervous, the server generates a message in a calm tone.

[0410] The terminal receives the voice message sent from the server and conveys it to the user, allowing the user to travel safely while listening to the voice guidance.

[0411] Examples:

[0412] Imagine a user is walking on a train platform. The user wears a head-mounted display, and the wearable camera captures video of the platform. The device sends the video data to a server via 5G communication. The server receives GPS data and identifies the user's current location. The server calls the Google Maps API to obtain map information for the station platform. The server uses generative AI to detect that the user is approaching the edge of the platform, and generates a voice message saying, "You are approaching the edge of the platform. Please be careful." If the emotion engine determines that the user's stress level is low, the message is delivered in a calm tone. The device delivers the voice message to the user, allowing them to move safely.

[0413] Example prompt sentence:

[0414] "As a user walks along a station platform, a wearable camera captures video of the platform, and the device transmits the video data to a server via 5G. The server receives GPS data and identifies the user's current location. The server then calls the Google Maps API to obtain map information for the station platform. Using generative AI, the server detects when the user is approaching the edge of the platform and generates a voice message to alert the user. If the emotion engine determines that the user's stress level is low, the message is delivered in a calming tone."

[0415] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0416] Step 1:

[0417] The user wears a head-mounted display, and a wearable camera captures high-resolution images of the surrounding environment.

[0418] Input: The user's surroundings

[0419] Output: High-resolution video data

[0420] Specific operation: The camera continues to capture video data in real time.

[0421] Step 2:

[0422] The device receives video data from the wearable camera and transmits it to a server using 5G communication.

[0423] Input: Video data from a wearable camera

[0424] Output: Video data sent to the server

[0425] Specific operation: The device uses 5G communication to transfer data at high speed with low latency.

[0426] Step 3:

[0427] The server receives the GPS data sent from the device and identifies the user's current location.

[0428] Input: GPS data from the device

[0429] Output: Current location of the user

[0430] What happens: The server analyzes the GPS data and determines the user's exact location.

[0431] Step 4:

[0432] The server calls the map information system and obtains map information about the area around the user's current location.

[0433] Input: current location of the user

[0434] Output: Map information of the surrounding area

[0435] Specific operation: The server uses the map information system to obtain detailed map information.

[0436] Step 5:

[0437] The server uses generative AI to analyze the video data and map information.

[0438] Input: Video data, map information

[0439] Output: Analysis results (obstacles and route guidance information)

[0440] Specific operation: The server uses generative AI to generate obstacles and route guidance information around the user.

[0441] Step 6:

[0442] The server uses a natural language feedback system to convert the analysis results into a voice message.

[0443] Input: Analysis results

[0444] Output: Voice message

[0445] Specific operation: The server converts the analysis results into natural language and generates a voice message in a format that is easy for the user to understand.

[0446] Step 7:

[0447] The server uses an emotion engine to analyze the user's stress level and emotional state and adjust the content and tone of the feedback.

[0448] Input: User's emotional state

[0449] Output: Modified voice message

[0450] Specific behavior: The server uses an emotion engine to adjust the content and tone of the feedback depending on the user's emotional state.

[0451] Step 8:

[0452] The terminal receives the voice message sent from the server and conveys it to the user.

[0453] Input: Voice message from the server

[0454] Output: Audio feedback to the user

[0455] Specific operation: The device plays a voice message, allowing the user to travel safely while listening to the voice guidance.

[0456] (Application example 1)

[0457] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0458] When visually impaired people use self-driving vehicles, they need a system that can help them get in and out of the vehicle safely and travel to their destination. However, current technology makes it difficult for visually impaired people to accurately grasp their surroundings and provide them with real-time information to travel safely. There is also a lack of technology that provides appropriate feedback that takes into account the user's emotional state. This poses a challenge in ensuring the safety and convenience of self-driving vehicles for visually impaired people.

[0459] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as low-latency, real-time information utilizing 5G communication; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted persons; means for analyzing the user's emotional state using an emotion engine and providing appropriate feedback; and means for supporting visually impaired persons in safely getting in and out of autonomous vehicles. This enables visually impaired persons to travel safely and efficiently when using autonomous vehicles.

[0460] "Visually impaired persons" refer to people who have visual impairments and are unable to fully utilize visual information.

[0461] A "wearable camera" is a camera that can be worn by the user, and is primarily used by attaching it to the head or body.

[0462] A "location information system" is a system that uses technology such as GPS to determine a user's current location.

[0463] A "map information system" is a system that provides geographical information, including detailed map data such as roads and buildings.

[0464] "Generative AI" is a system that uses artificial intelligence technology to generate new information and analytical results from input data.

[0465] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0466] "Natural language feedback" is a means of providing information to users using natural language such as voice or text.

[0467] An "emotion engine" is a system that analyzes the user's emotional state and provides appropriate feedback based on the results.

[0468] An "autonomous vehicle" is a vehicle that drives itself automatically using artificial intelligence and sensor technology.

[0469] "Analyzing space" means identifying the surrounding environment and the location of obstacles based on acquired image information and location information, and providing the user with the necessary information.

[0470] "Timely risk identification" means detecting surrounding risks in real time and providing information to users promptly.

[0471] "Providing feedback" means giving the user necessary information or instructions via voice or text.

[0472] The system for implementing this invention supports safe and efficient travel for visually impaired people when using autonomous vehicles. The system is composed of a wearable camera, a location information system, a map information system, generative AI, 5G communications, a natural language feedback system, and an emotion engine.

[0473] System configuration

[0474] 1. Wearable Camera:

[0475] It is a camera worn on the head of a visually impaired person that captures the surrounding environment in real time.

[0476] 2. Location Information Systems:

[0477] A GPS module is used to determine the current location of the visually impaired person.

[0478] 3. Map Information System:

[0479] It provides highly accurate map data and calculates the optimal route from your current location to your destination.

[0480] 4. Generative AI:

[0481] It analyzes image information acquired by a wearable camera and recognizes the surrounding environment and obstacles.

[0482] Based on the analysis results, the optimal route is generated to enable visually impaired people to travel safely.

[0483] 5. 5G communications:

[0484] It enables high-speed, low-latency communication, sending and receiving information in real time.

[0485] 6. Natural Language Feedback System:

[0486] It provides real-time audio feedback to visually impaired people, helping them move around safely.

[0487] 7. Emotion Engine:

[0488] It analyzes the user's emotional state and provides appropriate feedback to reduce stress and anxiety.

[0489] System Operation

[0490] The server uses generative AI to analyze image information acquired by the wearable camera and recognize the surrounding environment and obstacles. It uses a location information system and map information system to calculate the optimal route from the current location to the destination. It utilizes 5G communications to provide real-time voice feedback to visually impaired people. It uses an emotion engine to analyze the user's emotional state and provide appropriate feedback, allowing visually impaired people to travel safely.

[0491] Specific examples

[0492] For example, when a visually impaired person gets into an autonomous vehicle, a wearable camera will analyze the surrounding environment, and if there is an obstacle, a voice message will be sent saying, "There is an obstacle ahead. Please be careful." Also, when the vehicle arrives at its destination, the GPS and map information system will be used to say, "You have arrived at your destination. Please get off the vehicle."

[0493] Prompt Sentence Examples

[0494] Develop an application that assists visually impaired people when getting in and out of autonomous vehicles. This application will combine a wearable camera, a location information system, a map information system, generative AI, 5G communications, a natural language feedback system, and an emotion engine. Specifically, the wearable camera will capture images of the surrounding environment, and the generative AI will analyze the images to identify obstacles and provide route guidance. GPS will be used to identify the current location, and in conjunction with the map information system, the application will guide the user to their destination. Real-time voice feedback will be provided to visually impaired people to help them navigate safely. The application will analyze the user's emotional state and provide appropriate feedback to reduce stress and anxiety.

[0495] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0496] Step 1:

[0497] The server receives image information captured in real time from the wearable camera. The input is video data from the camera, and the output is preprocessed image data that is passed to the generative AI model. Specifically, the server divides the video data into frames, removes noise, and adjusts the resolution.

[0498] Step 2:

[0499] The server inputs the preprocessed image data into the generative AI model, which analyzes the surrounding environment and obstacles. The input is the preprocessed image data, and the output is the analysis results, which include environmental information and obstacle location data. Specifically, the generative AI model analyzes the image data and uses object recognition algorithms to identify obstacles and important landmarks.

[0500] Step 3:

[0501] The server receives current location data obtained from the location information system. The input is location data from the GPS module, and the output is current location data passed to the map information system. Specifically, the server analyzes the GPS data and extracts latitude and longitude information.

[0502] Step 4:

[0503] The server uses a map information system to calculate the optimal route from the current location to the destination. The input is current location data and destination data, and the output is optimal route information. Specifically, the server refers to a map database and calculates the shortest route using a route calculation algorithm.

[0504] Step 5:

[0505] The server uses 5G communications to send analysis results and route information to the visually impaired person's device in real time. The input is the analysis results and route information, and the output is feedback data sent to the visually impaired person's device. Specifically, the server divides the data into packets and transmits them with low latency.

[0506] Step 6:

[0507] The device converts the received feedback data into speech using a natural language feedback system and provides it to the visually impaired. The input is feedback data, and the output is speech feedback. Specifically, the device inputs text data into a speech synthesis engine to generate speech data.

[0508] Step 7:

[0509] The server uses an emotion engine to analyze the user's emotional state and provide appropriate feedback. The input is the user's biometric and behavioral data, and the output is feedback data based on the emotional state. Specifically, the server analyzes data from biometric sensors and evaluates the user's stress and anxiety levels.

[0510] Step 8:

[0511] The device receives feedback data from the emotion engine and provides appropriate voice feedback to the visually impaired. The input is the feedback data from the emotion engine, and the output is voice feedback. Specifically, the device inputs the analysis results of the emotion engine into a voice synthesis engine to generate a voice message that gives the user a sense of security.

[0512] Example 2

[0513] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0514] For visually impaired people to travel safely and efficiently, they need a system that can acquire information about their surroundings and their location in real time, and provide optimal routes based on that information. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there are a lack of concrete means for visually impaired people to travel independently without the help of sighted people. This makes it difficult for visually impaired people to travel safely, and they are subject to many restrictions in their daily lives.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0516] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; means for generating an optimal route based on the information analyzed by the generative AI and providing this to the user via a terminal; and means for the user to receive information for safe movement through audio guidance and vibration feedback. This enables visually impaired people to obtain information about their surrounding environment in real time and move safely and efficiently.

[0517] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining visual information.

[0518] A "wearable camera" refers to a camera device that can be worn by a user and captures the surrounding environment from the user's point of view.

[0519] A "location information system" refers to a system that uses technology such as GPS to accurately obtain a user's current location.

[0520] "Map information system" refers to a system that provides highly accurate map information corresponding to the user's location.

[0521] "Generative AI" refers to artificial intelligence that analyzes space based on acquired image information, location information, and map information, and generates the optimal route.

[0522] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables low-latency, high-speed data communication.

[0523] "Natural language feedback" refers to providing information to users in language that humans can understand.

[0524] An "optimal route" refers to the most suitable travel route for a visually impaired person to reach their destination safely and efficiently.

[0525] "Terminal" refers to a device used by a user that receives information from a server and provides it to the user.

[0526] "Audio guide" refers to a means of providing information to a user using audio.

[0527] "Vibration feedback" refers to a means of providing information to a user by causing a device to vibrate.

[0528] This invention is a system for enabling visually impaired people to move around safely and efficiently, and is realized by combining technologies such as wearable cameras, location information systems, map information systems, generative AI, 5G communications, terminals, audio guides, and vibration feedback.

[0529] First, the user wears a wearable camera, which captures images of the surrounding environment from the user's point of view in real time. The device acquires the image information sent from the wearable camera and sends it to the server.

[0530] The server uses a location information system to accurately obtain the user's current location, and also uses a map information system to obtain highly accurate map information corresponding to the user's location.

[0531] The server sends the acquired image information, location information, and map information to a generative AI (e.g., OpenAI GPT-4), which uses this information to analyze the space and generate the optimal route for the visually impaired person.

[0532] The generated route information is sent from the server to the device, which then provides the received route information to the user. The user can then travel safely along the optimal route through audio guidance and vibration feedback.

[0533] As a concrete example, consider a scenario in which a user travels from the station to their home. The user wears a wearable camera on the station platform and takes pictures of their surroundings. The device acquires image information from the station platform and sends it to a server. The server uses GPS to acquire the user's current location (station platform). The server uses a map API to acquire highly accurate map information about the area around the station. Based on this information, a generative AI generates the optimal route from the station to their home. The server sends the generated route information to the device, and the device instructs the user via voice guidance, saying, "Leave the platform, turn right, and use the elevator."

[0534] Example prompt sentence:

[0535] "Generate the optimal route for a visually impaired person to move safely within the park. Please perform your analysis based on the following information: Image information: {Image data}, Location information: {Latitude and longitude}, Map information: {Map data}"

[0536] In this way, visually impaired people can obtain information about their surroundings in real time and move around safely and efficiently.

[0537] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0538] Step 1:

[0539] A user wears a wearable camera and captures the surrounding environment.

[0540] Input: Real-time video from the user's point of view

[0541] Output: Image information captured by the wearable camera

[0542] Specific operation: The user wears a wearable camera on their head or chest and starts walking. The camera continuously captures the surrounding environment from the user's point of view.

[0543] Step 2:

[0544] The terminal acquires the captured image information and sends it to the server.

[0545] Input: Image information sent from the wearable camera

[0546] Output: Image information sent to the server

[0547] Specific operation: The device receives image information sent in real time from the wearable camera and transmits it to the server with low latency using 5G communication.

[0548] Step 3:

[0549] The server obtains the user's precise location information from a location information system.

[0550] Input: Location data from a location information system

[0551] Output: Current location of the user (latitude and longitude)

[0552] Specific operation: The server accesses the location information system to obtain the user's current location. The obtained location information is stored on the server in the form of latitude and longitude.

[0553] Step 4:

[0554] The server obtains highly accurate map information from the map information system.

[0555] Input: Current location of the user (latitude and longitude)

[0556] Output: High-precision map information corresponding to the user's location

[0557] Specific operation: The server accesses the map information system and obtains detailed map information corresponding to the user's current location. The map information includes information on roads, buildings, obstacles, etc.

[0558] Step 5:

[0559] The server sends image information, location information, and map information to the generative AI.

[0560] Input: Image information, location information, map information

[0561] Output: Data sent to the generative AI

[0562] Specific operation: The server sends the acquired image information, location information, and map information as a single dataset to the generative AI, which is then ready to begin analysis.

[0563] Step 6:

[0564] Based on this information, generative AI analyzes the space and generates the optimal route.

[0565] Input: Image information, location information, map information

[0566] Output: Optimal route information

[0567] How it works: The generative AI analyzes the transmitted data and generates the optimal route for the visually impaired. The analysis includes detecting obstacles and selecting a safe route.

[0568] Step 7:

[0569] The server transmits the generated route information to the terminal.

[0570] Input: Optimal route information from generative AI

[0571] Output: Route information sent to the terminal

[0572] Specific operation: The server sends the optimal route information received from the generative AI to the terminal. The terminal receives this information and prepares to provide it to the user.

[0573] Step 8:

[0574] The terminal provides the route information to the user.

[0575] Input: Route information sent from the server

[0576] Output: Route information provided to the user

[0577] Specific operation: The device provides the received route information to the user through voice guidance and vibration feedback, allowing the user to travel safely based on this feedback.

[0578] (Application example 2)

[0579] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0580] Visually impaired people face challenges in navigating physical stores safely and efficiently when searching for products or moving to specific locations. It is also difficult for visually impaired people to understand their surroundings and identify risks in a timely manner. For this reason, there is a demand for a system to assist visually impaired people in navigating physical stores.

[0581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0582] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; and means for providing navigation through audio guidance and tactile feedback when visually impaired people search for products in physical stores. This enables visually impaired people to move safely and efficiently within physical stores, understand their surrounding environment, and assess risks in a timely manner.

[0583] "Visually impaired persons" refer to people who have visual impairments and have difficulty using visual information.

[0584] A "wearable camera" is a camera device that can be worn on the body and is used to capture images of the surrounding environment from the perspective of a visually impaired person.

[0585] A "location information system" is a system that provides accurate location information using GPS or other technologies.

[0586] A "map information system" is a system that provides highly accurate map information corresponding to a specific location.

[0587] "Generative AI" is an artificial intelligence that analyzes space based on acquired image information, location information, and map information, and generates the optimal route.

[0588] "5G communication" refers to the fifth generation mobile communication system, a technology that provides low latency and excellent real-time communication.

[0589] "Means of providing feedback in natural language" refers to a means of conveying information analyzed by generative AI to users in natural language.

[0590] "Sighted person" refers to a person who has no visual impairment and can use normal visual information.

[0591] "Guidance to destination" refers to providing a route for a visually impaired person to reach their destination.

[0592] "Timely understanding of risks in the surrounding environment" refers to visually impaired people being able to recognize risks in their surrounding environment in real time.

[0593] A "physical store" is a store that exists in a physical location and is a facility for selling products.

[0594] An "audio guide" is a means of providing information in audio form to visually impaired people.

[0595] "Haptic feedback" is a means of providing information to visually impaired people through the sense of touch.

[0596] "Means for providing navigation" refers to means for providing route guidance to help visually impaired people reach their destination.

[0597] This invention is a system that provides navigation through audio guidance and haptic feedback to visually impaired people as they move safely and efficiently through a physical store and search for products. Specific embodiments for realizing this system are described below.

[0598] System Configuration

[0599] Hardware

[0600] Smart glasses: Equipped with a wearable camera and GPS module, they capture images of the surrounding environment from the perspective of the visually impaired and acquire location information.

[0601] Server: Runs generative AI and analyzes image, location, and map information to generate the optimal route.

[0602] Communications network: Utilizing 5G communications, data communications will be performed with low latency and excellent real-time performance.

[0603] software

[0604] OpenCV: A library for image processing.

[0605] requests: A library for sending HTTP requests.

[0606] geopy: A library for working with geolocation data.

[0607] Processing flow

[0608] 1. Image acquisition: The camera in the smart glasses captures the surrounding environment and acquires image data.

[0609] 2. Location information acquisition: The GPS module acquires the current location information.

[0610] 3. Map information acquisition: Obtain highly accurate map information based on location information.

[0611] 4. Optimal route generation: The acquired image information, location information, and map information are sent to the server, and the generative AI generates the optimal route.

[0612] 5. Audio guide generation: Generate audio guide based on the generated route and provide it to visually impaired people.

[0613] 6. Haptic Feedback: Provide haptic feedback when needed to help visually impaired people navigate safely.

[0614] Specific examples

[0615] When a visually impaired person searches for a specific product in a physical store, the smart glasses capture real-time images of the surrounding environment and the GPS module acquires their location. This data is sent to a server, and generative AI generates an optimal route. The generated route is provided to the visually impaired person as an audio guide, and haptic feedback is also provided if necessary.

[0616] Prompt Sentence Examples

[0617] "Create an AI model that generates the optimal route for a visually impaired person to find a product in a physical store. As input, use images from a wearable camera, location information from a GPS, and high-precision map information. As output, generate an audio guide."

[0618] In this way, the present invention provides a system that greatly assists visually impaired people in safely and efficiently navigating and locating products within a brick-and-mortar store.

[0619] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0620] Step 1:

[0621] Image acquisition

[0622] The device (smart glasses) uses a wearable camera to capture the surrounding environment from the point of view of the visually impaired person. The input is real-time video from the camera, and the output is captured image data, which is used in subsequent processing steps.

[0623] Step 2:

[0624] Location information acquisition

[0625] The device (smart glasses) acquires its current location information using a GPS module. The input is a GPS signal, and the output is precise location information including latitude and longitude. This location information is used to acquire map information and generate routes.

[0626] Step 3:

[0627] Get map information

[0628] The server obtains highly accurate map information based on the acquired location information. The input is location information including latitude and longitude, and the output is detailed map data corresponding to that location. This map data is used for route analysis by generative AI.

[0629] Step 4:

[0630] Optimal Route Generation

[0631] The server sends the acquired image information, location information, and map information to the generative AI to generate the optimal route. The input is image data, location information, and map data, and the output is optimal route information for the visually impaired. The generative AI analyzes this data and calculates a route that the visually impaired can travel safely and efficiently.

[0632] Step 5:

[0633] Audio guide generation

[0634] The server generates an audio guide based on the generated route information. The input is the optimal route information, and the output is an audio guide provided to the visually impaired person. This audio guide includes instructions for the visually impaired person to reach their destination.

[0635] Step 6:

[0636] haptic feedback

[0637] The device (smart glasses) provides haptic feedback as needed. The input is optimal route information and real-time environmental information, and the output is haptic feedback for the visually impaired. This haptic feedback provides additional information to enable the visually impaired to navigate safely.

[0638] In this way, a system is realized that allows visually impaired people to move safely and efficiently within a physical store and navigate through it using audio guidance and haptic feedback as they search for products.

[0639] Example 3

[0640] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0641] For visually impaired people to travel safely and efficiently, it is necessary to obtain information about the surrounding environment and route in real time and receive appropriate feedback. However, conventional systems are unable to respond appropriately when visually impaired people feel anxious, and do not ensure sufficient freedom of movement or safety. In addition, there are issues with the difficulty of guiding people to their destination or identifying risks in the surrounding environment in a timely manner without the intervention of sighted people.

[0642] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for analyzing space using a generative AI model based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system; means for incorporating user emotional information acquired from an emotion engine into the analysis; means for providing feedback to the user in natural language as low-latency, real-time information utilizing 5G communication; means for providing guidance to a destination and timely identification of risks in the surrounding environment without the intervention of sighted people; and means for adjusting the content of the feedback based on the user's emotional information. This allows visually impaired people to receive appropriate feedback even when they feel anxious, significantly improving their freedom of movement and safety.

[0643] A "wearable camera" is a portable camera device that can be worn by a user and is a device for acquiring image information of the surroundings in real time.

[0644] A "location information system" is a system for accurately identifying a user's current location, and generally refers to the GPS (Global Positioning System).

[0645] A "map information system" is a system that provides geographical information, including highly accurate map data for providing a user's current location and route information to a destination.

[0646] A "generative AI model" is an algorithm or model that uses artificial intelligence to analyze data and generate results for a specific task.

[0647] An "emotion engine" is a system for recognizing a user's emotional state and analyzing that information, and is used to provide appropriate feedback based on the user's emotions.

[0648] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0649] "Feedback in natural language" means providing feedback using language that is commonly used in daily life in order to provide information in a form that is easy for users to understand.

[0650] "Analyzing space" means understanding the surrounding environment and routes based on acquired image information, location information, and map information, and generating appropriate instructions and information.

[0651] "Timely risk identification" means recognizing in real time the risks that users may face while traveling and taking appropriate measures.

[0652] This invention is a system for safe and efficient mobility for visually impaired people, utilizing wearable cameras, location information systems, map information systems, generative AI models, emotion engines, and 5G communications.

[0653] Server Roles

[0654] The server receives image information sent from the wearable camera. Specifically, it acquires image data in real time from the wearable camera (e.g., a portable camera device) worn by the user. It also receives data from location information systems and map information systems (e.g., a geographic information system). Based on this data, the server uses a generative AI model (e.g., an artificial intelligence algorithm) to analyze the space and generate optimal route information for the visually impaired person.

[0655] In addition, the server also incorporates user emotional information obtained from an emotion engine (e.g., a system that recognizes the user's emotional state) into the analysis. For example, if the user is feeling anxious, the generative AI model will generate more detailed route information, allowing the user to travel with peace of mind.

[0656] Device Role

[0657] The device receives route information sent from the server in real time via 5G communication. 5G communication is a technology that enables high-speed, low-latency communication, allowing visually impaired people to quickly receive the information they need. The device then feeds the received route information back to the user via an audio output device (e.g., earphones). The feedback is provided in natural language and in a format that is easy for visually impaired people to understand. For example, instructions might be given such as, "There is an obstacle five meters ahead. Move two meters to the right."

[0658] User Roles

[0659] Users wear a wearable camera and receive real-time route information while moving. The emotion engine recognizes the user's emotions and adjusts the feedback accordingly. For example, if the user is feeling anxious, the emotion engine conveys that information to the generative AI model, which then generates more detailed route information. This allows users to be guided to their destination and to be aware of risks in the surrounding environment in a timely manner without the need for a sighted person.

[0660] Specific examples

[0661] Hardware and software used

[0662] Wearable cameras (e.g., handheld camera devices)

[0663] Location information systems (e.g. GPS)

[0664] Map information systems (e.g., geographical information systems)

[0665] Generative AI models (e.g., artificial intelligence algorithms)

[0666] Emotion engines (e.g., systems that recognize a user's emotional state)

[0667] 5G communications infrastructure

[0668] Prompt Sentence Examples

[0669] "If the user is concerned, generate a detailed route. Current location: Latitude 35.6895, Longitude 139.6917. Obstacle ahead."

[0670] This system enables visually impaired people to be guided to their destination and to grasp risks in the surrounding environment in a timely manner without the intervention of sighted people. This significantly improves the freedom and safety of movement for visually impaired people. The flow of the identification process in Example 3 will be explained using Figure 21.

[0671] Step 1: Receiving data

[0672] The server receives image information sent from the wearable camera. The input is real-time image data from the wearable camera, and the output is image data stored in the server. Specifically, the server analyzes the data stream from the camera, extracts image frames, and stores them.

[0673] Step 2: Receiving location and map information

[0674] The server receives data from a GPS and a map information system. The input is the current location data from the GPS and the map data from the map information system, and the output is the location information and map information stored in the server. Specifically, the server analyzes the GPS data, identifies the user's current location, and combines it with the map data.

[0675] Step 3: Acquiring emotional information

[0676] The server obtains the user's emotional information from the emotion engine. The input is the emotional data from the emotion engine, and the output is the emotional information stored in the server. Specifically, the server analyzes the data from the emotion engine and determines whether the user is feeling anxious.

[0677] Step 4: Spatial analysis and route generation

[0678] The server uses a generative AI model to analyze the space based on the received image information, location information, map information, and emotion information, and generates route information. The input is image data, location data, map data, and emotion data, and the output is the generated route information. Specifically, the server sends a prompt to the generative AI model to generate detailed route information. For example, "If the user feels anxious, please generate detailed route information. The current location is latitude 35.6895, longitude 139.6917. There is an obstacle ahead."

[0679] Step 5: Prepare for feedback

[0680] The server converts the generated route information into natural language and prepares feedback in a form that is easy for visually impaired people to understand. The input is the generated route information, and the output is feedback information expressed in natural language. Specifically, the server analyzes the route information and generates instructions such as, "There is an obstacle 5 meters ahead. Please move 2 meters to the right."

[0681] Step 6: Submit your feedback

[0682] The server transmits the prepared feedback information to the terminal via 5G communication. The input is the feedback information expressed in natural language, and the output is the feedback information transmitted to the terminal. Specifically, the server transmits the feedback information to the terminal with low latency using the 5G communication infrastructure.

[0683] Step 7: Receiving and Providing Feedback

[0684] The terminal receives feedback information transmitted from the server and provides it to the user through an audio output device. The input is the feedback information transmitted from the server, and the output is the feedback information provided in audio. Specifically, the terminal converts the received feedback information into audio using speech synthesis technology and transmits it to the user through earphones.

[0685] Step 8: Move users

[0686] The user receives voice feedback from the device and moves according to the instructions. The input is the feedback information provided by voice, and the output is the user's movement behavior. Specifically, the user moves according to instructions such as "There is an obstacle 5 meters ahead. Please move 2 meters to the right."

[0687] (Application example 3)

[0688] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0689] When visually impaired people use self-driving vehicles, it is difficult to provide them with real-time guidance to their destination and information about risks in the surrounding environment. In addition, it is difficult for them to travel safely because the feedback cannot be adjusted according to their emotional state.

[0690] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as low-latency, real-time information utilizing 5G communication; means for providing guidance to a destination and timely risk assessment of the surrounding environment without the intervention of sighted persons; means for recognizing the emotional state of the visually impaired person and adjusting the feedback content according to that emotion; and means for being installed in an autonomous vehicle and providing guidance to a destination and risk information of the surrounding environment in real time when a visually impaired person uses an autonomous vehicle. This enables visually impaired people to travel to their destination safely when using an autonomous vehicle.

[0691] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining visual information.

[0692] A "wearable camera" is a camera device that can be worn on the body and is a device for acquiring image information from the user's point of view.

[0693] A "location information system" is a system that uses technology such as GPS to accurately obtain a user's current location.

[0694] A "map information system" is a system that provides geographical information, including highly accurate map data.

[0695] "Generative AI" is an artificial intelligence system that performs spatial analysis and route generation based on acquired data.

[0696] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0697] "Feedback in natural language" refers to providing information in natural language that is easy for users to understand.

[0698] "Sighted person" refers to a person who has no visual impairment and can normally obtain visual information.

[0699] "Emotional state" refers to a user's current emotional or psychological state.

[0700] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0701] "Providing in real time" means providing information immediately without delay.

[0702] The system for implementing this invention provides real-time guidance to a visually impaired person using an automated driving vehicle to guide them to their destination and provide information about risks in the surrounding environment. Specific embodiments of this system are described below.

[0703] System configuration

[0704] 1. Hardware:

[0705] Wearable camera: A device worn by visually impaired people to obtain image information of their surroundings.

[0706] GPS module: A device that accurately determines the current location of a visually impaired person.

[0707] 5G communication module: A device that enables high-speed, low-latency communication.

[0708] Autonomous vehicle: A vehicle that uses artificial intelligence and sensor technology to drive autonomously.

[0709] 2. Software:

[0710] Generative AI: An artificial intelligence system that analyzes space and generates optimal routes based on image information acquired by a wearable camera, location information from a GPS module, and highly accurate map information from a map information system.

[0711] Emotion recognition engine: Software that recognizes the emotional state of visually impaired people and provides that information to generative AI.

[0712] Natural language processing engine: Software that provides feedback in natural language on generated route information and risk information.

[0713] Map information system: A system that provides highly accurate map data.

[0714] Data processing and calculation

[0715] 1. Image Analysis:

[0716] Generative AI analyzes image information obtained from the wearable camera and recognizes surrounding obstacles and terrain.

[0717] For example, if there is an obstacle ahead, the distance and direction of the obstacle can be identified.

[0718] 2. Location information acquisition:

[0719] The current location is obtained from the GPS module and linked to the map information system to determine the current location.

[0720] 3. Route generation:

[0721] The generative AI generates the optimal route based on the acquired image information, location information, and map information.

[0722] The level of detail of route information is adjusted according to the emotional state of the visually impaired person.

[0723] 4. Feedback:

[0724] A natural language processing engine provides feedback in natural language on the generated route and risk information.

[0725] For example, it provides instructions such as, "There is an obstacle 5 meters ahead. Move 2 meters to the right."

[0726] Specific examples

[0727] If a visually impaired person is in an autonomous vehicle and encounters an obstacle ahead while heading to their destination, the system will provide voice instructions such as, "There is an obstacle 5 meters ahead. Please move 2 meters to the right." Additionally, if the visually impaired person feels anxious, the emotion recognition engine will convey that information to the generative AI, which will then generate more detailed route information.

[0728] Prompt Sentence Examples

[0729] "If the user feels anxious, generate more detailed route information. For example, provide instructions like, 'There is an obstacle 5 meters ahead. Go 2 meters to the right.'"

[0730] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0731] Step 1:

[0732] A wearable camera acquires image information of the visually impaired person's surroundings. The input is image data from the camera, and the output is raw data for image analysis. The camera captures images in real time from the visually impaired person's point of view.

[0733] Step 2:

[0734] The GPS module obtains the current location of the visually impaired person. The input is the GPS signal and the output is the coordinate data of the current location. The GPS module receives signals from satellites and identifies the exact location of the visually impaired person.

[0735] Step 3:

[0736] The server obtains highly accurate map data from the map information system. The input is the coordinate data of the current location, and the output is map information of the surrounding area. The server accesses the map information system and obtains detailed map data based on the current location.

[0737] Step 4:

[0738] The generative AI analyzes space based on image information from the wearable camera, location information from the GPS module, and map information from the map information system. The inputs are image data, location data, and map data, and the output is spatial information as the analysis result. The generative AI integrates this data and recognizes surrounding obstacles and terrain.

[0739] Step 5:

[0740] The emotion recognition engine analyzes image information acquired from a wearable camera to recognize the emotional state of the visually impaired person. The input is image data, and the output is emotional state data. The emotion recognition engine analyzes the facial expressions and movements of the visually impaired person to identify their current emotional state.

[0741] Step 6:

[0742] The generative AI receives emotional state data from the emotion recognition engine and adjusts the level of detail in the route information. The input is emotional state data, and the output is adjusted route information. The generative AI generates more detailed route information according to the emotional state of the visually impaired person.

[0743] Step 7:

[0744] The natural language processing engine provides feedback in natural language on the generated route and risk information. The input is route and risk information, and the output is a feedback message in natural language. The natural language processing engine provides the information in a form that is easy for visually impaired people to understand.

[0745] Step 8:

[0746] The 5G communication module sends feedback messages to the visually impaired in real time. The input is the feedback message, and the output is audio feedback to the visually impaired. The 5G communication module sends feedback with low latency, allowing the visually impaired to receive the information instantly.

[0747] Step 9:

[0748] A visually impaired person receives feedback and navigates safely in an autonomous vehicle. The input is audio feedback, and the output is safe navigation for the visually impaired person. The visually impaired person navigates to their destination based on the information provided.

[0749] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0750] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0751] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0752] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0753] [Second embodiment]

[0754] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0755] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0756] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0757] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0758] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0759] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0760] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0761] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0762] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0763] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0764] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0765] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0766] "Example 1"

[0767] In one embodiment of the present invention, a wearable camera for visually impaired people is used to capture images of the surrounding environment. This wearable camera is part of a head-mounted display that can be worn on the head of a visually impaired person.

[0768] "Example 2"

[0769] The captured image information is then sent to a generative AI system along with precise location information obtained from a location information system and highly accurate map information obtained from a map information system. Based on this information, the generative AI system analyzes the space and generates the optimal route for the visually impaired person to travel.

[0770] "Example 3"

[0771] The generated route information is fed back to the visually impaired person in real time using 5G communications. The feedback is provided in natural language and in a format that is easy for the visually impaired to understand. For example, instructions might be given such as, "There is an obstacle five meters ahead. Move two meters to the right."

[0772] "Example 4"

[0773] This system will enable visually impaired people to receive guidance to their destination and to be aware of risks in their surroundings in a timely manner without the need for sighted people, greatly improving the freedom and safety of movement for visually impaired people.

[0774] The processing flow of each embodiment will be described below.

[0775] "Example 1"

[0776] Step 1: A visually impaired person wears a head-mounted display that incorporates a wearable camera to capture images of the surrounding environment from the visually impaired person's perspective.

[0777] Step 2: Image information from the wearable camera is analyzed in conjunction with a location information system and a map information system. The location information system uses GPS and other technologies to provide the visually impaired person with accurate location information, and the map information system provides highly accurate map information corresponding to that location. Step 3: The generative AI analyzes the space based on this information and generates the optimal route for the visually impaired person to travel.

[0778] "Example 2"

[0779] Step 1: The generated route information is fed back to the visually impaired person in real time using 5G communications.

[0780] Step 2: Feedback is given in natural language and in a form that is easy for a visually impaired person to understand, such as "There is an obstacle 5 meters ahead. Move 2 meters to the right."

[0781] Step 3: The visually impaired person can navigate according to this feedback and reach their destination safely.

[0782] Example 1

[0783] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0784] For visually impaired people to travel safely and efficiently, it is important to grasp information about their surroundings in real time and receive appropriate feedback. However, conventional systems do not provide enough information for visually impaired people to accurately recognize their surroundings, and they often require assistance from sighted people. In addition, due to the lack of real-time and low latency, it is difficult to grasp risks in a timely manner.

[0785] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communication; means for preprocessing video data and performing object recognition and text reading using a generative AI model; means for generating the analysis results as audio data and converting it into audio using speech synthesis software; means for transmitting the generated audio data to the visually impaired person's head-mounted display to notify them by audio; and means for providing guidance to the visually impaired person and timely identification of risks in the surrounding environment without the intervention of a sighted person. This enables the visually impaired person to accurately recognize their surrounding environment in real time and move safely and efficiently.

[0786] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0787] A "wearable camera" is a camera that can be worn by the user and is used as part of a head-mounted display worn on the head of a visually impaired person.

[0788] A "location information system" refers to a system that uses technology such as GPS to accurately identify a user's current location.

[0789] A "map information system" refers to a system that provides highly accurate map data and provides surrounding geographic information in conjunction with the user's location information.

[0790] "Generative AI" refers to a system that uses artificial intelligence technology to generate useful information from input data.

[0791] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0792] "Feedback in natural language" refers to conveying analysis results and information to users in natural language that is easy for humans to understand.

[0793] "Preprocessing" refers to processing that is performed before data analysis, and includes processes such as noise removal and resolution adjustment.

[0794] "Object recognition" refers to a technology that detects specific objects from image data and identifies their type and location.

[0795] "Text reading" refers to the technology of extracting character information from image data and recognizing it as text data.

[0796] "Speech synthesis software" refers to software for converting text data into voice data.

[0797] A "head-mounted display" is a display device worn by the user on the head, and has an audio output function for visually impaired people.

[0798] "Timely risk understanding" refers to analyzing surrounding environmental information in real time and providing users with immediate risk information.

[0799] This invention is a system that enables visually impaired people to recognize their surroundings in real time and move safely and efficiently. This system is implemented using hardware and software such as a wearable camera, a location information system, a map information system, a generative AI model, 5G communications, and speech synthesis software.

[0800] The server receives video data sent from a wearable camera worn on the head of a visually impaired person. This camera is used by visually impaired people to capture their surroundings. The server uses an image processing library such as OpenCV to preprocess the received video data. Preprocessing includes noise removal and resolution adjustment.

[0801] The server then inputs the preprocessed video data into a generative AI model, which has capabilities such as object recognition and text reading. For example, an object detection model is used to recognize objects in the video. The generative AI model generates information that helps visually impaired people understand their surroundings.

[0802] The server uses speech synthesis software to generate voice data based on the analysis results obtained from the generative AI model. The analysis results are converted into voice in a natural language that is easy for visually impaired people to understand. For example, voice data such as "There is a pedestrian ahead" or "The traffic light is red" is generated.

[0803] The generated voice data is transmitted in real time with low latency via 5G communications to the visually impaired person's head-mounted display. The visually impaired person can then understand the analysis results through a built-in speaker in the head-mounted display. This enables the visually impaired person to receive guidance to their destination and be aware of risks in their surrounding environment in a timely manner, without the need for a sighted person.

[0804] As a concrete example, consider a scene in which a visually impaired person is walking around town. The video captured by the wearable camera includes traffic lights, pedestrians, and vehicles. The server receives this video data and uses a generative AI model to analyze the color of the traffic lights, the location of pedestrians, and the movement of vehicles. The analysis results are generated as audio data, and the visually impaired person is provided with information such as "The traffic light is red," "There is a pedestrian ahead," and "A car is approaching from the right."

[0805] Example prompt sentence:

[0806] "Imagine a scene where a visually impaired person is walking around town. Explain a system that analyzes video data captured by a wearable camera and provides audio notifications about traffic light colors, pedestrian locations, and vehicle movements."

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] The device uses a wearable camera worn on the head of the visually impaired person to capture images of the surrounding environment in real time. The input is the video data captured by the camera, and the output is the captured video data. Specifically, the camera captures video at 30 frames per second.

[0810] Step 2:

[0811] The terminal transmits the captured video data to the server. The input is the captured video data, and the output is the video data transmitted to the server. In concrete terms, the terminal compresses the video data and transmits it to the server.

[0812] Step 3:

[0813] The server preprocesses the received video data. The input is the video data sent from the device, and the output is the preprocessed video data. Specifically, the server uses an image processing library such as OpenCV to remove noise and adjust the resolution. For example, it adjusts the resolution of the video data to 640x480 pixels and applies a noise filter.

[0814] Step 4:

[0815] The server inputs the preprocessed video data into the generative AI model. The input is the preprocessed video data, and the output is the analysis results by the generative AI model. Specifically, the server uses the object detection model to recognize objects in the video. For example, it analyzes the color of traffic lights, the position of pedestrians, and the movement of vehicles.

[0816] Step 5:

[0817] The server generates the analysis results obtained from the generative AI model as voice data. The input is the analysis results from the generative AI model, and the output is voice data. Specifically, the server uses voice synthesis software to convert the analysis results into voice. For example, it generates voice data such as "There is a pedestrian ahead" or "The traffic light is red."

[0818] Step 6:

[0819] The server transmits the generated voice data to the terminal. The input is the generated voice data, and the output is the voice data transmitted to the terminal. In concrete terms, the server compresses the voice data and transmits it to the terminal.

[0820] Step 7:

[0821] The device plays the received audio data. The input is the audio data sent from the server, and the output is the audio heard by the visually impaired person. Specifically, the device decodes the received audio data and plays it back from the speaker built into the head-mounted display. This allows the visually impaired person to recognize information about their surrounding environment through audio.

[0822] (Application example 1)

[0823] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0824] For visually impaired people to move around safely, it is important to accurately grasp their surroundings and receive appropriate feedback. However, conventional technologies have made it difficult for visually impaired people to recognize surrounding dangers in real time and take appropriate action. In addition, there is a lack of systems that enable visually impaired people to move around independently without the assistance of sighted people, which has hindered their independence.

[0825] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0826] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; and means for detecting dangerous situations and obstacles using a generative AI model and notifying the user through audio and haptic feedback, thereby enabling visually impaired people to recognize surrounding dangers in real time and move safely.

[0827] "Visually impaired persons" refer to people who have visual impairments and are unable to obtain sufficient visual information.

[0828] A "wearable camera" is a camera that can be worn by the user, and is primarily a device attached to a head-mounted display.

[0829] A "location information system" refers to a system that uses technology such as GPS to accurately identify a user's current location.

[0830] A "map information system" is a system that provides geographical information and includes highly accurate map data.

[0831] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate new information and predictions.

[0832] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0833] "Natural language feedback" refers to providing information in natural language that is easy for users to understand.

[0834] "Sighted person" refers to a person who has no visual impairment and can normally obtain visual information.

[0835] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task.

[0836] "Dangerous Situation" means any environment or situation that may potentially pose a hazard to the User.

[0837] "Obstacle" means any object or structure that impedes the user's movement.

[0838] "Audio feedback" refers to a means of providing information to a user using audio.

[0839] "Haptic feedback" refers to a means of providing information to a user through the sense of touch.

[0840] A system for implementing this invention includes a wearable camera worn by a visually impaired person, a location information system, a map information system, a generative AI model, 5G communication, audio feedback, and haptic feedback.

[0841] System configuration

[0842] 1. Wearable camera: Built into a head-mounted display worn by visually impaired people, it captures the surrounding environment in real time.

[0843] 2. Location systems: Using technologies such as GPS to pinpoint a user's exact location.

[0844] 3. Map information system: Provides highly accurate map data and links it to the user's location information.

[0845] 4. Generative AI model: Analyzes image information acquired by a wearable camera and detects dangerous situations and obstacles.

[0846] 5. 5G communications: Realizing high-speed, low-latency communications, processing and transmitting information in real time.

[0847] 6. Audio feedback: Notify the user of detected information via audio.

[0848] 7. Haptic feedback: Providing information to the user through touch when needed.

[0849] Program processing

[0850] The server receives image information from the wearable camera and analyzes it using a generative AI model. This analysis detects dangerous situations and obstacles. The detection results are sent to the user's device in real time via 5G communication. The user's device then provides information to the user through audio and haptic feedback.

[0851] Hardware and software used

[0852] Hardware: Head-mounted display, smartphone, GPS module

[0853] Software: OpenCV (image processing library), TensorFlow (generative AI model framework), pyttsx3 (voice synthesis library)

[0854] Specific examples

[0855] When a visually impaired person is walking down a sidewalk and sees an obstacle ahead, a wearable camera captures the image. A generative AI model analyzes the image and issues a voice notification saying, "There is an obstacle ahead." This notification is sent in real time via 5G communication, allowing the user to respond immediately.

[0856] Prompt Sentence Examples

[0857] "Detect if there are any obstacles ahead."

[0858] This prompt is used to instruct the generative AI model on a specific task.

[0859] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0860] Step 1:

[0861] The user puts on the wearable camera and starts the system.

[0862] Input: System startup via user interaction

[0863] Output: The camera begins capturing the surrounding environment in real time.

[0864] Specific operation: A user puts on the head-mounted display and turns on the system. The camera starts up and starts capturing video.

[0865] Step 2:

[0866] The video data acquired by the camera is sent to the server.

[0867] Input: Real-time video captured by a camera

[0868] Output: Video data sent to the server

[0869] Specific operation: The video data captured by the camera is sent to the server via 5G communication.

[0870] Step 3:

[0871] The video data received by the server is input into the generative AI model for analysis.

[0872] Input: Video data sent to the server

[0873] Output: Analysis results (detection of dangerous situations and obstacles)

[0874] How it works: The server preprocesses the video data it receives using OpenCV and then inputs it into a generative AI model trained with TensorFlow. The model analyzes the video and detects dangerous situations and obstacles.

[0875] Step 4:

[0876] The server sends the analysis results to the user's device via 5G communication.

[0877] Input: Analysis results from generative AI model

[0878] Output: Analysis results sent to the user's device

[0879] Specific operation: The server sends the analysis results to the user's smartphone in real time.

[0880] Step 5:

[0881] The user's device receives the analysis results and generates audio and haptic feedback.

[0882] Input: Analysis results sent from the server

[0883] Output: Audio and haptic feedback

[0884] Specific operation: The user's smartphone receives the analysis results and generates audio feedback using pyttsx3, and also generates haptic feedback if necessary.

[0885] Step 6:

[0886] The user receives audio and haptic feedback and takes appropriate action.

[0887] Input: Audio and haptic feedback

[0888] Output: User action (e.g., avoiding obstacles)

[0889] Specific actions: The user receives audio and haptic feedback and takes appropriate action, such as avoiding an obstacle ahead.

[0890] In this way, visually impaired people can recognize dangers around them in real time and move around safely.

[0891] Example 2

[0892] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0893] For visually impaired people to travel safely and efficiently, they need a system that can grasp information about their surroundings and their location in real time and guide them to the appropriate route. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there is insufficient support for visually impaired people to travel independently, and they often need the help of sighted people. This limits the freedom of movement of visually impaired people.

[0894] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image information captured by a user, means for acquiring accurate location information from a location information system, means for acquiring highly accurate map information from a map information system, means for transmitting the acquired image information, location information, and map information to a generative AI, means for the generative AI to analyze space based on this information and generate an optimal route for the visually impaired person to travel, means for transmitting the generated optimal route information to the user's terminal, and means for the user's terminal to display the received route information to the user. This enables the visually impaired person to grasp information about their surrounding environment in real time and travel safely and efficiently.

[0895] "User" refers to a visually impaired person who uses the system to take images and receive route information.

[0896] "Image information" refers to visual data captured by a user using a terminal.

[0897] A "location information system" refers to a system that uses technology such as GPS to determine a person's current location.

[0898] "Precise location information" refers to latitude and longitude data obtained from a location information system.

[0899] "Map information system" refers to a system that provides geographical information.

[0900] "High-precision map information" refers to detailed geographic data obtained from a map information system.

[0901] "Generative AI" refers to artificial intelligence that analyzes space based on the data it receives and generates the optimal route.

[0902] "Analyzing space" refers to the process of understanding the surrounding environment based on image information, location information, and map information.

[0903] An "optimal route" refers to a route that allows visually impaired people to travel safely and efficiently.

[0904] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[0905] "Route information" refers to data regarding travel routes generated by generative AI.

[0906] "Displaying" refers to the terminal providing the route information to the user visually or audibly.

[0907] This invention is a system for enabling visually impaired people to travel safely and efficiently, in which a generative AI generates an optimal route based on image information taken by the user, accurate location information obtained from a location information system, and highly accurate map information obtained from a map information system.

[0908] Hardware and software used

[0909] Location information system: GPS module

[0910] Map Information System: Geographical Information API

[0911] Generative AI: Artificial intelligence models (e.g., OpenAI GPT-4)

[0912] System configuration

[0913] server

[0914] The server receives image information taken by the user on the device. Specifically, the image data is sent to the server via an HTTP POST request. Next, the server obtains the current location information from the location information system. This is done by sending an API request to the GPS module and receiving latitude and longitude data. The server then obtains highly accurate map information from the map information system. This is done by sending a request to the geographic information provision API and receiving map data based on the specified location information.

[0915] The server sends the acquired image information, location information, and map information to the generative AI. This data is converted into JSON format and a POST request is sent to the generative AI's API endpoint. The generative AI analyzes the space based on the received information and generates the optimal route for the visually impaired person to travel. The generated optimal route information is sent from the server to the user's device.

[0916] Terminal

[0917] The device sends the image information taken by the user to the server. Specifically, the device's camera app captures the image and sends it to the server via an HTTP POST request. The device then displays the optimal route information received from the server to the user. The device application analyzes the route information and provides visual or audio guidance to the user.

[0918] User

[0919] The user takes an image using the device. Specifically, they open the camera app on their smartphone and capture the desired scene. Next, the user moves based on the optimal route information displayed on the device. They safely travel to their destination while checking the device's screen and audio guidance.

[0920] Specific examples

[0921] The user takes an image with their smartphone. The smartphone (device) sends the captured image to the server via an HTTP POST request. The server obtains the current location information from the GPS module and obtains highly accurate map information using a geographic information provision API. The server sends the image information, location information, and map information to the generative AI. The generative AI analyzes the space based on this information and generates the optimal route for the visually impaired person to travel. The server then sends the generated optimal route information to the smartphone. The smartphone displays the optimal route information to the user and provides visual and audio guidance.

[0922] Prompt Sentence Examples

[0923] "Generate the optimal route for visually impaired people based on image information taken by the user, location information obtained from GPS, and map information obtained from the map information providing API."

[0924] In this way, the system helps visually impaired people navigate safely and efficiently.

[0925] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0926] Step 1:

[0927] The server receives image information taken by the user on the device. Specifically, the user takes an image using the camera app on their smartphone, and the image data is sent to the server via an HTTP POST request. The input is the image information taken by the user, and the output is the image data stored on the server.

[0928] Step 2:

[0929] The server obtains precise location information from the location information system. Specifically, the server sends an API request to the GPS module and receives the current latitude and longitude data. The input is the location request from the GPS module, and the output is the latitude and longitude data stored on the server.

[0930] Step 3:

[0931] The server obtains highly accurate map information from the map information system. Specifically, the server sends a request to the geographic information API and receives map data based on the specified location information. The input is the request to the geographic information API, and the output is highly accurate map information stored on the server.

[0932] Step 4:

[0933] The server sends the acquired image information, location information, and map information to the generative AI. Specifically, it converts this data into JSON format and sends a POST request to the generative AI's API endpoint. The input is the image information, location information, and map information, and the output is the data sent to the generative AI.

[0934] Step 5:

[0935] Generative AI analyzes the space based on the information it receives and generates the optimal route for visually impaired people to travel. Specifically, generative AI combines image analysis algorithms with map data to calculate a route that takes into account obstacles, road width, etc. The inputs are image information, location information, and map information, and the output is information about the optimal route.

[0936] Step 6:

[0937] The server sends the generated optimal route information to the user's device. Specifically, it sends the route information received from the generative AI to the device in JSON format. The input is the optimal route information from the generative AI, and the output is the route information sent to the device.

[0938] Step 7:

[0939] The terminal displays the optimal route information received from the server to the user. Specifically, the terminal application analyzes the route information and guides the user visually or audibly. The input is the optimal route information from the server, and the output is the route information displayed to the user.

[0940] Step 8:

[0941] The user travels based on the optimal route information displayed on the device. Specifically, the user travels safely to their destination while checking the device's screen and voice guidance. The input is the route information displayed on the device, and the output is the user's travel behavior.

[0942] (Application example 2)

[0943] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0944] It is extremely difficult for visually impaired people to navigate safely and efficiently within a physical store. With conventional technology, visually impaired people require the assistance of sighted people to understand their surroundings and find the optimal route to their destination. It is also difficult to provide real-time information, and risk assessment during travel is insufficient. This limits the independent mobility of visually impaired people.

[0945] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0946] In this invention, the server includes: means for analyzing space using a generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communication; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; means for capturing images of the store interior using a camera mounted on smart glasses and for the generative AI to provide an optimal route in real time based on the location information and map information; and means for the generative AI to convert the captured image information into a list format, acquire the location information and map information in JSON format, and calculate the optimal route. This enables visually impaired people to move safely and efficiently within physical stores.

[0947] "Visually impaired people" refers to people who have visual impairments and have difficulty using visual information.

[0948] A "wearable camera" refers to a camera device that can be worn on the body.

[0949] "Image information" refers to visual data captured by a photographing device such as a camera.

[0950] A "location information system" refers to a system that uses technology such as GPS to determine the location of a specific location, such as its latitude and longitude.

[0951] A "map information system" is a system that provides geographical information and includes highly accurate map data.

[0952] "Generative AI" refers to an artificial intelligence system that analyzes and generates information based on acquired data.

[0953] "Analyzing space" refers to understanding the surrounding environment and the structure of space based on acquired image information and location information.

[0954] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[0955] "Real-time" refers to the property of data acquisition and processing occurring immediately and without delay.

[0956] "Providing feedback in natural language" means providing information in language that is easy for users to understand.

[0957] "Sighted people" refers to people who are not visually impaired and can use visual information.

[0958] "Guidance to a destination" refers to providing a route or method for a user to reach a destination.

[0959] "Timely risk assessment of the surrounding environment" means immediately recognizing potential dangers in the surrounding environment and informing users.

[0960] "Smart glasses" are eyeglass-type wearable devices equipped with functions such as a camera and display.

[0961] "List format" refers to a format in which data is represented as a list (array).

[0962] "JSON format" is an abbreviation for JavaScript Object Notation, and refers to a lightweight data exchange format that represents data in text format.

[0963] "Calculating the optimal route" refers to calculating the most efficient and safe route for a user to reach their destination.

[0964] The system for implementing this invention helps visually impaired people navigate safely and efficiently within a physical store. The system uses hardware and software such as wearable cameras, location information systems, map information systems, generative AI, 5G communications, and smart glasses.

[0965] First, the user puts on the smart glasses. The smart glasses are equipped with a camera that captures images of the store in real time. The captured image information is sent to the generative AI along with accurate location information obtained from a location information system and highly accurate map information obtained from a map information system.

[0966] Based on this information, the generative AI analyzes the space and generates the optimal route for the visually impaired. The generated route information is fed back to the user in real time with low latency using 5G communications. The feedback is provided in natural language in a format that is easy for the user to understand.

[0967] Specifically, the generative AI converts captured image information into a list format and obtains location and map information in JSON format. This allows it to calculate the optimal route and provide it to the user. For example, when a user moves through a supermarket, the generative AI analyzes the location of shelves, the width of aisles, etc., and suggests the safest and most efficient route.

[0968] An example of a prompt is, "Generate the optimal route for a visually impaired person to move safely within a supermarket. Your current location is latitude 35.6895, longitude 139.6917. Use generative AI to provide the optimal route based on image information within the store and highly accurate map information."

[0969] This system allows visually impaired people to be guided to their destination and to be aware of risks in their surroundings in a timely manner without relying on sighted people, thereby enabling them to move around independently.

[0970] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0971] Step 1:

[0972] A user wears smart glasses and moves around the store. The camera on the smart glasses captures images in real time.

[0973] Input: Real-time images of the store

[0974] Output: Captured image information

[0975] How it works: The camera in the smart glasses continuously captures images of the in-store environment and generates image data.

[0976] Step 2:

[0977] The device obtains precise location information from the location information system.

[0978] Input: Current location request

[0979] Output: Latitude and longitude location

[0980] What happens: The device uses a location information system such as GPS to determine the user's current location and obtains latitude and longitude data.

[0981] Step 3:

[0982] The terminal obtains highly accurate map information from the map information system.

[0983] Input: Current location

[0984] Output: High-precision map information

[0985] Specific operation: The device sends its current location information to the map information system and obtains detailed map data corresponding to that location.

[0986] Step 4:

[0987] The server sends the captured image information, location information, and map information to the generative AI.

[0988] Input: Image information, location information, map information

[0989] Output: Sending data to generative AI

[0990] Specific operation: The device sends the image information, location information, and map information it has acquired to the server, and the server then sends this data to the generative AI.

[0991] Step 5:

[0992] The generative AI analyzes the space based on the data it receives and generates the optimal route for visually impaired people to move around.

[0993] Input: Image information, location information, map information

[0994] Output: Optimal route information

[0995] Specific operation: The generative AI converts image information into a list format, analyzes location information and map information in JSON format, and calculates the optimal route.

[0996] Step 6:

[0997] The server uses 5G communications to provide the optimal route information generated to the user in real time.

[0998] Input: Optimal route information

[0999] Output: Natural language feedback

[1000] Specific operation: The server converts the optimal route information received from the generative AI into natural language and sends it to the user's smart glasses via 5G communication.

[1001] Step 7:

[1002] Users receive optimal route information provided in natural language through smart glasses, allowing them to move safely and efficiently within the store.

[1003] Input: Natural language feedback

[1004] Output: User movement

[1005] Specific operation: The user moves around the store following the generated optimal route while listening to audio guidance from the smart glasses.

[1006] Example 3

[1007] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1008] For visually impaired people to move safely and efficiently, they need to be able to grasp information about their surroundings and the location of obstacles in real time, and receive appropriate instructions. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there are limited means for visually impaired people to move independently without the assistance of sighted people. This has restricted the freedom and safety of their movement.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1010] In this invention, the server includes: means for analyzing space using a generative AI model based on environmental information acquired by a sensor device, accurate location information from a location information system, and highly accurate map information from a map information system; means for providing feedback to a user in natural language as information with low latency and excellent real-time performance using a communication network; means for providing guidance to a destination and timely risk assessment of the surrounding environment without the intervention of a sighted person; means for generating instructions in natural language by inputting prompt sentences into the generative AI model; and means for communicating the generated instructions to a user using a voice output function. This enables visually impaired people to grasp information about their surrounding environment in real time and move safely and efficiently.

[1011] A "sensor device" is a device for acquiring environmental information and is used to detect the situation around a visually impaired person in real time.

[1012] A "location information system" is a system that accurately identifies the current location of visually impaired people and provides location data using technologies such as GPS.

[1013] A "map information system" is a system that provides highly accurate map information and is used to calculate travel routes for visually impaired people.

[1014] A "generative AI model" is an artificial intelligence model that analyzes space based on collected data and generates instructions in natural language that are easy for visually impaired people to understand.

[1015] A "communications network" is a network for transmitting information in real time with low latency, and includes high-speed communication technologies such as 5G.

[1016] "Means for providing feedback in natural language" refers to means for providing instructions generated by a generative AI model in a form that is easy for visually impaired people to understand, and includes voice output functions, etc.

[1017] A "prompt sentence" is a sentence that is input into a generative AI model and serves as the basis for generating specific instructions for visually impaired people.

[1018] The "voice output function" is a function for conveying generated natural language instructions as voice to visually impaired people.

[1019] This invention is a system for enabling visually impaired people to move around safely and efficiently, and is implemented using sensor devices, a location information system, a map information system, a generative AI model, a communication network, and a voice output function.

[1020] The server first obtains environmental information from a sensor device. The sensor device is used to detect the visually impaired person's surroundings in real time. The server then uses a location information system to determine the visually impaired person's current location. Location information systems use technologies such as GPS. The server also obtains highly accurate map information using a map information system and calculates the visually impaired person's travel route.

[1021] The server uses this data to analyze the space using a generative AI model. The generative AI model generates instructions in natural language that are easy for visually impaired people to understand based on the collected data. At this time, a prompt sentence is input to the generative AI model. For example, the following prompt sentence can be used:

[1022] "The user is currently at the entrance to a park. There is a bench 5 meters ahead. Instruct the user to move 2 meters to the right."

[1023] Based on this prompt, the generative AI model generates the instruction, "There is a bench 5 meters ahead. Please move 2 meters to the right."

[1024] The server then transmits the generated natural language instructions to the visually impaired person's device via a communications network using high-speed communications technology such as 5G. The device then communicates the received instructions to the visually impaired person using its voice output function. For example, the device may play back a voice instruction such as, "There is a bench five meters ahead. Please move two meters to the right."

[1025] The user can move safely by following the voice instructions. This system enables the visually impaired person to be guided to their destination and to be aware of risks in the surrounding environment in a timely manner without the intervention of a sighted person. This significantly improves the freedom and safety of movement for the visually impaired person. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[1026] Step 1: Data collection

[1027] The server acquires environmental information from the sensor devices. The input is real-time data from the sensor devices. Specifically, the server receives the data sent from the sensor devices and detects the situation around the visually impaired person. The output is environmental information data.

[1028] Step 2: Obtaining location information

[1029] The server uses a location information system to identify the current location of the visually impaired person. GPS data is used as input. Specifically, the server analyzes the GPS data to identify the exact location of the visually impaired person. The output is the current location data.

[1030] Step 3: Get map information

[1031] The server uses a map information system to obtain highly accurate map information. The input is a map database. Specifically, the server obtains the necessary map information from the map database and obtains basic data for calculating the travel route of a visually impaired person. The output is map information data.

[1032] Step 4: Data analysis

[1033] The server analyzes the space using a generative AI model based on the collected environmental information, current location data, and map information data. The inputs are environmental information data, current location data, and map information data. Specifically, the server inputs this data into the generative AI model and performs analysis to generate instructions in natural language that are easy for visually impaired people to understand. The output is analysis result data.

[1034] Step 5: Natural Language Generation

[1035] The server inputs a prompt sentence into the generative AI model to generate natural language instructions. The inputs include analysis result data and the prompt sentence. In concrete terms, the server inputs the following prompt sentence into the generative AI model: "The user's current location is the entrance to the park. There is a bench 5 meters ahead. Please instruct the user to move 2 meters to the right." The generative AI model generates natural language instructions based on this prompt sentence. The output is natural language instruction data.

[1036] Step 6: Feedback

[1037] The server sends the generated natural language instructions to the visually impaired person's device via a communication network. The input is natural language instruction data. Specifically, the server sends the instruction data to the device using 5G communication. The output is the instruction data sent to the device.

[1038] Step 7: Audio Output

[1039] The device uses its voice output function to communicate the received instruction data to the visually impaired person. The input is the instruction data sent to the device. Specifically, the device converts the instruction data into voice using voice synthesis technology and plays it back to the visually impaired person. The output is the voice instruction that was communicated to the visually impaired person.

[1040] Step 8: User Behavior

[1041] The user moves safely according to voice instructions. The input is the voice instructions. The specific action is for the user to follow the voice instructions and move in the specified direction. The output is the user's safe movement.

[1042] (Application example 3)

[1043] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1044] When visually impaired people use self-driving vehicles, they face the challenge of having difficulty grasping route information to their destination and risk information about the surrounding environment in real time. Furthermore, in order for visually impaired people to travel safely, they often need the assistance of sighted people, which limits their freedom of movement. There is a need to solve these challenges and enable visually impaired people to use self-driving vehicles safely and freely.

[1045] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1046] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted persons; means for being installed in autonomous vehicles and providing real-time route information and risk information of the surrounding environment to enable visually impaired persons to use autonomous vehicles safely; and means including a speech synthesis engine for providing voice feedback. This enables visually impaired persons to understand route information and risk information of the surrounding environment in real time when using autonomous vehicles and to travel safely and freely.

[1047] "Visually impaired people" refers to people who have visual impairments and have difficulty using visual information.

[1048] A "wearable camera" refers to a camera device that can be worn on the body.

[1049] A "location information system" refers to a system that provides accurate location information using GPS or other technologies.

[1050] "Map information system" refers to a system that provides highly accurate map information.

[1051] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate information for specific purposes.

[1052] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[1053] "Means for providing feedback in natural language" refers to means for providing information to users using natural language.

[1054] "Sighted people" refers to people who are not visually impaired and can use visual information.

[1055] An "autonomous vehicle" refers to a vehicle that can drive itself using artificial intelligence and sensor technology.

[1056] "Route information" refers to information about the route and direction to a destination.

[1057] "Risk information about the surrounding environment" refers to information about obstacles and dangerous situations that may be encountered during travel.

[1058] A "speech synthesis engine" refers to technology that converts text information into speech.

[1059] This invention is a system that enables visually impaired people to travel safely and freely when using self-driving vehicles by understanding route information and risk information about the surrounding environment in real time.

[1060] System configuration

[1061] The system consists of the following main components:

[1062] 1. Wearable camera: Worn by visually impaired people, it captures image information of the surrounding area.

[1063] 2. Location information system: Provides accurate location information using GPS etc.

[1064] 3. Map information system: Provides highly accurate map information.

[1065] 4. Generative AI: Analyzes space based on image information, location information, and map information obtained from a wearable camera, and generates optimal route information.

[1066] 5. 5G communications: Provides low-latency, real-time communications.

[1067] 6. Speech synthesis engine: Provides generated route and risk information as voice feedback in natural language.

[1068] Program processing

[1069] The server analyzes image information acquired from the wearable camera and integrates data obtained from the location information system and map information system. Based on this data, the generative AI generates optimal route information for the visually impaired. The generated route information is fed back to the visually impaired in real time using 5G communications. A speech synthesis engine is used for voice feedback, which is provided in natural language that is easy for the visually impaired to understand.

[1070] Hardware and software used

[1071] Hardware: Smartphones, wearable cameras, GPS devices

[1072] Software: Python, requests library, pyttsx3 library

[1073] Specific examples

[1074] For example, when a visually impaired person is using a self-driving vehicle, they may be given feedback such as:

[1075] "Turn left at the next intersection."

[1076] "There are pedestrians ahead, please be careful."

[1077] "We're 500 meters away from our destination."

[1078] Prompt Sentence Examples

[1079] An example of a prompt to be input to the generative AI model is as follows:

[1080] Develop an application that provides real-time voice feedback on route information and risk information about the surrounding environment to visually impaired people using self-driving vehicles. Route information is obtained from an API using 5G communications, and the pyttsx3 library is used for voice feedback. Specific examples of feedback include "Turn left at the next intersection," and "There are pedestrians ahead, so please be careful."

[1081] In this way, applications can be realized that allow visually impaired people to safely use autonomous vehicles.

[1082] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1083] Step 1:

[1084] The server receives image information from the wearable camera in real time. The input is image data from the wearable camera, and the output is image data for analysis. This image data captures the environment around the visually impaired person in real time.

[1085] Step 2:

[1086] The server obtains precise location information from a location information system. The input is location data from a GPS device, and the output is the current precise location information. This location information is used to determine the current location of the visually impaired person.

[1087] Step 3:

[1088] The server obtains high-precision map information from the map information system. The input is map data from the map information system, and the output is map information for analysis. This map information is used to calculate routes for visually impaired people.

[1089] Step 4:

[1090] The server uses generative AI to integrate the acquired image information, location information, and map information and analyze the space. The input is image data, location information, and map information, and the output is optimal route information. This analysis generates the optimal route for visually impaired people to travel.

[1091] Step 5:

[1092] The server uses 5G communications to send the generated route information to the visually impaired person's device in real time. The input is the optimal route information, and the output is the route information sent to the visually impaired person's device. 5G communications enable low-latency, real-time communications.

[1093] Step 6:

[1094] The device uses a speech synthesis engine to provide natural language voice feedback of the received route information. The input is route information and the output is voice feedback. The speech synthesis engine provides information in a format that is easy for visually impaired people to understand.

[1095] Step 7:

[1096] The user receives voice feedback and follows instructions to operate the autonomous vehicle. The input is voice feedback and the output is the user's actions. This allows the visually impaired to move around safely and freely.

[1097] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1098] "Example 1"

[1099] This invention is a system for safe mobility for the visually impaired. This system is composed of a wearable camera, a location information system, a map information system, generative AI, 5G communication, a natural language feedback system, and an emotion engine.

[1100] "Example 2"

[1101] The wearable camera captures the surrounding environment from the perspective of the visually impaired person. The location information system uses GPS and other technologies to provide the visually impaired person with accurate location information, and the map information system provides highly accurate map information corresponding to that location.

[1102] "Example 3"

[1103] The generative AI analyzes the space based on image information from the wearable camera and information from the location information system and map information system. It also incorporates the user's emotional information obtained from the emotion engine. For example, if the user is feeling anxious, the generative AI will generate more detailed route information.

[1104] "Example 4"

[1105] The generated route information is fed back to the visually impaired in real time using 5G communications, and the feedback is provided in natural language in a format that is easy for the visually impaired to understand.

[1106] "Example 5"

[1107] The emotion engine recognizes the emotions of the visually impaired person and adjusts the feedback accordingly. For example, if the visually impaired person is feeling anxious, the emotion engine conveys that information to the generative AI, which then generates more detailed route information.

[1108] The processing flow of each embodiment will be described below.

[1109] "Example 1"

[1110] Step 1: A visually impaired person wears a wearable camera.

[1111] Step 2: The wearable camera captures the surrounding environment and sends the image information to the generative AI.

[1112] Step 3: The location information system acquires the location information of the visually impaired person and sends that information to Generation System A.

[1113] Send to I.

[1114] Step 4: The map information system obtains map information based on the location information and sends that information to the generative AI.

[1115] Step 5: The emotion engine acquires the emotion information of the visually impaired person and sends that information to the generative AI.

[1116] Step 6: The generative AI analyzes the space based on the received information and generates optimal route information.

[1117] Step 7: The generated route information is fed back to the visually impaired person in real time via 5G communication. Feedback is provided in natural language.

[1118] "Example 2"

[1119] Step 1: A visually impaired person puts on the wearable camera and starts moving.

[1120] Step 2: The wearable camera captures the surrounding environment and sends the image information to the generative AI.

[1121] Step 3: The location information system acquires the location information of the visually impaired person and sends that information to the generative AI.

[1122] Step 4: The map information system obtains map information based on the location information and sends that information to the generative AI.

[1123] Step 5: The emotion engine acquires the emotion information of the visually impaired person and sends that information to the generative AI.

[1124] Step 6: The generative AI analyzes the space based on the received information and generates optimal route information.

[1125] Step 7: The generated route information is fed back to the visually impaired person in real time via 5G communication. The feedback is provided in natural language and in a format that is easy for the visually impaired to understand.

[1126] Step 8: The visually impaired person continues to move according to the feedback. During this time, the emotion engine monitors the visually impaired person's emotions in real time and sends this information to the generative AI.

[1127] Step 9: The generative AI takes emotional information into account, updates the route information as needed, and provides feedback to the visually impaired user.

[1128] Example 1

[1129] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1130] For visually impaired people to travel safely, they need to accurately understand their surroundings and select an appropriate route. However, with conventional technology, it was difficult for visually impaired people to understand their surroundings and select an appropriate route on their own. In addition, there were times when real-time information provision was delayed, making it difficult to ensure the safety of visually impaired people. Furthermore, feedback based on the emotional state of visually impaired people was not provided, which could cause stress and anxiety.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1132] In this invention, the server includes: a means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; a means for providing feedback to users in natural language using 5G communications for low latency and excellent real-time information; and a means for analyzing the user's emotional state using an emotion engine and adjusting the content and tone of the feedback. This enables visually impaired people to accurately grasp their surroundings and provide information in real time to move safely. Furthermore, providing feedback according to the user's emotional state reduces stress and anxiety, allowing them to move with peace of mind.

[1133] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining information visually.

[1134] A "wearable camera" is a camera that can be worn by a user, typically on the head, and refers to a device that captures images of the surrounding environment.

[1135] A "location information system" refers to a system that uses GPS or other devices to determine a user's current location.

[1136] "Map information system" refers to a system that provides map data and acquires information about the user's current location and surrounding environment.

[1137] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate useful information for users.

[1138] "5G communication" refers to the fifth-generation mobile communication system, a technology that transmits data at high speeds and with low latency.

[1139] "Natural language feedback" refers to the means of converting analysis results into natural language and providing information to users in voice or text.

[1140] An "emotion engine" refers to a system that analyzes a user's emotional state and provides feedback according to that state.

[1141] "Means for analyzing space" refers to technology that analyzes the surrounding environment based on image information, location information, and map information obtained by a wearable camera.

[1142] "Information with excellent real-time characteristics" refers to information that is provided immediately with little delay.

[1143] "Means for adjusting the content and tone of feedback" refers to technology that changes the content of the information provided and the tone of the voice depending on the user's emotional state.

[1144] This invention is a support system for the safe movement of visually impaired people. The system is composed of a wearable camera, a location information system, a map information system, generative AI, 5G communication, a natural language feedback system, and an emotion engine.

[1145] The user wears a head-mounted display on their head, and a wearable camera captures high-resolution images of the surrounding environment. The camera continues to capture video data in real time. The device receives the video data from the wearable camera and sends it to a server using 5G communication. Using 5G communication enables high-speed data transfer with low latency.

[1146] The server receives the GPS data sent from the device and identifies the user's current location. The server then calls the map information system to obtain map information around the user's current location. The map information includes details of roads, buildings, obstacles, etc.

[1147] The server uses generative AI to analyze video data and map information. The server generates information about obstacles around the user and route guidance. For example, the server generates information such as, "There are stairs ahead. Please be careful."

[1148] The server uses a natural language feedback system to convert the analysis results into a voice message. The voice message is generated in a format that is easy for the user to understand. The server uses an emotion engine to analyze the user's stress level and emotional state and adjust the content and tone of the feedback. For example, if the user is nervous, the server generates a message in a calm tone.

[1149] The terminal receives the voice message sent from the server and conveys it to the user, allowing the user to travel safely while listening to the voice guidance.

[1150] Examples:

[1151] Imagine a user is walking on a train platform. The user wears a head-mounted display, and the wearable camera captures video of the platform. The device sends the video data to a server via 5G communication. The server receives GPS data and identifies the user's current location. The server calls the Google Maps API to obtain map information for the station platform. The server uses generative AI to detect that the user is approaching the edge of the platform, and generates a voice message saying, "You are approaching the edge of the platform. Please be careful." If the emotion engine determines that the user's stress level is low, the message is delivered in a calm tone. The device delivers the voice message to the user, allowing them to move safely.

[1152] Example prompt sentence:

[1153] "As a user walks along a station platform, a wearable camera captures video of the platform, and the device transmits the video data to a server via 5G. The server receives GPS data and identifies the user's current location. The server then calls the Google Maps API to obtain map information for the station platform. Using generative AI, the server detects when the user is approaching the edge of the platform and generates a voice message to alert the user. If the emotion engine determines that the user's stress level is low, the message is delivered in a calming tone."

[1154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1155] Step 1:

[1156] The user wears a head-mounted display, and a wearable camera captures high-resolution images of the surrounding environment.

[1157] Input: The user's surroundings

[1158] Output: High-resolution video data

[1159] Specific operation: The camera continues to capture video data in real time.

[1160] Step 2:

[1161] The device receives video data from the wearable camera and transmits it to a server using 5G communication.

[1162] Input: Video data from a wearable camera

[1163] Output: Video data sent to the server

[1164] Specific operation: The device uses 5G communication to transfer data at high speed with low latency.

[1165] Step 3:

[1166] The server receives the GPS data sent from the device and identifies the user's current location.

[1167] Input: GPS data from the device

[1168] Output: Current location of the user

[1169] What happens: The server analyzes the GPS data and determines the user's exact location.

[1170] Step 4:

[1171] The server calls the map information system and obtains map information about the area around the user's current location.

[1172] Input: current location of the user

[1173] Output: Map information of the surrounding area

[1174] Specific operation: The server uses the map information system to obtain detailed map information.

[1175] Step 5:

[1176] The server uses generative AI to analyze the video data and map information.

[1177] Input: Video data, map information

[1178] Output: Analysis results (obstacles and route guidance information)

[1179] Specific operation: The server uses generative AI (e.g., OpenAI's GPT-4) to generate obstacles and route guidance information around the user.

[1180] Step 6:

[1181] The server uses a natural language feedback system to convert the analysis results into a voice message.

[1182] Input: Analysis results

[1183] Output: Voice message

[1184] Specific operation: The server converts the analysis results into natural language and generates a voice message in a format that is easy for the user to understand.

[1185] Step 7:

[1186] The server uses an emotion engine to analyze the user's stress level and emotional state and adjust the content and tone of the feedback.

[1187] Input: User's emotional state

[1188] Output: Modified voice message

[1189] Specific behavior: The server uses an emotion engine to adjust the content and tone of the feedback depending on the user's emotional state.

[1190] Step 8:

[1191] The terminal receives the voice message sent from the server and conveys it to the user.

[1192] Input: Voice message from the server

[1193] Output: Audio feedback to the user

[1194] Specific operation: The device plays a voice message, allowing the user to travel safely while listening to the voice guidance.

[1195] (Application example 1)

[1196] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1197] When visually impaired people use self-driving vehicles, they need a system that can help them get in and out of the vehicle safely and travel to their destination. However, current technology makes it difficult for visually impaired people to accurately grasp their surroundings and provide them with real-time information to travel safely. There is also a lack of technology that provides appropriate feedback that takes into account the user's emotional state. This poses a challenge in ensuring the safety and convenience of self-driving vehicles for visually impaired people.

[1198] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as low-latency, real-time information utilizing 5G communication; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted persons; means for analyzing the user's emotional state using an emotion engine and providing appropriate feedback; and means for supporting visually impaired persons in safely getting in and out of autonomous vehicles. This enables visually impaired persons to travel safely and efficiently when using autonomous vehicles.

[1199] "Visually impaired persons" refer to people who have visual impairments and are unable to fully utilize visual information.

[1200] A "wearable camera" is a camera that can be worn by the user, and is primarily used by attaching it to the head or body.

[1201] A "location information system" is a system that uses technology such as GPS to determine a user's current location.

[1202] A "map information system" is a system that provides geographical information, including detailed map data such as roads and buildings.

[1203] "Generative AI" is a system that uses artificial intelligence technology to generate new information and analytical results from input data.

[1204] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[1205] "Natural language feedback" is a means of providing information to users using natural language such as voice or text.

[1206] An "emotion engine" is a system that analyzes the user's emotional state and provides appropriate feedback based on the results.

[1207] An "autonomous vehicle" is a vehicle that drives itself automatically using artificial intelligence and sensor technology.

[1208] "Analyzing space" means identifying the surrounding environment and the location of obstacles based on acquired image information and location information, and providing the user with the necessary information.

[1209] "Timely risk identification" means detecting surrounding risks in real time and providing information to users promptly.

[1210] "Providing feedback" means giving the user necessary information or instructions via voice or text.

[1211] The system for implementing this invention supports safe and efficient travel for visually impaired people when using autonomous vehicles. The system is composed of a wearable camera, a location information system, a map information system, generative AI, 5G communications, a natural language feedback system, and an emotion engine.

[1212] System configuration

[1213] 1. Wearable Camera:

[1214] It is a camera worn on the head of a visually impaired person that captures the surrounding environment in real time.

[1215] 2. Location Information Systems:

[1216] A GPS module is used to determine the current location of the visually impaired person.

[1217] 3. Map Information System:

[1218] It provides highly accurate map data and calculates the optimal route from your current location to your destination.

[1219] 4. Generative AI:

[1220] It analyzes image information acquired by a wearable camera and recognizes the surrounding environment and obstacles.

[1221] Based on the analysis results, the optimal route is generated to enable visually impaired people to travel safely.

[1222] 5. 5G communications:

[1223] It enables high-speed, low-latency communication, sending and receiving information in real time.

[1224] 6. Natural Language Feedback System:

[1225] It provides real-time audio feedback to visually impaired people, helping them move around safely.

[1226] 7. Emotion Engine:

[1227] It analyzes the user's emotional state and provides appropriate feedback to reduce stress and anxiety.

[1228] System Operation

[1229] The server uses generative AI to analyze image information acquired by the wearable camera and recognize the surrounding environment and obstacles. It uses a location information system and map information system to calculate the optimal route from the current location to the destination. It utilizes 5G communications to provide real-time voice feedback to visually impaired people. It uses an emotion engine to analyze the user's emotional state and provide appropriate feedback, allowing visually impaired people to travel safely.

[1230] Specific examples

[1231] For example, when a visually impaired person gets into an autonomous vehicle, a wearable camera will analyze the surrounding environment, and if there is an obstacle, a voice message will be sent saying, "There is an obstacle ahead. Please be careful." Also, when the vehicle arrives at its destination, the GPS and map information system will be used to say, "You have arrived at your destination. Please get off the vehicle."

[1232] Prompt Sentence Examples

[1233] Develop an application that assists visually impaired people when getting in and out of autonomous vehicles. This application will combine a wearable camera, a location information system, a map information system, generative AI, 5G communications, a natural language feedback system, and an emotion engine. Specifically, the wearable camera will capture images of the surrounding environment, and the generative AI will analyze the images to identify obstacles and provide route guidance. GPS will be used to identify the current location, and in conjunction with the map information system, the application will guide the user to their destination. Real-time voice feedback will be provided to visually impaired people to help them navigate safely. The application will analyze the user's emotional state and provide appropriate feedback to reduce stress and anxiety.

[1234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1235] Step 1:

[1236] The server receives image information captured in real time from the wearable camera. The input is video data from the camera, and the output is preprocessed image data that is passed to the generative AI model. Specifically, the server divides the video data into frames, removes noise, and adjusts the resolution.

[1237] Step 2:

[1238] The server inputs the preprocessed image data into the generative AI model, which analyzes the surrounding environment and obstacles. The input is the preprocessed image data, and the output is the analysis results, which include environmental information and obstacle location data. Specifically, the generative AI model analyzes the image data and uses object recognition algorithms to identify obstacles and important landmarks.

[1239] Step 3:

[1240] The server receives current location data obtained from the location information system. The input is location data from the GPS module, and the output is current location data passed to the map information system. Specifically, the server analyzes the GPS data and extracts latitude and longitude information.

[1241] Step 4:

[1242] The server uses a map information system to calculate the optimal route from the current location to the destination. The input is current location data and destination data, and the output is optimal route information. Specifically, the server refers to a map database and calculates the shortest route using a route calculation algorithm.

[1243] Step 5:

[1244] The server uses 5G communications to send analysis results and route information to the visually impaired person's device in real time. The input is the analysis results and route information, and the output is feedback data sent to the visually impaired person's device. Specifically, the server divides the data into packets and transmits them with low latency.

[1245] Step 6:

[1246] The device converts the received feedback data into speech using a natural language feedback system and provides it to the visually impaired. The input is feedback data, and the output is speech feedback. Specifically, the device inputs text data into a speech synthesis engine to generate speech data.

[1247] Step 7:

[1248] The server uses an emotion engine to analyze the user's emotional state and provide appropriate feedback. The input is the user's biometric and behavioral data, and the output is feedback data based on the emotional state. Specifically, the server analyzes data from biometric sensors and evaluates the user's stress and anxiety levels.

[1249] Step 8:

[1250] The device receives feedback data from the emotion engine and provides appropriate voice feedback to the visually impaired. The input is the feedback data from the emotion engine, and the output is voice feedback. Specifically, the device inputs the analysis results of the emotion engine into a voice synthesis engine to generate a voice message that gives the user a sense of security.

[1251] Example 2

[1252] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1253] For visually impaired people to travel safely and efficiently, they need a system that can acquire information about their surroundings and their location in real time, and provide optimal routes based on that information. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there are a lack of concrete means for visually impaired people to travel independently without the help of sighted people. This makes it difficult for visually impaired people to travel safely, and they are subject to many restrictions in their daily lives.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1255] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; means for generating an optimal route based on the information analyzed by the generative AI and providing this to the user via a terminal; and means for the user to receive information for safe movement through audio guidance and vibration feedback. This enables visually impaired people to obtain information about their surrounding environment in real time and move safely and efficiently.

[1256] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining visual information.

[1257] A "wearable camera" refers to a camera device that can be worn by a user and captures the surrounding environment from the user's point of view.

[1258] A "location information system" refers to a system that uses technology such as GPS to accurately obtain a user's current location.

[1259] "Map information system" refers to a system that provides highly accurate map information corresponding to the user's location.

[1260] "Generative AI" refers to artificial intelligence that analyzes space based on acquired image information, location information, and map information, and generates the optimal route.

[1261] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables low-latency, high-speed data communication.

[1262] "Natural language feedback" refers to providing information to users in language that humans can understand.

[1263] An "optimal route" refers to the most suitable travel route for a visually impaired person to reach their destination safely and efficiently.

[1264] "Terminal" refers to a device used by a user that receives information from a server and provides it to the user.

[1265] "Audio guide" refers to a means of providing information to a user using audio.

[1266] "Vibration feedback" refers to a means of providing information to a user by causing a device to vibrate.

[1267] This invention is a system for enabling visually impaired people to move around safely and efficiently, and is realized by combining technologies such as wearable cameras, location information systems, map information systems, generative AI, 5G communications, terminals, audio guides, and vibration feedback.

[1268] First, the user wears a wearable camera, which captures images of the surrounding environment from the user's point of view in real time. The device acquires the image information sent from the wearable camera and sends it to the server.

[1269] The server uses a location information system to accurately obtain the user's current location, and also uses a map information system to obtain highly accurate map information corresponding to the user's location.

[1270] The server sends the acquired image information, location information, and map information to a generative AI (e.g., OpenAI GPT-4), which uses this information to analyze the space and generate the optimal route for the visually impaired person.

[1271] The generated route information is sent from the server to the device, which then provides the received route information to the user. The user can then travel safely along the optimal route through audio guidance and vibration feedback.

[1272] As a concrete example, consider a scenario in which a user travels from the station to their home. The user wears a wearable camera on the station platform and takes pictures of their surroundings. The device acquires image information from the station platform and sends it to a server. The server uses GPS to acquire the user's current location (station platform). The server uses a map API to acquire highly accurate map information about the area around the station. Based on this information, a generative AI generates the optimal route from the station to their home. The server sends the generated route information to the device, and the device instructs the user via voice guidance, saying, "Leave the platform, turn right, and use the elevator."

[1273] Example prompt sentence:

[1274] "Generate the optimal route for a visually impaired person to move safely within the park. Please perform your analysis based on the following information: Image information: {Image data}, Location information: {Latitude and longitude}, Map information: {Map data}"

[1275] In this way, visually impaired people can obtain information about their surroundings in real time and move around safely and efficiently.

[1276] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1277] Step 1:

[1278] A user wears a wearable camera and captures the surrounding environment.

[1279] Input: Real-time video from the user's point of view

[1280] Output: Image information captured by the wearable camera

[1281] Specific operation: The user wears a wearable camera on their head or chest and starts walking. The camera continuously captures the surrounding environment from the user's point of view.

[1282] Step 2:

[1283] The terminal acquires the captured image information and sends it to the server.

[1284] Input: Image information sent from the wearable camera

[1285] Output: Image information sent to the server

[1286] Specific operation: The device receives image information sent in real time from the wearable camera and transmits it to the server with low latency using 5G communication.

[1287] Step 3:

[1288] The server obtains the user's precise location information from a location information system.

[1289] Input: Location data from a location information system

[1290] Output: Current location of the user (latitude and longitude)

[1291] Specific operation: The server accesses the location information system to obtain the user's current location. The obtained location information is stored on the server in the form of latitude and longitude.

[1292] Step 4:

[1293] The server obtains highly accurate map information from the map information system.

[1294] Input: Current location of the user (latitude and longitude)

[1295] Output: High-precision map information corresponding to the user's location

[1296] Specific operation: The server accesses the map information system and obtains detailed map information corresponding to the user's current location. The map information includes information on roads, buildings, obstacles, etc.

[1297] Step 5:

[1298] The server sends image information, location information, and map information to the generative AI.

[1299] Input: Image information, location information, map information

[1300] Output: Data sent to the generative AI

[1301] Specific operation: The server sends the acquired image information, location information, and map information as a single dataset to the generative AI, which is then ready to begin analysis.

[1302] Step 6:

[1303] Based on this information, generative AI analyzes the space and generates the optimal route.

[1304] Input: Image information, location information, map information

[1305] Output: Optimal route information

[1306] How it works: The generative AI analyzes the transmitted data and generates the optimal route for the visually impaired. The analysis includes detecting obstacles and selecting a safe route.

[1307] Step 7:

[1308] The server transmits the generated route information to the terminal.

[1309] Input: Optimal route information from generative AI

[1310] Output: Route information sent to the terminal

[1311] Specific operation: The server sends the optimal route information received from the generative AI to the terminal. The terminal receives this information and prepares to provide it to the user.

[1312] Step 8:

[1313] The terminal provides the route information to the user.

[1314] Input: Route information sent from the server

[1315] Output: Route information provided to the user

[1316] Specific operation: The device provides the received route information to the user through voice guidance and vibration feedback, allowing the user to travel safely based on this feedback.

[1317] (Application example 2)

[1318] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1319] Visually impaired people face challenges in navigating physical stores safely and efficiently when searching for products or moving to specific locations. It is also difficult for visually impaired people to understand their surroundings and identify risks in a timely manner. For this reason, there is a demand for a system to assist visually impaired people in navigating physical stores.

[1320] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1321] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; and means for providing navigation through audio guidance and tactile feedback when visually impaired people search for products in physical stores. This enables visually impaired people to move safely and efficiently within physical stores, understand their surrounding environment, and assess risks in a timely manner.

[1322] "Visually impaired persons" refer to people who have visual impairments and have difficulty using visual information.

[1323] A "wearable camera" is a camera device that can be worn on the body and is used to capture images of the surrounding environment from the perspective of a visually impaired person.

[1324] A "location information system" is a system that provides accurate location information using GPS or other technologies.

[1325] A "map information system" is a system that provides highly accurate map information corresponding to a specific location.

[1326] "Generative AI" is an artificial intelligence that analyzes space based on acquired image information, location information, and map information, and generates the optimal route.

[1327] "5G communication" refers to the fifth generation mobile communication system, a technology that provides low latency and excellent real-time communication.

[1328] "Means of providing feedback in natural language" refers to a means of conveying information analyzed by generative AI to users in natural language.

[1329] "Sighted person" refers to a person who has no visual impairment and can use normal visual information.

[1330] "Guidance to destination" refers to providing a route for a visually impaired person to reach their destination.

[1331] "Timely understanding of risks in the surrounding environment" refers to visually impaired people being able to recognize risks in their surrounding environment in real time.

[1332] A "physical store" is a store that exists in a physical location and is a facility for selling products.

[1333] An "audio guide" is a means of providing information in audio form to visually impaired people.

[1334] "Haptic feedback" is a means of providing information to visually impaired people through the sense of touch.

[1335] "Means for providing navigation" refers to means for providing route guidance to help visually impaired people reach their destination.

[1336] This invention is a system that provides navigation through audio guidance and haptic feedback to visually impaired people as they move safely and efficiently through a physical store and search for products. Specific embodiments for realizing this system are described below.

[1337] System Configuration

[1338] Hardware

[1339] Smart glasses: Equipped with a wearable camera and GPS module, they capture images of the surrounding environment from the perspective of the visually impaired and acquire location information.

[1340] Server: Runs generative AI and analyzes image, location, and map information to generate the optimal route.

[1341] Communications network: Utilizing 5G communications, data communications will be performed with low latency and excellent real-time performance.

[1342] software

[1343] OpenCV: A library for image processing.

[1344] requests: A library for sending HTTP requests.

[1345] geopy: A library for working with geolocation data.

[1346] Processing flow

[1347] 1. Image acquisition: The camera in the smart glasses captures the surrounding environment and acquires image data.

[1348] 2. Location information acquisition: The GPS module acquires the current location information.

[1349] 3. Map information acquisition: Obtain highly accurate map information based on location information.

[1350] 4. Optimal route generation: The acquired image information, location information, and map information are sent to the server, and the generative AI generates the optimal route.

[1351] 5. Audio guide generation: Generate audio guide based on the generated route and provide it to visually impaired people.

[1352] 6. Haptic Feedback: Provide haptic feedback when needed to help visually impaired people navigate safely.

[1353] Specific examples

[1354] When a visually impaired person searches for a specific product in a physical store, the smart glasses capture real-time images of the surrounding environment and the GPS module acquires their location. This data is sent to a server, and generative AI generates an optimal route. The generated route is provided to the visually impaired person as an audio guide, and haptic feedback is also provided if necessary.

[1355] Prompt Sentence Examples

[1356] "Create an AI model that generates the optimal route for a visually impaired person to find a product in a physical store. As input, use images from a wearable camera, location information from a GPS, and high-precision map information. As output, generate an audio guide."

[1357] In this way, the present invention provides a system that greatly assists visually impaired people in safely and efficiently navigating and locating products within a brick-and-mortar store.

[1358] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1359] Step 1:

[1360] Image acquisition

[1361] The device (smart glasses) uses a wearable camera to capture the surrounding environment from the point of view of the visually impaired person. The input is real-time video from the camera, and the output is captured image data, which is used in subsequent processing steps.

[1362] Step 2:

[1363] Location information acquisition

[1364] The device (smart glasses) acquires its current location information using a GPS module. The input is a GPS signal, and the output is precise location information including latitude and longitude. This location information is used to acquire map information and generate routes.

[1365] Step 3:

[1366] Get map information

[1367] The server obtains highly accurate map information based on the acquired location information. The input is location information including latitude and longitude, and the output is detailed map data corresponding to that location. This map data is used for route analysis by generative AI.

[1368] Step 4:

[1369] Optimal Route Generation

[1370] The server sends the acquired image information, location information, and map information to the generative AI to generate the optimal route. The input is image data, location information, and map data, and the output is optimal route information for the visually impaired. The generative AI analyzes this data and calculates a route that the visually impaired can travel safely and efficiently.

[1371] Step 5:

[1372] Audio guide generation

[1373] The server generates an audio guide based on the generated route information. The input is the optimal route information, and the output is an audio guide provided to the visually impaired person. This audio guide includes instructions for the visually impaired person to reach their destination.

[1374] Step 6:

[1375] haptic feedback

[1376] The device (smart glasses) provides haptic feedback as needed. The input is optimal route information and real-time environmental information, and the output is haptic feedback for the visually impaired. This haptic feedback provides additional information to enable the visually impaired to navigate safely.

[1377] In this way, a system is realized that allows visually impaired people to move safely and efficiently within a physical store and navigate through it using audio guidance and haptic feedback as they search for products.

[1378] Example 3

[1379] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1380] For visually impaired people to travel safely and efficiently, it is necessary to obtain information about the surrounding environment and route in real time and receive appropriate feedback. However, conventional systems are unable to respond appropriately when visually impaired people feel anxious, and do not ensure sufficient freedom of movement or safety. In addition, there are issues with the difficulty of guiding people to their destination or identifying risks in the surrounding environment in a timely manner without the intervention of sighted people.

[1381] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for analyzing space using a generative AI model based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system; means for incorporating user emotional information acquired from an emotion engine into the analysis; means for providing feedback to the user in natural language as low-latency, real-time information utilizing 5G communication; means for providing guidance to a destination and timely identification of risks in the surrounding environment without the intervention of sighted people; and means for adjusting the content of the feedback based on the user's emotional information. This allows visually impaired people to receive appropriate feedback even when they feel anxious, significantly improving their freedom of movement and safety.

[1382] A "wearable camera" is a portable camera device that can be worn by a user and is a device for acquiring image information of the surroundings in real time.

[1383] A "location information system" is a system for accurately identifying a user's current location, and generally refers to the GPS (Global Positioning System).

[1384] A "map information system" is a system that provides geographical information, including highly accurate map data for providing a user's current location and route information to a destination.

[1385] A "generative AI model" is an algorithm or model that uses artificial intelligence to analyze data and generate results for a specific task.

[1386] An "emotion engine" is a system for recognizing a user's emotional state and analyzing that information, and is used to provide appropriate feedback based on the user's emotions.

[1387] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[1388] "Feedback in natural language" means providing feedback using language that is commonly used in daily life in order to provide information in a form that is easy for users to understand.

[1389] "Analyzing space" means understanding the surrounding environment and routes based on acquired image information, location information, and map information, and generating appropriate instructions and information.

[1390] "Timely risk identification" means recognizing in real time the risks that users may face while traveling and taking appropriate measures.

[1391] This invention is a system for safe and efficient mobility for visually impaired people, utilizing wearable cameras, location information systems, map information systems, generative AI models, emotion engines, and 5G communications.

[1392] Server Roles

[1393] The server receives image information sent from the wearable camera. Specifically, it acquires image data in real time from the wearable camera (e.g., a portable camera device) worn by the user. It also receives data from location information systems and map information systems (e.g., a geographic information system). Based on this data, the server uses a generative AI model (e.g., an artificial intelligence algorithm) to analyze the space and generate optimal route information for the visually impaired person.

[1394] In addition, the server also incorporates user emotional information obtained from an emotion engine (e.g., a system that recognizes the user's emotional state) into the analysis. For example, if the user is feeling anxious, the generative AI model will generate more detailed route information, allowing the user to travel with peace of mind.

[1395] Device Role

[1396] The device receives route information sent from the server in real time via 5G communication. 5G communication is a technology that enables high-speed, low-latency communication, allowing visually impaired people to quickly receive the information they need. The device then feeds the received route information back to the user via an audio output device (e.g., earphones). The feedback is provided in natural language and in a format that is easy for visually impaired people to understand. For example, instructions might be given such as, "There is an obstacle five meters ahead. Move two meters to the right."

[1397] User Roles

[1398] Users wear a wearable camera and receive real-time route information while moving. The emotion engine recognizes the user's emotions and adjusts the feedback accordingly. For example, if the user is feeling anxious, the emotion engine conveys that information to the generative AI model, which then generates more detailed route information. This allows users to be guided to their destination and to be aware of risks in the surrounding environment in a timely manner without the need for a sighted person.

[1399] Specific examples

[1400] Hardware and software used

[1401] Wearable cameras (e.g., handheld camera devices)

[1402] Location Information System

[1403] Map information systems (e.g., geographical information systems)

[1404] Generative AI models (e.g., artificial intelligence algorithms)

[1405] Emotion engines (e.g., systems that recognize a user's emotional state)

[1406] 5G communications infrastructure

[1407] Prompt Sentence Examples

[1408] "If the user is concerned, generate a detailed route. Current location: Latitude 35.6895, Longitude 139.6917. Obstacle ahead."

[1409] This system enables visually impaired people to be guided to their destination and to grasp risks in the surrounding environment in a timely manner without the intervention of sighted people. This significantly improves the freedom and safety of movement for visually impaired people. The flow of the identification process in Example 3 will be explained using Figure 21.

[1410] Step 1: Receiving data

[1411] The server receives image information sent from the wearable camera. The input is real-time image data from the wearable camera, and the output is image data stored in the server. Specifically, the server analyzes the data stream from the camera, extracts image frames, and stores them.

[1412] Step 2: Receiving location and map information

[1413] The server receives data from a GPS and a map information system. The input is the current location data from the GPS and the map data from the map information system, and the output is the location information and map information stored in the server. Specifically, the server analyzes the GPS data, identifies the user's current location, and combines it with the map data.

[1414] Step 3: Acquiring emotional information

[1415] The server obtains the user's emotional information from the emotion engine. The input is the emotional data from the emotion engine, and the output is the emotional information stored in the server. Specifically, the server analyzes the data from the emotion engine and determines whether the user is feeling anxious.

[1416] Step 4: Spatial analysis and route generation

[1417] The server uses a generative AI model to analyze the space based on the received image information, location information, map information, and emotion information, and generates route information. The input is image data, location data, map data, and emotion data, and the output is the generated route information. Specifically, the server sends a prompt to the generative AI model to generate detailed route information. For example, "If the user feels anxious, please generate detailed route information. The current location is latitude 35.6895, longitude 139.6917. There is an obstacle ahead."

[1418] Step 5: Prepare for feedback

[1419] The server converts the generated route information into natural language and prepares feedback in a form that is easy for visually impaired people to understand. The input is the generated route information, and the output is feedback information expressed in natural language. Specifically, the server analyzes the route information and generates instructions such as, "There is an obstacle 5 meters ahead. Please move 2 meters to the right."

[1420] Step 6: Submit your feedback

[1421] The server transmits the prepared feedback information to the terminal via 5G communication. The input is the feedback information expressed in natural language, and the output is the feedback information transmitted to the terminal. Specifically, the server transmits the feedback information to the terminal with low latency using the 5G communication infrastructure.

[1422] Step 7: Receiving and Providing Feedback

[1423] The terminal receives feedback information transmitted from the server and provides it to the user through an audio output device. The input is the feedback information transmitted from the server, and the output is the feedback information provided in audio. Specifically, the terminal converts the received feedback information into audio using speech synthesis technology and transmits it to the user through earphones.

[1424] Step 8: Move users

[1425] The user receives voice feedback from the device and moves according to the instructions. The input is the feedback information provided by voice, and the output is the user's movement behavior. Specifically, the user moves according to instructions such as "There is an obstacle 5 meters ahead. Please move 2 meters to the right."

[1426] (Application example 3)

[1427] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1428] When visually impaired people use self-driving vehicles, it is difficult to provide them with real-time guidance to their destination and information about risks in the surrounding environment. In addition, it is difficult to adjust the feedback content according to the visually impaired person's emotional state, making it difficult for them to travel safely.

[1429] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as low-latency, real-time information using 5G communication; means for providing guidance to a destination and timely risk assessment of the surrounding environment without the intervention of sighted persons; means for recognizing the emotional state of the visually impaired person and adjusting the feedback content according to that emotion; and means for being installed in an autonomous vehicle and providing guidance to a destination and risk information of the surrounding environment in real time when a visually impaired person uses an autonomous vehicle. This enables visually impaired people to travel to their destination safely when using an autonomous vehicle.

[1430] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining visual information.

[1431] A "wearable camera" is a camera device that can be worn on the body and is a device for acquiring image information from the user's point of view.

[1432] A "location information system" is a system that uses technology such as GPS to accurately obtain a user's current location.

[1433] A "map information system" is a system that provides geographical information, including highly accurate map data.

[1434] "Generative AI" is an artificial intelligence system that performs spatial analysis and route generation based on acquired data.

[1435] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[1436] "Feedback in natural language" refers to providing information in natural language that is easy for users to understand.

[1437] "Sighted person" refers to a person who has no visual impairment and can normally obtain visual information.

[1438] "Emotional state" refers to a user's current emotional or psychological state.

[1439] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1440] "Providing in real time" means providing information immediately without delay.

[1441] The system for implementing this invention provides real-time guidance to a visually impaired person using an automated driving vehicle, as well as information about risks in the surrounding environment. Specific embodiments of this system are described below.

[1442] System configuration

[1443] 1. Hardware:

[1444] Wearable camera: A device worn by visually impaired people to obtain image information of their surroundings.

[1445] GPS module: A device that accurately determines the current location of a visually impaired person.

[1446] 5G communication module: A device that enables high-speed, low-latency communication.

[1447] Autonomous vehicle: A vehicle that uses artificial intelligence and sensor technology to drive autonomously.

[1448] 2. Software:

[1449] Generative AI: An artificial intelligence system that analyzes space and generates optimal routes based on image information acquired by a wearable camera, location information from a GPS module, and highly accurate map information from a map information system.

[1450] Emotion recognition engine: Software that recognizes the emotional state of visually impaired people and provides that information to generative AI.

[1451] Natural language processing engine: Software that provides feedback in natural language on generated route information and risk information.

[1452] Map information system: A system that provides highly accurate map data.

[1453] Data processing and calculation

[1454] 1. Image Analysis:

[1455] Generative AI analyzes image information obtained from the wearable camera and recognizes surrounding obstacles and terrain.

[1456] For example, if there is an obstacle ahead, the distance and direction of the obstacle can be identified.

[1457] 2. Location information acquisition:

[1458] The current location is obtained from the GPS module and linked to the map information system to determine the current location.

[1459] 3. Route generation:

[1460] The generative AI generates the optimal route based on the acquired image information, location information, and map information.

[1461] The level of detail of route information is adjusted according to the emotional state of the visually impaired person.

[1462] 4. Feedback:

[1463] A natural language processing engine provides feedback in natural language on the generated route and risk information.

[1464] For example, it provides instructions such as, "There is an obstacle 5 meters ahead. Move 2 meters to the right."

[1465] Specific examples

[1466] If a visually impaired person is in an autonomous vehicle and encounters an obstacle ahead while heading to their destination, the system will provide voice instructions such as, "There is an obstacle 5 meters ahead. Please move 2 meters to the right." Additionally, if the visually impaired person feels anxious, the emotion recognition engine will convey that information to the generative AI, which will then generate more detailed route information.

[1467] Prompt Sentence Examples

[1468] "If the user feels anxious, generate more detailed route information. For example, provide instructions like, 'There is an obstacle 5 meters ahead. Go 2 meters to the right.'"

[1469] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1470] Step 1:

[1471] A wearable camera acquires image information of the visually impaired person's surroundings. The input is image data from the camera, and the output is raw data for image analysis. The camera captures images in real time from the visually impaired person's point of view.

[1472] Step 2:

[1473] The GPS module obtains the current location of the visually impaired person. The input is the GPS signal and the output is the coordinate data of the current location. The GPS module receives signals from satellites and identifies the exact location of the visually impaired person.

[1474] Step 3:

[1475] The server obtains highly accurate map data from the map information system. The input is the coordinate data of the current location, and the output is map information of the surrounding area. The server accesses the map information system and obtains detailed map data based on the current location.

[1476] Step 4:

[1477] The generative AI analyzes space based on image information from the wearable camera, location information from the GPS module, and map information from the map information system. The inputs are image data, location data, and map data, and the output is spatial information as the analysis result. The generative AI integrates this data and recognizes surrounding obstacles and terrain.

[1478] Step 5:

[1479] The emotion recognition engine analyzes image information acquired from a wearable camera to recognize the emotional state of the visually impaired person. The input is image data, and the output is emotional state data. The emotion recognition engine analyzes the facial expressions and movements of the visually impaired person to identify their current emotional state.

[1480] Step 6:

[1481] The generative AI receives emotional state data from the emotion recognition engine and adjusts the level of detail in the route information. The input is emotional state data, and the output is adjusted route information. The generative AI generates more detailed route information according to the emotional state of the visually impaired person.

[1482] Step 7:

[1483] The natural language processing engine provides feedback in natural language on the generated route and risk information. The input is route and risk information, and the output is a feedback message in natural language. The natural language processing engine provides the information in a form that is easy for visually impaired people to understand.

[1484] Step 8:

[1485] The 5G communication module sends feedback messages to the visually impaired in real time. The input is the feedback message, and the output is audio feedback to the visually impaired. The 5G communication module sends feedback with low latency, allowing the visually impaired to receive the information instantly.

[1486] Step 9:

[1487] A visually impaired person receives feedback and navigates safely within an autonomous vehicle. The input is audio feedback, and the output is safe navigation for the visually impaired person. The visually impaired person navigates to their destination based on the information provided.

[1488] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1489] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1490] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1491] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1492] [Third embodiment]

[1493] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1494] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1496] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1500] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1503] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1504] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1505] "Example 1"

[1506] In one embodiment of the present invention, a wearable camera for visually impaired people is used to capture images of the surrounding environment. This wearable camera is part of a head-mounted display that can be worn on the head of a visually impaired person.

[1507] "Example 2"

[1508] The captured image information is then sent to a generative AI system along with precise location information obtained from a location information system and highly accurate map information obtained from a map information system. Based on this information, the generative AI system analyzes the space and generates the optimal route for the visually impaired person to travel.

[1509] "Example 3"

[1510] The generated route information is fed back to the visually impaired person in real time using 5G communications. The feedback is provided in natural language and in a format that is easy for the visually impaired to understand. For example, instructions might be given such as, "There is an obstacle five meters ahead. Move two meters to the right."

[1511] "Example 4"

[1512] This system will enable visually impaired people to receive guidance to their destination and to be aware of risks in their surroundings in a timely manner without the need for sighted people, greatly improving the freedom and safety of movement for visually impaired people.

[1513] The processing flow of each embodiment will be described below.

[1514] "Example 1"

[1515] Step 1: A visually impaired person wears a head-mounted display that incorporates a wearable camera to capture images of the surrounding environment from the visually impaired person's perspective.

[1516] Step 2: Image information from the wearable camera is analyzed in conjunction with a location information system and a map information system. The location information system uses GPS and other technologies to provide the visually impaired person with accurate location information, and the map information system provides highly accurate map information corresponding to that location. Step 3: The generative AI analyzes the space based on this information and generates the optimal route for the visually impaired person to travel.

[1517] "Example 2"

[1518] Step 1: The generated route information is fed back to the visually impaired person in real time using 5G communications.

[1519] Step 2: Feedback is given in natural language and in a form that is easy for a visually impaired person to understand, such as "There is an obstacle 5 meters ahead. Move 2 meters to the right."

[1520] Step 3: The visually impaired person can navigate according to this feedback and reach their destination safely.

[1521] Example 1

[1522] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1523] For visually impaired people to travel safely and efficiently, it is important to grasp information about their surroundings in real time and receive appropriate feedback. However, conventional systems do not provide enough information for visually impaired people to accurately recognize their surroundings, and they often require assistance from sighted people. In addition, due to the lack of real-time and low latency, it is difficult to grasp risks in a timely manner.

[1524] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired persons; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communication; means for preprocessing video data and performing object recognition and text reading using a generative AI model; means for generating the analysis results as audio data and converting it into audio using speech synthesis software; means for transmitting the generated audio data to the visually impaired person's head-mounted display to notify them by audio; and means for providing guidance to the visually impaired person and timely identification of risks in the surrounding environment without the intervention of a sighted person. This enables the visually impaired person to accurately recognize their surrounding environment in real time and move safely and efficiently.

[1525] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1526] A "wearable camera" is a camera that can be worn by the user and is used as part of a head-mounted display worn on the head of a visually impaired person.

[1527] A "location information system" refers to a system that uses technology such as GPS to accurately identify a user's current location.

[1528] A "map information system" refers to a system that provides highly accurate map data and provides surrounding geographic information in conjunction with the user's location information.

[1529] "Generative AI" refers to a system that uses artificial intelligence technology to generate useful information from input data.

[1530] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[1531] "Feedback in natural language" refers to conveying analysis results and information to users in natural language that is easy for humans to understand.

[1532] "Preprocessing" refers to processing that is performed before data analysis, and includes processes such as noise removal and resolution adjustment.

[1533] "Object recognition" refers to a technology that detects specific objects from image data and identifies their type and location.

[1534] "Text reading" refers to the technology of extracting character information from image data and recognizing it as text data.

[1535] "Speech synthesis software" refers to software for converting text data into voice data.

[1536] A "head-mounted display" is a display device worn by the user on the head, and has an audio output function for visually impaired people.

[1537] "Timely risk understanding" refers to analyzing surrounding environmental information in real time and providing users with immediate risk information.

[1538] This invention is a system that enables visually impaired people to recognize their surroundings in real time and move safely and efficiently. This system is implemented using hardware and software such as a wearable camera, a location information system, a map information system, a generative AI model, 5G communications, and speech synthesis software.

[1539] The server receives video data sent from a wearable camera worn on the head of a visually impaired person. This camera is used by visually impaired people to capture their surroundings. The server uses an image processing library such as OpenCV to preprocess the received video data. Preprocessing includes noise removal and resolution adjustment.

[1540] The server then inputs the preprocessed video data into a generative AI model, which has capabilities such as object recognition and text reading. For example, an object detection model is used to recognize objects in the video. The generative AI model generates information that helps visually impaired people understand their surroundings.

[1541] The server uses speech synthesis software to generate voice data based on the analysis results obtained from the generative AI model. The analysis results are converted into voice in a natural language that is easy for visually impaired people to understand. For example, voice data such as "There is a pedestrian ahead" or "The traffic light is red" is generated.

[1542] The generated voice data is transmitted in real time with low latency via 5G communications to the visually impaired person's head-mounted display. The visually impaired person can then understand the analysis results through a built-in speaker in the head-mounted display. This enables the visually impaired person to receive guidance to their destination and be aware of risks in their surrounding environment in a timely manner, without the need for a sighted person.

[1543] As a concrete example, consider a scene in which a visually impaired person is walking around town. The video captured by the wearable camera includes traffic lights, pedestrians, and vehicles. The server receives this video data and uses a generative AI model to analyze the color of the traffic lights, the location of pedestrians, and the movement of vehicles. The analysis results are generated as audio data, and the visually impaired person is provided with information such as "The traffic light is red," "There is a pedestrian ahead," and "A car is approaching from the right."

[1544] Example prompt sentence:

[1545] "Imagine a scene where a visually impaired person is walking around town. Explain a system that analyzes video data captured by a wearable camera and provides audio notifications about traffic light colors, pedestrian locations, and vehicle movements."

[1546] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1547] Step 1:

[1548] The device uses a wearable camera worn on the head of the visually impaired person to capture images of the surrounding environment in real time. The input is the video data captured by the camera, and the output is the captured video data. Specifically, the camera captures video at 30 frames per second.

[1549] Step 2:

[1550] The terminal transmits the captured video data to the server. The input is the captured video data, and the output is the video data transmitted to the server. In concrete terms, the terminal compresses the video data and transmits it to the server.

[1551] Step 3:

[1552] The server preprocesses the received video data. The input is the video data sent from the device, and the output is the preprocessed video data. Specifically, the server uses an image processing library such as OpenCV to remove noise and adjust the resolution. For example, it adjusts the resolution of the video data to 640x480 pixels and applies a noise filter.

[1553] Step 4:

[1554] The server inputs the preprocessed video data into the generative AI model. The input is the preprocessed video data, and the output is the analysis results by the generative AI model. Specifically, the server uses the object detection model to recognize objects in the video. For example, it analyzes the color of traffic lights, the position of pedestrians, and the movement of vehicles.

[1555] Step 5:

[1556] The server generates the analysis results obtained from the generative AI model as voice data. The input is the analysis results from the generative AI model, and the output is voice data. Specifically, the server uses voice synthesis software to convert the analysis results into voice. For example, it generates voice data such as "There is a pedestrian ahead" or "The traffic light is red."

[1557] Step 6:

[1558] The server transmits the generated voice data to the terminal. The input is the generated voice data, and the output is the voice data transmitted to the terminal. In concrete terms, the server compresses the voice data and transmits it to the terminal.

[1559] Step 7:

[1560] The device plays the received audio data. The input is the audio data sent from the server, and the output is the audio heard by the visually impaired person. Specifically, the device decodes the received audio data and plays it back from the speaker built into the head-mounted display. This allows the visually impaired person to recognize information about their surrounding environment through audio.

[1561] (Application example 1)

[1562] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1563] For visually impaired people to move around safely, it is important to accurately grasp their surroundings and receive appropriate feedback. However, conventional technologies have made it difficult for visually impaired people to recognize surrounding dangers in real time and take appropriate action. In addition, there is a lack of systems that enable visually impaired people to move around independently without the assistance of sighted people, which has hindered their independence.

[1564] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1565] In this invention, the server includes: means for analyzing space using generative AI based on image information acquired by a wearable camera, accurate location information from a location information system, and highly accurate map information from a map information system for visually impaired people; means for providing feedback to users in natural language as information with low latency and excellent real-time performance using 5G communications; means for providing guidance to destinations and timely risk assessment of the surrounding environment without the intervention of sighted people; and means for detecting dangerous situations and obstacles using a generative AI model and notifying the user through audio and haptic feedback, thereby enabling visually impaired people to recognize surrounding dangers in real time and move safely.

[1566] "Visually impaired persons" refer to people who have visual impairments and are unable to obtain sufficient visual information.

[1567] A "wearable camera" is a camera that can be worn by the user, and is primarily a device attached to a head-mounted display.

[1568] A "location information system" refers to a system that uses technology such as GPS to accurately identify a user's current location.

[1569] A "map information system" is a system that provides geographical information and includes highly accurate map data.

[1570] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate new information and predictions.

[1571] "5G communication" refers to the fifth-generation mobile communication system, a technology that enables high-speed, low-latency communication.

[1572] "Natural language feedback" refers to providing information in natural language that is easy for users to understand.

[1573] "Sighted person" refers to a person who has no visual impairment and can normally obtain visual information.

[1574] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task.

[1575] "Dangerous Situation" means any environment or situation that may potentially pose a hazard to the User.

[1576] "Obstacle" means any object or structure that impedes the user's movement.

[1577] "Audio feedback" refers to a means of providing information to a user using audio.

[1578] "Haptic feedback" refers to a means of providing information to a user through the sense of touch.

[1579] A system for implementing this invention includes a wearable camera worn by a visually impaired person, a location information system, a map information system, a generative AI model, 5G communication, audio feedback, and haptic feedback.

[1580] System configuration

[1581] 1. Wearable camera: Built into a head-mounted display worn by visually impaired people, it captures the surrounding environment in real time.

[1582] 2. Location systems: Using technologies such as GPS to pinpoint a user's exact location.

[1583] 3. Map information system: Provides highly accurate map data and links it to the user's location information.

[1584] 4. Generative AI model: Analyzes image information acquired by a wearable camera and detects dangerous situations and obstacles.

[1585] 5. 5G communications: Realizing high-speed, low-latency communications, processing and transmitting information in real time.

[1586] 6. Audio feedback: Notify the user of detected information via audio.

[1587] 7. Haptic feedback: Providing information to the user through touch when needed.

[1588] Program processing

[1589] The server receives image information from the wearable camera and analyzes it using a generative AI model. This analysis detects dangerous situations and obstacles. The detection results are sent to the user's device in real time via 5G communication. The user's device then provides information to the user through audio and haptic feedback.

[1590] Hardware and software used

[1591] Hardware: Head-mounted display, smartphone, GPS module

[1592] Software: OpenCV (image processing library), TensorFlow (generative AI model framework), pyttsx3 (voice synthesis library)

[1593] Specific examples

[1594] When a visually impaired person is walking down a sidewalk and sees an obstacle ahead, a wearable camera captures the image. A generative AI model analyzes the image and issues a voice notification saying, "There is an obstacle ahead." This notification is sent in real time via 5G communication, allowing the user to respond immediately.

[1595] Prompt Sentence Examples

[1596] "Detect if there are any obstacles ahead."

[1597] This prompt is used to instruct the generative AI model on a specific task.

[1598] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1599] Step 1:

[1600] The user puts on the wearable camera and starts the system.

[1601] Input: System startup via user interaction

[1602] Output: The camera begins capturing the surrounding environment in real time.

[1603] Specific operation: A user puts on the head-mounted display and turns on the system. The camera starts up and starts capturing video.

[1604] Step 2:

[1605] The video data acquired by the camera is sent to the server.

[1606] Input: Real-time video captured by a camera

[1607] Output: Video data sent to the server

[1608] Specific operation: The video data captured by the camera is sent to the server via 5G communication.

[1609] Step 3:

[1610] The video data received by the server is input into the generative AI model for analysis.

[1611] Input: Video data sent to the server

[1612] Output: Analysis results (detection of dangerous situations and obstacles)

[1613] How it works: The server preprocesses the video data it receives using OpenCV and then inputs it into a generative AI model trained with TensorFlow. The model analyzes the video and detects dangerous situations and obstacles.

[1614] Step 4:

[1615] The server sends the analysis results to the user's device via 5G communication.

[1616] Input: Analysis results from generative AI model

[1617] Output: Analysis results sent to the user's device

[1618] Specific operation: The server sends the analysis results to the user's smartphone in real time.

[1619] Step 5:

[1620] The user's device receives the analysis results and generates audio and haptic feedback.

[1621] Input: Analysis results sent from the server

[1622] Output: Audio and haptic feedback

[1623] Specific operation: The user's smartphone receives the analysis results and generates audio feedback using pyttsx3, and also generates haptic feedback if necessary.

[1624] Step 6:

[1625] The user receives audio and haptic feedback and takes appropriate action.

[1626] Input: Audio and haptic feedback

[1627] Output: User action (e.g., avoiding obstacles)

[1628] Specific actions: The user receives audio and haptic feedback and takes appropriate action, such as avoiding an obstacle ahead.

[1629] In this way, visually impaired people can recognize dangers around them in real time and move around safely.

[1630] Example 2

[1631] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1632] For visually impaired people to travel safely and efficiently, they need a system that can grasp information about their surroundings and their location in real time and guide them to the appropriate route. However, conventional systems often take time to acquire and analyze information, and lack real-time capabilities. In addition, there is insufficient support for visually impaired people to travel independently, and they often need the help of sighted people. This limits the freedom of movement of visually impaired people.

[1633] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image information captured by a user, means for acquiring accurate location information from a location information system, means for acquiring highly accurate map information from a map information system, means for transmitting the acquired image information, location information, and map information to a generative AI, means for the generative AI to analyze space based on this information and generate an optimal route for the visually impaired person to travel, means for transmitting the generated optimal route information to the user's terminal, and means for the user's terminal to display the received route information to the user. This enables the visually impaired person to grasp information about their surrounding environment in real time and travel safely and efficiently.

[1634] "User" refers to a visually impaired person who uses the system to take images and receive route information.

[1635] "Image information" refers to visual data captured by a user using a terminal.

[1636] A "location information system" refers to a system that uses technology such as GPS to determine a person's current location.

[1637] "Precise location information" refers to latitude and longitude data obtained from a location information system.

[1638] "Map information system" refers to a system that provides geographical information.

[1639] "High-precision map information" refers to detailed geographic data obtained from a map information system.

[1640] "Generative AI" refers to artificial intelligence that analyzes space based on the data it receives and generates the optimal route.

[1641] "Analyzing space" refers to the process of understanding the surrounding environment based on image information, location information, and map information.

[1642] An "optimal route" refers to a route that allows visually impaired people to travel safely and efficiently.

[1643] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[1644] "Route information" refers to data regarding travel routes generated by generative AI.

[1645] "Displaying" refers to the terminal providing the route information to the user visually or audibly.

[1646] This invention is a system for enabling visually impaired people to travel safely and efficiently, in which a generative AI generates an optimal route based on image information taken by the user, accurate location information obtained from a location information system, and highly accurate map information obtained from a map information system.

[1647] Hardware and software used

[1648] Location information system: GPS module

[1649] Map Information System: Geographical Information API

[1650] Generative AI: Artificial intelligence models (e.g., OpenAI GPT-4)

[1651] System configuration

[1652] server

[1653] The server receives image information taken by the user on the device. Specifically, the image data is sent to the server via an HTTP POST request. Next, the server obtains the current location information from the location information system. This is done by sending an API request to the GPS module and receiving latitude and longitude data. The server then obtains highly accurate map information from the map information system. This is done by sending a request to the geographic information provision API and receiving map data based on the specified location information.

[1654] The server sends the acquired image information, location information, and map information to the generative AI. This data is converted into JSON format and a POST request is sent to the generative AI's API endpoint. The generative AI analyzes the space based on the received information and generates the optimal route for the visually impaired person to travel. The generated optimal route information is sent from the server to the user's device.

[1655] Terminal

[1656] The device sends the image information taken by the user to the server. Specifically, the device's camera app captures the image and sends it to the server via an HTTP POST request. The device then displays the optimal route information received from the server to the user. The device application analyzes the route information and provides visual or audio guidance to the user.

[1657] User

[1658] The user takes an image using the device. Specifically, they open the camera app on their smartphone and capture the desired scene. Next, the user moves based on the optimal route information displayed on the device. They safely travel to their destination while checking the device's screen and audio guidance.

[1659] Specific examples

[1660] The user takes an image with their smartphone. The smartphone (device) sends the captured image to the server via an HTTP POST request. The server obtains the current location information from the GPS module and obtains highly accurate map information using a geographic information provision API. The server sends the image information, location information, and map information to the generative AI. The generative AI analyzes the space based on this information and generates the optimal route for the visually impaired person to travel. The server then sends the generated optimal route information to the smartphone. The smartphone displays the optimal route information to the user and provides visual and audio guidance.

[1661] Prompt Sentence Examples

[1662] "Generate the optimal route for visually impaired people based on image information taken by the user, location information obtained from GPS, and map information obtained from the map information providing API."

[1663] In this way, the system helps visually impaired people navigate safely and efficiently.

[1664] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1665] Step 1:

[1666] The server receives image information taken by the user on the device. Specifically, the user takes an image using the camera app on their smartphone, and the image data is sent to the server via an HTTP POST request. The input is the image information taken by the user, and the output is the image data stored on the server.

[1667] Step 2:

[1668] The server obtains precise location information from the location information system. Specifically, the server sends an API request to the GPS module and receives the current latitude and longitude data. The input is the location request from the GPS module, and the output is the latitude and longitude data stored on the server.

[1669] Step 3:

[1670] The server obtains highly accurate map information from the map information system. Specifically, the server sends a request to the geographic information API and receives map data based on the specified location information. The input is the request to the geographic information API, and the output is highly accurate map information stored on the server.

[1671] Step 4:

[1672] The server sends the acquired image information, location information, and map information to the generative AI. Specifically, it converts this data into JSON format and sends a POST request to the generative AI's API endpoint. The input is the image information, location information, and map information, and the output is the data sent to ...

Claims

1. A means for analyzing space using generative AI based on image information acquired by a wearable camera, location information from a location information system, and map information from a map information system; a means for generating information for recognizing a surrounding environment based on information obtained by the means for analyzing the space; A means for providing feedback to a user in natural language by voice using 5G communication to provide information for recognizing the surrounding environment; A system including:

2. The system according to claim 1, wherein the generation system AI generates a route suitable for travel based on information for recognizing the surrounding environment, and the feedback means further provides feedback of the route to the user by voice in natural language.

3. The system of claim 2 , further comprising: a wearable camera that recognizes an emotional state of a user of the wearable camera; and, if the emotional state is recognized to be anxiety, provides the user with route information that is more detailed than the route via voice feedback in natural language.

Images