System

Abstract

To enable a user to know the situation in front of him / her in real time.SOLUTION: A system comprises: means for notifying a user walking with a mobile information terminal hanging from the neck of the situation in front of him / her by voice, and analyzing video data related to the situation around the user acquired by an imaging device of the mobile information terminal with image recognition software to identify the positions of objects and people; means for generating an explanatory text in a natural language using a generative AI model on the basis of the analysis result of the portable information terminal; means for converting the text generated by the portable information terminal into the voice using voice synthesis software; and means for allowing the portable information terminal to reproduce the converted voice file and outputting the voice through a speaker of the mobile information terminal in real time during user movement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When out and about, visually impaired people are constantly exposed to dangers such as traffic accidents, railroad crossing accidents, falls from train platforms, etc. To protect visually impaired people from these dangers, it is necessary to know the situation in front of them in real time. [Means for solving the problem]

[0005] This invention provides a means for visually impaired people to walk with a smartphone hanging around their neck and be notified of the situation in front of them by voice, a means for recognizing the surrounding situation using the smartphone's camera and converting that information into voice, and a means for transmitting that voice information to the visually impaired in real time. This allows the visually impaired to know the situation in front of them in real time and protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15]FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] This invention is a system that notifies visually impaired people of the situation in front of them by voice by hanging a smartphone around their neck. Specifically, the system uses the smartphone's camera to recognize the surrounding situation and converts that information into voice. This voice information is transmitted to the visually impaired in real time.

[0029] "Example 2"

[0030] The audio information provides specific information about the situation, such as "There is a railroad crossing ahead," "Three people are walking towards you from the left," "There is a telephone pole ahead to the right," etc. This allows visually impaired people to know the situation in front of them in real time, and helps protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0031] "Example 3"

[0032] As a specific example of use, if a visually impaired person is standing on a train platform, their smartphone will provide audio information such as, "The end of the platform is in front of you. There are stairs on your right." This will help the visually impaired person avoid the risk of falling off the platform. Also, while walking, the smartphone can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: A visually impaired person wears a smartphone around their neck.

[0036] Step 2: The smartphone camera recognizes the surroundings.

[0037] Step 3: Convert the recognized information into speech.

[0038] Step 4: Transmit audio information to the visually impaired in real time.

[0039] "Example 2"

[0040] Step 1: Voice information is generated. This information indicates specific situations such as "There is a railroad crossing ahead," "Three people are walking towards you from the left," and "There is a telephone pole ahead to the right."

[0041] Step 2: The visually impaired person listens to the audio information and understands the situation in front of them.

[0042] Step 3: The visually impaired person adjusts their behavior based on the audio information to protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0043] "Example 3"

[0044] Step 1: When a visually impaired person is standing on a station platform, their smartphone will communicate information via voice, such as, "The end of the platform is in front of you. There are stairs on your right."

[0045] Step 2: Visually impaired people listen to the audio information and avoid the risk of falling off the platform.

[0046] Step 3: While walking, information such as "A bicycle is approaching from the front left" is provided to prevent collisions.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] For the visually impaired to travel safely, it is important to be able to grasp the situation around them in real time. However, conventional technology has made it difficult for the visually impaired to accurately and quickly recognize the situation around them. In particular, there has been a lack of means to provide information to help them avoid dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms. This has hindered the safe travels of the visually impaired.

[0050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0051] In this invention, the server includes means for analyzing video data transmitted from the mobile information terminal and recognizing objects and text, means for generating the recognized information as text data, and means for converting the text data into audio data. This allows visually impaired people to use their mobile information terminals to receive audio information about their surroundings in real time. This allows visually impaired people to avoid dangers such as traffic accidents, railroad crossing accidents, and falls from train platforms, and to travel safely.

[0052] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0053] A "personal digital assistant" is a portable electronic device that is equipped with communication functions and an imaging device.

[0054] "Imaging device" refers to a device for capturing images, such as a camera or sensor.

[0055] "Video data" refers to data that digitally represents visual information captured by an imaging device.

[0056] A "server" refers to a computer system that processes and manages data over a network.

[0057] "Analysis" refers to the process of examining data in detail to understand its structure and content.

[0058] "Object recognition" refers to a technology that identifies specific objects within video data and determines their location and type.

[0059] "Text data" refers to data that represents character information in digital form.

[0060] "Audio data" refers to data that represents audio information in digital form.

[0061] "Speech synthesis" refers to the technology of generating voice data based on text data.

[0062] "Real-time" refers to processing and transmission occurring immediately without delay.

[0063] "User" refers to a visually impaired person who uses this system.

[0064] This invention is a system that uses a portable information terminal hung around the neck of a visually impaired person to provide audio information about the situation in front of them. Specifically, the system uses the imaging device of the portable information terminal to recognize the surrounding situation and converts that information into audio. This audio information is transmitted to the visually impaired person in real time.

[0065] The server receives video data sent from the mobile information terminal. The server uses computer vision technology to analyze the video data. Specifically, it uses libraries such as TENSORFLOW (registered trademark) and OpenCV to recognize objects and text in the video. The server generates the recognized information as text data. Next, the server uses speech synthesis software to convert this text data into speech. Specifically, it uses a text-to-speech API or a speech synthesis engine.

[0066] The terminal (mobile information terminal) receives the voice data sent from the server and transmits it to the user in real time through a speaker, allowing visually impaired people to hear what is happening in front of them.

[0067] As a concrete example, consider a scene where a user is walking through town. The device's camera captures a traffic light ahead, and the device sends this image to a server. The server analyzes the image and recognizes that the traffic light is red. The server generates text saying, "There is a red light ahead. Please stop," and converts this into audio data. The audio data is sent to the device, which then tells the user through a speaker, "There is a red light ahead. Please stop."

[0068] Example prompt sentence:

[0069] "Please create a program that uses the imaging device of a mobile information terminal to recognize the situation ahead and notify the user by voice that the traffic light is red."

[0070] This system allows visually impaired people to travel safely, avoiding dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0071] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0072] Step 1:

[0073] The device captures the surroundings in real time using a built-in image capture device. The input is the image data from the image capture device, and the output is the captured image data. Specifically, the device continues to capture images at a constant frame rate.

[0074] Step 2:

[0075] The terminal transmits the captured video data to a server via the Internet. The input is the captured video data, and the output is the video data transmitted to the server. Specifically, the terminal transmits the video data in a compressed format (e.g., H.264).

[0076] Step 3:

[0077] The server receives the video data transmitted from the terminal. The input is the video data transmitted from the terminal, and the output is the received video data. In concrete operation, the server receives the data via the network.

[0078] Step 4:

[0079] The server analyzes the received video data and recognizes objects and text. The input is the received video data, and the output is the recognized objects and text information. Specifically, the server performs the analysis using computer vision libraries such as TensorFlow and OpenCV.

[0080] Step 5:

[0081] The server generates text data based on the recognized objects and text information. The input is the recognized objects and text information, and the output is the generated text data. Specifically, the server generates text such as "There is a red light ahead. Please stop."

[0082] Step 6:

[0083] The server converts the generated text data into voice data. The input is the generated text data, and the output is the generated voice data. Specifically, the server generates voice data using a Text-to-Speech API or a voice synthesis engine.

[0084] Step 7:

[0085] The server sends the generated audio data to the terminal via the Internet. The input is the generated audio data, and the output is the audio data sent to the terminal. Specifically, the server sends the audio data in a compressed format (e.g., MP3).

[0086] Step 8:

[0087] The terminal plays back the received voice data and transmits it to the user. The input is the received voice data, and the output is the voice information transmitted to the user. In concrete terms, the terminal plays back voice through a speaker and tells the user, "There is a red light ahead. Please stop."

[0088] (Application example 1)

[0089] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Visually impaired people face difficulties in understanding their surroundings when searching for products in physical stores or navigating specific locations. There is also a lack of support for visually impaired people to navigate safely and find the products they need.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0092] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk around with the information processing device hanging around their neck, means for recognizing the surrounding situation using an imaging device in the information processing device and converting that information into voice, means for transmitting that voice information to the visually impaired person in real time, and means for recognizing the surrounding situation when the visually impaired person is searching for a product or moving to a specific location in a physical store and transmitting that information by voice in real time, thereby enabling the visually impaired person to move safely within the physical store and find the products they need.

[0093] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0094] "Information processing device" refers to an electronic device that has the function of inputting, processing, and outputting data.

[0095] An "imaging device" refers to a device that converts light into an electrical signal and generates image data.

[0096] "Surroundings" refers to the state of the objects and environment around a visually impaired person.

[0097] "Means for converting into audio" refers to technology or devices that convert image data or other information into audio data.

[0098] "Real-time" refers to processing and transmission occurring immediately without delay.

[0099] "Brick and mortar store" refers to a store that exists in a physical location and offers goods and services.

[0100] "Products" refers to items sold in physical stores.

[0101] "Specific location" refers to the specific location to which the visually impaired person wishes to travel.

[0102] The system for carrying out this invention provides a function that notifies a visually impaired person of the situation in front of them by voice when they walk while wearing an information processing device around their neck. A specific embodiment of this system will be described below.

[0103] 1. System Configuration

[0104] The system consists of the following main components:

[0105] Information processing devices: Portable electronic devices such as smartphones and tablets.

[0106] Imaging device: A camera built into an information processing device.

[0107] Audio output device: A speaker built into the information processing device or a connected earphone.

[0108] Software: Applications for image recognition, speech synthesis, and real-time processing.

[0109] 2. Program Processing

[0110] The server implements the system using the following means:

[0111] Image recognition means: Analyzes video data captured by an imaging device and recognizes the surrounding situation. Specifically, object recognition is performed using machine learning libraries such as TensorFlow.

[0112] Speech synthesis means: Converts the recognized information into speech data. Specifically, it uses a speech synthesis library such as pyttsx3.

[0113] Real-time communication: Audio data is transmitted to the visually impaired in real time, allowing them to instantly understand their surroundings.

[0114] 3. Hardware and Software Used

[0115] Hardware: Smartphone camera, speakers, earphones.

[0116] Software: OpenCV (image processing library), TensorFlow (machine learning library), pyttsx3 (speech synthesis library).

[0117] 4. Specific Examples

[0118] For example, if a visually impaired person is looking for "milk" in a physical store, the camera in the information processing device will scan the shelves, and if "milk" is recognized, a voice message will be sent saying, "Milk found." In this way, visually impaired people can safely move around the physical store and find the products they need.

[0119] 5. Examples of prompts

[0120] An example of a prompt to input to a generative AI model is as follows:

[0121] "Design a navigation system to help visually impaired people find products in a physical store. The system uses a smartphone camera to recognize the surroundings and communicates that information in real time via voice."

[0122] In this way, a system can be realized that helps visually impaired people shop more smoothly in physical stores.

[0123] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0124] Step 1:

[0125] The server activates the imaging device of the information processing device and acquires surrounding video data. The input is video data from the camera, and the output is the acquired video frames. Specifically, the camera continuously captures video and generates frames in real time.

[0126] Step 2:

[0127] The server preprocesses the captured video frames. The input is the video frame captured in step 1, and the output is the preprocessed image data. Specifically, the server resizes the video frame and performs noise removal and color correction.

[0128] Step 3:

[0129] The server inputs the preprocessed image data into an object recognition model to recognize the surrounding situation. The input is the preprocessed image data, and the output is the label and location information of the recognized object. Specifically, it uses TensorFlow's object recognition model to detect objects in the image and identify their label and location.

[0130] Step 4:

[0131] The server converts the information of the recognized object into audio data. The input is the label and location information of the recognized object, and the output is audio data. Specifically, it uses the pyttsx3 speech synthesis library to convert the text information into audio.

[0132] Step 5:

[0133] The server transmits the generated voice data to the visually impaired person in real time. The input is voice data, and the output is voice information that the visually impaired person hears. Specifically, the voice is played through the speaker or earphones of the information processing device.

[0134] Step 6:

[0135] Users navigate and search for products within a physical store based on voice information. The input is voice information transmitted from the server, and the output is the user's actions. Specifically, the user follows the voice guidance to walk and find the desired product or location.

[0136] Example 2

[0137] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] For the visually impaired to travel safely, it is important to be able to grasp the situation around them in real time. However, with conventional technology, it is difficult for the visually impaired to accurately and quickly recognize their surroundings, which has led to risks such as traffic accidents, railroad crossing accidents, and falls from station platforms. For this reason, there is a need for a system that enables the visually impaired to grasp the situation around them in real time and travel safely.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0140] In this invention, the server includes: means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck; means for recognizing the surrounding situation using an imaging device of the mobile device and converting that information into voice; means for transmitting the voice information to the visually impaired person in real time; means for the mobile device to acquire surrounding images and distance information using an imaging device and a distance measuring device; means for the mobile device to analyze the acquired video data using image recognition software and identify the positions of objects and people; means for the mobile device to generate a natural language explanatory text using a generative AI model based on the analysis results; means for the mobile device to convert the generated text into voice using voice synthesis software; and means for the mobile device to play the generated voice file and output the voice from a speaker. This allows the visually impaired person to grasp the surrounding situation in real time and move around safely.

[0141] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0142] A "mobile terminal" is a portable electronic device that has communication and computing capabilities.

[0143] "Imaging device" refers to a device, such as a camera or sensor, that receives light and generates image data.

[0144] "Distance measuring device" refers to a sensor or device used to measure the distance to an object.

[0145] "Image recognition software" refers to a program that analyzes image data and identifies objects and people.

[0146] "Generative AI model" refers to a model that uses artificial intelligence to generate natural language text from data.

[0147] "Speech synthesis software" refers to a program for converting text data into voice data.

[0148] "Audio information" refers to information transmitted as sound, and includes content intended to inform visually impaired people of their surroundings.

[0149] "Real-time" refers to processing and transmission occurring immediately without delay.

[0150] A "speaker" refers to a device that converts audio signals into sound waves and outputs them.

[0151] This invention is a system that enables visually impaired people to grasp their surroundings in real time and move around safely. Specific embodiments of this system will be described below.

[0152] Hardware and software used

[0153] Hardware: Mobile devices (smartphones and tablets), imaging devices (cameras), distance measurement devices (distance sensors), speakers

[0154] Software: Image recognition software (e.g., OpenCV), speech synthesis software (e.g., Google® Text-to-Speech), generative AI models (e.g., GPT-4®)

[0155] System Overview

[0156] This system allows visually impaired people to walk around with a mobile device hanging from their neck and receive audio notifications of the situation in front of them. The mobile device's imaging device and distance measurement device are used to capture images of the surrounding area and distance information, and the data is analyzed and provided to the user as audio information.

[0157] Data Acquisition

[0158] The device activates the camera to capture images of what is ahead in real time, while the distance sensor measures the distance to objects, allowing visually impaired people to have a detailed understanding of their surroundings while walking.

[0159] Data analysis

[0160] The image data acquired by the device is analyzed using image recognition software (OpenCV) to identify the location of objects and people. For example, it can recognize from the camera image that "three people are walking to the left and in front." Based on the results of this analysis, a generative AI model (GPT-4) is used to generate a natural language description.

[0161] Status text generation

[0162] The device inputs the analysis results as prompts into the generative AI model to generate appropriate explanatory text. An example of a prompt is "Please explain the situation recognized from the camera image. Example: There is a railroad crossing in front of you, three people are walking from the front left, and there is a utility pole in front of you on the right."

[0163] Speech synthesis

[0164] The device converts the generated text into speech using speech synthesis software (Google Text-to-Speech). The generated audio file is high-quality and adjusted to be easy for the user to hear.

[0165] Providing audio information

[0166] The device plays the generated audio file and outputs the audio through the speaker at an appropriate volume, allowing the user to accurately understand their surroundings.

[0167] Specific examples

[0168] When a user is walking on the sidewalk, the device will announce, "Three people are walking towards you from the left." Or, when the user is on a train platform, the device will announce, "There is a utility pole ahead of you on the right." This allows visually impaired people to understand their surroundings in real time and move around safely.

[0169] This system allows visually impaired people to travel safely, avoiding dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0170] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0171] Step 1:

[0172] The device activates the camera and captures the image in front of it in real time. The input is the image data from the camera, and the output is the captured image frames. Specifically, the device captures images at 30 frames per second and stores them in its internal memory.

[0173] Step 2:

[0174] The device activates the distance sensor and measures the distance to the object. The input is distance data from the distance sensor, and the output is the measured distance information. Specifically, the device measures the distance 10 times per second and synchronizes it with the video frame.

[0175] Step 3:

[0176] The device analyzes the video data it acquires using image recognition software (OpenCV) to identify the location of objects and people. The input is the captured video frame, and the output is the location information of the object or person. Specifically, the device detects the contours of objects in each frame and lists the type and location of the identified objects.

[0177] Step 4:

[0178] Based on the analysis results, the device uses a generative AI model (GPT-4) to generate a natural language description. The input is the location information of objects and people, and the output is the generated natural language description. Specifically, the device inputs a prompt statement such as "Please explain the situation recognized from the camera footage. Example: There is a railroad crossing in front of you, three people are walking from the front left, there is a utility pole in front of you on the right" into the generative AI model, and an appropriate description is generated.

[0179] Step 5:

[0180] The device converts the generated text into speech using speech synthesis software (Google Text-to-Speech). The input is the generated natural language description, and the output is an audio file. Specifically, the device inputs the generated text into the speech synthesis software, which generates a high-quality audio file.

[0181] Step 6:

[0182] The device plays the generated audio file and outputs the audio from the speaker. The input is the audio file and the output is the audio from the speaker. Specifically, the device plays the audio file and outputs the audio from the speaker at an appropriate volume.

[0183] (Application example 2)

[0184] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] For visually impaired people to travel safely, it is important to understand the surrounding situation in real time. However, current technology lacks the means to properly recognize the surrounding situation and provide audio information when visually impaired people use self-driving vehicles. This poses the issue of insufficient safety when visually impaired people use self-driving vehicles.

[0186] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a visually impaired person walking with a mobile terminal hanging around their neck to be notified of the situation in front of them by voice; means for recognizing the surrounding situation using the camera of the mobile terminal and converting the information into voice; means for transmitting the voice information to the visually impaired person in real time; means for collecting surrounding situation data from the sensors of the autonomous vehicle; means for analyzing the data and identifying objects; means for analyzing the positional relationship and movement of the identified objects and extracting important information; means for converting the extracted information into voice information; and means for providing the voice information to the visually impaired person through an on-board speaker. This enables a visually impaired person using an autonomous vehicle to grasp the surrounding situation in real time and travel safely.

[0187] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0188] A "mobile device" is a portable electronic device, including a smartphone, tablet, etc.

[0189] A "camera" is a device that converts light into an electrical signal and captures images or videos.

[0190] "Audio information" is information that is transmitted using sound and is used to inform visually impaired people of their surroundings.

[0191] "Real-time" refers to processing and transmission occurring immediately without delay.

[0192] An "autonomous vehicle" is a vehicle that can drive autonomously without the need for driver operation.

[0193] A "sensor" is a device that detects a physical phenomenon and converts it into an electrical signal.

[0194] "Data" is a collection of information obtained through observation or measurement.

[0195] "Analysis" is the process of examining data in detail to clarify its meaning and structure.

[0196] An "object" refers to something that has a concrete form and exists in space.

[0197] The "positional relationship" is a relationship that indicates how multiple objects are arranged in space.

[0198] "Motion" refers to an object changing its position over time.

[0199] "Extraction" refers to extracting the necessary parts from a large amount of information.

[0200] An "in-vehicle speaker" is a device installed inside a vehicle to output sound.

[0201] The system for implementing this invention enables visually impaired people to understand the surrounding situation in real time and travel safely when using an autonomous vehicle. Specific embodiments of this system are described below.

[0202] Hardware and software used

[0203] Hardware: Autonomous vehicle sensors (cameras, LiDAR, radar), on-board computers, mobile devices, and on-board speakers

[0204] Software: Image recognition software (OpenCV), speech synthesis software (Google Text-to-Speech API), generative AI model (GPT-4)

[0205] Data processing and calculation

[0206] 1. Data collection: Collecting surrounding situation data from the autonomous vehicle's sensors (cameras, LiDAR, radar), which provides information about objects and obstacles around the vehicle.

[0207] 2. Image Recognition: The collected data is analyzed using image recognition software (OpenCV) to identify objects (people, cars, utility poles, etc.), thereby extracting information that is important for visually impaired people.

[0208] 3. Situational Assessment: Using a generative AI model (GPT-4), the system analyzes the relative positions and movements of identified objects to extract information that is important to the visually impaired, providing them with information to navigate safely.

[0209] 4. Speech generation: The extracted information is converted into speech using speech synthesis software (Google Text-to-Speech API), allowing visually impaired people to understand their surroundings in real time.

[0210] 5. Audio output: The generated audio information is provided to the visually impaired through the vehicle's speakers, allowing them to navigate safely within the autonomous vehicle.

[0211] Specific examples

[0212] For example, when a vehicle approaches an intersection, the following prompt sentence is input into the generative AI model:

[0213] Example prompt sentence:

[0214] "The vehicle is currently approaching an intersection. Three pedestrians are crossing from the left. A bicycle is approaching on the right. There is a traffic light ahead. Please provide this information in a voice message that is easy to understand for visually impaired people."

[0215] Based on this prompt, the generative AI model generates the following audio information:

[0216] "You are approaching an intersection. Three pedestrians are crossing from the left. A bicyclist is approaching on the right. There is a traffic light ahead."

[0217] This audio information is provided to the visually impaired through the vehicle's speakers, allowing them to understand their surroundings in real time and move around safely.

[0218] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0219] Step 1:

[0220] The server collects surrounding situation data from the autonomous vehicle's sensors (camera, LiDAR, radar). The input is raw data from the sensors, and the output is the collected surrounding situation data. Specifically, the sensors scan the vehicle's surroundings and detect the position and movement of objects.

[0221] Step 2:

[0222] The server analyzes the collected surrounding situation data using image recognition software (OpenCV) to identify objects. The input is the surrounding situation data collected in step 1, and the output is a list of identified objects. Specifically, the image recognition algorithm analyzes the data and identifies objects such as people, cars, and utility poles.

[0223] Step 3:

[0224] The server uses a generative AI model (GPT-4) to analyze the relative positions and movements of the identified objects and extract information that is important to visually impaired people. The input is a list of objects identified in step 2, and the output is a list of important information. Specifically, the generative AI model analyzes the relative positions and movements of objects and selects information that is important to visually impaired people.

[0225] Step 4:

[0226] The server converts the extracted information into audio information using speech synthesis software (Google Text-to-Speech API). The input is the list of important information extracted in step 3, and the output is the generated audio information. Specifically, the speech synthesis software converts the text information into audio data.

[0227] Step 5:

[0228] The server provides the generated audio information to the visually impaired person through the car's speaker. The input is the audio information generated in step 4, and the output is the audio information provided to the visually impaired person. Specifically, the audio data is played through the car's speaker, informing the visually impaired person of the surrounding situation.

[0229] Example 3

[0230] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] For visually impaired people to travel safely, they need a way to accurately grasp their surroundings in real time. However, conventional systems have had difficulty accurately recognizing the surroundings and transmitting appropriate voice information. In particular, the provision of information to help them avoid dangers on train platforms and while walking has been insufficient.

[0232] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0233] In this invention, the server includes means for recognizing the surrounding situation using the camera of the mobile device and sending the information to the central processing unit, means for the central processing unit to recognize objects using video analysis software and generate a natural language explanatory text using a generative AI model, and means for sending the explanatory text to the mobile device and converting it into voice using voice synthesis software, thereby enabling visually impaired people to accurately grasp the surrounding situation in real time and move around safely.

[0234] "Visually impaired persons" refer to people who have visual impairments and have difficulty using visual information to understand their surroundings.

[0235] A "mobile terminal" is a portable electronic device that is equipped with a camera and communication functions.

[0236] "Capture device" refers to a device that includes a camera and sensors for capturing images.

[0237] "Central processing unit" refers to a computer or server that analyzes received data and performs the necessary processing.

[0238] "Video analysis software" refers to a program that analyzes captured video and recognizes objects and situations.

[0239] A "generative AI model" refers to an artificial intelligence model that generates natural language explanations based on input data.

[0240] "Natural language descriptions" refer to text that verbally describes the surroundings in a way that is easy for visually impaired people to understand.

[0241] "Speech synthesis software" refers to a program for converting text data into speech.

[0242] "Audio Information" refers to audio data generated by speech synthesis software.

[0243] This invention is a support system for visually impaired people to move around safely, and is implemented using a mobile device, a central processing unit, video analysis software, a generative AI model, and speech synthesis software.

[0244] First, the user activates the camera on their mobile device and captures images of their surroundings. The mobile device then transmits the captured images to a central processing unit in real time. The central processing unit then analyzes the received images using video analysis software (e.g., OpenCV) and recognizes objects and situations.

[0245] Next, the server inputs information about the recognized object into a generative AI model (e.g., GPT-4). The generative AI model generates a natural language description based on the input information. For example, it generates a description such as, "The edge of the platform is in front of you. There are stairs on your right."

[0246] The generated description is sent from the server to the mobile device, which then converts it into speech using speech synthesis software (e.g., Google Text-to-Speech). The converted speech is then transmitted to the user through the mobile device's speaker.

[0247] As a concrete example, consider the case where a user is standing on a train platform. The camera on the mobile device recognizes the edge of the platform and stairs and sends this information to the central processing unit. The central processing unit analyzes the image and uses a generative AI model to generate a description such as, "The edge of the platform is in front of you. There are stairs on your right." The description is sent to the mobile device, converted into audio, and conveyed to the user.

[0248] The same applies when a user is walking and the camera on the mobile device detects an approaching bicycle. The central processing unit generates a description such as "A bicycle is approaching from the front left" and sends it to the mobile device. The mobile device then converts the description into voice and conveys it to the user.

[0249] An example of a prompt is as follows:

[0250] "Generate natural language descriptions to describe the surroundings to a visually impaired person standing on a train platform. The camera footage shows the edge of the platform and a staircase."

[0251] This system enables visually impaired people to accurately grasp the surrounding situation in real time and move around safely. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0252] Step 1:

[0253] A user activates the camera on their mobile device and captures a video of their surroundings. The input is the real-time video captured by the camera. The output is the captured video data. Specifically, the user opens the camera app on their mobile device and points the camera at their surroundings.

[0254] Step 2:

[0255] The terminal transmits the captured video data to the central processing unit. The input is the video data captured by the camera of the mobile terminal. The output is the video data transmitted to the central processing unit. In concrete terms, the terminal uploads the video data to a server via the Internet.

[0256] Step 3:

[0257] The server analyzes the received video data using video analysis software. The input is the video data sent from the device. The output is information about the analyzed objects and situation. Specifically, the server uses video analysis software (e.g., OpenCV) to recognize objects in the video (e.g., the edge of the platform, stairs, bicycles, etc.).

[0258] Step 4:

[0259] The server inputs information about the recognized object into a generative AI model and generates a natural language description. The input is information about the object recognized by video analysis software. The output is the generated natural language description. Specifically, the server inputs the object information into a generative AI model (e.g., GPT-4) and generates a description such as, "The edge of the platform is in front of you. There are stairs on your right."

[0260] Step 5:

[0261] The server sends the generated explanation to the terminal. The input is the natural language explanation generated by the generative AI model. The output is the explanation sent to the terminal. Specifically, the server sends the explanation in text format to the terminal.

[0262] Step 6:

[0263] The device converts the received description into speech using speech synthesis software. The input is the description in natural language sent from the server. The output is the description converted into speech. Specifically, the device uses speech synthesis software (e.g., Google Text-to-Speech) to convert the description into speech.

[0264] Step 7:

[0265] The device transmits the explanatory text converted into voice to the user. The input is the explanatory text converted into voice. The output is the audio information transmitted to the user. In concrete terms, the device plays the audio information through a speaker to inform the user of the surrounding situation.

[0266] (Application example 3)

[0267] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] For visually impaired people to travel safely, it is important to accurately grasp their surroundings. However, with conventional technology, it is difficult for visually impaired people to grasp their surroundings in real time, which poses risks such as traffic accidents and falls from train platforms. In addition, there is a lack of means to communicate specific situations by voice, and the provision of information to ensure the safety of visually impaired people is insufficient.

[0269] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0270] In this invention, the server includes means for audio-informing a visually impaired person of the situation in front of them as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into audio, means for transmitting the audio information to the visually impaired person in real time, means for analyzing images from the camera device in the mobile device and detecting surrounding objects using an object detection model, and means for providing audio guidance in accordance with the detected objects, thereby enabling the visually impaired person to grasp the surrounding situation in real time and move around safely.

[0271] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0272] A "mobile terminal" is a portable electronic device that has communication and photographing functions.

[0273] "Image capture device" refers to a hardware component used to capture video or images.

[0274] An "object detection model" refers to an algorithm or program that uses machine learning and artificial intelligence technology to detect specific objects in video or images.

[0275] "Voice guidance" refers to a means of conveying information using voice.

[0276] "Real-time" refers to processing and transmission occurring immediately without delay.

[0277] "Surrounding circumstances" refers to the state of the objects, people, and environment around a visually impaired person.

[0278] "Audio information" refers to information that is conveyed using sound.

[0279] "Analysis" refers to examining data or information in detail to clarify its structure and meaning.

[0280] This invention is a system for enabling the visually impaired to move around safely, and provides real-time audio guidance of the surrounding situation using a mobile terminal. A specific embodiment of this system will be described below.

[0281] First, the visually impaired user walks around with a mobile device hanging from their neck. This mobile device is equipped with a camera that can capture images of the surroundings in real time. The images captured by the camera are analyzed by an object detection model built into the mobile device. This object detection model is built using machine learning frameworks such as TensorFlow, and is able to detect objects in the images with high accuracy.

[0282] Information about detected objects is communicated to the user as voice guidance. The voice guidance is generated using a speech synthesis engine such as pyttsx3 and provided to the user in real time. For example, if the user is standing on a train platform, the system will provide voice guidance such as, "The end of the platform is in front of you. There are stairs on your right." Also, while walking, the system can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[0283] The specific processing of this system involves first acquiring video from the mobile device's camera and inputting that video into an object detection model. The object detection model detects objects in the video and identifies their location and type. Next, audio guidance is generated based on the detected objects and conveyed to the user. This series of processes is performed in real time, allowing the user to always be aware of the latest surrounding situation.

[0284] For example, if a visually impaired person is standing on a train platform, the system will provide audio information such as, "The end of the platform is in front of you. There are stairs on your right." Also, while walking, the system can provide information such as, "A bicycle is approaching from the left," preventing collisions.

[0285] Examples of prompts to be input to a generative AI model include:

[0286] "I want to develop an application that provides audio guidance about the surroundings to visually impaired people standing on a train platform. The application captures images in real time using the camera on the mobile device, detects surrounding objects using an object detection model, and uses a voice engine to provide audio guidance to the user. Specifically, the application provides information about the edge of the platform, stairs, bicycles, etc."

[0287] In this way, the present invention provides an effective means for safe transportation for the visually impaired.

[0288] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0289] Step 1:

[0290] The terminal uses a camera to capture images of the surroundings in real time.

[0291] Input: Surrounding image

[0292] Output: Captured video data

[0293] Specific operation: The device's camera continuously captures video and stores the video data in memory.

[0294] Step 2:

[0295] The terminal inputs the captured video data into an object detection model.

[0296] Input: Captured video data

[0297] Output: Object detection results

[0298] Specific operation: The device uses a machine learning framework such as TensorFlow to input video data into an object detection model and detect objects in the video.

[0299] Step 3:

[0300] The device analyzes the object detection results and identifies the location and type of the detected object.

[0301] Input: Object detection results

[0302] Output: Object location and type information

[0303] Specific operation: The device analyzes the data obtained from the object detection model and identifies the location coordinates and type of each object (e.g., person, bicycle, stairs, etc.).

[0304] Step 4:

[0305] The terminal generates voice guidance based on the information of the identified object.

[0306] Input: Object location and type information

[0307] Output: Generated voice guidance

[0308] Specific operation: The device uses a speech synthesis engine such as pyttsx3 to generate voice guidance based on the information about the identified object. For example, it creates a voice message such as "A bicycle is approaching from the front left."

[0309] Step 5:

[0310] The terminal transmits the generated voice guidance to the user in real time.

[0311] Input: Generated voice prompt

[0312] Output: Audio information transmitted to the user

[0313] Specific operation: The terminal plays the generated voice guidance through the speaker to the user in real time, and the user can understand the surrounding situation by listening to the voice guidance.

[0314] This series of processes enables visually impaired users to grasp the situation around them in real time and move around safely.

[0315] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0316] "Example 1"

[0317] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system includes a means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a smartphone around their neck, a means for recognizing the surrounding situation using the smartphone's camera and converting that information into voice, and a means for transmitting voice information to the visually impaired person in real time. Furthermore, this system includes an emotion engine that recognizes the user's emotions.

[0318] "Example 2"

[0319] The emotion engine recognizes the user's emotion from the tone of their voice, volume, speaking style, etc. For example, when a user is angry, their voice tone is high, their volume is loud, and they speak quickly. The emotion engine recognizes these characteristics and determines the user's emotion as "anger."

[0320] "Example 3"

[0321] The system adjusts the way voice information is transmitted based on the user's emotions recognized by the emotion engine. For example, if the user is angry, the system reduces the amount of voice information transmitted. On the other hand, if the user is calm, the system transmits voice information normally. This makes it possible to provide services that take the user's emotions into consideration.

[0322] The processing flow of each embodiment will be described below.

[0323] "Example 1"

[0324] Step 1: A visually impaired person hangs the smartphone around their neck and activates the system.

[0325] Step 2: The system uses the smartphone's camera to recognize the surroundings and converts that information into audio.

[0326] Step 3: The system transmits audio information to the visually impaired person in real time.

[0327] Step 4: The system activates the emotion engine to recognize the user's emotion.

[0328] "Example 2"

[0329] Step 1: The emotion engine recognizes the emotion from the user's voice tone, volume, speaking style, etc. Step 2: The emotion engine communicates the recognized emotion to the system.

[0330] "Example 3"

[0331] Step 1: The system adjusts how it delivers audio information based on the emotional information received from the emotion engine.

[0332] Step 2: The system conveys the adjusted audio information to the visually impaired person.

[0333] Example 1

[0334] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] For visually impaired people to walk safely, they need to be able to grasp their surroundings in real time and take appropriate action. However, conventional systems were unable to accurately recognize the surrounding situation and communicate it via voice, nor were they able to provide feedback based on the user's emotional state. As a result, visually impaired people were unable to respond appropriately when they felt anxious or in an emergency, which could lead to accidents or danger.

[0336] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0337] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into voice, means for transmitting the voice information to the visually impaired person in real time, means including an emotion engine for recognizing the user's emotions, and means for providing appropriate feedback based on the user's emotions, thereby enabling the visually impaired person to accurately grasp the surrounding situation and further receive appropriate feedback according to their own emotional state.

[0338] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0339] A "mobile terminal" is a portable electronic device that has communication and photography functions.

[0340] "Capture device" refers to a device that includes a camera or sensor for capturing images or video.

[0341] "Audio information" refers to data obtained by converting acquired information into audio.

[0342] "Real-time" refers to processing and transmission occurring immediately without delay.

[0343] An "emotion engine" refers to software or algorithms that analyze and recognize a user's emotional state.

[0344] "Feedback" refers to the response or information that a system provides to a user.

[0345] This invention is a system that visually impaired people can wear around their necks using a mobile device to receive audio notifications of the situation in front of them. The system uses the mobile device's camera to recognize the surroundings and converts that information into audio. It also includes an emotion engine that recognizes the user's emotions and provides appropriate feedback based on the user's emotions.

[0346] Hardware and software used

[0347] Mobile device: A portable electronic device such as a smartphone.

[0348] Camera: Built-in camera on smartphone.

[0349] Image recognition software: Image analysis tools such as TensorFlow and OpenCV.

[0350] Text-to-speech software: Text-to-speech tools such as Google Text-to-Speech.

[0351] Emotion engine: An emotion analysis tool such as Affectiva or Microsoft® Azure® Emotion API.

[0352] Explanation of program processing

[0353] 1. Starting the camera and acquiring images

[0354] The device activates the smartphone camera and captures images of the surrounding area in real time.

[0355] Example: A smartphone camera continuously captures the scenery in front of it.

[0356] 2. Image Recognition

[0357] The images captured by the device are analyzed using TensorFlow and OpenCV to recognize objects and obstacles.

[0358] Example: A TensorFlow model analyzes an image to detect pedestrians and cars ahead.

[0359] 3. Audio conversion

[0360] The device converts the information it recognizes into text, which is then converted into audio using Google Text-to-Speech.

[0361] Example: The text "There is a pedestrian ahead" is generated and converted to speech.

[0362] 4. Audio Output

[0363] The device transmits the generated audio to the user in real time using Bluetooth earphones or the built-in speaker.

[0364] Example: Your smartphone will announce, "There is a pedestrian ahead."

[0365] 5. Emotion recognition

[0366] The terminal uses an emotion engine to recognize the user's emotion.

[0367] Example: A smartphone camera captures a user's face, and an emotion engine analyzes the user's facial expressions to detect emotions such as anxiety or relief.

[0368] 6. Emotional Feedback

[0369] The device provides appropriate feedback based on the user's emotions.

[0370] Example: An emotion engine detects a user's anxiety and provides a reassuring voice message such as "It's okay, there are no obstacles ahead."

[0371] Examples and prompts

[0372] Example: When a visually impaired person is walking with a smartphone hanging around their neck, if there is a pedestrian ahead, the smartphone will announce with a voice, "There is a pedestrian ahead." Also, if the user is feeling anxious, the emotion engine will detect this and provide a reassuring message, "It's okay, there are no obstacles ahead."

[0373] Example prompt sentence:

[0374] Explain how a visually impaired person walking with a smartphone around their neck can be notified by voice if there is a pedestrian ahead. Also, explain how a reassuring message can be provided if the user is feeling anxious.

[0375] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0376] Step 1:

[0377] Starting the camera and acquiring images

[0378] The device activates the smartphone camera and captures images of the surrounding area in real time.

[0379] Input: Smartphone camera is activated.

[0380] Data processing: The camera continuously captures images.

[0381] Output: Acquired image data.

[0382] Specific operation: The smartphone's camera app will automatically start up and take a picture of the scenery in front of it every second.

[0383] Step 2:

[0384] Image Recognition

[0385] The images captured by the device are analyzed using image recognition software (e.g., TensorFlow or OpenCV) to recognize objects and obstacles.

[0386] Input: Acquired image data.

[0387] Data computation: A TensorFlow model analyzes the image and applies an object detection algorithm.

[0388] Output: Information about recognized objects and obstacles.

[0389] How it works: The TensorFlow model analyzes images and detects pedestrians and cars ahead.

[0390] Step 3:

[0391] Audio conversion

[0392] The device converts the information it recognizes into text, which is then converted into speech using speech synthesis software (e.g., Google Text-to-Speech).

[0393] Input: Information about recognized objects and obstacles.

[0394] Data processing: Convert the recognized information into text and then convert it into speech using a speech synthesis engine.

[0395] Output: Audio data.

[0396] Specific behavior: The text "There is a pedestrian ahead" is generated and converted into speech.

[0397] Step 4:

[0398] Audio Output

[0399] The device transmits the generated audio to the user in real time using Bluetooth earphones or the built-in speaker.

[0400] Input: Audio data.

[0401] Data processing: Sends audio data to Bluetooth earphones or built-in speakers.

[0402] Output: The audio transmitted to the user.

[0403] Specific action: The smartphone will announce "There is a pedestrian ahead."

[0404] Step 5:

[0405] emotion recognition

[0406] The device uses an emotion engine (e.g., Affectiva or Microsoft Azure Emotion API) to recognize the user's emotions.

[0407] Input: A face image of the user taken with a smartphone camera.

[0408] Data calculation: The emotion engine analyzes the user's facial expressions and recognizes their emotional state.

[0409] Output: Information about the user's emotional state.

[0410] Specific operation: The smartphone camera takes a picture of the user's face, and the emotion engine analyzes the user's facial expressions to detect emotions such as anxiety or relief.

[0411] Step 6:

[0412] Emotion-based feedback

[0413] The device provides appropriate feedback based on the user's emotions.

[0414] Input: User's emotional state information.

[0415] Data processing: Generate appropriate voice messages based on emotional state.

[0416] Output: Audio feedback delivered to the user.

[0417] What it does: The emotion engine detects the user's anxiety and provides reassuring voice messages such as "It's okay, there are no obstacles ahead."

[0418] (Application example 1)

[0419] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0420] There is a lack of support systems to help visually impaired people stay safe in self-driving vehicles. In particular, there is a need for systems that can recognize the situation inside and outside the vehicle in real time and provide appropriate feedback. It is also important to recognize the emotions of visually impaired people and provide appropriate feedback based on that.

[0421] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes: means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing an information processing device around their neck; means for recognizing the surrounding situation using an imaging device of the information processing device and converting that information into voice; means for transmitting the voice information to the visually impaired person in real time; emotion recognition means for recognizing the user's emotion; means for providing appropriate feedback based on the emotion recognition means; and means for recognizing the situation inside and outside the vehicle and converting that information into voice for transmission. This enables visually impaired people to stay safely inside self-driving vehicles.

[0422] An "information processing device" is a device that inputs, processes, and outputs data, and specifically refers to portable electronic devices such as smartphones and tablets.

[0423] An "imaging device" is a device that converts light into an electrical signal to generate an image, and specifically refers to a camera or video camera.

[0424] "Means for converting into speech" refers to technology that converts text or data into an audio signal, and specifically includes speech synthesis engines and text-to-speech software.

[0425] "Means of real-time transmission" refers to technologies that transmit information instantly without delay, and specifically includes streaming technologies and real-time communication protocols.

[0426] "Emotion recognition means" refers to technology that analyzes and recognizes emotions from a user's facial expressions and voice, and specifically includes emotion recognition algorithms and machine learning models.

[0427] "Means for providing feedback" refers to technology that provides appropriate information or instructions depending on the user's behavior or state, and specifically includes audio guidance and vibration feedback devices.

[0428] "Means for recognizing the situation inside and outside the vehicle" refers to technology that detects and analyzes the environment and situation inside and outside the vehicle, and specifically includes situational awareness systems that use sensors and cameras.

[0429] The system for implementing this invention is a support system for visually impaired people to stay safely in autonomous vehicles. The system uses hardware and software such as an information processing device, an imaging device, a speech synthesis engine, an emotion recognition algorithm, and a real-time communication protocol.

[0430] The server provides a means for visually impaired people to walk around with an information processing device around their neck, and to receive audio notifications of the situation in front of them. Information processing devices include smartphones and tablets. The smartphone camera is used as the imaging device. The camera captures the surrounding environment and processes the video data in real time.

[0431] The server analyzes the video data acquired from the imaging device and recognizes the surrounding situation. Specifically, it uses an image processing library such as OpenCV to perform face detection and obstacle detection. The recognized information is converted into voice using a voice synthesis engine (e.g., pyttsx3). The voice information is transmitted to the visually impaired person in real time.

[0432] Furthermore, the server recognizes the user's emotions using an emotion recognition means. It uses an emotion recognition algorithm (e.g., EmotionRecognizer) to analyze emotions from the user's facial expressions and voice. Based on the analyzed emotion information, it provides appropriate feedback. The feedback is provided through audio guidance or a vibration feedback device.

[0433] To recognize the situation inside and outside the vehicle, the server includes a situational awareness system using sensors and cameras, which can detect and analyze the environment and situation inside and outside the vehicle, for example, detecting obstacles and recognizing road signs.

[0434] As a concrete example, consider a scenario in which a visually impaired person is riding in an autonomous vehicle. By providing real-time audio information about the situation inside and outside the vehicle, the visually impaired person can travel safely. An example of a prompt sentence to be input to the generative AI model is as follows:

[0435] "I want to develop a support system to help visually impaired people stay safe in self-driving vehicles. I will create a system that uses a smartphone camera to recognize the situation inside and outside the vehicle, converts that information into voice, and transmits it in real time. I will also combine it with an emotion engine to recognize the user's emotions and provide appropriate feedback."

[0436] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0437] Step 1:

[0438] The server acquires video data from the camera of the information processing device (smartphone). The input is real-time video data from the camera, and the output is video frames. Specifically, the camera captures the surrounding situation and sends the video data to the server.

[0439] Step 2:

[0440] The server analyzes the acquired video data using an image processing library (OpenCV). The input is the video frame, and the output is the recognized objects (e.g., faces, obstacles, road signs). Specifically, the server converts the video frame to grayscale and performs face and obstacle detection.

[0441] Step 3:

[0442] The server converts the recognized object information into speech using a speech synthesis engine (pyttsx3). The input is the recognized object information, and the output is speech data. Specifically, the server converts the recognized object information into text, and then inputs the text into the speech synthesis engine to generate speech data.

[0443] Step 4:

[0444] The server transmits the generated voice data to the information processing device in real time and conveys it to the visually impaired person. The input is voice data, and the output is voice information heard by the visually impaired person. Specifically, the voice data is transmitted to the information processing device using streaming technology.

[0445] Step 5:

[0446] The server analyzes the user's emotions using an emotion recognition algorithm (EmotionRecognizer). The input is the user's facial expression and voice data, and the output is the recognized emotion information. Specifically, the server extracts the user's face area from the camera image and inputs it into the emotion recognition algorithm to analyze the emotion.

[0447] Step 6:

[0448] The server provides appropriate feedback based on the recognized emotional information. The input is the recognized emotional information, and the output is feedback information (e.g., audio guidance, vibration feedback). Specific operations include determining the feedback content according to the emotional information and transmitting it to the user via audio guidance or a vibration feedback device.

[0449] Step 7:

[0450] The server recognizes the situation inside and outside the vehicle using sensors and cameras and analyzes that information. The input is data from the sensors and cameras, and the output is analyzed situation information. Specifically, it collects data from the sensors and cameras and analyzes it using situation recognition algorithms.

[0451] Step 8:

[0452] The server converts the analyzed situation information into speech using a speech synthesis engine and transmits it to the visually impaired person. The input is the analyzed situation information and the output is speech data. Specifically, the situation information is converted into text, and the text is input into the speech synthesis engine to generate speech data, which is then transmitted to the visually impaired person.

[0453] Example 2

[0454] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0455] For visually impaired people to travel safely, it is important to understand their surroundings in real time. However, conventional systems make it difficult for visually impaired people to accurately and quickly recognize their surroundings, exposing them to risks such as traffic accidents, railroad crossing accidents, and falls from train platforms. Furthermore, there is a lack of means to recognize the emotions of visually impaired people and provide appropriate feedback accordingly, which makes it difficult to increase the user's sense of security.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0457] In this invention, the server includes: a means for a visually impaired person walking with a mobile device hanging around their neck to be notified of the situation in front of them by voice; a means for recognizing the surrounding situation using a camera in the mobile device and converting that information into voice; a means for transmitting the voice information to the visually impaired person in real time; a means for capturing the user's voice and analyzing the voice data to recognize emotions; and a means for generating appropriate feedback based on the recognized emotions and providing it to the user. This enables the visually impaired person to accurately and quickly grasp the situation around them and avoid dangers such as traffic accidents, railroad crossing accidents, and falls from train platforms. Furthermore, by recognizing the user's emotions and providing feedback according to them, the user's sense of security can be enhanced.

[0458] "Visually impaired" refers to a person who has a visual impairment and is unable to fully utilize visual information.

[0459] A "mobile terminal" is a portable electronic device that has communication functions and the ability to execute various applications.

[0460] "Capture device" refers to a device for capturing video, including cameras and video cameras.

[0461] "Audio information" refers to information that is communicated using voice, and refers to a voice message that is generated using voice synthesis technology.

[0462] "Real-time" refers to processing and transmission occurring immediately without delay.

[0463] "User" refers to anyone who uses the system, and includes in particular the visually impaired.

[0464] "Audio data" refers to data that records audio in digital format.

[0465] "Means for recognizing emotions" refers to technology for analyzing a user's emotions from voice data and determining a specific emotional state.

[0466] "Feedback" refers to responses or information provided by a system to a user, and particularly includes appropriate responses according to the user's emotions.

[0467] "Surrounding circumstances" refers to the state of objects, people, and the environment around the user.

[0468] MODE FOR CARRYING OUT THE INVENTION

[0469] This invention provides a voice information providing system for visually impaired people to move around safely, and includes a function for recognizing the user's emotions and providing appropriate feedback. Specific embodiments of this system will be described below.

[0470] Hardware and software used

[0471] Hardware: Mobile devices (smartphones, etc.), imaging devices (cameras), audio input devices (microphones), audio output devices (speakers)

[0472] Software: Image recognition software (e.g., OpenCV), voice recognition software (e.g., Google Speech-to-Text API), emotion recognition engine (e.g., IBM Watson® Tone Analyzer)

[0473] System Operation Overview

[0474] 1. The camera captures the image

[0475] The camera captures the surroundings in real time, for example, when a user is walking down a sidewalk, the camera captures the image in front of them.

[0476] 2. The device sends the video data to the server

[0477] The device sends the captured video data to a server, which then transmits the data over the Internet.

[0478] 3. The server analyzes the video data

[0479] The server analyzes the received video data using image recognition software (e.g., OpenCV), extracting information such as "three people are walking towards you from the left front."

[0480] 4. The server converts the analysis results into audio information

[0481] The server converts the analysis results into voice information using natural language processing technology, for example, generating voice information such as "Three people are walking towards you from the left."

[0482] 5. The server sends the audio information to the device.

[0483] The server sends the generated voice information to the terminal, and the data is transmitted via the Internet.

[0484] 6. The device provides voice information to the user

[0485] The device will then provide the received voice information to the user through the speaker, for example, an audio notification saying, "Three people are walking towards you from the left."

[0486] 7. The microphone captures the user's voice

[0487] A microphone captures the user's voice in real time, for example when the user asks, "Where are you now?"

[0488] 8. The device sends the audio data to the server

[0489] The device sends the captured audio data to a server, which then transmits the data over the Internet.

[0490] 9. The server analyzes the voice data and recognizes emotions.

[0491] The server converts the received voice data into text using speech recognition software (e.g., Google Speech-to-Text API), and then analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Tone Analyzer). For example, it detects "anxiety" from the tone and volume of the user's voice.

[0492] 10. The server generates emotion-based feedback

[0493] The server generates appropriate feedback based on the analyzed emotions, such as "Don't worry, the roads are safe."

[0494] 11. The server sends feedback to the device

[0495] The server generates feedback and sends it to the device, with the data transmitted over the internet.

[0496] 12. Device provides feedback to the user

[0497] The device will provide the received feedback to the user through the speaker, for example, by announcing "Don't worry, the roads are safe."

[0498] Specific examples

[0499] When a user is walking on the sidewalk, a camera captures video of what is ahead and the device sends the video data to a server. The server analyzes the video data, extracts information such as "Three people are walking towards you from the left," and converts it into audio information. The server sends the generated audio information to the device, which notifies the user through a speaker that "Three people are walking towards you from the left." When the user asks "Where are you now?", the microphone captures the audio and the device sends the audio data to the server. The server analyzes the audio data and detects "anxiety" from the tone and volume of the user's voice. The server generates feedback such as "Don't worry, the roads are safe" and sends it to the device. The device notifies the user through a speaker that "Don't worry, the roads are safe."

[0500] Prompt Sentence Examples

[0501] "When a user is walking down the sidewalk, the camera captures an image of what is ahead, and image recognition software analyzes it to determine, 'Three people are walking towards you from the left.' Voice recognition software converts the user's question, 'Where are you now?' into text, and an emotion recognition engine detects 'anxiety' from the tone and volume of the user's voice. The server integrates this information and generates audio information, such as, 'Three people are walking towards you from the left. Don't worry, the road is safe,' which is communicated to the user through the speaker."

[0502] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0503] Step 1:

[0504] The camera captures the image

[0505] The camera captures the surroundings in real time, for example, when a user is walking down a sidewalk, the camera captures the image in front of them.

[0506] Input: Surrounding image

[0507] Output: Captured video data

[0508] Step 2:

[0509] The device sends the video data to the server

[0510] The device sends the captured video data to a server, which then transmits the data over the Internet.

[0511] Input: Captured video data

[0512] Output: Video data sent to the server

[0513] Step 3:

[0514] The server analyzes the video data

[0515] The server analyzes the received video data using image recognition software (e.g., OpenCV), extracting information such as "three people are walking towards you from the left front."

[0516] Input: Video data sent to the server

[0517] Output: Analyzed situation information (e.g., three people are walking from the front left)

[0518] Step 4:

[0519] The server converts the analysis results into audio information

[0520] The server converts the analysis results into voice information using natural language processing technology, for example, generating voice information such as "Three people are walking towards you from the left."

[0521] Input: Parsed status information

[0522] Output: Generated audio information

[0523] Step 5:

[0524] The server sends the voice information to the device.

[0525] The server sends the generated voice information to the terminal, and the data is transmitted via the Internet.

[0526] Input: Generated audio information

[0527] Output: Audio information sent to the device

[0528] Step 6:

[0529] The terminal provides voice information to the user

[0530] The device will then provide the received voice information to the user through the speaker, for example, an audio notification saying, "Three people are walking towards you from the left."

[0531] Input: Audio information sent to the device

[0532] Output: Audio information provided to the user

[0533] Step 7:

[0534] A microphone captures the user's voice

[0535] A microphone captures the user's voice in real time, for example when the user asks, "Where are you now?"

[0536] Input: User's voice

[0537] Output: Captured audio data

[0538] Step 8:

[0539] The device sends the audio data to the server.

[0540] The device sends the captured audio data to a server, which then transmits the data over the Internet.

[0541] Input: Captured audio data

[0542] Output: Audio data sent to the server

[0543] Step 9:

[0544] The server analyzes the voice data and recognizes emotions.

[0545] The server converts the received voice data into text using speech recognition software (e.g., Google Speech-to-Text API), and then analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Tone Analyzer). For example, it detects "anxiety" from the tone and volume of the user's voice.

[0546] Input: Audio data sent to the server

[0547] Output: Parsed emotion information (e.g., anxiety)

[0548] Step 10:

[0549] The server generates emotion-based feedback

[0550] The server generates appropriate feedback based on the analyzed emotions, such as "Don't worry, the roads are safe."

[0551] Input: Parsed emotion information

[0552] Output: The generated feedback

[0553] Step 11:

[0554] The server sends feedback to the device

[0555] The server generates feedback and sends it to the device, with the data transmitted over the internet.

[0556] Input: Generated feedback

[0557] Output: Feedback sent to the device

[0558] Step 12:

[0559] The device provides feedback to the user

[0560] The device will provide the received feedback to the user through the speaker, for example, by announcing "Don't worry, the roads are safe."

[0561] Input: Feedback sent to the device

[0562] Output: Feedback provided to the user

[0563] (Application example 2)

[0564] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0565] When visually impaired people use self-driving vehicles, they lack the means to understand their surroundings in real time and ride safely. There is also a need for a system that can recognize the emotions of visually impaired people and provide appropriate feedback to give them a sense of security.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0567] In this invention, the server includes: a means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing an information terminal around their neck; a means for recognizing the surrounding situation using a camera device of the information terminal and converting that information into voice; a means for transmitting the voice information to the visually impaired person in real time; a means for acquiring the user's voice and recognizing their emotion; and a means for providing appropriate voice feedback based on the recognized emotion. This allows visually impaired people to understand the surrounding situation in real time when using an autonomous vehicle and ride safely. Furthermore, providing feedback according to the user's emotion can give them a sense of security.

[0568] "Visually impaired" refers to a person who has a visual impairment and is unable to fully utilize visual information.

[0569] An "information terminal" is an electronic device for processing and communicating data, including smartphones and tablets.

[0570] "Photography device" refers to a device for capturing images or videos, such as a camera or sensor.

[0571] "Means for converting into voice" refers to technology or devices for converting acquired information into voice data and transmitting it to the user.

[0572] "Means of real-time communication" refers to technologies and devices that convey acquired information to users immediately and without delay.

[0573] The "means for acquiring the user's voice" refers to a technique or device for collecting the user's speech using a microphone or the like.

[0574] "Means for recognizing emotions" refers to technology or devices for analyzing and identifying emotions from a user's voice, facial expressions, etc.

[0575] "Means for providing appropriate audio feedback" refers to techniques and devices for generating and conveying appropriate audio messages to a user based on the recognized emotion.

[0576] A system for implementing this invention includes a means for a visually impaired person to walk with an information terminal hanging around their neck and be notified of the situation in front of them by voice, a means for recognizing the surrounding situation using a camera device in the information terminal and converting that information into voice, a means for transmitting the voice information to the visually impaired person in real time, a means for acquiring the user's voice and recognizing their emotions, and a means for providing appropriate voice feedback based on the recognized emotions.

[0577] Hardware and software used

[0578] Hardware: Smartphone or head-mounted display, microphone, vehicle sensors

[0579] Software: speech_recognition library, pyttsx3 library, emotion recognition engine (e.g. emotion_recognition library), vehicle sensor interface (e.g. vehicle_sensors library)

[0580] System Operation

[0581] 1. The information terminal uses a camera to recognize the surrounding situation. For example, it obtains information such as whether there is a railroad crossing ahead or if someone is walking towards the left front.

[0582] 2. The information terminal converts the acquired situational information into audio and conveys it to the visually impaired person in real time, providing audio information such as "There is a railroad crossing ahead," or "Three people are walking towards you from the left."

[0583] 3. The information terminal uses a microphone to capture the user's voice and converts what the user says into text using voice recognition technology.

[0584] 4. The emotion recognition engine analyzes and identifies the user's emotion from the acquired voice data. For example, if the user says "I'm scared," the emotion recognition engine will recognize "fear."

[0585] 5. The information terminal generates appropriate voice feedback based on the recognized emotion and conveys it to the user, for example, "Don't worry. I'm driving safely."

[0586] Specific examples

[0587] When the vehicle approaches a railroad crossing, the information terminal will issue a voice notification saying, "There is a railroad crossing ahead. Please be careful."

[0588] If the user says "I'm scared," the emotion recognition engine will recognize the "fear," and the information terminal will provide voice feedback saying, "Don't worry. I'm driving safely."

[0589] Prompt Sentence Examples

[0590] User says: "Scary"

[0591] Prompt the generative AI model: "The user says they're scared. Please provide appropriate feedback."

[0592] Generative AI model output: "Don't worry, I'm driving safely."

[0593] In this way, a system can be realized that supports visually impaired people in safely using self-driving vehicles.

[0594] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0595] Step 1:

[0596] The device recognizes the surrounding situation using a camera. Image data is acquired from the camera as input. Data is processed using an image recognition algorithm to identify the location of obstacles and people ahead. Recognized situation information (e.g., "There is a railroad crossing ahead," "Three people are walking towards you from the left") is generated as output.

[0597] Step 2:

[0598] The device converts the acquired situation information into voice and transmits it to the visually impaired person in real time. As input, it receives the situation information generated in step 1. As data processing, it uses a voice synthesis engine (e.g., the pyttsx3 library) to convert the text information into voice data. As output, it generates voice information and transmits it to the user through a speaker.

[0599] Step 3:

[0600] The device acquires the user's voice using a microphone. As input, it acquires voice data from the microphone. As data processing, it converts the voice data into text using a speech recognition engine (e.g., speech_recognition library). As output, it generates text data based on the user's speech.

[0601] Step 4:

[0602] The device uses an emotion recognition engine to analyze and identify the user's emotion from the acquired voice data. As input, it receives the text data generated in step 3. As data processing, it uses an emotion recognition algorithm (e.g., emotion_recognition library) to identify the emotion from the text data. As output, it generates recognized emotion information (e.g., "fear" or "anger").

[0603] Step 5:

[0604] The device generates appropriate voice feedback based on the recognized emotion and transmits it to the user. As input, it receives the emotion information generated in step 4. As data processing, it generates a feedback message according to the emotion and converts it into voice data using a voice synthesis engine. As output, it generates voice feedback and transmits it to the user through a speaker.

[0605] In this way, a system can be realized that supports visually impaired people in safely using self-driving vehicles.

[0606] Example 3

[0607] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0608] For visually impaired people to travel safely, it is necessary to grasp the situation around them in real time and take appropriate action. However, conventional systems do not provide information according to the user's emotional state, and are unable to reduce the user's stress and anxiety. In addition, there is a lack of means to communicate specific situations in detail, making it difficult for visually impaired people to fully avoid the dangers they face.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0610] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into voice, means for transmitting the voice information to the visually impaired person in real time, means for the mobile device to analyze the user's emotions using an emotion analysis device, and means for adjusting the method of transmitting the voice information based on the user's emotions.This makes it possible to provide information for the visually impaired to move safely in real time and to transmit information appropriately according to the user's emotional state.

[0611] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining visual information.

[0612] A "mobile terminal" is a portable electronic device that has communication and photographing functions.

[0613] "Image capture device" refers to hardware used to capture images or video.

[0614] "Audio information" refers to audio information generated using speech synthesis technology.

[0615] "Real-time" refers to processing and transmission occurring immediately without delay.

[0616] An "emotion analysis device" refers to hardware and software for analyzing a user's emotional state from their facial expressions and voice.

[0617] "User emotion" refers to the user's psychological state or mood.

[0618] The "method of transmitting audio information" refers to the means or method of transmitting audio information to the user.

[0619] This invention is a system that provides real-time information for the safe movement of visually impaired people and transmits appropriate information according to the emotional state of the user. Specific embodiments of this system are described below.

[0620] First, the user walks around with the mobile device hanging from their neck. The mobile device is equipped with a camera and an emotion analysis device. When the user launches the application, the camera captures images of the surroundings in real time, and the sensors collect information on distance and obstacles. This data is temporarily stored in the mobile device.

[0621] Next, the device sends the captured video and sensor data to a server. The data is encrypted and transmitted securely. The server then uses computer vision technology (e.g., OpenCV) and voice recognition technology (e.g., voice recognition API) to analyze the received data. This allows the location of the platform edge, stairs, bicycles, etc. to be identified.

[0622] The analyzed information is sent from the server to the device. The device then uses speech synthesis software (e.g., speech synthesis API) to convert the analysis results into audio information. For example, audio information such as "The end of the platform is in front of you. There are stairs on your right." is generated. The device then transmits the generated audio information to the user. The user can listen to the audio information through earphones or speakers.

[0623] Furthermore, the device uses an emotion analyzer (e.g., emotion analysis SDK) to analyze the user's facial expressions through the camera and recognize the user's emotional state. For example, it identifies whether the user is angry or calm. Based on the user's emotional state, the device adjusts the way it transmits audio information. If the user is angry, it reduces the amount of audio information transmitted, and if the user is calm, it transmits audio information normally.

[0624] As a concrete example, consider a visually impaired person standing on a train platform. The user's mobile device uses a camera and sensors to capture the surroundings and sends the data to a server. The server uses OpenCV to analyze the video data and identify the location of the edge of the platform and stairs. It then uses a speech synthesis API to generate audio information such as, "The edge of the platform is in front of you. There are stairs to your right." The user can move safely by listening to this audio information.

[0625] Additionally, if a user is walking and sees a bicycle approaching from the left front, the mobile device's camera and sensors capture the information and send it to the server. The server analyzes the data and uses a speech synthesis API to generate audio information such as "A bicycle is approaching from the left front." By listening to this audio information, the user can prevent a collision accident.

[0626] Example prompts to input to the generative AI model:

[0627] "Please explain in detail the processing steps of a system that will provide audible information about the surroundings to a visually impaired person standing on a train platform. Please also specify the names of the hardware and software used."

[0628] This system allows the visually impaired to move around safely and improves the quality of their daily lives. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0629] Step 1:

[0630] The user starts the mobile terminal and starts the dedicated application.

[0631] Input: The application is started by a user action.

[0632] Output: Application launch complete, camera and sensors start working.

[0633] Specific behavior: A user taps the application icon on their mobile device, the application launches, and the camera and sensors automatically start working, ready to capture the surroundings.

[0634] Step 2:

[0635] The device uses its camera and sensors to capture the surroundings.

[0636] Input: Surrounding images and distance information obtained from cameras and sensors.

[0637] Output: Captured video and sensor data.

[0638] How it works: The device's camera captures images of the surroundings in real time, and sensors collect distance and obstacle information, which is then temporarily stored on the device.

[0639] Step 3:

[0640] The device sends the captured data to the server.

[0641] Input: Captured video and sensor data.

[0642] Output: The data sent to the server.

[0643] Specific operation: The device encrypts the captured video data and sensor data and securely transmits them to the server. The server receives this data.

[0644] Step 4:

[0645] The server analyzes the received data and identifies the surrounding situation.

[0646] Input: Video data and sensor data sent to the server.

[0647] Output: Parsed surrounding information.

[0648] Specific operation: The server analyzes video data using computer vision technology (e.g., OpenCV) and audio data using speech recognition technology (e.g., speech recognition API), thereby identifying the location of platform edges, stairs, bicycles, etc.

[0649] Step 5:

[0650] The server sends the analysis results to the device.

[0651] Input: Parsed surrounding information.

[0652] Output: Analysis results sent to the device.

[0653] Specific operation: The server sends the analysis results to the device, including information on identified obstacles and the movement route.

[0654] Step 6:

[0655] The device converts the analysis results into voice information using speech synthesis software.

[0656] Input: Analysis results received from the server.

[0657] Output: The generated audio information.

[0658] Specific operation: The device uses speech synthesis software (e.g., speech synthesis API) to convert the analysis results into speech information. For example, speech information such as "The end of the platform is ahead of you. There are stairs on your right." is generated.

[0659] Step 7:

[0660] The terminal conveys the audio information to the user.

[0661] Input: The generated audio information.

[0662] Output: The audio information conveyed to the user.

[0663] Specific operation: The terminal transmits the generated voice information to the user, who can listen to the voice information through earphones or speakers.

[0664] Step 8:

[0665] The terminal uses an emotion analyzer to analyze the user's emotions.

[0666] Input: User's facial expression data.

[0667] Output: The analyzed emotional state of the user.

[0668] Specific operation: The device uses an emotion analyzer (e.g., Emotion Analysis SDK) to analyze the user's facial expressions through the camera and recognize the user's emotional state. For example, it can determine whether the user is angry or calm.

[0669] Step 9:

[0670] The terminal adjusts the way audio information is delivered based on the user's emotions.

[0671] Input: Parsed user emotional state.

[0672] Output: The method of conveying the adjusted audio information.

[0673] Specific operation: The device adjusts the way it transmits voice information based on the user's emotional state. If the user is angry, it will transmit voice information less, and if the user is calm, it will transmit voice information normally.

[0674] (Application example 3)

[0675] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0676] For visually impaired people to travel safely, it is important to accurately grasp their surroundings and obtain information at the appropriate time. However, with conventional technology, it is difficult for visually impaired people to grasp their surroundings in real time while walking or riding in an autonomous vehicle, and information transmission based on the user's emotions is not possible. This has left visually impaired people at risk of being involved in traffic accidents or falling from train platforms.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0678] In this invention, the server includes means for audibly informing a visually impaired person of the situation in front of them as they walk with a smartphone hanging around their neck, means for recognizing the surrounding situation using the camera of the smartphone and converting that information into audio, means for transmitting the audio information to the visually impaired person in real time, means for recognizing the user's emotions and adjusting the method of transmitting the audio information based on the emotions, and means for audibly transmitting the situation around the vehicle in an autonomous vehicle. This enables a visually impaired person to grasp the surrounding situation in real time while walking or riding in an autonomous vehicle, and to transmit appropriate information according to the user's emotions.

[0679] "Visually impaired persons" refer to people who have visual impairments and are unable to obtain sufficient visual information.

[0680] A "smartphone" is a type of mobile phone, a portable information terminal with a variety of functions, including internet connectivity, application use, and camera functionality.

[0681] A "camera" is a device that converts light into an electrical signal and captures images or videos.

[0682] "Audio information" is information that is transmitted using sound and is used to inform visually impaired people of their surroundings.

[0683] "Real-time" refers to processing and transmission occurring immediately without delay.

[0684] "Emotion recognition" is a technology that determines a user's emotional state from facial expressions, voice, etc.

[0685] An "autonomous vehicle" is a vehicle that uses artificial intelligence and sensor technology to drive autonomously without a human driver.

[0686] "Surrounding circumstances" refers to the state of objects, people, and the environment around a visually impaired person or an autonomous vehicle.

[0687] The "method of transmitting audio information" refers to the means or method of transmitting audio information to the user.

[0688] This invention is a system for enabling visually impaired people to move around safely, and uses hardware and software such as smartphones, cameras, speech synthesis engines, emotion recognition engines, and self-driving vehicles.

[0689] System configuration

[0690] 1. Smartphone: A smartphone worn around the neck by the visually impaired. Equipped with a camera, it captures the surroundings.

[0691] 2. Camera: Built into the smartphone, it captures images in real time.

[0692] 3. Speech synthesis engine: Converts information recognized from captured video into speech and conveys it to visually impaired people.

[0693] 4. Emotion recognition engine: Recognizes emotions from the user's facial expressions and voice and adjusts how voice information is conveyed.

[0694] 5. Self-driving vehicles: When a visually impaired person is riding in the vehicle, the situation around the vehicle is communicated to them via voice.

[0695] Program processing

[0696] The server uses the smartphone's camera to recognize the surrounding situation and converts that information into voice. It uses a speech synthesis engine (e.g., pyttsx3) to transmit voice information to visually impaired people in real time. It also uses an emotion recognition engine (e.g., EmotionRecognizer) to recognize the user's emotions and adjust the way voice information is transmitted based on those emotions. In autonomous vehicles, the situation around the vehicle is communicated through voice.

[0697] Specific examples

[0698] For example, if a visually impaired person is standing on a train platform, their smartphone will provide audio information such as, "The end of the platform is ahead of you. There are stairs on your right." This will help the visually impaired person avoid the risk of falling off the platform. Additionally, while walking, the smartphone can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[0699] In self-driving vehicles, when the vehicle approaches an intersection, it provides information such as "A vehicle is approaching from the right" to ensure the user's safety. Furthermore, if the user is angry, it slows down the voice speed to convey the information discreetly.

[0700] Prompt Sentence Examples

[0701] An example of a prompt to input to a generative AI model is as follows:

[0702] Develop an application that provides audio feedback to visually impaired passengers in autonomous vehicles about the vehicle's surroundings, including the ability to adjust the audio feedback based on the user's emotions.

[0703] In this way, a system can be realized that allows visually impaired people to move around safely.

[0704] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0705] Step 1:

[0706] The server uses the smartphone's camera to capture the surroundings. The input is real-time video from the camera, and the output is video data. Specifically, the camera continuously captures frames and generates video data.

[0707] Step 2:

[0708] The server analyzes the acquired video data using an image processing algorithm (e.g., OpenCV) to recognize objects and people. The input is the video data obtained in step 1, and the output is information about the recognized objects and people. Specifically, the server extracts features from the video data and inputs them into an object recognition model to obtain the recognition results.

[0709] Step 3:

[0710] The server converts the information about the recognized objects and people into voice information. The input is the recognition result obtained in step 2, and the output is voice information. Specifically, it uses a voice synthesis engine (e.g., pyttsx3) to convert the text information into voice.

[0711] Step 4:

[0712] The server transmits audio information to the visually impaired person in real time. The input is the audio information obtained in step 3, and the output is the audio transmitted to the visually impaired person. Specifically, the audio is played through the smartphone speaker.

[0713] Step 5:

[0714] The server uses the smartphone's camera and microphone to recognize the user's emotions. The input is camera footage and audio data, and the output is the user's emotional state. Specifically, it uses an emotion recognition engine (e.g., EmotionRecognizer) to determine emotions from facial expressions and voice.

[0715] Step 6:

[0716] The server adjusts the way the voice information is delivered based on the user's emotions. The input is the emotional state obtained in step 5, and the output is the adjusted voice information. Specific actions include changing the voice speed and tone.

[0717] Step 7:

[0718] In an autonomous vehicle, the server recognizes the situation around the vehicle using cameras and sensors and transmits that information via voice. The input is the vehicle's sensor data and the output is voice information. Specifically, the server analyzes the data obtained from the vehicle's sensors, generates voice information using a voice synthesis engine, and plays it through the vehicle's speakers.

[0719] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0720] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0721] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0722] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0723] [Second embodiment]

[0724] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0725] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure.

[0727] The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0728] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0729] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0730] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0731] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0732] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0733] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0734] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0735] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0736] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0737] "Example 1"

[0738] This invention is a system that notifies visually impaired people of the situation in front of them by voice by hanging a smartphone around their neck. Specifically, the system uses the smartphone's camera to recognize the surrounding situation and converts that information into voice. This voice information is transmitted to the visually impaired in real time.

[0739] "Example 2"

[0740] The audio information provides specific information about the situation, such as "There is a railroad crossing ahead," "Three people are walking towards you from the left," "There is a telephone pole ahead to the right," etc. This allows visually impaired people to know the situation in front of them in real time, and helps protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0741] "Example 3"

[0742] As a specific example of use, if a visually impaired person is standing on a train platform, their smartphone will provide audio information such as, "The end of the platform is in front of you. There are stairs on your right." This will help the visually impaired person avoid the risk of falling off the platform. Also, while walking, the smartphone can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[0743] The processing flow of each embodiment will be described below.

[0744] "Example 1"

[0745] Step 1: A visually impaired person wears a smartphone around their neck.

[0746] Step 2: The smartphone camera recognizes the surroundings.

[0747] Step 3: Convert the recognized information into speech.

[0748] Step 4: Transmit audio information to the visually impaired in real time.

[0749] "Example 2"

[0750] Step 1: Voice information is generated. This information indicates specific situations such as "There is a railroad crossing ahead," "Three people are walking towards you from the left," and "There is a telephone pole ahead to the right."

[0751] Step 2: The visually impaired person listens to the audio information and understands the situation in front of them.

[0752] Step 3: The visually impaired person adjusts their behavior based on the audio information to protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0753] "Example 3"

[0754] Step 1: When a visually impaired person is standing on a station platform, their smartphone will communicate information via voice, such as, "The end of the platform is in front of you. There are stairs on your right."

[0755] Step 2: Visually impaired people listen to the audio information and avoid the risk of falling off the platform.

[0756] Step 3: While walking, information such as "A bicycle is approaching from the front left" is provided to prevent collisions.

[0757] Example 1

[0758] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0759] For the visually impaired to travel safely, it is important to be able to grasp the situation around them in real time. However, conventional technology has made it difficult for the visually impaired to accurately and quickly recognize the situation around them. In particular, there has been a lack of means to provide information to help them avoid dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms. This has hindered the safe travels of the visually impaired.

[0760] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0761] In this invention, the server includes means for analyzing video data transmitted from the mobile information terminal and recognizing objects and text, means for generating the recognized information as text data, and means for converting the text data into audio data. This allows visually impaired people to use their mobile information terminals to receive audio information about their surroundings in real time. This allows visually impaired people to avoid dangers such as traffic accidents, railroad crossing accidents, and falls from train platforms, and to travel safely.

[0762] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0763] A "personal digital assistant" is a portable electronic device that is equipped with communication functions and an imaging device.

[0764] "Imaging device" refers to a device for capturing images, such as a camera or sensor.

[0765] "Video data" refers to data that digitally represents visual information captured by an imaging device.

[0766] A "server" refers to a computer system that processes and manages data over a network.

[0767] "Analysis" refers to the process of examining data in detail to understand its structure and content.

[0768] "Object recognition" refers to a technology that identifies specific objects within video data and determines their location and type.

[0769] "Text data" refers to data that represents character information in digital form.

[0770] "Audio data" refers to data that represents audio information in digital form.

[0771] "Speech synthesis" refers to the technology of generating voice data based on text data.

[0772] "Real-time" refers to processing and transmission occurring immediately without delay.

[0773] "User" refers to a visually impaired person who uses this system.

[0774] This invention is a system that uses a portable information terminal hung around the neck of a visually impaired person to provide audio information about the situation in front of them. Specifically, the system uses the imaging device of the portable information terminal to recognize the surrounding situation and converts that information into audio. This audio information is transmitted to the visually impaired person in real time.

[0775] The server receives the video data sent from the mobile information terminal. The server uses computer vision technology to analyze the video data. Specifically, it uses libraries such as TensorFlow and OpenCV to recognize objects and text in the video. The server generates the recognized information as text data. Next, the server uses speech synthesis software to convert this text data into speech. Specifically, it uses a text-to-speech API or a speech synthesis engine.

[0776] The terminal (mobile information terminal) receives the voice data sent from the server and transmits it to the user in real time through a speaker, allowing visually impaired people to hear what is happening in front of them.

[0777] As a concrete example, consider a scene where a user is walking through town. The device's camera captures a traffic light ahead, and the device sends this image to a server. The server analyzes the image and recognizes that the traffic light is red. The server generates text saying, "There is a red light ahead. Please stop," and converts this into audio data. The audio data is sent to the device, which then tells the user through a speaker, "There is a red light ahead. Please stop."

[0778] Example prompt sentence:

[0779] "Please create a program that uses the imaging device of a mobile information terminal to recognize the situation ahead and notify the user by voice that the traffic light is red."

[0780] This system allows visually impaired people to travel safely, avoiding dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0781] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0782] Step 1:

[0783] The device captures the surroundings in real time using a built-in image capture device. The input is the image data from the image capture device, and the output is the captured image data. Specifically, the device continues to capture images at a constant frame rate.

[0784] Step 2:

[0785] The terminal transmits the captured video data to a server via the Internet. The input is the captured video data, and the output is the video data transmitted to the server. Specifically, the terminal transmits the video data in a compressed format (e.g., H.264).

[0786] Step 3:

[0787] The server receives the video data transmitted from the terminal. The input is the video data transmitted from the terminal, and the output is the received video data. In concrete operation, the server receives the data via the network.

[0788] Step 4:

[0789] The server analyzes the received video data and recognizes objects and text. The input is the received video data, and the output is the recognized objects and text information. Specifically, the server performs the analysis using computer vision libraries such as TensorFlow and OpenCV.

[0790] Step 5:

[0791] The server generates text data based on the recognized objects and text information. The input is the recognized objects and text information, and the output is the generated text data. Specifically, the server generates text such as "There is a red light ahead. Please stop."

[0792] Step 6:

[0793] The server converts the generated text data into voice data. The input is the generated text data, and the output is the generated voice data. Specifically, the server generates voice data using a Text-to-Speech API or a voice synthesis engine.

[0794] Step 7:

[0795] The server sends the generated audio data to the terminal via the Internet. The input is the generated audio data, and the output is the audio data sent to the terminal. Specifically, the server sends the audio data in a compressed format (e.g., MP3).

[0796] Step 8:

[0797] The terminal plays back the received voice data and transmits it to the user. The input is the received voice data, and the output is the voice information transmitted to the user. In concrete terms, the terminal plays back voice through a speaker and tells the user, "There is a red light ahead. Please stop."

[0798] (Application example 1)

[0799] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0800] Visually impaired people face difficulties in understanding their surroundings when searching for products in physical stores or navigating specific locations. There is also a lack of support for visually impaired people to navigate safely and find the products they need.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0802] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk around with the information processing device hanging around their neck, means for recognizing the surrounding situation using an imaging device in the information processing device and converting that information into voice, means for transmitting that voice information to the visually impaired person in real time, and means for recognizing the surrounding situation when the visually impaired person is searching for a product or moving to a specific location in a physical store and transmitting that information by voice in real time, thereby enabling the visually impaired person to move safely within the physical store and find the products they need.

[0803] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0804] "Information processing device" refers to an electronic device that has the function of inputting, processing, and outputting data.

[0805] An "imaging device" refers to a device that converts light into an electrical signal and generates image data.

[0806] "Surroundings" refers to the state of the objects and environment around a visually impaired person.

[0807] "Means for converting into audio" refers to technology or devices that convert image data or other information into audio data.

[0808] "Real-time" refers to processing and transmission occurring immediately without delay.

[0809] "Brick and mortar store" refers to a store that exists in a physical location and offers goods and services.

[0810] "Products" refers to items sold in physical stores.

[0811] "Specific location" refers to the specific location to which the visually impaired person wishes to travel.

[0812] The system for carrying out this invention provides a function that notifies a visually impaired person of the situation in front of them by voice when they walk while wearing an information processing device around their neck. A specific embodiment of this system will be described below.

[0813] 1. System Configuration

[0814] The system consists of the following main components:

[0815] Information processing devices: Portable electronic devices such as smartphones and tablets.

[0816] Imaging device: A camera built into an information processing device.

[0817] Audio output device: A speaker built into the information processing device or a connected earphone.

[0818] Software: Applications for image recognition, speech synthesis, and real-time processing.

[0819] 2. Program Processing

[0820] The server implements the system using the following means:

[0821] Image recognition means: Analyzes video data captured by an imaging device and recognizes the surrounding situation. Specifically, object recognition is performed using machine learning libraries such as TensorFlow.

[0822] Speech synthesis means: Converts the recognized information into speech data. Specifically, it uses a speech synthesis library such as pyttsx3.

[0823] Real-time communication: Audio data is transmitted to the visually impaired in real time, allowing them to instantly understand their surroundings.

[0824] 3. Hardware and Software Used

[0825] Hardware: Smartphone camera, speakers, earphones.

[0826] Software: OpenCV (image processing library), TensorFlow (machine learning library), pyttsx3 (speech synthesis library).

[0827] 4. Specific Examples

[0828] For example, if a visually impaired person is looking for "milk" in a physical store, the camera in the information processing device will scan the shelves, and if "milk" is recognized, a voice message will be sent saying, "Milk found." In this way, visually impaired people can safely move around the physical store and find the products they need.

[0829] 5. Examples of prompts

[0830] An example of a prompt to input to a generative AI model is as follows:

[0831] "Design a navigation system to help visually impaired people find products in a physical store. The system uses a smartphone camera to recognize the surroundings and communicates that information in real time via voice."

[0832] In this way, a system can be realized that helps visually impaired people shop more smoothly in physical stores.

[0833] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0834] Step 1:

[0835] The server activates the imaging device of the information processing device and acquires surrounding video data. The input is video data from the camera, and the output is the acquired video frames. Specifically, the camera continuously captures video and generates frames in real time.

[0836] Step 2:

[0837] The server preprocesses the captured video frames. The input is the video frame captured in step 1, and the output is the preprocessed image data. Specifically, the server resizes the video frame and performs noise removal and color correction.

[0838] Step 3:

[0839] The server inputs the preprocessed image data into an object recognition model to recognize the surrounding situation. The input is the preprocessed image data, and the output is the label and location information of the recognized object. Specifically, it uses TensorFlow's object recognition model to detect objects in the image and identify their label and location.

[0840] Step 4:

[0841] The server converts the information of the recognized object into audio data. The input is the label and location information of the recognized object, and the output is audio data. Specifically, it uses the pyttsx3 speech synthesis library to convert the text information into audio.

[0842] Step 5:

[0843] The server transmits the generated voice data to the visually impaired person in real time. The input is voice data, and the output is voice information that the visually impaired person hears. Specifically, the voice is played through the speaker or earphones of the information processing device.

[0844] Step 6:

[0845] Users navigate and search for products within a physical store based on voice information. The input is voice information transmitted from the server, and the output is the user's actions. Specifically, the user follows the voice guidance to walk and find the desired product or location.

[0846] Example 2

[0847] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0848] For the visually impaired to travel safely, it is important to be able to grasp the situation around them in real time. However, with conventional technology, it is difficult for the visually impaired to accurately and quickly recognize their surroundings, which has led to risks such as traffic accidents, railroad crossing accidents, and falls from station platforms. For this reason, there is a need for a system that enables the visually impaired to grasp the situation around them in real time and travel safely.

[0849] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0850] In this invention, the server includes: means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck; means for recognizing the surrounding situation using an imaging device of the mobile device and converting that information into voice; means for transmitting the voice information to the visually impaired person in real time; means for the mobile device to acquire surrounding images and distance information using an imaging device and a distance measuring device; means for the mobile device to analyze the acquired video data using image recognition software and identify the positions of objects and people; means for the mobile device to generate a natural language explanatory text using a generative AI model based on the analysis results; means for the mobile device to convert the generated text into voice using voice synthesis software; and means for the mobile device to play the generated voice file and output the voice from a speaker. This allows the visually impaired person to grasp the surrounding situation in real time and move around safely.

[0851] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0852] A "mobile terminal" is a portable electronic device that has communication and computing capabilities.

[0853] "Imaging device" refers to a device, such as a camera or sensor, that receives light and generates image data.

[0854] "Distance measuring device" refers to a sensor or device used to measure the distance to an object.

[0855] "Image recognition software" refers to a program that analyzes image data and identifies objects and people.

[0856] "Generative AI model" refers to a model that uses artificial intelligence to generate natural language text from data.

[0857] "Speech synthesis software" refers to a program for converting text data into voice data.

[0858] "Audio information" refers to information transmitted as sound, and includes content intended to inform visually impaired people of their surroundings.

[0859] "Real-time" refers to processing and transmission occurring immediately without delay.

[0860] A "speaker" refers to a device that converts audio signals into sound waves and outputs them.

[0861] This invention is a system that enables visually impaired people to grasp their surroundings in real time and move around safely. Specific embodiments of this system will be described below.

[0862] Hardware and software used

[0863] Hardware: Mobile devices (smartphones and tablets), imaging devices (cameras), distance measurement devices (distance sensors), speakers

[0864] Software: Image recognition software (e.g., OpenCV), speech synthesis software (e.g., Google Text-to-Speech), generative AI models (e.g., GPT-4)

[0865] System Overview

[0866] This system allows visually impaired people to walk around with a mobile device hanging from their neck and receive audio notifications of the situation in front of them. The mobile device's imaging device and distance measurement device are used to capture images of the surrounding area and distance information, and the data is analyzed and provided to the user as audio information.

[0867] Data Acquisition

[0868] The device activates the camera to capture images of what is ahead in real time, while the distance sensor measures the distance to objects, allowing visually impaired people to have a detailed understanding of their surroundings while walking.

[0869] Data analysis

[0870] The image data acquired by the device is analyzed using image recognition software (OpenCV) to identify the location of objects and people. For example, it can recognize from the camera image that "three people are walking to the left and in front." Based on the results of this analysis, a generative AI model (GPT-4) is used to generate a natural language description.

[0871] Status text generation

[0872] The device inputs the analysis results as prompts into the generative AI model to generate appropriate explanatory text. An example of a prompt is "Please explain the situation recognized from the camera image. Example: There is a railroad crossing in front of you, three people are walking from the front left, and there is a utility pole in front of you on the right."

[0873] Speech synthesis

[0874] The device converts the generated text into speech using speech synthesis software (Google Text-to-Speech). The generated audio file is high-quality and adjusted to be easy for the user to hear.

[0875] Providing audio information

[0876] The device plays the generated audio file and outputs the audio through the speaker at an appropriate volume, allowing the user to accurately understand their surroundings.

[0877] Specific examples

[0878] When a user is walking on the sidewalk, the device will announce, "Three people are walking towards you from the left." Or, when the user is on a train platform, the device will announce, "There is a utility pole ahead of you on the right." This allows visually impaired people to understand their surroundings in real time and move around safely.

[0879] This system allows visually impaired people to travel safely, avoiding dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[0880] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0881] Step 1:

[0882] The device activates the camera and captures the image in front of it in real time. The input is the image data from the camera, and the output is the captured image frames. Specifically, the device captures images at 30 frames per second and stores them in its internal memory.

[0883] Step 2:

[0884] The device activates the distance sensor and measures the distance to the object. The input is distance data from the distance sensor, and the output is the measured distance information. Specifically, the device measures the distance 10 times per second and synchronizes it with the video frame.

[0885] Step 3:

[0886] The device analyzes the video data it acquires using image recognition software (OpenCV) to identify the location of objects and people. The input is the captured video frame, and the output is the location information of the object or person. Specifically, the device detects the contours of objects in each frame and lists the type and location of the identified objects.

[0887] Step 4:

[0888] Based on the analysis results, the device uses a generative AI model (GPT-4) to generate a natural language description. The input is the location information of objects and people, and the output is the generated natural language description. Specifically, the device inputs a prompt statement such as "Please explain the situation recognized from the camera footage. Example: There is a railroad crossing in front of you, three people are walking from the front left, there is a utility pole in front of you on the right" into the generative AI model, and an appropriate description is generated.

[0889] Step 5:

[0890] The device converts the generated text into speech using speech synthesis software (Google Text-to-Speech). The input is the generated natural language description, and the output is an audio file. Specifically, the device inputs the generated text into the speech synthesis software, which generates a high-quality audio file.

[0891] Step 6:

[0892] The device plays the generated audio file and outputs the audio from the speaker. The input is the audio file and the output is the audio from the speaker. Specifically, the device plays the audio file and outputs the audio from the speaker at an appropriate volume.

[0893] (Application example 2)

[0894] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0895] For visually impaired people to travel safely, it is important to understand the surrounding situation in real time. However, current technology lacks the means to properly recognize the surrounding situation and provide audio information when visually impaired people use self-driving vehicles. This poses the issue of insufficient safety when visually impaired people use self-driving vehicles.

[0896] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a visually impaired person walking with a mobile terminal hanging around their neck to be notified of the situation in front of them by voice; means for recognizing the surrounding situation using the camera of the mobile terminal and converting the information into voice; means for transmitting the voice information to the visually impaired person in real time; means for collecting surrounding situation data from the sensors of the autonomous vehicle; means for analyzing the data and identifying objects; means for analyzing the positional relationship and movement of the identified objects and extracting important information; means for converting the extracted information into voice information; and means for providing the voice information to the visually impaired person through an on-board speaker. This enables a visually impaired person using an autonomous vehicle to grasp the surrounding situation in real time and travel safely.

[0897] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0898] A "mobile device" is a portable electronic device, including a smartphone, tablet, etc.

[0899] A "camera" is a device that converts light into an electrical signal and captures images or videos.

[0900] "Audio information" is information that is transmitted using sound and is used to inform visually impaired people of their surroundings.

[0901] "Real-time" refers to processing and transmission occurring immediately without delay.

[0902] An "autonomous vehicle" is a vehicle that can drive autonomously without the need for driver operation.

[0903] A "sensor" is a device that detects a physical phenomenon and converts it into an electrical signal.

[0904] "Data" is a collection of information obtained through observation or measurement.

[0905] "Analysis" is the process of examining data in detail to clarify its meaning and structure.

[0906] An "object" refers to something that has a concrete form and exists in space.

[0907] The "positional relationship" is a relationship that indicates how multiple objects are arranged in space.

[0908] "Motion" refers to an object changing its position over time.

[0909] "Extraction" refers to extracting the necessary parts from a large amount of information.

[0910] An "in-vehicle speaker" is a device installed inside a vehicle to output sound.

[0911] The system for implementing this invention enables visually impaired people to understand the surrounding situation in real time and travel safely when using an autonomous vehicle. Specific embodiments of this system are described below.

[0912] Hardware and software used

[0913] Hardware: Autonomous vehicle sensors (cameras, LiDAR, radar), on-board computers, mobile devices, and on-board speakers

[0914] Software: Image recognition software (OpenCV), speech synthesis software (Google Text-to-Speech API), generative AI model (GPT-4)

[0915] Data processing and calculation

[0916] 1. Data collection: Collecting surrounding situation data from the autonomous vehicle's sensors (cameras, LiDAR, radar), which provides information about objects and obstacles around the vehicle.

[0917] 2. Image Recognition: The collected data is analyzed using image recognition software (OpenCV) to identify objects (people, cars, utility poles, etc.), thereby extracting information that is important for visually impaired people.

[0918] 3. Situational Assessment: Using a generative AI model (GPT-4), the system analyzes the relative positions and movements of identified objects to extract information that is important to the visually impaired, providing them with information to navigate safely.

[0919] 4. Speech generation: The extracted information is converted into speech using speech synthesis software (Google Text-to-Speech API), allowing visually impaired people to understand their surroundings in real time.

[0920] 5. Audio output: The generated audio information is provided to the visually impaired through the vehicle's speakers, allowing them to navigate safely within the autonomous vehicle.

[0921] Specific examples

[0922] For example, when a vehicle approaches an intersection, the following prompt sentence is input into the generative AI model:

[0923] Example prompt sentence:

[0924] "The vehicle is currently approaching an intersection. Three pedestrians are crossing from the left. A bicycle is approaching on the right. There is a traffic light ahead. Please provide this information in a voice message that is easy to understand for visually impaired people."

[0925] Based on this prompt, the generative AI model generates the following audio information:

[0926] "You are approaching an intersection. Three pedestrians are crossing from the left. A bicyclist is approaching on the right. There is a traffic light ahead."

[0927] This audio information is provided to the visually impaired through the vehicle's speakers, allowing them to understand their surroundings in real time and move around safely.

[0928] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0929] Step 1:

[0930] The server collects surrounding situation data from the autonomous vehicle's sensors (camera, LiDAR, radar). The input is raw data from the sensors, and the output is the collected surrounding situation data. Specifically, the sensors scan the vehicle's surroundings and detect the position and movement of objects.

[0931] Step 2:

[0932] The server analyzes the collected surrounding situation data using image recognition software (OpenCV) to identify objects. The input is the surrounding situation data collected in step 1, and the output is a list of identified objects. Specifically, the image recognition algorithm analyzes the data and identifies objects such as people, cars, and utility poles.

[0933] Step 3:

[0934] The server uses a generative AI model (GPT-4) to analyze the relative positions and movements of the identified objects and extract information that is important to visually impaired people. The input is a list of objects identified in step 2, and the output is a list of important information. Specifically, the generative AI model analyzes the relative positions and movements of objects and selects information that is important to visually impaired people.

[0935] Step 4:

[0936] The server converts the extracted information into audio information using speech synthesis software (Google Text-to-Speech API). The input is the list of important information extracted in step 3, and the output is the generated audio information. Specifically, the speech synthesis software converts the text information into audio data.

[0937] Step 5:

[0938] The server provides the generated audio information to the visually impaired person through the car's speaker. The input is the audio information generated in step 4, and the output is the audio information provided to the visually impaired person. Specifically, the audio data is played through the car's speaker, informing the visually impaired person of the surrounding situation.

[0939] Example 3

[0940] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0941] For visually impaired people to travel safely, they need a way to accurately grasp their surroundings in real time. However, conventional systems have had difficulty accurately recognizing the surroundings and transmitting appropriate voice information. In particular, the provision of information to help them avoid dangers on train platforms and while walking has been insufficient.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0943] In this invention, the server includes means for recognizing the surrounding situation using the camera of the mobile device and sending the information to the central processing unit, means for the central processing unit to recognize objects using video analysis software and generate a natural language explanatory text using a generative AI model, and means for sending the explanatory text to the mobile device and converting it into voice using voice synthesis software, thereby enabling visually impaired people to accurately grasp the surrounding situation in real time and move around safely.

[0944] "Visually impaired persons" refer to people who have visual impairments and have difficulty using visual information to understand their surroundings.

[0945] A "mobile terminal" is a portable electronic device that is equipped with a camera and communication functions.

[0946] "Capture device" refers to a device that includes a camera and sensors for capturing images.

[0947] "Central processing unit" refers to a computer or server that analyzes received data and performs the necessary processing.

[0948] "Video analysis software" refers to a program that analyzes captured video and recognizes objects and situations.

[0949] A "generative AI model" refers to an artificial intelligence model that generates natural language explanations based on input data.

[0950] "Natural language descriptions" refer to text that verbally describes the surroundings in a way that is easy for visually impaired people to understand.

[0951] "Speech synthesis software" refers to a program for converting text data into speech.

[0952] "Audio Information" refers to audio data generated by speech synthesis software.

[0953] This invention is a support system for visually impaired people to move around safely, and is implemented using a mobile device, a central processing unit, video analysis software, a generative AI model, and speech synthesis software.

[0954] First, the user activates the camera on their mobile device and captures images of their surroundings. The mobile device then transmits the captured images to a central processing unit in real time. The central processing unit then analyzes the received images using video analysis software (e.g., OpenCV) and recognizes objects and situations.

[0955] Next, the server inputs information about the recognized object into a generative AI model (e.g., GPT-4). The generative AI model generates a natural language description based on the input information. For example, it generates a description such as, "The edge of the platform is in front of you. There are stairs on your right."

[0956] The generated description is sent from the server to the mobile device, which then converts it into speech using speech synthesis software (e.g., Google Text-to-Speech). The converted speech is then transmitted to the user through the mobile device's speaker.

[0957] As a concrete example, consider the case where a user is standing on a train platform. The camera on the mobile device recognizes the edge of the platform and stairs and sends this information to the central processing unit. The central processing unit analyzes the image and uses a generative AI model to generate a description such as, "The edge of the platform is in front of you. There are stairs on your right." The description is sent to the mobile device, converted into audio, and conveyed to the user.

[0958] The same applies when a user is walking and the camera on the mobile device detects an approaching bicycle. The central processing unit generates a description such as "A bicycle is approaching from the front left" and sends it to the mobile device. The mobile device then converts the description into voice and conveys it to the user.

[0959] An example of a prompt is as follows:

[0960] "Generate natural language descriptions to describe the surroundings to a visually impaired person standing on a train platform. The camera footage shows the edge of the platform and a staircase."

[0961] This system enables visually impaired people to accurately grasp the surrounding situation in real time and move around safely. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0962] Step 1:

[0963] A user activates the camera on their mobile device and captures a video of their surroundings. The input is the real-time video captured by the camera. The output is the captured video data. Specifically, the user opens the camera app on their mobile device and points the camera at their surroundings.

[0964] Step 2:

[0965] The terminal transmits the captured video data to the central processing unit. The input is the video data captured by the camera of the mobile terminal. The output is the video data transmitted to the central processing unit. In concrete terms, the terminal uploads the video data to a server via the Internet.

[0966] Step 3:

[0967] The server analyzes the received video data using video analysis software. The input is the video data sent from the device. The output is information about the analyzed objects and situation. Specifically, the server uses video analysis software (e.g., OpenCV) to recognize objects in the video (e.g., the edge of the platform, stairs, bicycles, etc.).

[0968] Step 4:

[0969] The server inputs information about the recognized object into a generative AI model and generates a natural language description. The input is information about the object recognized by video analysis software. The output is the generated natural language description. Specifically, the server inputs the object information into a generative AI model (e.g., GPT-4) and generates a description such as, "The edge of the platform is in front of you. There are stairs on your right."

[0970] Step 5:

[0971] The server sends the generated explanation to the terminal. The input is the natural language explanation generated by the generative AI model. The output is the explanation sent to the terminal. Specifically, the server sends the explanation in text format to the terminal.

[0972] Step 6:

[0973] The device converts the received description into speech using speech synthesis software. The input is the description in natural language sent from the server. The output is the description converted into speech. Specifically, the device uses speech synthesis software (e.g., Google Text-to-Speech) to convert the description into speech.

[0974] Step 7:

[0975] The device transmits the explanatory text converted into voice to the user. The input is the explanatory text converted into voice. The output is the audio information transmitted to the user. In concrete terms, the device plays the audio information through a speaker to inform the user of the surrounding situation.

[0976] (Application example 3)

[0977] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0978] For visually impaired people to travel safely, it is important to accurately grasp their surroundings. However, with conventional technology, it is difficult for visually impaired people to grasp their surroundings in real time, which poses risks such as traffic accidents and falls from train platforms. In addition, there is a lack of means to communicate specific situations by voice, and the provision of information to ensure the safety of visually impaired people is insufficient.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0980] In this invention, the server includes means for audio-informing a visually impaired person of the situation in front of them as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into audio, means for transmitting the audio information to the visually impaired person in real time, means for analyzing images from the camera device in the mobile device and detecting surrounding objects using an object detection model, and means for providing audio guidance in accordance with the detected objects, thereby enabling the visually impaired person to grasp the surrounding situation in real time and move around safely.

[0981] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[0982] A "mobile terminal" is a portable electronic device that has communication and photographing functions.

[0983] "Image capture device" refers to a hardware component used to capture video or images.

[0984] An "object detection model" refers to an algorithm or program that uses machine learning and artificial intelligence technology to detect specific objects in video or images.

[0985] "Voice guidance" refers to a means of conveying information using voice.

[0986] "Real-time" refers to processing and transmission occurring immediately without delay.

[0987] "Surrounding circumstances" refers to the state of the objects, people, and environment around a visually impaired person.

[0988] "Audio information" refers to information that is conveyed using sound.

[0989] "Analysis" refers to examining data or information in detail to clarify its structure and meaning.

[0990] This invention is a system for enabling the visually impaired to move around safely, and provides real-time audio guidance of the surrounding situation using a mobile terminal. A specific embodiment of this system will be described below.

[0991] First, the visually impaired user walks around with a mobile device hanging from their neck. This mobile device is equipped with a camera that can capture images of the surroundings in real time. The images captured by the camera are analyzed by an object detection model built into the mobile device. This object detection model is built using machine learning frameworks such as TensorFlow, and is able to detect objects in the images with high accuracy.

[0992] Information about detected objects is communicated to the user as voice guidance. The voice guidance is generated using a speech synthesis engine such as pyttsx3 and provided to the user in real time. For example, if the user is standing on a train platform, the system will provide voice guidance such as, "The end of the platform is in front of you. There are stairs on your right." Also, while walking, the system can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[0993] The specific processing of this system involves first acquiring video from the mobile device's camera and inputting that video into an object detection model. The object detection model detects objects in the video and identifies their location and type. Next, audio guidance is generated based on the detected objects and conveyed to the user. This series of processes is performed in real time, allowing the user to always be aware of the latest surrounding situation.

[0994] For example, if a visually impaired person is standing on a train platform, the system will provide audio information such as, "The end of the platform is in front of you. There are stairs on your right." Also, while walking, the system can provide information such as, "A bicycle is approaching from the left," preventing collisions.

[0995] Examples of prompts to be input to a generative AI model include:

[0996] "I want to develop an application that provides audio guidance about the surroundings to visually impaired people standing on a train platform. The application captures images in real time using the camera on the mobile device, detects surrounding objects using an object detection model, and uses a voice engine to provide audio guidance to the user. Specifically, the application provides information about the edge of the platform, stairs, bicycles, etc."

[0997] In this way, the present invention provides an effective means for safe transportation for the visually impaired.

[0998] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0999] Step 1:

[1000] The terminal uses a camera to capture images of the surroundings in real time.

[1001] Input: Surrounding image

[1002] Output: Captured video data

[1003] Specific operation: The device's camera continuously captures video and stores the video data in memory.

[1004] Step 2:

[1005] The terminal inputs the captured video data into an object detection model.

[1006] Input: Captured video data

[1007] Output: Object detection results

[1008] Specific operation: The device uses a machine learning framework such as TensorFlow to input video data into an object detection model and detect objects in the video.

[1009] Step 3:

[1010] The device analyzes the object detection results and identifies the location and type of the detected object.

[1011] Input: Object detection results

[1012] Output: Object location and type information

[1013] Specific operation: The device analyzes the data obtained from the object detection model and identifies the location coordinates and type of each object (e.g., person, bicycle, stairs, etc.).

[1014] Step 4:

[1015] The terminal generates voice guidance based on the information of the identified object.

[1016] Input: Object location and type information

[1017] Output: Generated voice guidance

[1018] Specific operation: The device uses a speech synthesis engine such as pyttsx3 to generate voice guidance based on the information about the identified object. For example, it creates a voice message such as "A bicycle is approaching from the front left."

[1019] Step 5:

[1020] The terminal transmits the generated voice guidance to the user in real time.

[1021] Input: Generated voice prompt

[1022] Output: Audio information transmitted to the user

[1023] Specific operation: The terminal plays the generated voice guidance through the speaker to the user in real time, and the user can understand the surrounding situation by listening to the voice guidance.

[1024] This series of processes enables visually impaired users to grasp the situation around them in real time and move around safely.

[1025] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1026] "Example 1"

[1027] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system includes a means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a smartphone around their neck, a means for recognizing the surrounding situation using the smartphone's camera and converting that information into voice, and a means for transmitting voice information to the visually impaired person in real time. Furthermore, this system includes an emotion engine that recognizes the user's emotions.

[1028] "Example 2"

[1029] The emotion engine recognizes the user's emotion from the tone of their voice, volume, speaking style, etc. For example, when a user is angry, their voice tone is high, their volume is loud, and they speak quickly. The emotion engine recognizes these characteristics and determines the user's emotion as "anger."

[1030] "Example 3"

[1031] The system adjusts the way voice information is transmitted based on the user's emotions recognized by the emotion engine. For example, if the user is angry, the system reduces the amount of voice information transmitted. On the other hand, if the user is calm, the system transmits voice information normally. This makes it possible to provide services that take the user's emotions into consideration.

[1032] The processing flow of each embodiment will be described below.

[1033] "Example 1"

[1034] Step 1: A visually impaired person hangs the smartphone around their neck and activates the system.

[1035] Step 2: The system uses the smartphone's camera to recognize the surroundings and converts that information into audio.

[1036] Step 3: The system transmits audio information to the visually impaired person in real time.

[1037] Step 4: The system activates the emotion engine to recognize the user's emotion.

[1038] "Example 2"

[1039] Step 1: The emotion engine recognizes the emotion from the user's voice tone, volume, speaking style, etc. Step 2: The emotion engine communicates the recognized emotion to the system.

[1040] "Example 3"

[1041] Step 1: The system adjusts how it delivers audio information based on the emotional information received from the emotion engine.

[1042] Step 2: The system conveys the adjusted audio information to the visually impaired person.

[1043] Example 1

[1044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1045] For visually impaired people to walk safely, they need to be able to grasp their surroundings in real time and take appropriate action. However, conventional systems were unable to accurately recognize the surrounding situation and communicate it via voice, nor were they able to provide feedback based on the user's emotional state. As a result, visually impaired people were unable to respond appropriately when they felt anxious or in an emergency, which could lead to accidents or danger.

[1046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1047] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into voice, means for transmitting the voice information to the visually impaired person in real time, means including an emotion engine for recognizing the user's emotions, and means for providing appropriate feedback based on the user's emotions, thereby enabling the visually impaired person to accurately grasp the surrounding situation and further receive appropriate feedback according to their own emotional state.

[1048] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1049] A "mobile terminal" is a portable electronic device that has communication and photography functions.

[1050] "Capture device" refers to a device that includes a camera or sensor for capturing images or video.

[1051] "Audio information" refers to data obtained by converting acquired information into audio.

[1052] "Real-time" refers to processing and transmission occurring immediately without delay.

[1053] An "emotion engine" refers to software or algorithms that analyze and recognize a user's emotional state.

[1054] "Feedback" refers to the response or information that a system provides to a user.

[1055] This invention is a system that visually impaired people can wear around their necks using a mobile device to receive audio notifications of the situation in front of them. The system uses the mobile device's camera to recognize the surroundings and converts that information into audio. It also includes an emotion engine that recognizes the user's emotions and provides appropriate feedback based on the user's emotions.

[1056] Hardware and software used

[1057] Mobile device: A portable electronic device such as a smartphone.

[1058] Camera: Built-in camera on smartphone.

[1059] Image recognition software: Image analysis tools such as TensorFlow and OpenCV.

[1060] Text-to-speech software: Text-to-speech tools such as Google Text-to-Speech.

[1061] Emotion engine: A sentiment analysis tool such as Affectiva or Microsoft Azure Emotion API.

[1062] Explanation of program processing

[1063] 1. Starting the camera and acquiring images

[1064] The device activates the smartphone camera and captures images of the surrounding area in real time.

[1065] Example: A smartphone camera continuously captures the scenery in front of it.

[1066] 2. Image Recognition

[1067] The images captured by the device are analyzed using TensorFlow and OpenCV to recognize objects and obstacles.

[1068] Example: A TensorFlow model analyzes an image to detect pedestrians and cars ahead.

[1069] 3. Audio conversion

[1070] The device converts the information it recognizes into text, which is then converted into audio using Google Text-to-Speech.

[1071] Example: The text "There is a pedestrian ahead" is generated and converted to speech.

[1072] 4. Audio Output

[1073] The device transmits the generated audio to the user in real time using Bluetooth earphones or the built-in speaker.

[1074] Example: Your smartphone will announce, "There is a pedestrian ahead."

[1075] 5. Emotion recognition

[1076] The terminal uses an emotion engine to recognize the user's emotion.

[1077] Example: A smartphone camera captures a user's face, and an emotion engine analyzes the user's facial expressions to detect emotions such as anxiety or relief.

[1078] 6. Emotional Feedback

[1079] The device provides appropriate feedback based on the user's emotions.

[1080] Example: An emotion engine detects a user's anxiety and provides a reassuring voice message such as "It's okay, there are no obstacles ahead."

[1081] Examples and prompts

[1082] Example: When a visually impaired person is walking with a smartphone hanging around their neck, if there is a pedestrian ahead, the smartphone will announce with a voice, "There is a pedestrian ahead." Also, if the user is feeling anxious, the emotion engine will detect this and provide a reassuring message, "It's okay, there are no obstacles ahead."

[1083] Example prompt sentence:

[1084] Explain how a visually impaired person walking with a smartphone around their neck can be notified by voice if there is a pedestrian ahead. Also, explain how a reassuring message can be provided if the user is feeling anxious.

[1085] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1086] Step 1:

[1087] Starting the camera and acquiring images

[1088] The device activates the smartphone camera and captures images of the surrounding area in real time.

[1089] Input: Smartphone camera is activated.

[1090] Data processing: The camera continuously captures images.

[1091] Output: Acquired image data.

[1092] Specific operation: The smartphone's camera app will automatically start up and take a picture of the scenery in front of it every second.

[1093] Step 2:

[1094] Image Recognition

[1095] The images captured by the device are analyzed using image recognition software (e.g., TensorFlow or OpenCV) to recognize objects and obstacles.

[1096] Input: Acquired image data.

[1097] Data computation: A TensorFlow model analyzes the image and applies an object detection algorithm.

[1098] Output: Information about recognized objects and obstacles.

[1099] How it works: The TensorFlow model analyzes images and detects pedestrians and cars ahead.

[1100] Step 3:

[1101] Audio conversion

[1102] The device converts the information it recognizes into text, which is then converted into speech using speech synthesis software (e.g., Google Text-to-Speech).

[1103] Input: Information about recognized objects and obstacles.

[1104] Data processing: Convert the recognized information into text and then convert it into speech using a speech synthesis engine.

[1105] Output: Audio data.

[1106] Specific behavior: The text "There is a pedestrian ahead" is generated and converted into speech.

[1107] Step 4:

[1108] Audio Output

[1109] The device transmits the generated audio to the user in real time using Bluetooth earphones or the built-in speaker.

[1110] Input: Audio data.

[1111] Data processing: Sends audio data to Bluetooth earphones or built-in speakers.

[1112] Output: The audio transmitted to the user.

[1113] Specific action: The smartphone will announce "There is a pedestrian ahead."

[1114] Step 5:

[1115] emotion recognition

[1116] The device uses an emotion engine (e.g., Affectiva or Microsoft Azure Emotion API) to recognize the user's emotions.

[1117] Input: A face image of the user taken with a smartphone camera.

[1118] Data calculation: The emotion engine analyzes the user's facial expressions and recognizes their emotional state.

[1119] Output: Information about the user's emotional state.

[1120] Specific operation: The smartphone camera takes a picture of the user's face, and the emotion engine analyzes the user's facial expressions to detect emotions such as anxiety or relief.

[1121] Step 6:

[1122] Emotion-based feedback

[1123] The device provides appropriate feedback based on the user's emotions.

[1124] Input: User's emotional state information.

[1125] Data processing: Generate appropriate voice messages based on emotional state.

[1126] Output: Audio feedback delivered to the user.

[1127] What it does: The emotion engine detects the user's anxiety and provides reassuring voice messages such as "It's okay, there are no obstacles ahead."

[1128] (Application example 1)

[1129] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1130] There is a lack of support systems to help visually impaired people stay safe in self-driving vehicles. In particular, there is a need for systems that can recognize the situation inside and outside the vehicle in real time and provide appropriate feedback. It is also important to recognize the emotions of visually impaired people and provide appropriate feedback based on that.

[1131] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes: means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing an information processing device around their neck; means for recognizing the surrounding situation using an imaging device of the information processing device and converting that information into voice; means for transmitting the voice information to the visually impaired person in real time; emotion recognition means for recognizing the user's emotion; means for providing appropriate feedback based on the emotion recognition means; and means for recognizing the situation inside and outside the vehicle and converting that information into voice for transmission. This enables visually impaired people to stay safely inside self-driving vehicles.

[1132] An "information processing device" is a device that inputs, processes, and outputs data, and specifically refers to portable electronic devices such as smartphones and tablets.

[1133] An "imaging device" is a device that converts light into an electrical signal to generate an image, and specifically refers to a camera or video camera.

[1134] "Means for converting into speech" refers to technology that converts text or data into an audio signal, and specifically includes speech synthesis engines and text-to-speech software.

[1135] "Means of real-time transmission" refers to technologies that transmit information instantly without delay, and specifically includes streaming technologies and real-time communication protocols.

[1136] "Emotion recognition means" refers to technology that analyzes and recognizes emotions from a user's facial expressions and voice, and specifically includes emotion recognition algorithms and machine learning models.

[1137] "Means for providing feedback" refers to technology that provides appropriate information or instructions depending on the user's behavior or state, and specifically includes audio guidance and vibration feedback devices.

[1138] "Means for recognizing the situation inside and outside the vehicle" refers to technology that detects and analyzes the environment and situation inside and outside the vehicle, and specifically includes situational awareness systems that use sensors and cameras.

[1139] The system for implementing this invention is a support system for visually impaired people to stay safely in autonomous vehicles. The system uses hardware and software such as an information processing device, an imaging device, a speech synthesis engine, an emotion recognition algorithm, and a real-time communication protocol.

[1140] The server provides a means for visually impaired people to walk around with an information processing device around their neck, and to receive audio notifications of the situation in front of them. Information processing devices include smartphones and tablets. The smartphone camera is used as the imaging device. The camera captures the surrounding environment and processes the video data in real time.

[1141] The server analyzes the video data acquired from the imaging device and recognizes the surrounding situation. Specifically, it uses an image processing library such as OpenCV to perform face detection and obstacle detection. The recognized information is converted into voice using a voice synthesis engine (e.g., pyttsx3). The voice information is transmitted to the visually impaired person in real time.

[1142] Furthermore, the server recognizes the user's emotions using an emotion recognition means. It uses an emotion recognition algorithm (e.g., EmotionRecognizer) to analyze emotions from the user's facial expressions and voice. Based on the analyzed emotion information, it provides appropriate feedback. The feedback is provided through audio guidance or a vibration feedback device.

[1143] To recognize the situation inside and outside the vehicle, the server includes a situational awareness system using sensors and cameras, which can detect and analyze the environment and situation inside and outside the vehicle, for example, detecting obstacles and recognizing road signs.

[1144] As a concrete example, consider a scenario in which a visually impaired person is riding in an autonomous vehicle. By providing real-time audio information about the situation inside and outside the vehicle, the visually impaired person can travel safely. An example of a prompt sentence to be input to the generative AI model is as follows:

[1145] "I want to develop a support system to help visually impaired people stay safe in self-driving vehicles. I will create a system that uses a smartphone camera to recognize the situation inside and outside the vehicle, converts that information into voice, and transmits it in real time. I will also combine it with an emotion engine to recognize the user's emotions and provide appropriate feedback."

[1146] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1147] Step 1:

[1148] The server acquires video data from the camera of the information processing device (smartphone). The input is real-time video data from the camera, and the output is video frames. Specifically, the camera captures the surrounding situation and sends the video data to the server.

[1149] Step 2:

[1150] The server analyzes the acquired video data using an image processing library (OpenCV). The input is the video frame, and the output is the recognized objects (e.g., faces, obstacles, road signs). Specifically, the server converts the video frame to grayscale and performs face and obstacle detection.

[1151] Step 3:

[1152] The server converts the recognized object information into speech using a speech synthesis engine (pyttsx3). The input is the recognized object information, and the output is speech data. Specifically, the server converts the recognized object information into text, and then inputs the text into the speech synthesis engine to generate speech data.

[1153] Step 4:

[1154] The server transmits the generated voice data to the information processing device in real time and conveys it to the visually impaired person. The input is voice data, and the output is voice information heard by the visually impaired person. Specifically, the voice data is transmitted to the information processing device using streaming technology.

[1155] Step 5:

[1156] The server analyzes the user's emotions using an emotion recognition algorithm (EmotionRecognizer). The input is the user's facial expression and voice data, and the output is the recognized emotion information. Specifically, the server extracts the user's face area from the camera image and inputs it into the emotion recognition algorithm to analyze the emotion.

[1157] Step 6:

[1158] The server provides appropriate feedback based on the recognized emotional information. The input is the recognized emotional information, and the output is feedback information (e.g., audio guidance, vibration feedback). Specific operations include determining the feedback content according to the emotional information and transmitting it to the user via audio guidance or a vibration feedback device.

[1159] Step 7:

[1160] The server recognizes the situation inside and outside the vehicle using sensors and cameras and analyzes that information. The input is data from the sensors and cameras, and the output is analyzed situation information. Specifically, it collects data from the sensors and cameras and analyzes it using situation recognition algorithms.

[1161] Step 8:

[1162] The server converts the analyzed situation information into speech using a speech synthesis engine and transmits it to the visually impaired person. The input is the analyzed situation information and the output is speech data. Specifically, the situation information is converted into text, and the text is input into the speech synthesis engine to generate speech data, which is then transmitted to the visually impaired person.

[1163] Example 2

[1164] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1165] For visually impaired people to travel safely, it is important to understand their surroundings in real time. However, conventional systems make it difficult for visually impaired people to accurately and quickly recognize their surroundings, exposing them to risks such as traffic accidents, railroad crossing accidents, and falls from train platforms. Furthermore, there is a lack of means to recognize the emotions of visually impaired people and provide appropriate feedback accordingly, which makes it difficult to increase the user's sense of security.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1167] In this invention, the server includes: a means for a visually impaired person walking with a mobile device hanging around their neck to be notified of the situation in front of them by voice; a means for recognizing the surrounding situation using a camera in the mobile device and converting that information into voice; a means for transmitting the voice information to the visually impaired person in real time; a means for capturing the user's voice and analyzing the voice data to recognize emotions; and a means for generating appropriate feedback based on the recognized emotions and providing it to the user. This enables the visually impaired person to accurately and quickly grasp the situation around them and avoid dangers such as traffic accidents, railroad crossing accidents, and falls from train platforms. Furthermore, by recognizing the user's emotions and providing feedback according to them, the user's sense of security can be enhanced.

[1168] "Visually impaired" refers to a person who has a visual impairment and is unable to fully utilize visual information.

[1169] A "mobile terminal" is a portable electronic device that has communication functions and the ability to execute various applications.

[1170] "Capture device" refers to a device for capturing video, including cameras and video cameras.

[1171] "Audio information" refers to information that is communicated using voice, and refers to a voice message that is generated using voice synthesis technology.

[1172] "Real-time" refers to processing and transmission occurring immediately without delay.

[1173] "User" refers to anyone who uses the system, and includes in particular the visually impaired.

[1174] "Audio data" refers to data that records audio in digital format.

[1175] "Means for recognizing emotions" refers to technology for analyzing a user's emotions from voice data and determining a specific emotional state.

[1176] "Feedback" refers to responses or information provided by a system to a user, and particularly includes appropriate responses according to the user's emotions.

[1177] "Surrounding circumstances" refers to the state of objects, people, and the environment around the user.

[1178] MODE FOR CARRYING OUT THE INVENTION

[1179] This invention provides a voice information providing system for visually impaired people to move around safely, and includes a function for recognizing the user's emotions and providing appropriate feedback. Specific embodiments of this system will be described below.

[1180] Hardware and software used

[1181] Hardware: Mobile devices (smartphones, etc.), imaging devices (cameras), audio input devices (microphones), audio output devices (speakers)

[1182] Software: Image recognition software (e.g., OpenCV), speech recognition software (e.g., Google Speech-to-Text API), emotion recognition engine (e.g., IBM Watson Tone Analyzer)

[1183] System Operation Overview

[1184] 1. The camera captures the image

[1185] The camera captures the surroundings in real time, for example, when a user is walking down a sidewalk, the camera captures the image in front of them.

[1186] 2. The device sends the video data to the server

[1187] The device sends the captured video data to a server, which then transmits the data over the Internet.

[1188] 3. The server analyzes the video data

[1189] The server analyzes the received video data using image recognition software (e.g., OpenCV), extracting information such as "three people are walking towards you from the left front."

[1190] 4. The server converts the analysis results into audio information

[1191] The server converts the analysis results into voice information using natural language processing technology, for example, generating voice information such as "Three people are walking towards you from the left."

[1192] 5. The server sends the audio information to the device.

[1193] The server sends the generated voice information to the terminal, and the data is transmitted via the Internet.

[1194] 6. The device provides voice information to the user

[1195] The device will then provide the received voice information to the user through the speaker, for example, an audio notification saying, "Three people are walking towards you from the left."

[1196] 7. The microphone captures the user's voice

[1197] A microphone captures the user's voice in real time, for example when the user asks, "Where are you now?"

[1198] 8. The device sends the audio data to the server

[1199] The device sends the captured audio data to a server, which then transmits the data over the Internet.

[1200] 9. The server analyzes the voice data and recognizes emotions.

[1201] The server converts the received voice data into text using speech recognition software (e.g., Google Speech-to-Text API), and then analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Tone Analyzer). For example, it detects "anxiety" from the tone and volume of the user's voice.

[1202] 10. The server generates emotion-based feedback

[1203] The server generates appropriate feedback based on the analyzed emotions, such as "Don't worry, the roads are safe."

[1204] 11. The server sends feedback to the device

[1205] The server generates feedback and sends it to the device, with the data transmitted over the internet.

[1206] 12. Device provides feedback to the user

[1207] The device will provide the received feedback to the user through the speaker, for example, by announcing "Don't worry, the roads are safe."

[1208] Specific examples

[1209] When a user is walking on the sidewalk, a camera captures video of what is ahead and the device sends the video data to a server. The server analyzes the video data, extracts information such as "Three people are walking towards you from the left," and converts it into audio information. The server sends the generated audio information to the device, which notifies the user through a speaker that "Three people are walking towards you from the left." When the user asks "Where are you now?", the microphone captures the audio and the device sends the audio data to the server. The server analyzes the audio data and detects "anxiety" from the tone and volume of the user's voice. The server generates feedback such as "Don't worry, the roads are safe" and sends it to the device. The device notifies the user through a speaker that "Don't worry, the roads are safe."

[1210] Prompt Sentence Examples

[1211] "When a user is walking down the sidewalk, the camera captures an image of what is ahead, and image recognition software analyzes it to determine, 'Three people are walking towards you from the left.' Voice recognition software converts the user's question, 'Where are you now?' into text, and an emotion recognition engine detects 'anxiety' from the tone and volume of the user's voice. The server integrates this information and generates audio information, such as, 'Three people are walking towards you from the left. Don't worry, the road is safe,' which is communicated to the user through the speaker."

[1212] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1213] Step 1:

[1214] The camera captures the image

[1215] The camera captures the surroundings in real time, for example, when a user is walking down a sidewalk, the camera captures the image in front of them.

[1216] Input: Surrounding image

[1217] Output: Captured video data

[1218] Step 2:

[1219] The device sends the video data to the server

[1220] The device sends the captured video data to a server, which then transmits the data over the Internet.

[1221] Input: Captured video data

[1222] Output: Video data sent to the server

[1223] Step 3:

[1224] The server analyzes the video data

[1225] The server analyzes the received video data using image recognition software (e.g., OpenCV), extracting information such as "three people are walking towards you from the left front."

[1226] Input: Video data sent to the server

[1227] Output: Analyzed situation information (e.g., three people are walking from the front left)

[1228] Step 4:

[1229] The server converts the analysis results into audio information

[1230] The server converts the analysis results into voice information using natural language processing technology, for example, generating voice information such as "Three people are walking towards you from the left."

[1231] Input: Parsed status information

[1232] Output: Generated audio information

[1233] Step 5:

[1234] The server sends the voice information to the device.

[1235] The server sends the generated voice information to the terminal, and the data is transmitted via the Internet.

[1236] Input: Generated audio information

[1237] Output: Audio information sent to the device

[1238] Step 6:

[1239] The terminal provides voice information to the user

[1240] The device will then provide the received voice information to the user through the speaker, for example, an audio notification saying, "Three people are walking towards you from the left."

[1241] Input: Audio information sent to the device

[1242] Output: Audio information provided to the user

[1243] Step 7:

[1244] A microphone captures the user's voice

[1245] A microphone captures the user's voice in real time, for example when the user asks, "Where are you now?"

[1246] Input: User's voice

[1247] Output: Captured audio data

[1248] Step 8:

[1249] The device sends the audio data to the server.

[1250] The device sends the captured audio data to a server, which then transmits the data over the Internet.

[1251] Input: Captured audio data

[1252] Output: Audio data sent to the server

[1253] Step 9:

[1254] The server analyzes the voice data and recognizes emotions.

[1255] The server converts the received voice data into text using speech recognition software (e.g., Google Speech-to-Text API), and then analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Tone Analyzer). For example, it detects "anxiety" from the tone and volume of the user's voice.

[1256] Input: Audio data sent to the server

[1257] Output: Parsed emotion information (e.g., anxiety)

[1258] Step 10:

[1259] The server generates emotion-based feedback

[1260] The server generates appropriate feedback based on the analyzed emotions, such as "Don't worry, the roads are safe."

[1261] Input: Parsed emotion information

[1262] Output: The generated feedback

[1263] Step 11:

[1264] The server sends feedback to the device

[1265] The server generates feedback and sends it to the device, with the data transmitted over the internet.

[1266] Input: Generated feedback

[1267] Output: Feedback sent to the device

[1268] Step 12:

[1269] The device provides feedback to the user

[1270] The device will provide the received feedback to the user through the speaker, for example, by announcing "Don't worry, the roads are safe."

[1271] Input: Feedback sent to the device

[1272] Output: Feedback provided to the user

[1273] (Application example 2)

[1274] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1275] When visually impaired people use self-driving vehicles, they lack the means to understand their surroundings in real time and ride safely. There is also a need for a system that can recognize the emotions of visually impaired people and provide appropriate feedback to give them a sense of security.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1277] In this invention, the server includes: a means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing an information terminal around their neck; a means for recognizing the surrounding situation using a camera device of the information terminal and converting that information into voice; a means for transmitting the voice information to the visually impaired person in real time; a means for acquiring the user's voice and recognizing their emotion; and a means for providing appropriate voice feedback based on the recognized emotion. This allows visually impaired people to understand the surrounding situation in real time when using an autonomous vehicle and ride safely. Furthermore, providing feedback according to the user's emotion can give them a sense of security.

[1278] "Visually impaired" refers to a person who has a visual impairment and is unable to fully utilize visual information.

[1279] An "information terminal" is an electronic device for processing and communicating data, including smartphones and tablets.

[1280] "Photography device" refers to a device for capturing images or videos, such as a camera or sensor.

[1281] "Means for converting into voice" refers to technology or devices for converting acquired information into voice data and transmitting it to the user.

[1282] "Means of real-time communication" refers to technologies and devices that convey acquired information to users immediately and without delay.

[1283] The "means for acquiring the user's voice" refers to a technique or device for collecting the user's speech using a microphone or the like.

[1284] "Means for recognizing emotions" refers to technology or devices for analyzing and identifying emotions from a user's voice, facial expressions, etc.

[1285] "Means for providing appropriate audio feedback" refers to techniques and devices for generating and conveying appropriate audio messages to a user based on the recognized emotion.

[1286] A system for implementing this invention includes a means for a visually impaired person to walk with an information terminal hanging around their neck and be notified of the situation in front of them by voice, a means for recognizing the surrounding situation using a camera device in the information terminal and converting that information into voice, a means for transmitting the voice information to the visually impaired person in real time, a means for acquiring the user's voice and recognizing their emotions, and a means for providing appropriate voice feedback based on the recognized emotions.

[1287] Hardware and software used

[1288] Hardware: Smartphone or head-mounted display, microphone, vehicle sensors

[1289] Software: speech_recognition library, pyttsx3 library, emotion recognition engine (e.g. emotion_recognition library), vehicle sensor interface (e.g. vehicle_sensors library)

[1290] System Operation

[1291] 1. The information terminal uses a camera to recognize the surrounding situation. For example, it obtains information such as whether there is a railroad crossing ahead or if someone is walking towards the left front.

[1292] 2. The information terminal converts the acquired situational information into audio and conveys it to the visually impaired person in real time, providing audio information such as "There is a railroad crossing ahead," or "Three people are walking towards you from the left."

[1293] 3. The information terminal uses a microphone to capture the user's voice and converts what the user says into text using voice recognition technology.

[1294] 4. The emotion recognition engine analyzes and identifies the user's emotion from the acquired voice data. For example, if the user says "I'm scared," the emotion recognition engine will recognize "fear."

[1295] 5. The information terminal generates appropriate voice feedback based on the recognized emotion and conveys it to the user, for example, "Don't worry. I'm driving safely."

[1296] Specific examples

[1297] When the vehicle approaches a railroad crossing, the information terminal will issue a voice notification saying, "There is a railroad crossing ahead. Please be careful."

[1298] If the user says "I'm scared," the emotion recognition engine will recognize the "fear," and the information terminal will provide voice feedback saying, "Don't worry. I'm driving safely."

[1299] Prompt Sentence Examples

[1300] User says: "Scary"

[1301] Prompt the generative AI model: "The user says they're scared. Please provide appropriate feedback."

[1302] Generative AI model output: "Don't worry, I'm driving safely."

[1303] In this way, a system can be realized that supports visually impaired people in safely using self-driving vehicles.

[1304] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1305] Step 1:

[1306] The device recognizes the surrounding situation using a camera. Image data is acquired from the camera as input. Data is processed using an image recognition algorithm to identify the location of obstacles and people ahead. Recognized situation information (e.g., "There is a railroad crossing ahead," "Three people are walking towards you from the left") is generated as output.

[1307] Step 2:

[1308] The device converts the acquired situation information into voice and transmits it to the visually impaired person in real time. As input, it receives the situation information generated in step 1. As data processing, it uses a voice synthesis engine (e.g., the pyttsx3 library) to convert the text information into voice data. As output, it generates voice information and transmits it to the user through a speaker.

[1309] Step 3:

[1310] The device acquires the user's voice using a microphone. As input, it acquires voice data from the microphone. As data processing, it converts the voice data into text using a speech recognition engine (e.g., speech_recognition library). As output, it generates text data based on the user's speech.

[1311] Step 4:

[1312] The device uses an emotion recognition engine to analyze and identify the user's emotion from the acquired voice data. As input, it receives the text data generated in step 3. As data processing, it uses an emotion recognition algorithm (e.g., emotion_recognition library) to identify the emotion from the text data. As output, it generates recognized emotion information (e.g., "fear" or "anger").

[1313] Step 5:

[1314] The device generates appropriate voice feedback based on the recognized emotion and transmits it to the user. As input, it receives the emotion information generated in step 4. As data processing, it generates a feedback message according to the emotion and converts it into voice data using a voice synthesis engine. As output, it generates voice feedback and transmits it to the user through a speaker.

[1315] In this way, a system can be realized that supports visually impaired people in safely using self-driving vehicles.

[1316] Example 3

[1317] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1318] For visually impaired people to travel safely, it is necessary to grasp the situation around them in real time and take appropriate action. However, conventional systems do not provide information according to the user's emotional state, and are unable to reduce the user's stress and anxiety. In addition, there is a lack of means to communicate specific situations in detail, making it difficult for visually impaired people to fully avoid the dangers they face.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1320] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into voice, means for transmitting the voice information to the visually impaired person in real time, means for the mobile device to analyze the user's emotions using an emotion analysis device, and means for adjusting the method of transmitting the voice information based on the user's emotions.This makes it possible to provide information for the visually impaired to move safely in real time and to transmit information appropriately according to the user's emotional state.

[1321] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining visual information.

[1322] A "mobile terminal" is a portable electronic device that has communication and photographing functions.

[1323] "Image capture device" refers to hardware used to capture images or video.

[1324] "Audio information" refers to audio information generated using speech synthesis technology.

[1325] "Real-time" refers to processing and transmission occurring immediately without delay.

[1326] An "emotion analysis device" refers to hardware and software for analyzing a user's emotional state from their facial expressions and voice.

[1327] "User emotion" refers to the user's psychological state or mood.

[1328] The "method of transmitting audio information" refers to the means or method of transmitting audio information to the user.

[1329] This invention is a system that provides real-time information for the safe movement of visually impaired people and transmits appropriate information according to the emotional state of the user. Specific embodiments of this system are described below.

[1330] First, the user walks around with the mobile device hanging from their neck. The mobile device is equipped with a camera and an emotion analysis device. When the user launches the application, the camera captures images of the surroundings in real time, and the sensors collect information on distance and obstacles. This data is temporarily stored in the mobile device.

[1331] Next, the device sends the captured video and sensor data to a server. The data is encrypted and transmitted securely. The server then uses computer vision technology (e.g., OpenCV) and voice recognition technology (e.g., voice recognition API) to analyze the received data. This allows the location of the platform edge, stairs, bicycles, etc. to be identified.

[1332] The analyzed information is sent from the server to the device. The device then uses speech synthesis software (e.g., speech synthesis API) to convert the analysis results into audio information. For example, audio information such as "The end of the platform is in front of you. There are stairs on your right." is generated. The device then transmits the generated audio information to the user. The user can listen to the audio information through earphones or speakers.

[1333] Furthermore, the device uses an emotion analyzer (e.g., emotion analysis SDK) to analyze the user's facial expressions through the camera and recognize the user's emotional state. For example, it identifies whether the user is angry or calm. Based on the user's emotional state, the device adjusts the way it transmits audio information. If the user is angry, it reduces the amount of audio information transmitted, and if the user is calm, it transmits audio information normally.

[1334] As a concrete example, consider a visually impaired person standing on a train platform. The user's mobile device uses a camera and sensors to capture the surroundings and sends the data to a server. The server uses OpenCV to analyze the video data and identify the location of the edge of the platform and stairs. It then uses a speech synthesis API to generate audio information such as, "The edge of the platform is in front of you. There are stairs to your right." The user can move safely by listening to this audio information.

[1335] Additionally, if a user is walking and sees a bicycle approaching from the left front, the mobile device's camera and sensors capture the information and send it to the server. The server analyzes the data and uses a speech synthesis API to generate audio information such as "A bicycle is approaching from the left front." By listening to this audio information, the user can prevent a collision accident.

[1336] Example prompts to input to the generative AI model:

[1337] "Please explain in detail the processing steps of a system that will provide audible information about the surroundings to a visually impaired person standing on a train platform. Please also specify the names of the hardware and software used."

[1338] This system allows the visually impaired to move around safely and improves the quality of their daily lives. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1339] Step 1:

[1340] The user starts the mobile terminal and starts the dedicated application.

[1341] Input: The application is started by a user action.

[1342] Output: Application launch complete, camera and sensors start working.

[1343] Specific behavior: A user taps the application icon on their mobile device, the application launches, and the camera and sensors automatically start working, ready to capture the surroundings.

[1344] Step 2:

[1345] The device uses its camera and sensors to capture the surroundings.

[1346] Input: Surrounding images and distance information obtained from cameras and sensors.

[1347] Output: Captured video and sensor data.

[1348] How it works: The device's camera captures images of the surroundings in real time, and sensors collect distance and obstacle information, which is then temporarily stored on the device.

[1349] Step 3:

[1350] The device sends the captured data to the server.

[1351] Input: Captured video and sensor data.

[1352] Output: The data sent to the server.

[1353] Specific operation: The device encrypts the captured video data and sensor data and securely transmits them to the server. The server receives this data.

[1354] Step 4:

[1355] The server analyzes the received data and identifies the surrounding situation.

[1356] Input: Video data and sensor data sent to the server.

[1357] Output: Parsed surrounding information.

[1358] Specific operation: The server analyzes video data using computer vision technology (e.g., OpenCV) and audio data using speech recognition technology (e.g., speech recognition API), thereby identifying the location of platform edges, stairs, bicycles, etc.

[1359] Step 5:

[1360] The server sends the analysis results to the device.

[1361] Input: Parsed surrounding information.

[1362] Output: Analysis results sent to the device.

[1363] Specific operation: The server sends the analysis results to the device, including information on identified obstacles and the movement route.

[1364] Step 6:

[1365] The device converts the analysis results into voice information using speech synthesis software.

[1366] Input: Analysis results received from the server.

[1367] Output: The generated audio information.

[1368] Specific operation: The device uses speech synthesis software (e.g., speech synthesis API) to convert the analysis results into speech information. For example, speech information such as "The end of the platform is ahead of you. There are stairs on your right." is generated.

[1369] Step 7:

[1370] The terminal conveys the audio information to the user.

[1371] Input: The generated audio information.

[1372] Output: The audio information conveyed to the user.

[1373] Specific operation: The terminal transmits the generated voice information to the user, who can listen to the voice information through earphones or speakers.

[1374] Step 8:

[1375] The terminal uses an emotion analyzer to analyze the user's emotions.

[1376] Input: User's facial expression data.

[1377] Output: The analyzed emotional state of the user.

[1378] Specific operation: The device uses an emotion analyzer (e.g., Emotion Analysis SDK) to analyze the user's facial expressions through the camera and recognize the user's emotional state. For example, it can determine whether the user is angry or calm.

[1379] Step 9:

[1380] The terminal adjusts the way audio information is delivered based on the user's emotions.

[1381] Input: Parsed user emotional state.

[1382] Output: The method of conveying the adjusted audio information.

[1383] Specific operation: The device adjusts the way it transmits voice information based on the user's emotional state. If the user is angry, it will transmit voice information less, and if the user is calm, it will transmit voice information normally.

[1384] (Application example 3)

[1385] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1386] For visually impaired people to travel safely, it is important to accurately grasp their surroundings and obtain information at the appropriate time. However, with conventional technology, it is difficult for visually impaired people to grasp their surroundings in real time while walking or riding in an autonomous vehicle, and information transmission based on the user's emotions is not possible. This has left visually impaired people at risk of being involved in traffic accidents or falling from train platforms.

[1387] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1388] In this invention, the server includes means for audibly informing a visually impaired person of the situation in front of them as they walk with a smartphone hanging around their neck, means for recognizing the surrounding situation using the camera of the smartphone and converting that information into audio, means for transmitting the audio information to the visually impaired person in real time, means for recognizing the user's emotions and adjusting the method of transmitting the audio information based on the emotions, and means for audibly transmitting the situation around the vehicle in an autonomous vehicle. This enables a visually impaired person to grasp the surrounding situation in real time while walking or riding in an autonomous vehicle, and to transmit appropriate information according to the user's emotions.

[1389] "Visually impaired persons" refer to people who have visual impairments and are unable to obtain sufficient visual information.

[1390] A "smartphone" is a type of mobile phone, a portable information terminal with a variety of functions, including internet connectivity, application use, and camera functionality.

[1391] A "camera" is a device that converts light into an electrical signal and captures images or videos.

[1392] "Audio information" is information that is transmitted using sound and is used to inform visually impaired people of their surroundings.

[1393] "Real-time" refers to processing and transmission occurring immediately without delay.

[1394] "Emotion recognition" is a technology that determines a user's emotional state from facial expressions, voice, etc.

[1395] An "autonomous vehicle" is a vehicle that uses artificial intelligence and sensor technology to drive autonomously without a human driver.

[1396] "Surrounding circumstances" refers to the state of objects, people, and the environment around a visually impaired person or an autonomous vehicle.

[1397] The "method of transmitting audio information" refers to the means or method of transmitting audio information to the user.

[1398] This invention is a system for enabling visually impaired people to move around safely, and uses hardware and software such as smartphones, cameras, speech synthesis engines, emotion recognition engines, and self-driving vehicles.

[1399] System configuration

[1400] 1. Smartphone: A smartphone worn around the neck by the visually impaired. Equipped with a camera, it captures the surroundings.

[1401] 2. Camera: Built into the smartphone, it captures images in real time.

[1402] 3. Speech synthesis engine: Converts information recognized from captured video into speech and conveys it to visually impaired people.

[1403] 4. Emotion recognition engine: Recognizes emotions from the user's facial expressions and voice and adjusts how voice information is conveyed.

[1404] 5. Self-driving vehicles: When a visually impaired person is riding in the vehicle, the situation around the vehicle is communicated to them via voice.

[1405] Program processing

[1406] The server uses the smartphone's camera to recognize the surrounding situation and converts that information into voice. It uses a speech synthesis engine (e.g., pyttsx3) to transmit voice information to visually impaired people in real time. It also uses an emotion recognition engine (e.g., EmotionRecognizer) to recognize the user's emotions and adjust the way voice information is transmitted based on those emotions. In autonomous vehicles, the situation around the vehicle is communicated through voice.

[1407] Specific examples

[1408] For example, if a visually impaired person is standing on a train platform, their smartphone will provide audio information such as, "The end of the platform is ahead of you. There are stairs on your right." This will help the visually impaired person avoid the risk of falling off the platform. Additionally, while walking, the smartphone can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[1409] In self-driving vehicles, when the vehicle approaches an intersection, it provides information such as "A vehicle is approaching from the right" to ensure the user's safety. Furthermore, if the user is angry, it slows down the voice speed to convey the information discreetly.

[1410] Prompt Sentence Examples

[1411] An example of a prompt to input to a generative AI model is as follows:

[1412] Develop an application that provides audio feedback to visually impaired passengers in autonomous vehicles about the vehicle's surroundings, including the ability to adjust the audio feedback based on the user's emotions.

[1413] In this way, a system can be realized that allows visually impaired people to move around safely.

[1414] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1415] Step 1:

[1416] The server uses the smartphone's camera to capture the surroundings. The input is real-time video from the camera, and the output is video data. Specifically, the camera continuously captures frames and generates video data.

[1417] Step 2:

[1418] The server analyzes the acquired video data using an image processing algorithm (e.g., OpenCV) to recognize objects and people. The input is the video data obtained in step 1, and the output is information about the recognized objects and people. Specifically, the server extracts features from the video data and inputs them into an object recognition model to obtain the recognition results.

[1419] Step 3:

[1420] The server converts the information about the recognized objects and people into voice information. The input is the recognition result obtained in step 2, and the output is voice information. Specifically, it uses a voice synthesis engine (e.g., pyttsx3) to convert the text information into voice.

[1421] Step 4:

[1422] The server transmits audio information to the visually impaired person in real time. The input is the audio information obtained in step 3, and the output is the audio transmitted to the visually impaired person. Specifically, the audio is played through the smartphone speaker.

[1423] Step 5:

[1424] The server uses the smartphone's camera and microphone to recognize the user's emotions. The input is camera footage and audio data, and the output is the user's emotional state. Specifically, it uses an emotion recognition engine (e.g., EmotionRecognizer) to determine emotions from facial expressions and voice.

[1425] Step 6:

[1426] The server adjusts the way the voice information is delivered based on the user's emotions. The input is the emotional state obtained in step 5, and the output is the adjusted voice information. Specific actions include changing the voice speed and tone.

[1427] Step 7:

[1428] In an autonomous vehicle, the server recognizes the situation around the vehicle using cameras and sensors and transmits that information via voice. The input is the vehicle's sensor data and the output is voice information. Specifically, the server analyzes the data obtained from the vehicle's sensors, generates voice information using a voice synthesis engine, and plays it through the vehicle's speakers.

[1429] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1430] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1431] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1432] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1433] [Third embodiment]

[1434] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1435] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1437] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1441] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1442] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1443] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1444] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1445] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1446] "Example 1"

[1447] This invention is a system that notifies visually impaired people of the situation in front of them by voice by hanging a smartphone around their neck. Specifically, the system uses the smartphone's camera to recognize the surrounding situation and converts that information into voice. This voice information is transmitted to the visually impaired in real time.

[1448] "Example 2"

[1449] The audio information provides specific information about the situation, such as "There is a railroad crossing ahead," "Three people are walking towards you from the left," "There is a telephone pole ahead to the right," etc. This allows visually impaired people to know the situation in front of them in real time, and helps protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[1450] "Example 3"

[1451] As a specific example of use, if a visually impaired person is standing on a train platform, their smartphone will provide audio information such as, "The end of the platform is in front of you. There are stairs on your right." This will help the visually impaired person avoid the risk of falling off the platform. Also, while walking, the smartphone can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[1452] The processing flow of each embodiment will be described below.

[1453] "Example 1"

[1454] Step 1: A visually impaired person wears a smartphone around their neck.

[1455] Step 2: The smartphone camera recognizes the surroundings.

[1456] Step 3: Convert the recognized information into speech.

[1457] Step 4: Transmit audio information to the visually impaired in real time.

[1458] "Example 2"

[1459] Step 1: Voice information is generated. This information indicates specific situations such as "There is a railroad crossing ahead," "Three people are walking towards you from the left," and "There is a telephone pole ahead to the right."

[1460] Step 2: The visually impaired person listens to the audio information and understands the situation in front of them.

[1461] Step 3: The visually impaired person adjusts their behavior based on the audio information to protect themselves from dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[1462] "Example 3"

[1463] Step 1: When a visually impaired person is standing on a station platform, their smartphone will communicate information via voice, such as, "The end of the platform is in front of you. There are stairs on your right."

[1464] Step 2: Visually impaired people listen to the audio information and avoid the risk of falling off the platform.

[1465] Step 3: While walking, information such as "A bicycle is approaching from the front left" is provided to prevent collisions.

[1466] Example 1

[1467] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1468] For the visually impaired to travel safely, it is important to be able to grasp the situation around them in real time. However, conventional technology has made it difficult for the visually impaired to accurately and quickly recognize the situation around them. In particular, there has been a lack of means to provide information to help them avoid dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms. This has hindered the safe travels of the visually impaired.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1470] In this invention, the server includes means for analyzing video data transmitted from the mobile information terminal and recognizing objects and text, means for generating the recognized information as text data, and means for converting the text data into audio data. This allows visually impaired people to use their mobile information terminals to receive audio information about their surroundings in real time. This allows visually impaired people to avoid dangers such as traffic accidents, railroad crossing accidents, and falls from train platforms, and to travel safely.

[1471] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1472] A "personal digital assistant" is a portable electronic device that is equipped with communication functions and an imaging device.

[1473] "Imaging device" refers to a device for capturing images, such as a camera or sensor.

[1474] "Video data" refers to data that digitally represents visual information captured by an imaging device.

[1475] A "server" refers to a computer system that processes and manages data over a network.

[1476] "Analysis" refers to the process of examining data in detail to understand its structure and content.

[1477] "Object recognition" refers to a technology that identifies specific objects within video data and determines their location and type.

[1478] "Text data" refers to data that represents character information in digital form.

[1479] "Audio data" refers to data that represents audio information in digital form.

[1480] "Speech synthesis" refers to the technology of generating voice data based on text data.

[1481] "Real-time" refers to processing and transmission occurring immediately without delay.

[1482] "User" refers to a visually impaired person who uses this system.

[1483] This invention is a system that uses a portable information terminal hung around the neck of a visually impaired person to provide audio information about the situation in front of them. Specifically, the system uses the imaging device of the portable information terminal to recognize the surrounding situation and converts that information into audio. This audio information is transmitted to the visually impaired person in real time.

[1484] The server receives the video data sent from the mobile information terminal. The server uses computer vision technology to analyze the video data. Specifically, it uses libraries such as TensorFlow and OpenCV to recognize objects and text in the video. The server generates the recognized information as text data. Next, the server uses speech synthesis software to convert this text data into speech. Specifically, it uses a text-to-speech API or a speech synthesis engine.

[1485] The terminal (mobile information terminal) receives the voice data sent from the server and transmits it to the user in real time through a speaker, allowing visually impaired people to hear what is happening in front of them.

[1486] As a concrete example, consider a scene where a user is walking through town. The device's camera captures a traffic light ahead, and the device sends this image to a server. The server analyzes the image and recognizes that the traffic light is red. The server generates text saying, "There is a red light ahead. Please stop," and converts this into audio data. The audio data is sent to the device, which then tells the user through a speaker, "There is a red light ahead. Please stop."

[1487] Example prompt sentence:

[1488] "Please create a program that uses the imaging device of a mobile information terminal to recognize the situation ahead and notify the user by voice that the traffic light is red."

[1489] This system allows visually impaired people to travel safely, avoiding dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[1490] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1491] Step 1:

[1492] The device captures the surroundings in real time using a built-in image capture device. The input is the image data from the image capture device, and the output is the captured image data. Specifically, the device continues to capture images at a constant frame rate.

[1493] Step 2:

[1494] The terminal transmits the captured video data to a server via the Internet. The input is the captured video data, and the output is the video data transmitted to the server. Specifically, the terminal transmits the video data in a compressed format (e.g., H.264).

[1495] Step 3:

[1496] The server receives the video data transmitted from the terminal. The input is the video data transmitted from the terminal, and the output is the received video data. In concrete operation, the server receives the data via the network.

[1497] Step 4:

[1498] The server analyzes the received video data and recognizes objects and text. The input is the received video data, and the output is the recognized objects and text information. Specifically, the server performs the analysis using computer vision libraries such as TensorFlow and OpenCV.

[1499] Step 5:

[1500] The server generates text data based on the recognized objects and text information. The input is the recognized objects and text information, and the output is the generated text data. Specifically, the server generates text such as "There is a red light ahead. Please stop."

[1501] Step 6:

[1502] The server converts the generated text data into voice data. The input is the generated text data, and the output is the generated voice data. Specifically, the server generates voice data using a Text-to-Speech API or a voice synthesis engine.

[1503] Step 7:

[1504] The server sends the generated audio data to the terminal via the Internet. The input is the generated audio data, and the output is the audio data sent to the terminal. Specifically, the server sends the audio data in a compressed format (e.g., MP3).

[1505] Step 8:

[1506] The terminal plays back the received voice data and transmits it to the user. The input is the received voice data, and the output is the voice information transmitted to the user. In concrete terms, the terminal plays back voice through a speaker and tells the user, "There is a red light ahead. Please stop."

[1507] (Application example 1)

[1508] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1509] Visually impaired people face difficulties in understanding their surroundings when searching for products in physical stores or navigating specific locations. There is also a lack of support for visually impaired people to navigate safely and find the products they need.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1511] In this invention, the server includes means for notifying a visually impaired person of the situation in front of them by voice as they walk around with the information processing device hanging around their neck, means for recognizing the surrounding situation using an imaging device in the information processing device and converting that information into voice, means for transmitting that voice information to the visually impaired person in real time, and means for recognizing the surrounding situation when the visually impaired person is searching for a product or moving to a specific location in a physical store and transmitting that information by voice in real time, thereby enabling the visually impaired person to move safely within the physical store and find the products they need.

[1512] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1513] "Information processing device" refers to an electronic device that has the function of inputting, processing, and outputting data.

[1514] An "imaging device" refers to a device that converts light into an electrical signal and generates image data.

[1515] "Surroundings" refers to the state of the objects and environment around a visually impaired person.

[1516] "Means for converting into audio" refers to technology or devices that convert image data or other information into audio data.

[1517] "Real-time" refers to processing and transmission occurring immediately without delay.

[1518] "Brick and mortar store" refers to a store that exists in a physical location and offers goods and services.

[1519] "Products" refers to items sold in physical stores.

[1520] "Specific location" refers to the specific location to which the visually impaired person wishes to travel.

[1521] The system for carrying out this invention provides a function that notifies a visually impaired person of the situation in front of them by voice when they walk while wearing an information processing device around their neck. A specific embodiment of this system will be described below.

[1522] 1. System Configuration

[1523] The system consists of the following main components:

[1524] Information processing devices: Portable electronic devices such as smartphones and tablets.

[1525] Imaging device: A camera built into an information processing device.

[1526] Audio output device: A speaker built into the information processing device or a connected earphone.

[1527] Software: Applications for image recognition, speech synthesis, and real-time processing.

[1528] 2. Program Processing

[1529] The server implements the system using the following means:

[1530] Image recognition means: Analyzes video data captured by an imaging device and recognizes the surrounding situation. Specifically, object recognition is performed using machine learning libraries such as TensorFlow.

[1531] Speech synthesis means: Converts the recognized information into speech data. Specifically, it uses a speech synthesis library such as pyttsx3.

[1532] Real-time communication: Audio data is transmitted to the visually impaired in real time, allowing them to instantly understand their surroundings.

[1533] 3. Hardware and Software Used

[1534] Hardware: Smartphone camera, speakers, earphones.

[1535] Software: OpenCV (image processing library), TensorFlow (machine learning library), pyttsx3 (speech synthesis library).

[1536] 4. Specific Examples

[1537] For example, if a visually impaired person is looking for "milk" in a physical store, the camera in the information processing device will scan the shelves, and if "milk" is recognized, a voice message will be sent saying, "Milk found." In this way, visually impaired people can safely move around the physical store and find the products they need.

[1538] 5. Examples of prompts

[1539] An example of a prompt to input to a generative AI model is as follows:

[1540] "Design a navigation system to help visually impaired people find products in a physical store. The system uses a smartphone camera to recognize the surroundings and communicates that information in real time via voice."

[1541] In this way, a system can be realized that helps visually impaired people shop more smoothly in physical stores.

[1542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1543] Step 1:

[1544] The server activates the imaging device of the information processing device and acquires surrounding video data. The input is video data from the camera, and the output is the acquired video frames. Specifically, the camera continuously captures video and generates frames in real time.

[1545] Step 2:

[1546] The server preprocesses the captured video frames. The input is the video frame captured in step 1, and the output is the preprocessed image data. Specifically, the server resizes the video frame and performs noise removal and color correction.

[1547] Step 3:

[1548] The server inputs the preprocessed image data into an object recognition model to recognize the surrounding situation. The input is the preprocessed image data, and the output is the label and location information of the recognized object. Specifically, it uses TensorFlow's object recognition model to detect objects in the image and identify their label and location.

[1549] Step 4:

[1550] The server converts the information of the recognized object into audio data. The input is the label and location information of the recognized object, and the output is audio data. Specifically, it uses the pyttsx3 speech synthesis library to convert the text information into audio.

[1551] Step 5:

[1552] The server transmits the generated voice data to the visually impaired person in real time. The input is voice data, and the output is voice information that the visually impaired person hears. Specifically, the voice is played through the speaker or earphones of the information processing device.

[1553] Step 6:

[1554] Users navigate and search for products within a physical store based on voice information. The input is voice information transmitted from the server, and the output is the user's actions. Specifically, the user follows the voice guidance to walk and find the desired product or location.

[1555] Example 2

[1556] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1557] For the visually impaired to travel safely, it is important to be able to grasp the situation around them in real time. However, with conventional technology, it is difficult for the visually impaired to accurately and quickly recognize their surroundings, which has led to risks such as traffic accidents, railroad crossing accidents, and falls from station platforms. For this reason, there is a need for a system that enables the visually impaired to grasp the situation around them in real time and travel safely.

[1558] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1559] In this invention, the server includes: means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a mobile device around their neck; means for recognizing the surrounding situation using an imaging device of the mobile device and converting that information into voice; means for transmitting the voice information to the visually impaired person in real time; means for the mobile device to acquire surrounding images and distance information using an imaging device and a distance measuring device; means for the mobile device to analyze the acquired video data using image recognition software and identify the positions of objects and people; means for the mobile device to generate a natural language explanatory text using a generative AI model based on the analysis results; means for the mobile device to convert the generated text into voice using voice synthesis software; and means for the mobile device to play the generated voice file and output the voice from a speaker. This allows the visually impaired person to grasp the surrounding situation in real time and move around safely.

[1560] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1561] A "mobile terminal" is a portable electronic device that has communication and computing capabilities.

[1562] "Imaging device" refers to a device, such as a camera or sensor, that receives light and generates image data.

[1563] "Distance measuring device" refers to a sensor or device used to measure the distance to an object.

[1564] "Image recognition software" refers to a program that analyzes image data and identifies objects and people.

[1565] "Generative AI model" refers to a model that uses artificial intelligence to generate natural language text from data.

[1566] "Speech synthesis software" refers to a program for converting text data into voice data.

[1567] "Audio information" refers to information transmitted as sound, and includes content intended to inform visually impaired people of their surroundings.

[1568] "Real-time" refers to processing and transmission occurring immediately without delay.

[1569] A "speaker" refers to a device that converts audio signals into sound waves and outputs them.

[1570] This invention is a system that enables visually impaired people to grasp their surroundings in real time and move around safely. Specific embodiments of this system will be described below.

[1571] Hardware and software used

[1572] Hardware: Mobile devices (smartphones and tablets), imaging devices (cameras), distance measurement devices (distance sensors), speakers

[1573] Software: Image recognition software (e.g., OpenCV), speech synthesis software (e.g., Google Text-to-Speech), generative AI models (e.g., GPT-4)

[1574] System Overview

[1575] This system allows visually impaired people to walk around with a mobile device hanging from their neck and receive audio notifications of the situation in front of them. The mobile device's imaging device and distance measurement device are used to capture images of the surrounding area and distance information, and the data is analyzed and provided to the user as audio information.

[1576] Data Acquisition

[1577] The device activates the camera to capture images of what is ahead in real time, while the distance sensor measures the distance to objects, allowing visually impaired people to have a detailed understanding of their surroundings while walking.

[1578] Data analysis

[1579] The image data acquired by the device is analyzed using image recognition software (OpenCV) to identify the location of objects and people. For example, it can recognize from the camera image that "three people are walking to the left and in front." Based on the results of this analysis, a generative AI model (GPT-4) is used to generate a natural language description.

[1580] Status text generation

[1581] The device inputs the analysis results as prompts into the generative AI model to generate appropriate explanatory text. An example of a prompt is "Please explain the situation recognized from the camera image. Example: There is a railroad crossing in front of you, three people are walking from the front left, and there is a utility pole in front of you on the right."

[1582] Speech synthesis

[1583] The device converts the generated text into speech using speech synthesis software (Google Text-to-Speech). The generated audio file is high-quality and adjusted to be easy for the user to hear.

[1584] Providing audio information

[1585] The device plays the generated audio file and outputs the audio through the speaker at an appropriate volume, allowing the user to accurately understand their surroundings.

[1586] Specific examples

[1587] When a user is walking on the sidewalk, the device will announce, "Three people are walking towards you from the left." Or, when the user is on a train platform, the device will announce, "There is a utility pole ahead of you on the right." This allows visually impaired people to understand their surroundings in real time and move around safely.

[1588] This system allows visually impaired people to travel safely, avoiding dangers such as traffic accidents, railroad crossing accidents, and falls from station platforms.

[1589] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1590] Step 1:

[1591] The device activates the camera and captures the image in front of it in real time. The input is the image data from the camera, and the output is the captured image frames. Specifically, the device captures images at 30 frames per second and stores them in its internal memory.

[1592] Step 2:

[1593] The device activates the distance sensor and measures the distance to the object. The input is distance data from the distance sensor, and the output is the measured distance information. Specifically, the device measures the distance 10 times per second and synchronizes it with the video frame.

[1594] Step 3:

[1595] The device analyzes the video data it acquires using image recognition software (OpenCV) to identify the location of objects and people. The input is the captured video frame, and the output is the location information of the object or person. Specifically, the device detects the contours of objects in each frame and lists the type and location of the identified objects.

[1596] Step 4:

[1597] Based on the analysis results, the device uses a generative AI model (GPT-4) to generate a natural language description. The input is the location information of objects and people, and the output is the generated natural language description. Specifically, the device inputs a prompt statement such as "Please explain the situation recognized from the camera footage. Example: There is a railroad crossing in front of you, three people are walking from the front left, there is a utility pole in front of you on the right" into the generative AI model, and an appropriate description is generated.

[1598] Step 5:

[1599] The device converts the generated text into speech using speech synthesis software (Google Text-to-Speech). The input is the generated natural language description, and the output is an audio file. Specifically, the device inputs the generated text into the speech synthesis software, which generates a high-quality audio file.

[1600] Step 6:

[1601] The device plays the generated audio file and outputs the audio from the speaker. The input is the audio file and the output is the audio from the speaker. Specifically, the device plays the audio file and outputs the audio from the speaker at an appropriate volume.

[1602] (Application example 2)

[1603] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1604] For visually impaired people to travel safely, it is important to understand the surrounding situation in real time. However, current technology lacks the means to properly recognize the surrounding situation and provide audio information when visually impaired people use self-driving vehicles. This poses the issue of insufficient safety when visually impaired people use self-driving vehicles.

[1605] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a visually impaired person walking with a mobile terminal hanging around their neck to be notified of the situation in front of them by voice; means for recognizing the surrounding situation using the camera of the mobile terminal and converting the information into voice; means for transmitting the voice information to the visually impaired person in real time; means for collecting surrounding situation data from the sensors of the autonomous vehicle; means for analyzing the data and identifying objects; means for analyzing the positional relationship and movement of the identified objects and extracting important information; means for converting the extracted information into voice information; and means for providing the voice information to the visually impaired person through an on-board speaker. This enables a visually impaired person using an autonomous vehicle to grasp the surrounding situation in real time and travel safely.

[1606] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1607] A "mobile device" is a portable electronic device, including a smartphone, tablet, etc.

[1608] A "camera" is a device that converts light into an electrical signal and captures images or videos.

[1609] "Audio information" is information that is transmitted using sound and is used to inform visually impaired people of their surroundings.

[1610] "Real-time" refers to processing and transmission occurring immediately without delay.

[1611] An "autonomous vehicle" is a vehicle that can drive autonomously without the need for driver operation.

[1612] A "sensor" is a device that detects a physical phenomenon and converts it into an electrical signal.

[1613] "Data" is a collection of information obtained through observation or measurement.

[1614] "Analysis" is the process of examining data in detail to clarify its meaning and structure.

[1615] An "object" refers to something that has a concrete form and exists in space.

[1616] The "positional relationship" is a relationship that indicates how multiple objects are arranged in space.

[1617] "Motion" refers to an object changing its position over time.

[1618] "Extraction" refers to extracting the necessary parts from a large amount of information.

[1619] An "in-vehicle speaker" is a device installed inside a vehicle to output sound.

[1620] The system for implementing this invention enables visually impaired people to understand the surrounding situation in real time and travel safely when using an autonomous vehicle. Specific embodiments of this system are described below.

[1621] Hardware and software used

[1622] Hardware: Autonomous vehicle sensors (cameras, LiDAR, radar), on-board computers, mobile devices, and on-board speakers

[1623] Software: Image recognition software (OpenCV), speech synthesis software (Google Text-to-Speech API), generative AI model (GPT-4)

[1624] Data processing and calculation

[1625] 1. Data collection: Collecting surrounding situation data from the autonomous vehicle's sensors (cameras, LiDAR, radar), which provides information about objects and obstacles around the vehicle.

[1626] 2. Image Recognition: The collected data is analyzed using image recognition software (OpenCV) to identify objects (people, cars, utility poles, etc.), thereby extracting information that is important for visually impaired people.

[1627] 3. Situational Assessment: Using a generative AI model (GPT-4), the system analyzes the relative positions and movements of identified objects to extract information that is important to the visually impaired, providing them with information to navigate safely.

[1628] 4. Speech generation: The extracted information is converted into speech using speech synthesis software (Google Text-to-Speech API), allowing visually impaired people to understand their surroundings in real time.

[1629] 5. Audio output: The generated audio information is provided to the visually impaired through the vehicle's speakers, allowing them to navigate safely within the autonomous vehicle.

[1630] Specific examples

[1631] For example, when a vehicle approaches an intersection, the following prompt sentence is input into the generative AI model:

[1632] Example prompt sentence:

[1633] "The vehicle is currently approaching an intersection. Three pedestrians are crossing from the left. A bicycle is approaching on the right. There is a traffic light ahead. Please provide this information in a voice message that is easy to understand for visually impaired people."

[1634] Based on this prompt, the generative AI model generates the following audio information:

[1635] "You are approaching an intersection. Three pedestrians are crossing from the left. A bicyclist is approaching on the right. There is a traffic light ahead."

[1636] This audio information is provided to the visually impaired through the vehicle's speakers, allowing them to understand their surroundings in real time and move around safely.

[1637] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1638] Step 1:

[1639] The server collects surrounding situation data from the autonomous vehicle's sensors (camera, LiDAR, radar). The input is raw data from the sensors, and the output is the collected surrounding situation data. Specifically, the sensors scan the vehicle's surroundings and detect the position and movement of objects.

[1640] Step 2:

[1641] The server analyzes the collected surrounding situation data using image recognition software (OpenCV) to identify objects. The input is the surrounding situation data collected in step 1, and the output is a list of identified objects. Specifically, the image recognition algorithm analyzes the data and identifies objects such as people, cars, and utility poles.

[1642] Step 3:

[1643] The server uses a generative AI model (GPT-4) to analyze the relative positions and movements of the identified objects and extract information that is important to visually impaired people. The input is a list of objects identified in step 2, and the output is a list of important information. Specifically, the generative AI model analyzes the relative positions and movements of objects and selects information that is important to visually impaired people.

[1644] Step 4:

[1645] The server converts the extracted information into audio information using speech synthesis software (Google Text-to-Speech API). The input is the list of important information extracted in step 3, and the output is the generated audio information. Specifically, the speech synthesis software converts the text information into audio data.

[1646] Step 5:

[1647] The server provides the generated audio information to the visually impaired person through the car's speaker. The input is the audio information generated in step 4, and the output is the audio information provided to the visually impaired person. Specifically, the audio data is played through the car's speaker, informing the visually impaired person of the surrounding situation.

[1648] Example 3

[1649] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1650] For visually impaired people to travel safely, they need a way to accurately grasp their surroundings in real time. However, conventional systems have had difficulty accurately recognizing the surroundings and transmitting appropriate voice information. In particular, the provision of information to help them avoid dangers on train platforms and while walking has been insufficient.

[1651] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1652] In this invention, the server includes means for recognizing the surrounding situation using the camera of the mobile device and sending the information to the central processing unit, means for the central processing unit to recognize objects using video analysis software and generate a natural language explanatory text using a generative AI model, and means for sending the explanatory text to the mobile device and converting it into voice using voice synthesis software, thereby enabling visually impaired people to accurately grasp the surrounding situation in real time and move around safely.

[1653] "Visually impaired persons" refer to people who have visual impairments and have difficulty using visual information to understand their surroundings.

[1654] A "mobile terminal" is a portable electronic device that is equipped with a camera and communication functions.

[1655] "Capture device" refers to a device that includes a camera and sensors for capturing images.

[1656] "Central processing unit" refers to a computer or server that analyzes received data and performs the necessary processing.

[1657] "Video analysis software" refers to a program that analyzes captured video and recognizes objects and situations.

[1658] A "generative AI model" refers to an artificial intelligence model that generates natural language explanations based on input data.

[1659] "Natural language descriptions" refer to text that verbally describes the surroundings in a way that is easy for visually impaired people to understand.

[1660] "Speech synthesis software" refers to a program for converting text data into speech.

[1661] "Audio Information" refers to audio data generated by speech synthesis software.

[1662] This invention is a support system for visually impaired people to move around safely, and is implemented using a mobile device, a central processing unit, video analysis software, a generative AI model, and speech synthesis software.

[1663] First, the user activates the camera on their mobile device and captures images of their surroundings. The mobile device then transmits the captured images to a central processing unit in real time. The central processing unit then analyzes the received images using video analysis software (e.g., OpenCV) and recognizes objects and situations.

[1664] Next, the server inputs information about the recognized object into a generative AI model (e.g., GPT-4). The generative AI model generates a natural language description based on the input information. For example, it generates a description such as, "The edge of the platform is in front of you. There are stairs on your right."

[1665] The generated description is sent from the server to the mobile device, which then converts it into speech using speech synthesis software (e.g., Google Text-to-Speech). The converted speech is then transmitted to the user through the mobile device's speaker.

[1666] As a concrete example, consider the case where a user is standing on a train platform. The camera on the mobile device recognizes the edge of the platform and stairs and sends this information to the central processing unit. The central processing unit analyzes the image and uses a generative AI model to generate a description such as, "The edge of the platform is in front of you. There are stairs on your right." The description is sent to the mobile device, converted into audio, and conveyed to the user.

[1667] The same applies when a user is walking and the camera on the mobile device detects an approaching bicycle. The central processing unit generates a description such as "A bicycle is approaching from the front left" and sends it to the mobile device. The mobile device then converts the description into voice and conveys it to the user.

[1668] An example of a prompt is as follows:

[1669] "Generate natural language descriptions to describe the surroundings to a visually impaired person standing on a train platform. The camera footage shows the edge of the platform and a staircase."

[1670] This system enables visually impaired people to accurately grasp the surrounding situation in real time and move around safely. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1671] Step 1:

[1672] A user activates the camera on their mobile device and captures a video of their surroundings. The input is the real-time video captured by the camera. The output is the captured video data. Specifically, the user opens the camera app on their mobile device and points the camera at their surroundings.

[1673] Step 2:

[1674] The terminal transmits the captured video data to the central processing unit. The input is the video data captured by the camera of the mobile terminal. The output is the video data transmitted to the central processing unit. In concrete terms, the terminal uploads the video data to a server via the Internet.

[1675] Step 3:

[1676] The server analyzes the received video data using video analysis software. The input is the video data sent from the device. The output is information about the analyzed objects and situation. Specifically, the server uses video analysis software (e.g., OpenCV) to recognize objects in the video (e.g., the edge of the platform, stairs, bicycles, etc.).

[1677] Step 4:

[1678] The server inputs information about the recognized object into a generative AI model and generates a natural language description. The input is information about the object recognized by video analysis software. The output is the generated natural language description. Specifically, the server inputs the object information into a generative AI model (e.g., GPT-4) and generates a description such as, "The edge of the platform is in front of you. There are stairs on your right."

[1679] Step 5:

[1680] The server sends the generated explanation to the terminal. The input is the natural language explanation generated by the generative AI model. The output is the explanation sent to the terminal. Specifically, the server sends the explanation in text format to the terminal.

[1681] Step 6:

[1682] The device converts the received description into speech using speech synthesis software. The input is the description in natural language sent from the server. The output is the description converted into speech. Specifically, the device uses speech synthesis software (e.g., Google Text-to-Speech) to convert the description into speech.

[1683] Step 7:

[1684] The device transmits the explanatory text converted into voice to the user. The input is the explanatory text converted into voice. The output is the audio information transmitted to the user. In concrete terms, the device plays the audio information through a speaker to inform the user of the surrounding situation.

[1685] (Application example 3)

[1686] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1687] For visually impaired people to travel safely, it is important to accurately grasp their surroundings. However, with conventional technology, it is difficult for visually impaired people to grasp their surroundings in real time, which poses risks such as traffic accidents and falls from train platforms. In addition, there is a lack of means to communicate specific situations by voice, and the provision of information to ensure the safety of visually impaired people is insufficient.

[1688] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1689] In this invention, the server includes means for audio-informing a visually impaired person of the situation in front of them as they walk while wearing a mobile device around their neck, means for recognizing the surrounding situation using a camera device in the mobile device and converting that information into audio, means for transmitting the audio information to the visually impaired person in real time, means for analyzing images from the camera device in the mobile device and detecting surrounding objects using an object detection model, and means for providing audio guidance in accordance with the detected objects, thereby enabling the visually impaired person to grasp the surrounding situation in real time and move around safely.

[1690] "Visually impaired persons" refer to people who have visual impairments and have difficulty obtaining sufficient visual information.

[1691] A "mobile terminal" is a portable electronic device that has communication and photographing functions.

[1692] "Image capture device" refers to a hardware component used to capture video or images.

[1693] An "object detection model" refers to an algorithm or program that uses machine learning and artificial intelligence technology to detect specific objects in video or images.

[1694] "Voice guidance" refers to a means of conveying information using voice.

[1695] "Real-time" refers to processing and transmission occurring immediately without delay.

[1696] "Surrounding circumstances" refers to the state of the objects, people, and environment around a visually impaired person.

[1697] "Audio information" refers to information that is conveyed using sound.

[1698] "Analysis" refers to examining data or information in detail to clarify its structure and meaning.

[1699] This invention is a system for enabling the visually impaired to move around safely, and provides real-time audio guidance of the surrounding situation using a mobile terminal. A specific embodiment of this system will be described below.

[1700] First, the visually impaired user walks around with a mobile device hanging from their neck. This mobile device is equipped with a camera that can capture images of the surroundings in real time. The images captured by the camera are analyzed by an object detection model built into the mobile device. This object detection model is built using machine learning frameworks such as TensorFlow, and is able to detect objects in the images with high accuracy.

[1701] Information about detected objects is communicated to the user as voice guidance. The voice guidance is generated using a speech synthesis engine such as pyttsx3 and provided to the user in real time. For example, if the user is standing on a train platform, the system will provide voice guidance such as, "The end of the platform is in front of you. There are stairs on your right." Also, while walking, the system can provide information such as, "A bicycle is approaching from the left front," helping to prevent collisions.

[1702] The specific processing of this system involves first acquiring video from the mobile device's camera and inputting that video into an object detection model. The object detection model detects objects in the video and identifies their location and type. Next, audio guidance is generated based on the detected objects and conveyed to the user. This series of processes is performed in real time, allowing the user to always be aware of the latest surrounding situation.

[1703] For example, if a visually impaired person is standing on a train platform, the system will provide audio information such as, "The end of the platform is in front of you. There are stairs on your right." Also, while walking, the system can provide information such as, "A bicycle is approaching from the left," preventing collisions.

[1704] Examples of prompts to be input to a generative AI model include:

[1705] "I want to develop an application that provides audio guidance about the surroundings to visually impaired people standing on a train platform. The application captures images in real time using the camera on the mobile device, detects surrounding objects using an object detection model, and uses a voice engine to provide audio guidance to the user. Specifically, the application provides information about the edge of the platform, stairs, bicycles, etc."

[1706] In this way, the present invention provides an effective means for safe transportation for the visually impaired.

[1707] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1708] Step 1:

[1709] The terminal uses a camera to capture images of the surroundings in real time.

[1710] Input: Surrounding image

[1711] Output: Captured video data

[1712] Specific operation: The device's camera continuously captures video and stores the video data in memory.

[1713] Step 2:

[1714] The terminal inputs the captured video data into an object detection model.

[1715] Input: Captured video data

[1716] Output: Object detection results

[1717] Specific operation: The device uses a machine learning framework such as TensorFlow to input video data into an object detection model and detect objects in the video.

[1718] Step 3:

[1719] The device analyzes the object detection results and identifies the location and type of the detected object.

[1720] Input: Object detection results

[1721] Output: Object location and type information

[1722] Specific operation: The device analyzes the data obtained from the object detection model and identifies the location coordinates and type of each object (e.g., person, bicycle, stairs, etc.).

[1723] Step 4:

[1724] The terminal generates voice guidance based on the information of the identified object.

[1725] Input: Object location and type information

[1726] Output: Generated voice guidance

[1727] Specific operation: The device uses a speech synthesis engine such as pyttsx3 to generate voice guidance based on the information about the identified object. For example, it creates a voice message such as "A bicycle is approaching from the front left."

[1728] Step 5:

[1729] The terminal transmits the generated voice guidance to the user in real time.

[1730] Input: Generated voice prompt

[1731] Output: Audio information transmitted to the user

[1732] Specific operation: The terminal plays the generated voice guidance through the speaker to the user in real time, and the user can understand the surrounding situation by listening to the voice guidance.

[1733] This series of processes enables visually impaired users to grasp the situation around them in real time and move around safely.

[1734] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1735] "Example 1"

[1736] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system includes a means for notifying a visually impaired person of the situation in front of them by voice as they walk while wearing a smartphone around their neck, a means for recognizing the surrounding situation using the smartphone's camera and converting that information into voice, and a means for transmitting voice information to the visually impaired person in real time. Furthermore, this system includes an emotion engine that recognizes the user's emotions.

[1737] "Example 2"

[1738] The emotion engine recognizes the user's emotion from the tone of their voice, volume, speaking style, etc. For example, when a user is angry, their voice tone is high, their volume is loud, and they speak quickly. The emotion engine recognizes these characteristics and determines the user's emotion as "anger."

[1739] "Example 3"

[1740] The system adjusts the way voice information is transmitted based on the user's emotions recognized by the emotion engine. For example, if the user is angry, the system reduces the amount of voice information transmitted. On the other hand, if the user is calm, the system transmits voice information normally. This makes it possible to provide services that take the user's emotions into consideration.

[1741] The processing flow of each embodiment will be described below.

[1742] "Example 1"

[1743] Step 1: A visually impaired person hangs the smartphone around their neck and activates the system.

[1744] Step 2: The system uses the smartphone's camera to recognize the surroundings and converts that information into audio.

[1745] Step 3: The system transmits audio information to the visually impaired person in real time.

[1746] Step 4: The system activates the emotion engine to recognize the user's emotion.

[1747] "Example 2"

[1748] Step 1: The emotion engine recognizes the emotion from the user's voice tone, volume, speaking style, etc. Step 2: The emotion engine communicates the recognized emotion to the system.

[1749] "Example 3"

[1750] Step 1: The system adjusts how it delivers audio information based on the emotional information received from the emotion engine.

[1751] Step 2: The system conveys the adjust...

Claims

1. A means for notifying a situation in front of a user who is walking while hanging a portable information terminal around the user's neck by voice, comprising: a means for analyzing image data relating to the user's surroundings acquired by an imaging device of the mobile information terminal using image recognition software to identify the positions of objects and people; A means for generating text, which is an explanatory sentence in natural language, using a generation AI model based on the analysis result by the mobile information terminal; A means for converting the generated text into speech by using speech synthesis software in the mobile information terminal; a means for playing the converted audio file by the portable information terminal and outputting the audio from a speaker of the portable information terminal in real time while the user is moving; A system including:

2. The system according to claim 1 , wherein the output voice includes information on the positions of objects and people in front of the user, and is information that specifically indicates the situation around the user.

3. means for acquiring the voice of the user and analyzing data of the voice to recognize an emotion of the user; The method further includes a means for generating a text, which is a natural language explanatory text, using the generative AI model based on the recognized emotion and providing the text to the user. The system of claim 1 .

Images