System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring an image of surroundings by a camera mounted on a walking aid for a disabled person; means for analyzing the acquired image and specifying a position and a distance of an object; means for providing safety information to a user by voice based on the specified position and distance of the object; and means for transmitting the voice to the user in a wireless manner.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Visually impaired people often face various obstacles and dangers while traveling in their daily lives. In particular, in situations where they must instantly grasp their surroundings, such as on train platforms and at intersections, they need a safe means of providing information that does not rely on vision. However, conventional white canes alone make it difficult to understand the surroundings in detail, and they lack appropriate warnings to reduce the risk of accidents. The purpose of this invention is to solve these problems and enable visually impaired people to travel safely and with peace of mind. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for capturing surrounding images using a camera mounted on a walking aid for the disabled, a means for analyzing the captured images to identify the location and distance of objects, a means for providing audio safety information to the user based on the identified location and distance of the objects, and a means for wirelessly transmitting the audio to the user. Specifically, the camera captures surrounding images in real time, and an AI module analyzes the images to identify the location and distance of people and obstacles. The analysis results are converted into language using a natural language processing algorithm, and audio data is generated through a speech synthesis engine and transmitted to the user's earphones via Bluetooth. In this way, safety information is provided to the user in real time, supporting the mobility of the visually impaired.

[0006] "Mobility aids" are devices used by people with visual or physical disabilities to move around safely.

[0007] A "camera" is a device that captures video or images and outputs the data in digital format.

[0008] "Video" refers to continuous image data captured using a device such as a camera.

[0009] "Analysis" is the process of breaking down acquired data (e.g., video) and extracting specific information from it.

[0010] "Object" refers to anything that appears in the image captured by the camera, such as a person or an obstacle.

[0011] "Position" is information that indicates a specific location in space where an object exists.

[0012] "Distance" is information indicating the physical distance between the object and the camera.

[0013] "User" refers to a visually impaired person who uses this system.

[0014] "Speech" refers to data that has been converted into a form that can be heard by the ear.

[0015] A "wireless system" is a method of transmitting data using radio waves or other wireless communication means without using physical cables.

[0016] A "natural language processing algorithm" is a computational method that allows a computer to understand and generate human language.

[0017] A "speech synthesis engine" is a software or hardware system for converting text data into speech data.

[0018] "Bluetooth" is a short-range wireless communication technology that allows data to be sent and received wirelessly between devices. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, a Bluetooth module, and a Bluetooth earphone.

[0041] System configuration

[0042] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to a built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles, and sends the analysis results to a natural language processing module, which generates an appropriate verbal message. The speech synthesis engine then converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[0043] Program processing flow

[0044] Video capture and pre-processing

[0045] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[0046] Video data analysis

[0047] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[0048] Verbalizing information

[0049] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0050] Generating and transmitting audio data

[0051] The device passes the linguistic information from the natural language processing module to the speech synthesis engine to generate voice data, which is then transmitted to the user's earphones via the Bluetooth module.

[0052] User Notification

[0053] The user receives real-time audio information via a Bluetooth earphone, with specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[0054] Specific examples

[0055] When walking on the station platform

[0056] Consider a case where a user is walking on a train platform. At this time, the device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters ahead," or "There is a person three meters ahead." This information is converted into voice data and communicated to the user via Bluetooth earphones.

[0057] When crossing an intersection

[0058] When a user approaches an intersection, the device's camera captures the traffic light, and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones.

[0059] This system allows visually impaired people to know the situation around them in real time, enabling them to move around safely.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The device (a camera mounted on a white cane) captures images of the surroundings. The camera is designed to continuously collect image data and process it in real time.

[0063] Step 2:

[0064] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[0065] Step 3:

[0066] The device provides pre-processed video data to the AI ​​module, which then inputs the prepared data and begins object detection, specifically locating people and obstacles and measuring their distance.

[0067] Step 4:

[0068] The device recognizes specific objects such as traffic lights. The AI ​​module identifies the traffic light's status (red or green) and acquires that information. The location and status of the identified object are output as the analysis results.

[0069] Step 5:

[0070] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to be conveyed to the user.

[0071] Step 6:

[0072] The device passes the linguistic message to the speech synthesis engine, which generates voice data, which is immediately stored in temporary memory and moves on to the next process.

[0073] Step 7:

[0074] The device uses the Bluetooth module to send the generated audio data to the user's Bluetooth earphone. After confirming that the Bluetooth connection is established, the audio data is sent.

[0075] Step 8:

[0076] The user receives real-time voice information through a Bluetooth earphone. For example, the user is told, "There is a person 3 meters ahead," or "The end of the platform is 2 meters ahead." Based on this voice information, the user can move safely.

[0077] Through this series of processes, visually impaired people are notified of their surroundings via audio in real time, enabling them to move around safely.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] For visually impaired people to move around safely in their daily lives, it is essential that they be able to properly understand their surroundings. However, current walking aids do not provide enough information to achieve this, making it difficult to detect obstacles in real time or provide detailed information. The purpose of this invention is to solve these problems and enable visually impaired people to move around more safely and with peace of mind.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes means for capturing surrounding images using a camera mounted on a portable measuring device, means for preprocessing the captured images to remove noise and adjust resolution, means for analyzing the preprocessed images to identify the position and distance of an object, means for converting the analysis results of the identified object's position and distance into natural language, means for synthesizing the converted natural language information into voice data, and means for wirelessly transmitting the generated voice data to a user, thereby enabling visually impaired people to accurately grasp their surroundings in real time and move safely.

[0083] A "measuring device" is a device carried by a user for acquiring surrounding environmental data.

[0084] The "camera" is a device that is mounted on the measuring device and captures images of the surrounding area in real time.

[0085] "Preprocessing" refers to performing processes such as noise removal and resolution adjustment on the video data acquired by the imaging device.

[0086] "Analysis" refers to the application of calculations and algorithms to use pre-processed video data to determine information such as object location and distance.

[0087] "Conversion to natural language" refers to the process of converting the information obtained from the analysis results into words that are easy for humans to understand.

[0088] "Synthesis into voice data" is a process of generating voice from information converted into natural language.

[0089] A "wireless method" is a means of sending and receiving data using wireless communication technologies such as Bluetooth and Wi-Fi.

[0090] "User" refers to a visually impaired person who uses this system to walk safely.

[0091] "Safety information" refers to the surrounding situation, the location and distance of obstacles, and other related information necessary for the user to move safely.

[0092] The following describes a specific embodiment of the present invention. This system is a walking aid that enables the visually impaired to move around safely and with peace of mind. The system includes a camera mounted on a portable measuring device, hardware and software for preprocessing the images, a built-in AI module for object detection and analysis, a natural language processing module, a speech synthesis engine, a wireless communication module, and wireless earphones worn by the user.

[0093] Specifically, the portable measurement device is equipped with a small camera that captures images of the surroundings in real time using OpenCV. The captured images are preprocessed using image processing techniques such as Gaussian filtering. The preprocessed image data is then sent to the built-in AI module using AI frameworks such as TensorFlow and YOLO.

[0094] The AI ​​module identifies the location and distance of objects (people, obstacles, traffic lights, etc.) from the video data. For example, it uses the YOLO (You Only Look Once) algorithm to detect objects in the video in real time and calculate their location coordinates and distance. The results of this analysis are passed to a natural language processing module (e.g., spaCy or NLTK), which generates specific language messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0095] The generated linguistic message is then converted into voice data by a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly) and sent to wireless earphones via a Bluetooth module. By receiving this voice information in real time through the Bluetooth earphones, users can accurately grasp their surroundings and move safely.

[0096] Specific examples

[0097] When walking on the station platform

[0098] As a user walks along a station platform, the device's camera captures video of the surrounding area. After the video is preprocessed, an AI module detects when the edge of the platform is approaching or if a person is standing ahead. The analysis results are verbalized as "The edge of the platform is two meters away" or "There is a person three meters ahead." This verbalized information is then converted into audio data and transmitted to the user via Bluetooth earphones.

[0099] When crossing an intersection

[0100] When a user approaches an intersection, the device's camera captures the traffic light. After the video is preprocessed, the AI ​​module analyzes the traffic light status and determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and transmitted to the user's earphones.

[0101] Prompt Sentence Examples

[0102] By providing the following prompt to the generative AI model, it can generate a natural language message:

[0103] Example prompt 1:

[0104] Generate user-friendly messages based on the obstacle detection results.

[0105] Input: Distance to obstacle is 3 meters, object is a person

[0106] Output: There is a person 3 meters ahead

[0107] Example prompt 2:

[0108] Generate user-friendly messages based on the traffic light status.

[0109] Input: Traffic light status is red

[0110] Output: The light is red. Please wait until the light changes.

[0111] In this way, the system is designed to enable visually impaired people to accurately grasp their surroundings in real time.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The terminal captures the surroundings in real time using a camera mounted on a portable measuring device. The camera captures images at tens of frames per second and sends the image data to the next processing step.

[0115] Input: Surrounding image

[0116] Output: Raw captured video data

[0117] Step 2:

[0118] The device performs preprocessing on the captured video data. This preprocessing involves using a video processing library such as OpenCV to remove noise and adjust the resolution. For example, a Gaussian filter is applied to reduce image noise and convert the video to the optimal resolution.

[0119] Input: Raw captured video data

[0120] Output: Pre-processed video data

[0121] Step 3:

[0122] The device sends the preprocessed video data to the AI ​​module, which uses object detection algorithms such as TensorFlow and YOLO to identify objects in the video and determine their location and distance. For example, the YOLO algorithm can be used to generate bounding boxes for people and obstacles in the video and calculate distance from their center coordinates and size.

[0123] Input: Preprocessed video data

[0124] Output: Object position and distance information

[0125] Step 4:

[0126] The device passes the object position and distance information received from the AI ​​module to a natural language processing module. The natural language processing module (e.g., spaCy) is used to convert the analysis results into natural language. For example, the information "Person: 3 meters" is converted into a message such as "There is a person 3 meters ahead."

[0127] Input: Object position and distance information

[0128] Output: A natural language message

[0129] Step 5:

[0130] The device passes the natural language message generated by the natural language processing module to a speech synthesis engine. The speech synthesis engine (e.g., Google Text-to-Speech) is used to synthesize the text message into voice data. In this process, the text is converted into an audio file (e.g., WAV format).

[0131] Input: A natural language message

[0132] Output: Audio data

[0133] Step 6:

[0134] The terminal transmits the generated audio data to the wireless earphone via the Bluetooth module. The audio data is transferred to the earphone via Bluetooth communication and notified to the user in real time.

[0135] Input: Audio data

[0136] Output: Audio data sent to Bluetooth earphones

[0137] Step 7:

[0138] The user receives voice information transmitted through a Bluetooth earphone, and can confirm specific voice messages in real time, such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[0139] Input: Audio data sent to Bluetooth earphones

[0140] Output: User receives audio information

[0141] Through these steps, visually impaired people can accurately grasp their surroundings in real time and move around safely.

[0142] (Application example 1)

[0143] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0144] Traditionally, visually impaired people have relied primarily on white canes and guide dogs to understand their surroundings when traveling. However, these methods have limited ability to detect specific dangers, making it difficult to obtain real-time information about new obstacles and moving objects and take immediate action. Particularly at intersections and in complex road environments, there has been a lack of information provided to enable safe travel.

[0145] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0146] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for acquiring surrounding situation data of the mobile object and generating safety information based on the analysis results, thereby enabling visually impaired people to grasp the surrounding situation in real time and move around safely.

[0147] "Mobility aids for the disabled" are aids used by visually impaired people to move around safely.

[0148] A "camera" is a photographing device for capturing images of the surroundings.

[0149] The "means for acquiring images" refers to a means for collecting image data of the surroundings in real time using a camera.

[0150] The "means for analyzing video" is a means for analyzing acquired video data and identifying the position and distance of an object.

[0151] The "means for identifying the position and distance of an object" is a means for calculating the spatial position and distance of a specific object based on the results of video analysis.

[0152] The "means for providing safety information by voice" is a means for notifying the user by voice of information generated based on the analysis results.

[0153] "Means for wirelessly communicating to a user" refers to means for transmitting audio information to a user's device using wireless communication technology such as Bluetooth or Wi-Fi.

[0154] "Means for acquiring surrounding situation data of a moving body" refers to means for sensing the surrounding environment of an autonomous vehicle or the like and collecting that data.

[0155] The "means for generating safety information based on the analysis results" is a means for analyzing data acquired from a mobile object and generating safety information that is useful to the user.

[0156] The system embodying this invention is used as an aid for visually impaired people to move around safely. Specifically, the system includes a camera mounted on the walking aid, an AI module for video analysis, a module for natural language processing, a speech synthesis engine, a Bluetooth module, and Bluetooth earphones.

[0157] The device's built-in camera captures real-time video of the surrounding area. This video data is then sent to an AI module after pre-processing, such as adjusting the resolution and removing noise, and analyzed to determine the location and distance of objects. Specifically, it identifies people, obstacles, traffic light status, and other objects in the video.

[0158] The information analyzed by the AI ​​module is passed to the natural language processing module, which generates a verbal message to notify the user, such as "There is a person 3 meters ahead" or "The light is red, do not cross."

[0159] The generated linguistic message is sent to a speech synthesis engine and converted into voice data, which is then transmitted to a Bluetooth earphone via a Bluetooth module, providing real-time notification to the user.

[0160] Furthermore, this system also has the ability to acquire data on the surrounding environment of the moving object and generate safety information based on that data. For example, the system can analyze the surrounding environment data acquired by an autonomous vehicle and notify the user, "A vehicle is approaching, please be careful."

[0161] The hardware used includes a camera, an AI module, a natural language processing module, a speech synthesis engine, and a Bluetooth module, while the software required is an ObjectDetectionModel (object detection algorithm), an NLPModel (natural language generation algorithm), and a TTSEngine (speech synthesis engine).

[0162] Specific examples

[0163] For example, when a user is walking on a train platform, the AI ​​module detects the edge of the platform or people ahead from the images captured by the device's camera, and generates messages based on that information, such as "The edge of the platform is two meters ahead" or "There is a person three meters ahead." This allows the user to move safely. In addition, the camera recognizes the status of traffic lights at intersections, and when the light is red, a message is displayed saying, "The light is red. Please wait until the light changes." This system allows visually impaired people to understand their surroundings in real time and move safely.

[0164] Prompt Sentence Examples

[0165] It visually monitors the status of traffic lights and the surrounding environment and provides real-time voice guidance to help drivers cross the crosswalk safely. Notifications include, "The light is green. Please cross safely." For example, a vehicle camera captures the traffic light, and an AI model identifies that the light is green. This information is sent to a natural language processing module, which generates a voice message saying, "The light is green," and sends it to a Bluetooth earphone.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] The device uses a camera to capture images of the surroundings. The input is real-time image data, and the output is the raw image data.

[0169] Step 2:

[0170] Preprocesses the video data acquired by the device. Specifically, it removes noise and adjusts the resolution. The input is raw video data, and the output is preprocessed video data.

[0171] Step 3:

[0172] The preprocessed video data is sent to an AI module to detect objects and determine their location and distance. The input is the preprocessed video data, and the output is object location and distance information. Specifically, the AI ​​model recognizes people and obstacles in the video and calculates their respective positions and distances.

[0173] Step 4:

[0174] The device sends the object's position and distance information to the natural language processing module, which then generates a linguistic message to notify the user. The input is the object's position and distance information, and the output is a natural language message. Specifically, messages such as "There is a person 3 meters ahead" or "The light is red, do not cross" are generated.

[0175] Step 5:

[0176] The generated natural language message is passed to a speech synthesis engine and converted into audio data. The input is a natural language message, and the output is audio data. Specifically, the speech synthesis engine converts the text into an audio file.

[0177] Step 6:

[0178] The terminal transmits audio data to the user's Bluetooth earphone via the Bluetooth module. The input is audio data, and the output is audio notification to the Bluetooth earphone. Specifically, the Bluetooth module transmits audio data wirelessly to the earphone.

[0179] Step 7:

[0180] The user receives real-time audio information through a Bluetooth earphone. The input is the audio data sent to the Bluetooth earphone, and the output is audio notifications to the user. Specifically, the information conveyed to the user is "There is a person 3 meters ahead" or "The traffic light is red, do not cross."

[0181] Step 8:

[0182] The system acquires data on the surrounding situation of a moving object and generates safety information based on the analysis results. The input is the surrounding situation data from the moving object, and the output is safety information based on the analysis results. Specifically, it analyzes data acquired from the moving object's sensors and generates messages such as "A vehicle is approaching, please be careful."

[0183] Step 9:

[0184] The generated safety information is notified to the user as voice data. The input is a natural language message based on the analysis results, and the output is a safety information notification as voice data. Specifically, a voice synthesis engine converts the text into voice data and sends it to earphones via Bluetooth.

[0185] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0186] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[0187] System configuration

[0188] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to the built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles and sends the analysis results to the natural language processing module, which generates an appropriate verbal message. The emotion engine then recognizes the user's emotions and appropriately adjusts the voice information provided. Finally, the speech synthesis engine converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[0189] Program processing flow

[0190] Video capture and pre-processing

[0191] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[0192] Video data analysis

[0193] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[0194] Verbalizing information

[0195] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0196] User Emotion Recognition

[0197] Along with verbalized information, the device recognizes the user's emotional state through an emotion engine, which uses voice tone, speed, skin conductance, and other physiological data to determine the user's emotional state (e.g., stress level or relief).

[0198] Adjusting audio information

[0199] The device adjusts the voice information as needed based on the output of the emotion engine. If the user is under high stress, the system will provide information in a calm and reassuring tone. The frequency and intensity of voice warnings will also be adjusted according to the stress level.

[0200] Generating and transmitting audio data

[0201] The device passes the adjusted language message to a speech synthesis engine to generate voice data, which is then sent to the user's earphones via a Bluetooth module.

[0202] User Notification

[0203] The user receives real-time voice information through a Bluetooth earphone. For example, specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" are conveyed to the user. Based on this voice information, the user can move safely.

[0204] Specific examples

[0205] When walking on the station platform

[0206] Consider a case where a user is walking on a train platform. The device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing the user to travel with greater peace of mind.

[0207] When crossing an intersection

[0208] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[0209] This system allows visually impaired people to know what is happening around them in real time, enabling them to move around safely and with peace of mind.

[0210] The processing flow will be explained below.

[0211] Step 1:

[0212] The device (a camera mounted on a white cane) captures the surroundings. The camera continuously collects video data and passes it on to the next process in real time.

[0213] Step 2:

[0214] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[0215] Step 3:

[0216] The device provides preprocessed video data to the AI ​​module, which then uses the input data to run an object detection algorithm to identify the location and distance of people and obstacles. For example, it might detect a person standing three meters ahead and output that information.

[0217] Step 4:

[0218] The AI ​​module on the device identifies the state of a specific object (e.g., a traffic light). It detects that the traffic light is red and obtains that information.

[0219] Step 5:

[0220] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to convey to the user, such as "There is a person 3 meters ahead" or "The traffic light is red, please wait."

[0221] Step 6:

[0222] The terminal temporarily stores the generated linguistic message, which is then sent to the emotion engine.

[0223] Step 7:

[0224] The device operates an emotion engine to recognize the user's emotions. The emotion engine analyzes physiological data such as the user's voice tone and skin conductance to determine the user's emotional state (e.g., high stress state).

[0225] Step 8:

[0226] The device adjusts the verbal messages as needed based on the user's recognized emotional state, for example, making the tone of the voice information gentler and adjusting the frequency of the messages if the user is in a high-stress state.

[0227] Step 9:

[0228] The device passes the adjusted language message to a speech synthesis engine to generate speech data, which is stored in temporary memory.

[0229] Step 10:

[0230] The device uses the Bluetooth module to transmit the generated audio data to the user's Bluetooth earphone, and transmits the data only after verifying that the Bluetooth connection is established.

[0231] Step 11:

[0232] The user receives real-time audio information through a Bluetooth earphone, such as "There is a person 3 meters ahead," "The traffic light is red, please wait," or "It's okay, please proceed calmly."

[0233] This series of processes allows visually impaired people to know the situation around them in real time, allowing them to move around safely and with peace of mind.

[0234] Example 2

[0235] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] For people with disabilities, especially those with visual impairments, to walk safely, it is important to quickly and accurately grasp information about their surroundings. However, current walking aids lack a system that can efficiently analyze video data acquired by a camera and provide the user with appropriate audio information. They also lack the ability to adjust the audio according to the user's psychological state and stress level. This makes it difficult for users to move around with a sufficient sense of security.

[0237] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring surrounding video using a camera mounted on a walking aid for disabled persons; means for preprocessing the acquired video; means for analyzing the preprocessed video and identifying the position and distance of an object; means for converting information on the position and distance of the identified object into natural language; means for recognizing the user's emotional state based on the converted natural language information; means for adjusting voice information based on the recognized emotional state; means for converting the adjusted voice information into voice data; and means for wirelessly transmitting the voice data to the user. This allows the user to grasp information about the surrounding environment in real time and receive voice guidance according to the user's psychological state at that time, allowing for safer and more secure travel.

[0238] A "walking aid for the disabled" is a tool used by disabled people, such as those with visual impairments, to assist them in walking.

[0239] A "camera" is a device that captures images and acquires them as digital data.

[0240] "Video" is data that represents a series of still images linked together over time to create a motion image.

[0241] "Preprocessing" refers to the process of performing initial processing such as noise removal and resolution adjustment on the acquired raw data.

[0242] An "AI module" is a software or hardware unit that uses artificial intelligence technology to analyze data and make inferences.

[0243] The "position and distance of an object" refers to the spatial coordinates of an object detected by the camera and the linear distance from the camera to the object.

[0244] "Natural language processing" refers to converting analyzed data into natural language that humans can understand.

[0245] A "natural language processing module" is a software module that performs text analysis, generation, etc.

[0246] "Emotional state" refers to the user's psychological and physiological state, including stress levels, feelings of security, etc.

[0247] An "emotion engine" is software that has the ability to recognize a user's emotional state and adjust the system's response based on that.

[0248] "Adjusting audio information" refers to changing the tone, speed, and frequency of audio depending on the user's emotional state.

[0249] "Speech data" refers to a speech signal generated based on text information in a natural language.

[0250] A "speech synthesis engine" is a software or hardware unit that converts text data into speech.

[0251] A "wireless system" is a method of transmitting and receiving data using wireless communication technology.

[0252] A "Bluetooth earphone" is a device worn in the ear that receives audio data wirelessly using Bluetooth technology.

[0253] The system of the present invention includes a camera mounted on a walking aid for the disabled, an artificial intelligence (AI) module for video analysis, a module for natural language processing, a speech synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[0254] The device (a camera mounted on a white cane) captures the surroundings in real time. This video data is first preprocessed to remove noise and adjust the resolution. Specifically, a Gaussian filter is applied using OpenCV to remove noise, and the resolution is adjusted to 640x480 using a resize method.

[0255] After preprocessing, the video data is immediately sent to the device's AI module. The AI ​​module uses deep learning libraries such as TensorFlow to recognize objects in the video. Specifically, it uses SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) to detect people and obstacles and determine the location and distance of each object. The AI ​​module can also identify the color of traffic lights (red or green).

[0256] The analysis results are sent to the device's natural language processing (NLP) module, which translates the information into natural language. This module uses libraries such as SpaCy to generate messages like "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0257] The device sends the generated linguistic message to the emotion engine, which collects physiological data from skin conductance sensors and heart rate sensors to determine the user's stress level and sense of security. For example, high skin conductance is recognized as a state of high stress. This emotional state information is used to adjust the voice information.

[0258] The device adjusts the tone and intensity of messages generated by the natural language processing module based on the output of the emotion engine. Specifically, if the user is under high stress, the tone of the voice will be adjusted to a calmer tone and the frequency of messages will be reduced.

[0259] The adjusted language message is passed to a speech synthesis engine, which uses APIs such as Google Text-to-Speech to convert the text data into speech data, which is then sent via a Bluetooth module to the user's Bluetooth earphones.

[0260] In this way, users can receive real-time voice information through Bluetooth earphones. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" can be conveyed to the user as a voice message, allowing the user to move safely.

[0261] Specific examples

[0262] When walking on the station platform

[0263] As a user walks along a station platform, the device's camera captures images of the surrounding area, and the AI ​​module identifies when the edge of the platform is approaching or when a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing users to travel with greater peace of mind.

[0264] When crossing an intersection

[0265] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[0266] Example prompt sentence:

[0267] 1. Describe a scenario in which a user is walking along a train platform, and then describe how the system captures information about their surroundings and communicates it to the user.

[0268] 2. Describe a situation where a user is about to cross an intersection, and specify how the system captures the status of the traffic lights and notifies the user.

[0269] In this way, visually impaired people can learn about their surroundings in real time through the system, enabling them to move around safely and with peace of mind.

[0270] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0271] Specific processing flow of the system

[0272] Step 1:

[0273] The device (a camera mounted on a white cane) captures the surroundings in real time. The input is the surroundings video data, and the output is the captured raw video data. For example, the camera captures video at a rate of 30 frames per second with a resolution of 640x480 pixels. This video data is stored in temporary memory and goes to the next pre-processing step.

[0274] Step 2:

[0275] We perform preprocessing on raw video data acquired by the device camera. The input is raw video data, and the output is preprocessed video data. Specifically, we use OpenCV to remove noise using a Gaussian filter and apply it with the cv2.GaussianBlur function. We also use the cv2.resize function to convert it to 640x480 pixels to adjust the resolution.

[0276] Step 3:

[0277] Preprocessed video data is input into the device's AI module to identify the location and distance of objects. The input is preprocessed video data, and the output is object information (location and distance). Specifically, object detection is performed using TensorFlow's SSD or YOLO model, and the location and distance of each object is calculated. For example, the yolov3.detect method is used to identify people and obstacles and obtain their coordinates and distance.

[0278] Step 4:

[0279] Object information is input into the device's natural language processing module, and a linguistic message is generated. The input is object information, and the output is a natural language message. For example, SpaCy can be used to generate linguistic messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead." In this case, grammatically correct sentences are constructed based on the object's position and distance.

[0280] Step 5:

[0281] The device's emotion engine collects physiological data to recognize the user's emotional state. The input is the user's physiological data (skin conductance, heart rate, etc.), and the output is the user's emotional state (stress level and relief). Specifically, it receives data from the skin conductance sensor in real time and analyzes it to determine the stress level. For example, a high skin conductance value is recognized as a high stress state.

[0282] Step 6:

[0283] The device adjusts the voice information based on the output of the emotion engine and the message from the natural language processing module. The input is the emotional state and the natural language message, and the output is the adjusted voice message. Specifically, the tone and speed of the voice are adjusted, and information is provided in a calmer tone to users in a high-stress state. For example, the tone can be set lower using the voice.adjust_tone method.

[0284] Step 7:

[0285] The device's speech synthesis engine inputs the adjusted language message and generates voice data. The input is the adjusted language message, and the output is voice data. Specifically, the Google Text-to-Speech API is used to convert text to voice. For example, the gTTS(text) method of the gTTS library is used to generate the voice file.

[0286] Step 8:

[0287] The device sends the generated audio data to the user's earphone via the Bluetooth module. The input is the audio data, and the output is the audio sent to the user's earphone. Specifically, the audio data is sent using the send method of the Bluetooth module. For example, the bluetooth.send_audio(data) method is used.

[0288] Step 9:

[0289] The user receives real-time audio information from the system via a Bluetooth earphone. The input is the audio transmitted to the Bluetooth earphone, and the output is the audio information recognized by the user. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" is conveyed by voice. This allows the user to move safely.

[0290] (Application example 2)

[0291] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] When visually impaired people move around public spaces, the many obstacles and dangers they encounter make it difficult for them to navigate safely. Furthermore, existing walking aids provide information without taking into account the user's emotional state, which can sometimes prevent them from reducing stress and anxiety. Furthermore, even in self-driving vehicles, visually impaired people have limited means of understanding their surroundings, reducing safety. There is a need for a system that can solve these issues and enable users to travel safely and with peace of mind.

[0293] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0294] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for recognizing the emotional state of the user and adjusting the audio information. This enables visually impaired people to move around safely and with peace of mind by receiving safety information in real time and providing appropriate audio information according to their emotional state.

[0295] A "walking aid for the disabled" is an auxiliary tool used by visually impaired people to move around safely, and is a device that has the function of detecting obstacles and surrounding conditions.

[0296] A "camera" is a photographing device that captures images of the surroundings and provides image data in real time.

[0297] "Video analysis" is a process that identifies information such as the position and distance of an object based on acquired video data, and typically uses AI technology.

[0298] "Determining the position and distance of an object" means measuring and understanding the position of surrounding objects and the distance to them through video analysis.

[0299] "Safety information" is information about the surrounding situation provided to the user, including the position and distance of obstacles, the status of traffic lights, and the like.

[0300] "Audio provision" is a means of transmitting analyzed information to the user by voice, allowing visually impaired people to receive information aurally.

[0301] "Wireless" refers to a method of transmitting data using wireless communication, such as transmitting audio to earphones using technology such as Bluetooth.

[0302] "Emotional state recognition" refers to monitoring a user's psychological and physiological state to identify emotions such as stress or tension.

[0303] "Adjusting voice information" refers to changing the tone and content of a voice message based on the user's emotional state so that the user receives appropriate information.

[0304] A "system" is a collection of devices and programs that combine the above elements and function to achieve a specific purpose.

[0305] This invention applies a system installed in a walking aid for the disabled to an autonomous vehicle, enabling visually impaired people to travel safely and with peace of mind inside the vehicle. Specific embodiments for carrying out this invention are described below.

[0306] System configuration

[0307] The autonomous vehicle is equipped with a camera to help visually impaired people understand their surroundings, a video analysis AI, a natural language processing module, a speech synthesis engine, an emotion engine, and Bluetooth earphones.

[0308] Hardware and Software

[0309] Camera: The onboard camera captures real-time footage of the surroundings.

[0310] Video analysis AI: Uses TensorFlow and OpenCV to analyze captured video data and identify the position and distance of objects.

[0311] Natural Language Processing module: Uses NLTK and spaCy to generate appropriate messages based on the analysis results.

[0312] Speech synthesis engine: Uses the Google Text-to-Speech API and Amazon Polly to convert text messages into audio data.

[0313] Emotion Engine: Uses the Affectiva SDK to recognize the user's emotional state and adjusts the tone and content of the voice based on that information.

[0314] Bluetooth module and earphones: A commercially available Bluetooth module is used to wirelessly transmit audio data to the user's earphones.

[0315] Processing flow

[0316] The server captures real-time video data captured by the camera and pre-processes the data. The pre-processed video data is analyzed using TensorFlow and OpenCV to determine the location and distance of objects. The analysis results are sent to a natural language processing module using NLTK and spaCy, which generates appropriate messages to be presented to the user.

[0317] The generated messages are adjusted by an emotion engine (Affectiva SDK) based on the user's emotional state, for example, if the user is in a high stress state, the tone of the voice will be changed to a calmer one.

[0318] Finally, a speech synthesis engine (Google Text-to-Speech API or Amazon Polly) is used to convert the text message into audio data, which is then sent to the user's earphones via a Bluetooth module.

[0319] Specific examples

[0320] Example of approaching an intersection

[0321] The camera captures the intersection.

[0322] Video analysis AI identifies traffic light conditions and approaching pedestrians.

[0323] The natural language processing module generates messages such as "The traffic light is red" and "There is a pedestrian ahead."

[0324] The emotion engine recognizes when the user is in a high stress state.

[0325] Generates a tone-adjusted message: "Please wait calmly until the traffic light changes."

[0326] The generated audio data is transmitted to the user through a Bluetooth earphone.

[0327] Prompt Sentence Examples

[0328] "Identify the traffic light status from the video of the intersection captured by the in-car camera, and if the light is red and there are pedestrians, generate a message saying, 'The light is red. There are pedestrians ahead.' Also, if the user is in a high-stress state, adjust the message to a more reassuring tone."

[0329] This invention allows visually impaired people to grasp information about their surroundings in real time even while inside an autonomous vehicle, enabling them to travel with peace of mind.

[0330] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0331] Step 1:

[0332] The camera captures the surrounding image data in real time. The scene and objects captured by the camera are used as input. The captured image data is then output.

[0333] Step 2:

[0334] The server receives the captured video data and performs pre-processing, which includes noise removal, resolution adjustment, and color correction. The raw captured video data is used as input, and the pre-processed, clean video data is obtained as output.

[0335] Step 3:

[0336] The server inputs the preprocessed video data into a video analysis AI (TensorFlow and OpenCV) for object recognition. The AI ​​identifies the location and distance of objects such as pedestrians, vehicles, and traffic lights. The preprocessed video data is used as input, and data on the location and distance of objects is obtained as output.

[0337] Step 4:

[0338] The server inputs the results of the video analysis AI into a natural language processing module (NLTK and spaCy), which converts the analysis results into a text message. For example, messages such as "There is a pedestrian ahead" or "The traffic light is red" are generated. Data on the object's position and distance is used as input, and a text message in natural language is obtained as output.

[0339] Step 5:

[0340] The server inputs the generated text message into the emotion engine (Affectiva SDK) to monitor the user's emotional state. The emotion engine evaluates the user's stress level, comfort level, etc., and adjusts the tone and speed of the voice message. The text message and the user's emotion data are used as input, and the adjusted text message is obtained as output.

[0341] Step 6:

[0342] The server inputs the adjusted text message into a speech synthesis engine (such as Google Text-to-Speech API or Amazon Polly) and converts it into voice data. For example, a voice message such as "The traffic light is red. There is a pedestrian ahead" is generated. The adjusted text message is used as input, and voice data is obtained as output.

[0343] Step 7:

[0344] The server sends the generated voice data to the terminal (earphone) via the Bluetooth module, allowing the user to receive information as voice in real time. Voice data is used as input, and as output, a voice message is transmitted to the user through the Bluetooth earphone.

[0345] In this way, visually impaired people will be able to receive real-time safety information and understand their surroundings even while inside an autonomous vehicle.

[0346] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0347] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0348] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0349] [Second embodiment]

[0350] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0351] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0352] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0353] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0354] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0355] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0356] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0357] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0358] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0359] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0360] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0361] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0362] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, a Bluetooth module, and a Bluetooth earphone.

[0363] System configuration

[0364] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to a built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles, and sends the analysis results to a natural language processing module, which generates an appropriate verbal message. The speech synthesis engine then converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[0365] Program processing flow

[0366] Video capture and pre-processing

[0367] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[0368] Video data analysis

[0369] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[0370] Verbalizing information

[0371] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0372] Generating and transmitting audio data

[0373] The device passes the linguistic information from the natural language processing module to the speech synthesis engine to generate voice data, which is then transmitted to the user's earphones via the Bluetooth module.

[0374] User Notification

[0375] The user receives real-time audio information via a Bluetooth earphone, with specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[0376] Specific examples

[0377] When walking on the station platform

[0378] Consider a case where a user is walking on a train platform. At this time, the device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters ahead," or "There is a person three meters ahead." This information is converted into voice data and communicated to the user via Bluetooth earphones.

[0379] When crossing an intersection

[0380] When a user approaches an intersection, the device's camera captures the traffic light, and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones.

[0381] This system allows visually impaired people to know the situation around them in real time, enabling them to move around safely.

[0382] The processing flow will be explained below.

[0383] Step 1:

[0384] The device (a camera mounted on a white cane) captures images of the surroundings. The camera is designed to continuously collect image data and process it in real time.

[0385] Step 2:

[0386] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[0387] Step 3:

[0388] The device provides pre-processed video data to the AI ​​module, which then inputs the prepared data and begins object detection, specifically locating people and obstacles and measuring their distance.

[0389] Step 4:

[0390] The device recognizes specific objects such as traffic lights. The AI ​​module identifies the traffic light's status (red or green) and acquires that information. The location and status of the identified object are output as the analysis results.

[0391] Step 5:

[0392] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to be conveyed to the user.

[0393] Step 6:

[0394] The device passes the linguistic message to the speech synthesis engine, which generates voice data, which is immediately stored in temporary memory and moves on to the next process.

[0395] Step 7:

[0396] The device uses the Bluetooth module to send the generated audio data to the user's Bluetooth earphone. After confirming that the Bluetooth connection is established, the audio data is sent.

[0397] Step 8:

[0398] The user receives real-time voice information through a Bluetooth earphone. For example, the user is told, "There is a person 3 meters ahead," or "The end of the platform is 2 meters ahead." Based on this voice information, the user can move safely.

[0399] Through this series of processes, visually impaired people are notified of their surroundings via audio in real time, enabling them to move around safely.

[0400] Example 1

[0401] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0402] For visually impaired people to move around safely in their daily lives, it is essential that they be able to properly understand their surroundings. However, current walking aids do not provide enough information to achieve this, making it difficult to detect obstacles in real time or provide detailed information. The purpose of this invention is to solve these problems and enable visually impaired people to move around more safely and with peace of mind.

[0403] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0404] In this invention, the server includes means for capturing surrounding images using a camera mounted on a portable measuring device, means for preprocessing the captured images to remove noise and adjust resolution, means for analyzing the preprocessed images to identify the position and distance of an object, means for converting the analysis results of the identified object's position and distance into natural language, means for synthesizing the converted natural language information into voice data, and means for wirelessly transmitting the generated voice data to a user, thereby enabling visually impaired people to accurately grasp their surroundings in real time and move safely.

[0405] A "measuring device" is a device carried by a user for acquiring surrounding environmental data.

[0406] The "camera" is a device that is mounted on the measuring device and captures images of the surrounding area in real time.

[0407] "Preprocessing" refers to performing processes such as noise removal and resolution adjustment on the video data acquired by the imaging device.

[0408] "Analysis" refers to the application of calculations and algorithms to use pre-processed video data to determine information such as object location and distance.

[0409] "Conversion to natural language" refers to the process of converting the information obtained from the analysis results into words that are easy for humans to understand.

[0410] "Synthesis into voice data" is a process of generating voice from information converted into natural language.

[0411] A "wireless method" is a means of sending and receiving data using wireless communication technologies such as Bluetooth and Wi-Fi.

[0412] "User" refers to a visually impaired person who uses this system to walk safely.

[0413] "Safety information" refers to the surrounding situation, the location and distance of obstacles, and other related information necessary for the user to move safely.

[0414] The following describes a specific embodiment of the present invention. This system is a walking aid that enables the visually impaired to move around safely and with peace of mind. The system includes a camera mounted on a portable measuring device, hardware and software for preprocessing the images, a built-in AI module for object detection and analysis, a natural language processing module, a speech synthesis engine, a wireless communication module, and wireless earphones worn by the user.

[0415] Specifically, the portable measurement device is equipped with a small camera that captures images of the surroundings in real time using OpenCV. The captured images are preprocessed using image processing techniques such as Gaussian filtering. The preprocessed image data is then sent to the built-in AI module using AI frameworks such as TensorFlow and YOLO.

[0416] The AI ​​module identifies the location and distance of objects (people, obstacles, traffic lights, etc.) from the video data. For example, it uses the YOLO (You Only Look Once) algorithm to detect objects in the video in real time and calculate their location coordinates and distance. The results of this analysis are passed to a natural language processing module (e.g., spaCy or NLTK), which generates specific language messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0417] The generated linguistic message is then converted into voice data by a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly) and sent to wireless earphones via a Bluetooth module. By receiving this voice information in real time through the Bluetooth earphones, users can accurately grasp their surroundings and move safely.

[0418] Specific examples

[0419] When walking on the station platform

[0420] As a user walks along a station platform, the device's camera captures video of the surrounding area. After the video is preprocessed, an AI module detects when the edge of the platform is approaching or if a person is standing ahead. The analysis results are verbalized as "The edge of the platform is two meters away" or "There is a person three meters ahead." This verbalized information is then converted into audio data and transmitted to the user via Bluetooth earphones.

[0421] When crossing an intersection

[0422] When a user approaches an intersection, the device's camera captures the traffic light. After the video is preprocessed, the AI ​​module analyzes the traffic light status and determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and transmitted to the user's earphones.

[0423] Prompt Sentence Examples

[0424] By providing the following prompt to the generative AI model, it can generate a natural language message:

[0425] Example prompt 1:

[0426] Generate user-friendly messages based on the obstacle detection results.

[0427] Input: Distance to obstacle is 3 meters, object is a person

[0428] Output: There is a person 3 meters ahead

[0429] Example prompt 2:

[0430] Generate user-friendly messages based on the traffic light status.

[0431] Input: Traffic light status is red

[0432] Output: The light is red. Please wait until the light changes.

[0433] In this way, the system is designed to enable visually impaired people to accurately grasp their surroundings in real time.

[0434] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0435] Step 1:

[0436] The terminal captures the surroundings in real time using a camera mounted on a portable measuring device. The camera captures images at tens of frames per second and sends the image data to the next processing step.

[0437] Input: Surrounding image

[0438] Output: Raw captured video data

[0439] Step 2:

[0440] The device performs preprocessing on the captured video data. This preprocessing involves using a video processing library such as OpenCV to remove noise and adjust the resolution. For example, a Gaussian filter is applied to reduce image noise and convert the video to the optimal resolution.

[0441] Input: Raw captured video data

[0442] Output: Pre-processed video data

[0443] Step 3:

[0444] The device sends the preprocessed video data to the AI ​​module, which uses object detection algorithms such as TensorFlow and YOLO to identify objects in the video and determine their location and distance. For example, the YOLO algorithm can be used to generate bounding boxes for people and obstacles in the video and calculate distance from their center coordinates and size.

[0445] Input: Preprocessed video data

[0446] Output: Object position and distance information

[0447] Step 4:

[0448] The device passes the object position and distance information received from the AI ​​module to a natural language processing module. The natural language processing module (e.g., spaCy) is used to convert the analysis results into natural language. For example, the information "Person: 3 meters" is converted into a message such as "There is a person 3 meters ahead."

[0449] Input: Object position and distance information

[0450] Output: A natural language message

[0451] Step 5:

[0452] The device passes the natural language message generated by the natural language processing module to a speech synthesis engine. The speech synthesis engine (e.g., Google Text-to-Speech) is used to synthesize the text message into voice data. In this process, the text is converted into an audio file (e.g., WAV format).

[0453] Input: A natural language message

[0454] Output: Audio data

[0455] Step 6:

[0456] The terminal transmits the generated audio data to the wireless earphone via the Bluetooth module. The audio data is transferred to the earphone via Bluetooth communication and notified to the user in real time.

[0457] Input: Audio data

[0458] Output: Audio data sent to Bluetooth earphones

[0459] Step 7:

[0460] The user receives voice information transmitted through a Bluetooth earphone, and can confirm specific voice messages in real time, such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[0461] Input: Audio data sent to Bluetooth earphones

[0462] Output: User receives audio information

[0463] Through these steps, visually impaired people can accurately grasp their surroundings in real time and move around safely.

[0464] (Application example 1)

[0465] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] Traditionally, visually impaired people have relied primarily on white canes and guide dogs to understand their surroundings when traveling. However, these methods have limited ability to detect specific dangers, making it difficult to obtain real-time information about new obstacles and moving objects and take immediate action. Particularly at intersections and in complex road environments, there has been a lack of information provided to enable safe travel.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0468] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for acquiring surrounding situation data of the mobile object and generating safety information based on the analysis results, thereby enabling visually impaired people to grasp the surrounding situation in real time and move around safely.

[0469] "Mobility aids for the disabled" are aids used by visually impaired people to move around safely.

[0470] A "camera" is a photographing device for capturing images of the surroundings.

[0471] The "means for acquiring images" refers to a means for collecting image data of the surroundings in real time using a camera.

[0472] The "means for analyzing video" is a means for analyzing acquired video data and identifying the position and distance of an object.

[0473] The "means for identifying the position and distance of an object" is a means for calculating the spatial position and distance of a specific object based on the results of video analysis.

[0474] The "means for providing safety information by voice" is a means for notifying the user by voice of information generated based on the analysis results.

[0475] "Means for wirelessly communicating to a user" refers to means for transmitting audio information to a user's device using wireless communication technology such as Bluetooth or Wi-Fi.

[0476] "Means for acquiring surrounding situation data of a moving body" refers to means for sensing the surrounding environment of an autonomous vehicle or the like and collecting that data.

[0477] The "means for generating safety information based on the analysis results" is a means for analyzing data acquired from a mobile object and generating safety information that is useful to the user.

[0478] The system embodying this invention is used as an aid for visually impaired people to move around safely. Specifically, the system includes a camera mounted on the walking aid, an AI module for video analysis, a module for natural language processing, a speech synthesis engine, a Bluetooth module, and Bluetooth earphones.

[0479] The device's built-in camera captures real-time video of the surrounding area. This video data is then sent to an AI module after pre-processing, such as adjusting the resolution and removing noise, and analyzed to determine the location and distance of objects. Specifically, it identifies people, obstacles, traffic light status, and other objects in the video.

[0480] The information analyzed by the AI ​​module is passed to the natural language processing module, which generates a verbal message to notify the user, such as "There is a person 3 meters ahead" or "The light is red, do not cross."

[0481] The generated linguistic message is sent to a speech synthesis engine and converted into voice data, which is then transmitted to a Bluetooth earphone via a Bluetooth module, providing real-time notification to the user.

[0482] Furthermore, this system also has the ability to acquire data on the surrounding environment of the moving object and generate safety information based on that data. For example, the system can analyze the surrounding environment data acquired by an autonomous vehicle and notify the user, "A vehicle is approaching, please be careful."

[0483] The hardware used includes a camera, an AI module, a natural language processing module, a speech synthesis engine, and a Bluetooth module, while the software required is an ObjectDetectionModel (object detection algorithm), an NLPModel (natural language generation algorithm), and a TTSEngine (speech synthesis engine).

[0484] Specific examples

[0485] For example, when a user is walking on a train platform, the AI ​​module detects the edge of the platform or people ahead from the images captured by the device's camera, and generates messages based on that information, such as "The edge of the platform is two meters ahead" or "There is a person three meters ahead." This allows the user to move safely. In addition, the camera recognizes the status of traffic lights at intersections, and when the light is red, a message is displayed saying, "The light is red. Please wait until the light changes." This system allows visually impaired people to understand their surroundings in real time and move safely.

[0486] Prompt Sentence Examples

[0487] It visually monitors the status of traffic lights and the surrounding environment and provides real-time voice guidance to help drivers cross the crosswalk safely. Notifications include, "The light is green. Please cross safely." For example, a vehicle camera captures the traffic light, and an AI model identifies that the light is green. This information is sent to a natural language processing module, which generates a voice message saying, "The light is green," and sends it to a Bluetooth earphone.

[0488] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0489] Step 1:

[0490] The device uses a camera to capture images of the surroundings. The input is real-time image data, and the output is the raw image data.

[0491] Step 2:

[0492] Preprocesses the video data acquired by the device. Specifically, it removes noise and adjusts the resolution. The input is raw video data, and the output is preprocessed video data.

[0493] Step 3:

[0494] The preprocessed video data is sent to an AI module to detect objects and determine their location and distance. The input is the preprocessed video data, and the output is object location and distance information. Specifically, the AI ​​model recognizes people and obstacles in the video and calculates their respective positions and distances.

[0495] Step 4:

[0496] The device sends the object's position and distance information to the natural language processing module, which then generates a linguistic message to notify the user. The input is the object's position and distance information, and the output is a natural language message. Specifically, messages such as "There is a person 3 meters ahead" or "The light is red, do not cross" are generated.

[0497] Step 5:

[0498] The generated natural language message is passed to a speech synthesis engine and converted into audio data. The input is a natural language message, and the output is audio data. Specifically, the speech synthesis engine converts the text into an audio file.

[0499] Step 6:

[0500] The terminal transmits audio data to the user's Bluetooth earphone via the Bluetooth module. The input is audio data, and the output is audio notification to the Bluetooth earphone. Specifically, the Bluetooth module transmits audio data wirelessly to the earphone.

[0501] Step 7:

[0502] The user receives real-time audio information through a Bluetooth earphone. The input is the audio data sent to the Bluetooth earphone, and the output is audio notifications to the user. Specifically, the information conveyed to the user is "There is a person 3 meters ahead" or "The traffic light is red, do not cross."

[0503] Step 8:

[0504] The system acquires data on the surrounding situation of a moving object and generates safety information based on the analysis results. The input is the surrounding situation data from the moving object, and the output is safety information based on the analysis results. Specifically, it analyzes data acquired from the moving object's sensors and generates messages such as "A vehicle is approaching, please be careful."

[0505] Step 9:

[0506] The generated safety information is notified to the user as voice data. The input is a natural language message based on the analysis results, and the output is a safety information notification as voice data. Specifically, a voice synthesis engine converts the text into voice data and sends it to earphones via Bluetooth.

[0507] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0508] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[0509] System configuration

[0510] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to the built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles and sends the analysis results to the natural language processing module, which generates an appropriate verbal message. The emotion engine then recognizes the user's emotions and appropriately adjusts the voice information provided. Finally, the speech synthesis engine converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[0511] Program processing flow

[0512] Video capture and pre-processing

[0513] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[0514] Video data analysis

[0515] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[0516] Verbalizing information

[0517] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0518] User Emotion Recognition

[0519] Along with verbalized information, the device recognizes the user's emotional state through an emotion engine, which uses voice tone, speed, skin conductance, and other physiological data to determine the user's emotional state (e.g., stress level or relief).

[0520] Adjusting audio information

[0521] The device adjusts the voice information as needed based on the output of the emotion engine. If the user is under high stress, the system will provide information in a calm and reassuring tone. The frequency and intensity of voice warnings will also be adjusted according to the stress level.

[0522] Generating and transmitting audio data

[0523] The device passes the adjusted language message to a speech synthesis engine to generate voice data, which is then sent to the user's earphones via a Bluetooth module.

[0524] User Notification

[0525] The user receives real-time voice information through a Bluetooth earphone. For example, specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" are conveyed to the user. Based on this voice information, the user can move safely.

[0526] Specific examples

[0527] When walking on the station platform

[0528] Consider a case where a user is walking on a train platform. The device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing the user to travel with greater peace of mind.

[0529] When crossing an intersection

[0530] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[0531] This system allows visually impaired people to know what is happening around them in real time, enabling them to move around safely and with peace of mind.

[0532] The processing flow will be explained below.

[0533] Step 1:

[0534] The device (a camera mounted on a white cane) captures the surroundings. The camera continuously collects video data and passes it on to the next process in real time.

[0535] Step 2:

[0536] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[0537] Step 3:

[0538] The device provides preprocessed video data to the AI ​​module, which then uses the input data to run an object detection algorithm to identify the location and distance of people and obstacles. For example, it might detect a person standing three meters ahead and output that information.

[0539] Step 4:

[0540] The AI ​​module on the device identifies the state of a specific object (e.g., a traffic light). It detects that the traffic light is red and obtains that information.

[0541] Step 5:

[0542] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to convey to the user, such as "There is a person 3 meters ahead" or "The traffic light is red, please wait."

[0543] Step 6:

[0544] The terminal temporarily stores the generated linguistic message, which is then sent to the emotion engine.

[0545] Step 7:

[0546] The device operates an emotion engine to recognize the user's emotions. The emotion engine analyzes physiological data such as the user's voice tone and skin conductance to determine the user's emotional state (e.g., high stress state).

[0547] Step 8:

[0548] The device adjusts the verbal messages as needed based on the user's recognized emotional state, for example, making the tone of the voice information gentler and adjusting the frequency of the messages if the user is in a high-stress state.

[0549] Step 9:

[0550] The device passes the adjusted language message to a speech synthesis engine to generate speech data, which is stored in temporary memory.

[0551] Step 10:

[0552] The device uses the Bluetooth module to transmit the generated audio data to the user's Bluetooth earphone, and transmits the data only after verifying that the Bluetooth connection is established.

[0553] Step 11:

[0554] The user receives real-time audio information through a Bluetooth earphone, such as "There is a person 3 meters ahead," "The traffic light is red, please wait," or "It's okay, please proceed calmly."

[0555] This series of processes allows visually impaired people to know the situation around them in real time, allowing them to move around safely and with peace of mind.

[0556] Example 2

[0557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] For people with disabilities, especially those with visual impairments, to walk safely, it is important to quickly and accurately grasp information about their surroundings. However, current walking aids lack a system that can efficiently analyze video data acquired by a camera and provide the user with appropriate audio information. They also lack the ability to adjust the audio according to the user's psychological state and stress level. This makes it difficult for users to move around with a sufficient sense of security.

[0559] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring surrounding video using a camera mounted on a walking aid for disabled persons; means for preprocessing the acquired video; means for analyzing the preprocessed video and identifying the position and distance of an object; means for converting information on the position and distance of the identified object into natural language; means for recognizing the user's emotional state based on the converted natural language information; means for adjusting voice information based on the recognized emotional state; means for converting the adjusted voice information into voice data; and means for wirelessly transmitting the voice data to the user. This allows the user to grasp information about the surrounding environment in real time and receive voice guidance according to the user's psychological state at that time, allowing for safer and more secure travel.

[0560] A "walking aid for the disabled" is a tool used by disabled people, such as those with visual impairments, to assist them in walking.

[0561] A "camera" is a device that captures images and acquires them as digital data.

[0562] "Video" is data that represents a series of still images linked together over time to create a motion image.

[0563] "Preprocessing" refers to the process of performing initial processing such as noise removal and resolution adjustment on the acquired raw data.

[0564] An "AI module" is a software or hardware unit that uses artificial intelligence technology to analyze data and make inferences.

[0565] The "position and distance of an object" refers to the spatial coordinates of an object detected by the camera and the linear distance from the camera to the object.

[0566] "Natural language processing" refers to converting analyzed data into natural language that humans can understand.

[0567] A "natural language processing module" is a software module that performs text analysis, generation, etc.

[0568] "Emotional state" refers to the user's psychological and physiological state, including stress levels, feelings of security, etc.

[0569] An "emotion engine" is software that has the ability to recognize a user's emotional state and adjust the system's response based on that.

[0570] "Adjusting audio information" refers to changing the tone, speed, and frequency of audio depending on the user's emotional state.

[0571] "Speech data" refers to a speech signal generated based on text information in a natural language.

[0572] A "speech synthesis engine" is a software or hardware unit that converts text data into speech.

[0573] A "wireless system" is a method of transmitting and receiving data using wireless communication technology.

[0574] A "Bluetooth earphone" is a device worn in the ear that receives audio data wirelessly using Bluetooth technology.

[0575] The system of the present invention includes a camera mounted on a walking aid for the disabled, an artificial intelligence (AI) module for video analysis, a module for natural language processing, a speech synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[0576] The device (a camera mounted on a white cane) captures the surroundings in real time. This video data is first preprocessed to remove noise and adjust the resolution. Specifically, a Gaussian filter is applied using OpenCV to remove noise, and the resolution is adjusted to 640x480 using a resize method.

[0577] After preprocessing, the video data is immediately sent to the device's AI module. The AI ​​module uses deep learning libraries such as TensorFlow to recognize objects in the video. Specifically, it uses SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) to detect people and obstacles and determine the location and distance of each object. The AI ​​module can also identify the color of traffic lights (red or green).

[0578] The analysis results are sent to the device's natural language processing (NLP) module, which translates the information into natural language. This module uses libraries such as SpaCy to generate messages like "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0579] The device sends the generated linguistic message to the emotion engine, which collects physiological data from skin conductance sensors and heart rate sensors to determine the user's stress level and sense of security. For example, high skin conductance is recognized as a state of high stress. This emotional state information is used to adjust the voice information.

[0580] The device adjusts the tone and intensity of messages generated by the natural language processing module based on the output of the emotion engine. Specifically, if the user is under high stress, the tone of the voice will be adjusted to a calmer tone and the frequency of messages will be reduced.

[0581] The adjusted language message is passed to a speech synthesis engine, which uses APIs such as Google Text-to-Speech to convert the text data into speech data, which is then sent via a Bluetooth module to the user's Bluetooth earphones.

[0582] In this way, users can receive real-time voice information through Bluetooth earphones. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" can be conveyed to the user as a voice message, allowing the user to move safely.

[0583] Specific examples

[0584] When walking on the station platform

[0585] As a user walks along a station platform, the device's camera captures images of the surrounding area, and the AI ​​module identifies when the edge of the platform is approaching or when a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing users to travel with greater peace of mind.

[0586] When crossing an intersection

[0587] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[0588] Example prompt sentence:

[0589] 1. Describe a scenario in which a user is walking along a train platform, and then describe how the system captures information about their surroundings and communicates it to the user.

[0590] 2. Describe a situation where a user is about to cross an intersection, and specify how the system captures the status of the traffic lights and notifies the user.

[0591] In this way, visually impaired people can learn about their surroundings in real time through the system, enabling them to move around safely and with peace of mind.

[0592] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0593] Specific processing flow of the system

[0594] Step 1:

[0595] The device (a camera mounted on a white cane) captures the surroundings in real time. The input is the surroundings video data, and the output is the captured raw video data. For example, the camera captures video at a rate of 30 frames per second with a resolution of 640x480 pixels. This video data is stored in temporary memory and goes to the next pre-processing step.

[0596] Step 2:

[0597] We perform preprocessing on raw video data acquired by the device camera. The input is raw video data, and the output is preprocessed video data. Specifically, we use OpenCV to remove noise using a Gaussian filter and apply it with the cv2.GaussianBlur function. We also use the cv2.resize function to convert it to 640x480 pixels to adjust the resolution.

[0598] Step 3:

[0599] Preprocessed video data is input into the device's AI module to identify the location and distance of objects. The input is preprocessed video data, and the output is object information (location and distance). Specifically, object detection is performed using TensorFlow's SSD or YOLO model, and the location and distance of each object is calculated. For example, the yolov3.detect method is used to identify people and obstacles and obtain their coordinates and distance.

[0600] Step 4:

[0601] Object information is input into the device's natural language processing module, and a linguistic message is generated. The input is object information, and the output is a natural language message. For example, SpaCy can be used to generate linguistic messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead." In this case, grammatically correct sentences are constructed based on the object's position and distance.

[0602] Step 5:

[0603] The device's emotion engine collects physiological data to recognize the user's emotional state. The input is the user's physiological data (skin conductance, heart rate, etc.), and the output is the user's emotional state (stress level and relief). Specifically, it receives data from the skin conductance sensor in real time and analyzes it to determine the stress level. For example, a high skin conductance value is recognized as a high stress state.

[0604] Step 6:

[0605] The device adjusts the voice information based on the output of the emotion engine and the message from the natural language processing module. The input is the emotional state and the natural language message, and the output is the adjusted voice message. Specifically, the tone and speed of the voice are adjusted, and information is provided in a calmer tone to users in a high-stress state. For example, the tone can be set lower using the voice.adjust_tone method.

[0606] Step 7:

[0607] The device's speech synthesis engine inputs the adjusted language message and generates voice data. The input is the adjusted language message, and the output is voice data. Specifically, the Google Text-to-Speech API is used to convert text to voice. For example, the gTTS(text) method of the gTTS library is used to generate the voice file.

[0608] Step 8:

[0609] The device sends the generated audio data to the user's earphone via the Bluetooth module. The input is the audio data, and the output is the audio sent to the user's earphone. Specifically, the audio data is sent using the send method of the Bluetooth module. For example, the bluetooth.send_audio(data) method is used.

[0610] Step 9:

[0611] The user receives real-time audio information from the system via a Bluetooth earphone. The input is the audio transmitted to the Bluetooth earphone, and the output is the audio information recognized by the user. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" is conveyed by voice. This allows the user to move safely.

[0612] (Application example 2)

[0613] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] When visually impaired people move around public spaces, the many obstacles and dangers they encounter make it difficult for them to navigate safely. Furthermore, existing walking aids provide information without taking into account the user's emotional state, which can sometimes prevent them from reducing stress and anxiety. Furthermore, even in self-driving vehicles, visually impaired people have limited means of understanding their surroundings, reducing safety. There is a need for a system that can solve these issues and enable users to travel safely and with peace of mind.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0616] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for recognizing the emotional state of the user and adjusting the audio information. This enables visually impaired people to move around safely and with peace of mind by receiving safety information in real time and providing appropriate audio information according to their emotional state.

[0617] A "walking aid for the disabled" is an auxiliary tool used by visually impaired people to move around safely, and is a device that has the function of detecting obstacles and surrounding conditions.

[0618] A "camera" is a photographing device that captures images of the surroundings and provides image data in real time.

[0619] "Video analysis" is a process that identifies information such as the position and distance of an object based on acquired video data, and typically uses AI technology.

[0620] "Determining the position and distance of an object" means measuring and understanding the position of surrounding objects and the distance to them through video analysis.

[0621] "Safety information" is information about the surrounding situation provided to the user, including the position and distance of obstacles, the status of traffic lights, and the like.

[0622] "Audio provision" is a means of transmitting analyzed information to the user by voice, allowing visually impaired people to receive information aurally.

[0623] "Wireless" refers to a method of transmitting data using wireless communication, such as transmitting audio to earphones using technology such as Bluetooth.

[0624] "Emotional state recognition" refers to monitoring a user's psychological and physiological state to identify emotions such as stress or tension.

[0625] "Adjusting voice information" refers to changing the tone and content of a voice message based on the user's emotional state so that the user receives appropriate information.

[0626] A "system" is a collection of devices and programs that combine the above elements and function to achieve a specific purpose.

[0627] This invention applies a system installed in a walking aid for the disabled to an autonomous vehicle, enabling visually impaired people to travel safely and with peace of mind inside the vehicle. Specific embodiments for carrying out this invention are described below.

[0628] System configuration

[0629] The autonomous vehicle is equipped with a camera to help visually impaired people understand their surroundings, a video analysis AI, a natural language processing module, a speech synthesis engine, an emotion engine, and Bluetooth earphones.

[0630] Hardware and Software

[0631] Camera: The onboard camera captures real-time footage of the surroundings.

[0632] Video analysis AI: Uses TensorFlow and OpenCV to analyze captured video data and identify the position and distance of objects.

[0633] Natural Language Processing module: Uses NLTK and spaCy to generate appropriate messages based on the analysis results.

[0634] Speech synthesis engine: Uses the Google Text-to-Speech API and Amazon Polly to convert text messages into audio data.

[0635] Emotion Engine: Uses the Affectiva SDK to recognize the user's emotional state and adjusts the tone and content of the voice based on that information.

[0636] Bluetooth module and earphones: A commercially available Bluetooth module is used to wirelessly transmit audio data to the user's earphones.

[0637] Processing flow

[0638] The server captures real-time video data captured by the camera and pre-processes the data. The pre-processed video data is analyzed using TensorFlow and OpenCV to determine the location and distance of objects. The analysis results are sent to a natural language processing module using NLTK and spaCy, which generates appropriate messages to be presented to the user.

[0639] The generated messages are adjusted by an emotion engine (Affectiva SDK) based on the user's emotional state, for example, if the user is in a high stress state, the tone of the voice will be changed to a calmer one.

[0640] Finally, a speech synthesis engine (Google Text-to-Speech API or Amazon Polly) is used to convert the text message into audio data, which is then sent to the user's earphones via a Bluetooth module.

[0641] Specific examples

[0642] Example of approaching an intersection

[0643] The camera captures the intersection.

[0644] Video analysis AI identifies traffic light conditions and approaching pedestrians.

[0645] The natural language processing module generates messages such as "The traffic light is red" and "There is a pedestrian ahead."

[0646] The emotion engine recognizes when the user is in a high stress state.

[0647] Generates a tone-adjusted message: "Please wait calmly until the traffic light changes."

[0648] The generated audio data is transmitted to the user through a Bluetooth earphone.

[0649] Prompt Sentence Examples

[0650] "Identify the traffic light status from the video of the intersection captured by the in-car camera, and if the light is red and there are pedestrians, generate a message saying, 'The light is red. There are pedestrians ahead.' Also, if the user is in a high-stress state, adjust the message to a more reassuring tone."

[0651] This invention allows visually impaired people to grasp information about their surroundings in real time even while inside an autonomous vehicle, enabling them to travel with peace of mind.

[0652] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0653] Step 1:

[0654] The camera captures the surrounding image data in real time. The scene and objects captured by the camera are used as input. The captured image data is then output.

[0655] Step 2:

[0656] The server receives the captured video data and performs pre-processing, which includes noise removal, resolution adjustment, and color correction. The raw captured video data is used as input, and the pre-processed, clean video data is obtained as output.

[0657] Step 3:

[0658] The server inputs the preprocessed video data into a video analysis AI (TensorFlow and OpenCV) for object recognition. The AI ​​identifies the location and distance of objects such as pedestrians, vehicles, and traffic lights. The preprocessed video data is used as input, and data on the location and distance of objects is obtained as output.

[0659] Step 4:

[0660] The server inputs the results of the video analysis AI into a natural language processing module (NLTK and spaCy), which converts the analysis results into a text message. For example, messages such as "There is a pedestrian ahead" or "The traffic light is red" are generated. Data on the object's position and distance is used as input, and a text message in natural language is obtained as output.

[0661] Step 5:

[0662] The server inputs the generated text message into the emotion engine (Affectiva SDK) to monitor the user's emotional state. The emotion engine evaluates the user's stress level, comfort level, etc., and adjusts the tone and speed of the voice message. The text message and the user's emotion data are used as input, and the adjusted text message is obtained as output.

[0663] Step 6:

[0664] The server inputs the adjusted text message into a speech synthesis engine (such as Google Text-to-Speech API or Amazon Polly) and converts it into voice data. For example, a voice message such as "The traffic light is red. There is a pedestrian ahead" is generated. The adjusted text message is used as input, and voice data is obtained as output.

[0665] Step 7:

[0666] The server sends the generated voice data to the terminal (earphone) via the Bluetooth module, allowing the user to receive information as voice in real time. Voice data is used as input, and as output, a voice message is transmitted to the user through the Bluetooth earphone.

[0667] In this way, visually impaired people will be able to receive real-time safety information and understand their surroundings even while inside an autonomous vehicle.

[0668] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0669] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0670] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0671] [Third embodiment]

[0672] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0673] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0674] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0675] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0676] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0677] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0678] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0679] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0680] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0681] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0682] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0683] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0684] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, a Bluetooth module, and a Bluetooth earphone.

[0685] System configuration

[0686] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to a built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles, and sends the analysis results to a natural language processing module, which generates an appropriate verbal message. The speech synthesis engine then converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[0687] Program processing flow

[0688] Video capture and pre-processing

[0689] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[0690] Video data analysis

[0691] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[0692] Verbalizing information

[0693] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0694] Generating and transmitting audio data

[0695] The device passes the linguistic information from the natural language processing module to the speech synthesis engine to generate voice data, which is then transmitted to the user's earphones via the Bluetooth module.

[0696] User Notification

[0697] The user receives real-time audio information via a Bluetooth earphone, with specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[0698] Specific examples

[0699] When walking on the station platform

[0700] Consider a case where a user is walking on a train platform. At this time, the device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters ahead," or "There is a person three meters ahead." This information is converted into voice data and communicated to the user via Bluetooth earphones.

[0701] When crossing an intersection

[0702] When a user approaches an intersection, the device's camera captures the traffic light, and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones.

[0703] This system allows visually impaired people to know the situation around them in real time, enabling them to move around safely.

[0704] The processing flow will be explained below.

[0705] Step 1:

[0706] The device (a camera mounted on a white cane) captures images of the surroundings. The camera is designed to continuously collect image data and process it in real time.

[0707] Step 2:

[0708] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[0709] Step 3:

[0710] The device provides pre-processed video data to the AI ​​module, which then inputs the prepared data and begins object detection, specifically locating people and obstacles and measuring their distance.

[0711] Step 4:

[0712] The device recognizes specific objects such as traffic lights. The AI ​​module identifies the traffic light's status (red or green) and acquires that information. The location and status of the identified object are output as the analysis results.

[0713] Step 5:

[0714] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to be conveyed to the user.

[0715] Step 6:

[0716] The device passes the linguistic message to the speech synthesis engine, which generates voice data, which is immediately stored in temporary memory and moves on to the next process.

[0717] Step 7:

[0718] The device uses the Bluetooth module to send the generated audio data to the user's Bluetooth earphone. After confirming that the Bluetooth connection is established, the audio data is sent.

[0719] Step 8:

[0720] The user receives real-time voice information through a Bluetooth earphone. For example, the user is told, "There is a person 3 meters ahead," or "The end of the platform is 2 meters ahead." Based on this voice information, the user can move safely.

[0721] Through this series of processes, visually impaired people are notified of their surroundings via audio in real time, enabling them to move around safely.

[0722] Example 1

[0723] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0724] For visually impaired people to move around safely in their daily lives, it is essential that they be able to properly understand their surroundings. However, current walking aids do not provide enough information to achieve this, making it difficult to detect obstacles in real time or provide detailed information. The purpose of this invention is to solve these problems and enable visually impaired people to move around more safely and with peace of mind.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0726] In this invention, the server includes means for capturing surrounding images using a camera mounted on a portable measuring device, means for preprocessing the captured images to remove noise and adjust resolution, means for analyzing the preprocessed images to identify the position and distance of an object, means for converting the analysis results of the identified object's position and distance into natural language, means for synthesizing the converted natural language information into voice data, and means for wirelessly transmitting the generated voice data to a user, thereby enabling visually impaired people to accurately grasp their surroundings in real time and move safely.

[0727] A "measuring device" is a device carried by a user for acquiring surrounding environmental data.

[0728] The "camera" is a device that is mounted on the measuring device and captures images of the surrounding area in real time.

[0729] "Preprocessing" refers to performing processes such as noise removal and resolution adjustment on the video data acquired by the imaging device.

[0730] "Analysis" refers to the application of calculations and algorithms to use pre-processed video data to determine information such as object location and distance.

[0731] "Conversion to natural language" refers to the process of converting the information obtained from the analysis results into words that are easy for humans to understand.

[0732] "Synthesis into voice data" is a process of generating voice from information converted into natural language.

[0733] A "wireless method" is a means of sending and receiving data using wireless communication technologies such as Bluetooth and Wi-Fi.

[0734] "User" refers to a visually impaired person who uses this system to walk safely.

[0735] "Safety information" refers to the surrounding situation, the location and distance of obstacles, and other related information necessary for the user to move safely.

[0736] The following describes a specific embodiment of the present invention. This system is a walking aid that enables the visually impaired to move around safely and with peace of mind. The system includes a camera mounted on a portable measuring device, hardware and software for preprocessing the images, a built-in AI module for object detection and analysis, a natural language processing module, a speech synthesis engine, a wireless communication module, and wireless earphones worn by the user.

[0737] Specifically, the portable measurement device is equipped with a small camera that captures images of the surroundings in real time using OpenCV. The captured images are preprocessed using image processing techniques such as Gaussian filtering. The preprocessed image data is then sent to the built-in AI module using AI frameworks such as TensorFlow and YOLO.

[0738] The AI ​​module identifies the location and distance of objects (people, obstacles, traffic lights, etc.) from the video data. For example, it uses the YOLO (You Only Look Once) algorithm to detect objects in the video in real time and calculate their location coordinates and distance. The results of this analysis are passed to a natural language processing module (e.g., spaCy or NLTK), which generates specific language messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0739] The generated linguistic message is then converted into voice data by a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly) and sent to wireless earphones via a Bluetooth module. By receiving this voice information in real time through the Bluetooth earphones, users can accurately grasp their surroundings and move safely.

[0740] Specific examples

[0741] When walking on the station platform

[0742] As a user walks along a station platform, the device's camera captures video of the surrounding area. After the video is preprocessed, an AI module detects when the edge of the platform is approaching or if a person is standing ahead. The analysis results are verbalized as "The edge of the platform is two meters away" or "There is a person three meters ahead." This verbalized information is then converted into audio data and transmitted to the user via Bluetooth earphones.

[0743] When crossing an intersection

[0744] When a user approaches an intersection, the device's camera captures the traffic light. After the video is preprocessed, the AI ​​module analyzes the traffic light status and determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and transmitted to the user's earphones.

[0745] Prompt Sentence Examples

[0746] By providing the following prompt to the generative AI model, it can generate a natural language message:

[0747] Example prompt 1:

[0748] Generate user-friendly messages based on the obstacle detection results.

[0749] Input: Distance to obstacle is 3 meters, object is a person

[0750] Output: There is a person 3 meters ahead

[0751] Example prompt 2:

[0752] Generate user-friendly messages based on the traffic light status.

[0753] Input: Traffic light status is red

[0754] Output: The light is red. Please wait until the light changes.

[0755] In this way, the system is designed to enable visually impaired people to accurately grasp their surroundings in real time.

[0756] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0757] Step 1:

[0758] The terminal captures the surroundings in real time using a camera mounted on a portable measuring device. The camera captures images at tens of frames per second and sends the image data to the next processing step.

[0759] Input: Surrounding image

[0760] Output: Raw captured video data

[0761] Step 2:

[0762] The device performs preprocessing on the captured video data. This preprocessing involves using a video processing library such as OpenCV to remove noise and adjust the resolution. For example, a Gaussian filter is applied to reduce image noise and convert the video to the optimal resolution.

[0763] Input: Raw captured video data

[0764] Output: Pre-processed video data

[0765] Step 3:

[0766] The device sends the preprocessed video data to the AI ​​module, which uses object detection algorithms such as TensorFlow and YOLO to identify objects in the video and determine their location and distance. For example, the YOLO algorithm can be used to generate bounding boxes for people and obstacles in the video and calculate distance from their center coordinates and size.

[0767] Input: Preprocessed video data

[0768] Output: Object position and distance information

[0769] Step 4:

[0770] The device passes the object position and distance information received from the AI ​​module to a natural language processing module. The natural language processing module (e.g., spaCy) is used to convert the analysis results into natural language. For example, the information "Person: 3 meters" is converted into a message such as "There is a person 3 meters ahead."

[0771] Input: Object position and distance information

[0772] Output: A natural language message

[0773] Step 5:

[0774] The device passes the natural language message generated by the natural language processing module to a speech synthesis engine. The speech synthesis engine (e.g., Google Text-to-Speech) is used to synthesize the text message into voice data. In this process, the text is converted into an audio file (e.g., WAV format).

[0775] Input: A natural language message

[0776] Output: Audio data

[0777] Step 6:

[0778] The terminal transmits the generated audio data to the wireless earphone via the Bluetooth module. The audio data is transferred to the earphone via Bluetooth communication and notified to the user in real time.

[0779] Input: Audio data

[0780] Output: Audio data sent to Bluetooth earphones

[0781] Step 7:

[0782] The user receives voice information transmitted through a Bluetooth earphone, and can confirm specific voice messages in real time, such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[0783] Input: Audio data sent to Bluetooth earphones

[0784] Output: User receives audio information

[0785] Through these steps, visually impaired people can accurately grasp their surroundings in real time and move around safely.

[0786] (Application example 1)

[0787] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0788] Traditionally, visually impaired people have relied primarily on white canes and guide dogs to understand their surroundings when traveling. However, these methods have limited ability to detect specific dangers, making it difficult to obtain real-time information about new obstacles and moving objects and take immediate action. Particularly at intersections and in complex road environments, there has been a lack of information provided to enable safe travel.

[0789] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0790] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for acquiring surrounding situation data of the mobile object and generating safety information based on the analysis results, thereby enabling visually impaired people to grasp the surrounding situation in real time and move around safely.

[0791] "Mobility aids for the disabled" are aids used by visually impaired people to move around safely.

[0792] A "camera" is a photographing device for capturing images of the surroundings.

[0793] The "means for acquiring images" refers to a means for collecting image data of the surroundings in real time using a camera.

[0794] The "means for analyzing video" is a means for analyzing acquired video data and identifying the position and distance of an object.

[0795] The "means for identifying the position and distance of an object" is a means for calculating the spatial position and distance of a specific object based on the results of video analysis.

[0796] The "means for providing safety information by voice" is a means for notifying the user by voice of information generated based on the analysis results.

[0797] "Means for wirelessly communicating to a user" refers to means for transmitting audio information to a user's device using wireless communication technology such as Bluetooth or Wi-Fi.

[0798] "Means for acquiring surrounding situation data of a moving body" refers to means for sensing the surrounding environment of an autonomous vehicle or the like and collecting that data.

[0799] The "means for generating safety information based on the analysis results" is a means for analyzing data acquired from a mobile object and generating safety information that is useful to the user.

[0800] The system embodying this invention is used as an aid for visually impaired people to move around safely. Specifically, the system includes a camera mounted on the walking aid, an AI module for video analysis, a module for natural language processing, a speech synthesis engine, a Bluetooth module, and Bluetooth earphones.

[0801] The device's built-in camera captures real-time video of the surrounding area. This video data is then sent to an AI module after pre-processing, such as adjusting the resolution and removing noise, and analyzed to determine the location and distance of objects. Specifically, it identifies people, obstacles, traffic light status, and other objects in the video.

[0802] The information analyzed by the AI ​​module is passed to the natural language processing module, which generates a verbal message to notify the user, such as "There is a person 3 meters ahead" or "The light is red, do not cross."

[0803] The generated linguistic message is sent to a speech synthesis engine and converted into voice data, which is then transmitted to a Bluetooth earphone via a Bluetooth module, providing real-time notification to the user.

[0804] Furthermore, this system also has the ability to acquire data on the surrounding environment of the moving object and generate safety information based on that data. For example, the system can analyze the surrounding environment data acquired by an autonomous vehicle and notify the user, "A vehicle is approaching, please be careful."

[0805] The hardware used includes a camera, an AI module, a natural language processing module, a speech synthesis engine, and a Bluetooth module, while the software required is an ObjectDetectionModel (object detection algorithm), an NLPModel (natural language generation algorithm), and a TTSEngine (speech synthesis engine).

[0806] Specific examples

[0807] For example, when a user is walking on a train platform, the AI ​​module detects the edge of the platform or people ahead from the images captured by the device's camera, and generates messages based on that information, such as "The edge of the platform is two meters ahead" or "There is a person three meters ahead." This allows the user to move safely. In addition, the camera recognizes the status of traffic lights at intersections, and when the light is red, a message is displayed saying, "The light is red. Please wait until the light changes." This system allows visually impaired people to understand their surroundings in real time and move safely.

[0808] Prompt Sentence Examples

[0809] It visually monitors the status of traffic lights and the surrounding environment and provides real-time voice guidance to help drivers cross the crosswalk safely. Notifications include, "The light is green. Please cross safely." For example, a vehicle camera captures the traffic light, and an AI model identifies that the light is green. This information is sent to a natural language processing module, which generates a voice message saying, "The light is green," and sends it to a Bluetooth earphone.

[0810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0811] Step 1:

[0812] The device uses a camera to capture images of the surroundings. The input is real-time image data, and the output is the raw image data.

[0813] Step 2:

[0814] Preprocesses the video data acquired by the device. Specifically, it removes noise and adjusts the resolution. The input is raw video data, and the output is preprocessed video data.

[0815] Step 3:

[0816] The preprocessed video data is sent to an AI module to detect objects and determine their location and distance. The input is the preprocessed video data, and the output is object location and distance information. Specifically, the AI ​​model recognizes people and obstacles in the video and calculates their respective positions and distances.

[0817] Step 4:

[0818] The device sends the object's position and distance information to the natural language processing module, which then generates a linguistic message to notify the user. The input is the object's position and distance information, and the output is a natural language message. Specifically, messages such as "There is a person 3 meters ahead" or "The light is red, do not cross" are generated.

[0819] Step 5:

[0820] The generated natural language message is passed to a speech synthesis engine and converted into audio data. The input is a natural language message, and the output is audio data. Specifically, the speech synthesis engine converts the text into an audio file.

[0821] Step 6:

[0822] The terminal transmits audio data to the user's Bluetooth earphone via the Bluetooth module. The input is audio data, and the output is audio notification to the Bluetooth earphone. Specifically, the Bluetooth module transmits audio data wirelessly to the earphone.

[0823] Step 7:

[0824] The user receives real-time audio information through a Bluetooth earphone. The input is the audio data sent to the Bluetooth earphone, and the output is audio notifications to the user. Specifically, the information conveyed to the user is "There is a person 3 meters ahead" or "The traffic light is red, do not cross."

[0825] Step 8:

[0826] The system acquires data on the surrounding situation of a moving object and generates safety information based on the analysis results. The input is the surrounding situation data from the moving object, and the output is safety information based on the analysis results. Specifically, it analyzes data acquired from the moving object's sensors and generates messages such as "A vehicle is approaching, please be careful."

[0827] Step 9:

[0828] The generated safety information is notified to the user as voice data. The input is a natural language message based on the analysis results, and the output is a safety information notification as voice data. Specifically, a voice synthesis engine converts the text into voice data and sends it to earphones via Bluetooth.

[0829] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0830] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[0831] System configuration

[0832] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to the built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles and sends the analysis results to the natural language processing module, which generates an appropriate verbal message. The emotion engine then recognizes the user's emotions and appropriately adjusts the voice information provided. Finally, the speech synthesis engine converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[0833] Program processing flow

[0834] Video capture and pre-processing

[0835] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[0836] Video data analysis

[0837] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[0838] Verbalizing information

[0839] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0840] User Emotion Recognition

[0841] Along with verbalized information, the device recognizes the user's emotional state through an emotion engine, which uses voice tone, speed, skin conductance, and other physiological data to determine the user's emotional state (e.g., stress level or relief).

[0842] Adjusting audio information

[0843] The device adjusts the voice information as needed based on the output of the emotion engine. If the user is under high stress, the system will provide information in a calm and reassuring tone. The frequency and intensity of voice warnings will also be adjusted according to the stress level.

[0844] Generating and transmitting audio data

[0845] The device passes the adjusted language message to a speech synthesis engine to generate voice data, which is then sent to the user's earphones via a Bluetooth module.

[0846] User Notification

[0847] The user receives real-time voice information through a Bluetooth earphone. For example, specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" are conveyed to the user. Based on this voice information, the user can move safely.

[0848] Specific examples

[0849] When walking on the station platform

[0850] Consider a case where a user is walking on a train platform. The device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing the user to travel with greater peace of mind.

[0851] When crossing an intersection

[0852] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[0853] This system allows visually impaired people to know what is happening around them in real time, enabling them to move around safely and with peace of mind.

[0854] The processing flow will be explained below.

[0855] Step 1:

[0856] The device (a camera mounted on a white cane) captures the surroundings. The camera continuously collects video data and passes it on to the next process in real time.

[0857] Step 2:

[0858] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[0859] Step 3:

[0860] The device provides preprocessed video data to the AI ​​module, which then uses the input data to run an object detection algorithm to identify the location and distance of people and obstacles. For example, it might detect a person standing three meters ahead and output that information.

[0861] Step 4:

[0862] The AI ​​module on the device identifies the state of a specific object (e.g., a traffic light). It detects that the traffic light is red and obtains that information.

[0863] Step 5:

[0864] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to convey to the user, such as "There is a person 3 meters ahead" or "The traffic light is red, please wait."

[0865] Step 6:

[0866] The terminal temporarily stores the generated linguistic message, which is then sent to the emotion engine.

[0867] Step 7:

[0868] The device operates an emotion engine to recognize the user's emotions. The emotion engine analyzes physiological data such as the user's voice tone and skin conductance to determine the user's emotional state (e.g., high stress state).

[0869] Step 8:

[0870] The device adjusts the verbal messages as needed based on the user's recognized emotional state, for example, making the tone of the voice information gentler and adjusting the frequency of the messages if the user is in a high-stress state.

[0871] Step 9:

[0872] The device passes the adjusted language message to a speech synthesis engine to generate speech data, which is stored in temporary memory.

[0873] Step 10:

[0874] The device uses the Bluetooth module to transmit the generated audio data to the user's Bluetooth earphone, and transmits the data only after verifying that the Bluetooth connection is established.

[0875] Step 11:

[0876] The user receives real-time audio information through a Bluetooth earphone, such as "There is a person 3 meters ahead," "The traffic light is red, please wait," or "It's okay, please proceed calmly."

[0877] This series of processes allows visually impaired people to know the situation around them in real time, allowing them to move around safely and with peace of mind.

[0878] Example 2

[0879] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0880] For people with disabilities, especially those with visual impairments, to walk safely, it is important to quickly and accurately grasp information about their surroundings. However, current walking aids lack a system that can efficiently analyze video data acquired by a camera and provide the user with appropriate audio information. They also lack the ability to adjust the audio according to the user's psychological state and stress level. This makes it difficult for users to move around with a sufficient sense of security.

[0881] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring surrounding video using a camera mounted on a walking aid for disabled persons; means for preprocessing the acquired video; means for analyzing the preprocessed video and identifying the position and distance of an object; means for converting information on the position and distance of the identified object into natural language; means for recognizing the user's emotional state based on the converted natural language information; means for adjusting voice information based on the recognized emotional state; means for converting the adjusted voice information into voice data; and means for wirelessly transmitting the voice data to the user. This allows the user to grasp information about the surrounding environment in real time and receive voice guidance according to the user's psychological state at that time, allowing for safer and more secure travel.

[0882] A "walking aid for the disabled" is a tool used by disabled people, such as those with visual impairments, to assist them in walking.

[0883] A "camera" is a device that captures images and acquires them as digital data.

[0884] "Video" is data that represents a series of still images linked together over time to create a motion image.

[0885] "Preprocessing" refers to the process of performing initial processing such as noise removal and resolution adjustment on the acquired raw data.

[0886] An "AI module" is a software or hardware unit that uses artificial intelligence technology to analyze data and make inferences.

[0887] The "position and distance of an object" refers to the spatial coordinates of an object detected by the camera and the linear distance from the camera to the object.

[0888] "Natural language processing" refers to converting analyzed data into natural language that humans can understand.

[0889] A "natural language processing module" is a software module that performs text analysis, generation, etc.

[0890] "Emotional state" refers to the user's psychological and physiological state, including stress levels, feelings of security, etc.

[0891] An "emotion engine" is software that has the ability to recognize a user's emotional state and adjust the system's response based on that.

[0892] "Adjusting audio information" refers to changing the tone, speed, and frequency of audio depending on the user's emotional state.

[0893] "Speech data" refers to a speech signal generated based on text information in a natural language.

[0894] A "speech synthesis engine" is a software or hardware unit that converts text data into speech.

[0895] A "wireless system" is a method of transmitting and receiving data using wireless communication technology.

[0896] A "Bluetooth earphone" is a device worn in the ear that receives audio data wirelessly using Bluetooth technology.

[0897] The system of the present invention includes a camera mounted on a walking aid for the disabled, an artificial intelligence (AI) module for video analysis, a module for natural language processing, a speech synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[0898] The device (a camera mounted on a white cane) captures the surroundings in real time. This video data is first preprocessed to remove noise and adjust the resolution. Specifically, a Gaussian filter is applied using OpenCV to remove noise, and the resolution is adjusted to 640x480 using a resize method.

[0899] After preprocessing, the video data is immediately sent to the device's AI module. The AI ​​module uses deep learning libraries such as TensorFlow to recognize objects in the video. Specifically, it uses SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) to detect people and obstacles and determine the location and distance of each object. The AI ​​module can also identify the color of traffic lights (red or green).

[0900] The analysis results are sent to the device's natural language processing (NLP) module, which translates the information into natural language. This module uses libraries such as SpaCy to generate messages like "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[0901] The device sends the generated linguistic message to the emotion engine, which collects physiological data from skin conductance sensors and heart rate sensors to determine the user's stress level and sense of security. For example, high skin conductance is recognized as a state of high stress. This emotional state information is used to adjust the voice information.

[0902] The device adjusts the tone and intensity of messages generated by the natural language processing module based on the output of the emotion engine. Specifically, if the user is under high stress, the tone of the voice will be adjusted to a calmer tone and the frequency of messages will be reduced.

[0903] The adjusted language message is passed to a speech synthesis engine, which uses APIs such as Google Text-to-Speech to convert the text data into speech data, which is then sent via a Bluetooth module to the user's Bluetooth earphones.

[0904] In this way, users can receive real-time voice information through Bluetooth earphones. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" can be conveyed to the user as a voice message, allowing the user to move safely.

[0905] Specific examples

[0906] When walking on the station platform

[0907] As a user walks along a station platform, the device's camera captures images of the surrounding area, and the AI ​​module identifies when the edge of the platform is approaching or when a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing users to travel with greater peace of mind.

[0908] When crossing an intersection

[0909] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[0910] Example prompt sentence:

[0911] 1. Describe a scenario in which a user is walking along a train platform, and then describe how the system captures information about their surroundings and communicates it to the user.

[0912] 2. Describe a situation where a user is about to cross an intersection, and specify how the system captures the status of the traffic lights and notifies the user.

[0913] In this way, visually impaired people can learn about their surroundings in real time through the system, enabling them to move around safely and with peace of mind.

[0914] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0915] Specific processing flow of the system

[0916] Step 1:

[0917] The device (a camera mounted on a white cane) captures the surroundings in real time. The input is the surroundings video data, and the output is the captured raw video data. For example, the camera captures video at a rate of 30 frames per second with a resolution of 640x480 pixels. This video data is stored in temporary memory and goes to the next pre-processing step.

[0918] Step 2:

[0919] We perform preprocessing on raw video data acquired by the device camera. The input is raw video data, and the output is preprocessed video data. Specifically, we use OpenCV to remove noise using a Gaussian filter and apply it with the cv2.GaussianBlur function. We also use the cv2.resize function to convert it to 640x480 pixels to adjust the resolution.

[0920] Step 3:

[0921] Preprocessed video data is input into the device's AI module to identify the location and distance of objects. The input is preprocessed video data, and the output is object information (location and distance). Specifically, object detection is performed using TensorFlow's SSD or YOLO model, and the location and distance of each object is calculated. For example, the yolov3.detect method is used to identify people and obstacles and obtain their coordinates and distance.

[0922] Step 4:

[0923] Object information is input into the device's natural language processing module, and a linguistic message is generated. The input is object information, and the output is a natural language message. For example, SpaCy can be used to generate linguistic messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead." In this case, grammatically correct sentences are constructed based on the object's position and distance.

[0924] Step 5:

[0925] The device's emotion engine collects physiological data to recognize the user's emotional state. The input is the user's physiological data (skin conductance, heart rate, etc.), and the output is the user's emotional state (stress level and relief). Specifically, it receives data from the skin conductance sensor in real time and analyzes it to determine the stress level. For example, a high skin conductance value is recognized as a high stress state.

[0926] Step 6:

[0927] The device adjusts the voice information based on the output of the emotion engine and the message from the natural language processing module. The input is the emotional state and the natural language message, and the output is the adjusted voice message. Specifically, the tone and speed of the voice are adjusted, and information is provided in a calmer tone to users in a high-stress state. For example, the tone can be set lower using the voice.adjust_tone method.

[0928] Step 7:

[0929] The device's speech synthesis engine inputs the adjusted language message and generates voice data. The input is the adjusted language message, and the output is voice data. Specifically, the Google Text-to-Speech API is used to convert text to voice. For example, the gTTS(text) method of the gTTS library is used to generate the voice file.

[0930] Step 8:

[0931] The device sends the generated audio data to the user's earphone via the Bluetooth module. The input is the audio data, and the output is the audio sent to the user's earphone. Specifically, the audio data is sent using the send method of the Bluetooth module. For example, the bluetooth.send_audio(data) method is used.

[0932] Step 9:

[0933] The user receives real-time audio information from the system via a Bluetooth earphone. The input is the audio transmitted to the Bluetooth earphone, and the output is the audio information recognized by the user. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" is conveyed by voice. This allows the user to move safely.

[0934] (Application example 2)

[0935] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0936] When visually impaired people move around public spaces, the many obstacles and dangers they encounter make it difficult for them to navigate safely. Furthermore, existing walking aids provide information without taking into account the user's emotional state, which can sometimes prevent them from reducing stress and anxiety. Furthermore, even in self-driving vehicles, visually impaired people have limited means of understanding their surroundings, reducing safety. There is a need for a system that can solve these issues and enable users to travel safely and with peace of mind.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0938] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for recognizing the emotional state of the user and adjusting the audio information. This enables visually impaired people to move around safely and with peace of mind by receiving safety information in real time and providing appropriate audio information according to their emotional state.

[0939] A "walking aid for the disabled" is an auxiliary tool used by visually impaired people to move around safely, and is a device that has the function of detecting obstacles and surrounding conditions.

[0940] A "camera" is a photographing device that captures images of the surroundings and provides image data in real time.

[0941] "Video analysis" is a process that identifies information such as the position and distance of an object based on acquired video data, and typically uses AI technology.

[0942] "Determining the position and distance of an object" means measuring and understanding the position of surrounding objects and the distance to them through video analysis.

[0943] "Safety information" is information about the surrounding situation provided to the user, including the position and distance of obstacles, the status of traffic lights, and the like.

[0944] "Audio provision" is a means of transmitting analyzed information to the user by voice, allowing visually impaired people to receive information aurally.

[0945] "Wireless" refers to a method of transmitting data using wireless communication, such as transmitting audio to earphones using technology such as Bluetooth.

[0946] "Emotional state recognition" refers to monitoring a user's psychological and physiological state to identify emotions such as stress or tension.

[0947] "Adjusting voice information" refers to changing the tone and content of a voice message based on the user's emotional state so that the user receives appropriate information.

[0948] A "system" is a collection of devices and programs that combine the above elements and function to achieve a specific purpose.

[0949] This invention applies a system installed in a walking aid for the disabled to an autonomous vehicle, enabling visually impaired people to travel safely and with peace of mind inside the vehicle. Specific embodiments for carrying out this invention are described below.

[0950] System configuration

[0951] The autonomous vehicle is equipped with a camera to help visually impaired people understand their surroundings, a video analysis AI, a natural language processing module, a speech synthesis engine, an emotion engine, and Bluetooth earphones.

[0952] Hardware and Software

[0953] Camera: The onboard camera captures real-time footage of the surroundings.

[0954] Video analysis AI: Uses TensorFlow and OpenCV to analyze captured video data and identify the position and distance of objects.

[0955] Natural Language Processing module: Uses NLTK and spaCy to generate appropriate messages based on the analysis results.

[0956] Speech synthesis engine: Uses the Google Text-to-Speech API and Amazon Polly to convert text messages into audio data.

[0957] Emotion Engine: Uses the Affectiva SDK to recognize the user's emotional state and adjusts the tone and content of the voice based on that information.

[0958] Bluetooth module and earphones: A commercially available Bluetooth module is used to wirelessly transmit audio data to the user's earphones.

[0959] Processing flow

[0960] The server captures real-time video data captured by the camera and pre-processes the data. The pre-processed video data is analyzed using TensorFlow and OpenCV to determine the location and distance of objects. The analysis results are sent to a natural language processing module using NLTK and spaCy, which generates appropriate messages to be presented to the user.

[0961] The generated messages are adjusted by an emotion engine (Affectiva SDK) based on the user's emotional state, for example, if the user is in a high stress state, the tone of the voice will be changed to a calmer one.

[0962] Finally, a speech synthesis engine (Google Text-to-Speech API or Amazon Polly) is used to convert the text message into audio data, which is then sent to the user's earphones via a Bluetooth module.

[0963] Specific examples

[0964] Example of approaching an intersection

[0965] The camera captures the intersection.

[0966] Video analysis AI identifies traffic light conditions and approaching pedestrians.

[0967] The natural language processing module generates messages such as "The traffic light is red" and "There is a pedestrian ahead."

[0968] The emotion engine recognizes when the user is in a high stress state.

[0969] Generates a tone-adjusted message: "Please wait calmly until the traffic light changes."

[0970] The generated audio data is transmitted to the user through a Bluetooth earphone.

[0971] Prompt Sentence Examples

[0972] "Identify the traffic light status from the video of the intersection captured by the in-car camera, and if the light is red and there are pedestrians, generate a message saying, 'The light is red. There are pedestrians ahead.' Also, if the user is in a high-stress state, adjust the message to a more reassuring tone."

[0973] This invention allows visually impaired people to grasp information about their surroundings in real time even while inside an autonomous vehicle, enabling them to travel with peace of mind.

[0974] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0975] Step 1:

[0976] The camera captures the surrounding image data in real time. The scene and objects captured by the camera are used as input. The captured image data is then output.

[0977] Step 2:

[0978] The server receives the captured video data and performs pre-processing, which includes noise removal, resolution adjustment, and color correction. The raw captured video data is used as input, and the pre-processed, clean video data is obtained as output.

[0979] Step 3:

[0980] The server inputs the preprocessed video data into a video analysis AI (TensorFlow and OpenCV) for object recognition. The AI ​​identifies the location and distance of objects such as pedestrians, vehicles, and traffic lights. The preprocessed video data is used as input, and data on the location and distance of objects is obtained as output.

[0981] Step 4:

[0982] The server inputs the results of the video analysis AI into a natural language processing module (NLTK and spaCy), which converts the analysis results into a text message. For example, messages such as "There is a pedestrian ahead" or "The traffic light is red" are generated. Data on the object's position and distance is used as input, and a text message in natural language is obtained as output.

[0983] Step 5:

[0984] The server inputs the generated text message into the emotion engine (Affectiva SDK) to monitor the user's emotional state. The emotion engine evaluates the user's stress level, comfort level, etc., and adjusts the tone and speed of the voice message. The text message and the user's emotion data are used as input, and the adjusted text message is obtained as output.

[0985] Step 6:

[0986] The server inputs the adjusted text message into a speech synthesis engine (such as Google Text-to-Speech API or Amazon Polly) and converts it into voice data. For example, a voice message such as "The traffic light is red. There is a pedestrian ahead" is generated. The adjusted text message is used as input, and voice data is obtained as output.

[0987] Step 7:

[0988] The server sends the generated voice data to the terminal (earphone) via the Bluetooth module, allowing the user to receive information as voice in real time. Voice data is used as input, and as output, a voice message is transmitted to the user through the Bluetooth earphone.

[0989] In this way, visually impaired people will be able to receive real-time safety information and understand their surroundings even while inside an autonomous vehicle.

[0990] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0991] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0992] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0993] [Fourth embodiment]

[0994] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0995] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0996] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0997] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0998] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0999] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1000] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1001] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1002] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1003] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1004] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1005] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1006] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1007] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, a Bluetooth module, and a Bluetooth earphone.

[1008] System configuration

[1009] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to a built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles, and sends the analysis results to a natural language processing module, which generates an appropriate verbal message. The speech synthesis engine then converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[1010] Program processing flow

[1011] Video capture and pre-processing

[1012] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[1013] Video data analysis

[1014] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[1015] Verbalizing information

[1016] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[1017] Generating and transmitting audio data

[1018] The device passes the linguistic information from the natural language processing module to the speech synthesis engine to generate voice data, which is then transmitted to the user's earphones via the Bluetooth module.

[1019] User Notification

[1020] The user receives real-time audio information via a Bluetooth earphone, with specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[1021] Specific examples

[1022] When walking on the station platform

[1023] Consider a case where a user is walking on a train platform. At this time, the device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters ahead," or "There is a person three meters ahead." This information is converted into voice data and communicated to the user via Bluetooth earphones.

[1024] When crossing an intersection

[1025] When a user approaches an intersection, the device's camera captures the traffic light, and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones.

[1026] This system allows visually impaired people to know the situation around them in real time, enabling them to move around safely.

[1027] The processing flow will be explained below.

[1028] Step 1:

[1029] The device (a camera mounted on a white cane) captures images of the surroundings. The camera is designed to continuously collect image data and process it in real time.

[1030] Step 2:

[1031] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[1032] Step 3:

[1033] The device provides pre-processed video data to the AI ​​module, which then inputs the prepared data and begins object detection, specifically locating people and obstacles and measuring their distance.

[1034] Step 4:

[1035] The device recognizes specific objects such as traffic lights. The AI ​​module identifies the traffic light's status (red or green) and acquires that information. The location and status of the identified object are output as the analysis results.

[1036] Step 5:

[1037] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to be conveyed to the user.

[1038] Step 6:

[1039] The device passes the linguistic message to the speech synthesis engine, which generates voice data, which is immediately stored in temporary memory and moves on to the next process.

[1040] Step 7:

[1041] The device uses the Bluetooth module to send the generated audio data to the user's Bluetooth earphone. After confirming that the Bluetooth connection is established, the audio data is sent.

[1042] Step 8:

[1043] The user receives real-time voice information through a Bluetooth earphone. For example, the user is told, "There is a person 3 meters ahead," or "The end of the platform is 2 meters ahead." Based on this voice information, the user can move safely.

[1044] Through this series of processes, visually impaired people are notified of their surroundings via audio in real time, enabling them to move around safely.

[1045] Example 1

[1046] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1047] For visually impaired people to move around safely in their daily lives, it is essential that they be able to properly understand their surroundings. However, current walking aids do not provide enough information to achieve this, making it difficult to detect obstacles in real time or provide detailed information. The purpose of this invention is to solve these problems and enable visually impaired people to move around more safely and with peace of mind.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1049] In this invention, the server includes means for capturing surrounding images using a camera mounted on a portable measuring device, means for preprocessing the captured images to remove noise and adjust resolution, means for analyzing the preprocessed images to identify the position and distance of an object, means for converting the analysis results of the identified object's position and distance into natural language, means for synthesizing the converted natural language information into voice data, and means for wirelessly transmitting the generated voice data to a user, thereby enabling visually impaired people to accurately grasp their surroundings in real time and move safely.

[1050] A "measuring device" is a device carried by a user for acquiring surrounding environmental data.

[1051] The "camera" is a device that is mounted on the measuring device and captures images of the surrounding area in real time.

[1052] "Preprocessing" refers to performing processes such as noise removal and resolution adjustment on the video data acquired by the imaging device.

[1053] "Analysis" refers to the application of calculations and algorithms to use pre-processed video data to determine information such as object location and distance.

[1054] "Conversion to natural language" refers to the process of converting the information obtained from the analysis results into words that are easy for humans to understand.

[1055] "Synthesis into voice data" is a process of generating voice from information converted into natural language.

[1056] A "wireless method" is a means of sending and receiving data using wireless communication technologies such as Bluetooth and Wi-Fi.

[1057] "User" refers to a visually impaired person who uses this system to walk safely.

[1058] "Safety information" refers to the surrounding situation, the location and distance of obstacles, and other related information necessary for the user to move safely.

[1059] The following describes a specific embodiment of the present invention. This system is a walking aid that enables the visually impaired to move around safely and with peace of mind. The system includes a camera mounted on a portable measuring device, hardware and software for preprocessing the images, a built-in AI module for object detection and analysis, a natural language processing module, a speech synthesis engine, a wireless communication module, and wireless earphones worn by the user.

[1060] Specifically, the portable measurement device is equipped with a small camera that captures images of the surroundings in real time using OpenCV. The captured images are preprocessed using image processing techniques such as Gaussian filtering. The preprocessed image data is then sent to the built-in AI module using AI frameworks such as TensorFlow and YOLO.

[1061] The AI ​​module identifies the location and distance of objects (people, obstacles, traffic lights, etc.) from the video data. For example, it uses the YOLO (You Only Look Once) algorithm to detect objects in the video in real time and calculate their location coordinates and distance. The results of this analysis are passed to a natural language processing module (e.g., spaCy or NLTK), which generates specific language messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[1062] The generated linguistic message is then converted into voice data by a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly) and sent to wireless earphones via a Bluetooth module. By receiving this voice information in real time through the Bluetooth earphones, users can accurately grasp their surroundings and move safely.

[1063] Specific examples

[1064] When walking on the station platform

[1065] As a user walks along a station platform, the device's camera captures video of the surrounding area. After the video is preprocessed, an AI module detects when the edge of the platform is approaching or if a person is standing ahead. The analysis results are verbalized as "The edge of the platform is two meters away" or "There is a person three meters ahead." This verbalized information is then converted into audio data and transmitted to the user via Bluetooth earphones.

[1066] When crossing an intersection

[1067] When a user approaches an intersection, the device's camera captures the traffic light. After the video is preprocessed, the AI ​​module analyzes the traffic light status and determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and transmitted to the user's earphones.

[1068] Prompt Sentence Examples

[1069] By providing the following prompt to the generative AI model, it can generate a natural language message:

[1070] Example prompt 1:

[1071] Generate user-friendly messages based on the obstacle detection results.

[1072] Input: Distance to obstacle is 3 meters, object is a person

[1073] Output: There is a person 3 meters ahead

[1074] Example prompt 2:

[1075] Generate user-friendly messages based on the traffic light status.

[1076] Input: Traffic light status is red

[1077] Output: The light is red. Please wait until the light changes.

[1078] In this way, the system is designed to enable visually impaired people to accurately grasp their surroundings in real time.

[1079] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1080] Step 1:

[1081] The terminal captures the surroundings in real time using a camera mounted on a portable measuring device. The camera captures images at tens of frames per second and sends the image data to the next processing step.

[1082] Input: Surrounding image

[1083] Output: Raw captured video data

[1084] Step 2:

[1085] The device performs preprocessing on the captured video data. This preprocessing involves using a video processing library such as OpenCV to remove noise and adjust the resolution. For example, a Gaussian filter is applied to reduce image noise and convert the video to the optimal resolution.

[1086] Input: Raw captured video data

[1087] Output: Pre-processed video data

[1088] Step 3:

[1089] The device sends the preprocessed video data to the AI ​​module, which uses object detection algorithms such as TensorFlow and YOLO to identify objects in the video and determine their location and distance. For example, the YOLO algorithm can be used to generate bounding boxes for people and obstacles in the video and calculate distance from their center coordinates and size.

[1090] Input: Preprocessed video data

[1091] Output: Object position and distance information

[1092] Step 4:

[1093] The device passes the object position and distance information received from the AI ​​module to a natural language processing module. The natural language processing module (e.g., spaCy) is used to convert the analysis results into natural language. For example, the information "Person: 3 meters" is converted into a message such as "There is a person 3 meters ahead."

[1094] Input: Object position and distance information

[1095] Output: A natural language message

[1096] Step 5:

[1097] The device passes the natural language message generated by the natural language processing module to a speech synthesis engine. The speech synthesis engine (e.g., Google Text-to-Speech) is used to synthesize the text message into voice data. In this process, the text is converted into an audio file (e.g., WAV format).

[1098] Input: A natural language message

[1099] Output: Audio data

[1100] Step 6:

[1101] The terminal transmits the generated audio data to the wireless earphone via the Bluetooth module. The audio data is transferred to the earphone via Bluetooth communication and notified to the user in real time.

[1102] Input: Audio data

[1103] Output: Audio data sent to Bluetooth earphones

[1104] Step 7:

[1105] The user receives voice information transmitted through a Bluetooth earphone, and can confirm specific voice messages in real time, such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead."

[1106] Input: Audio data sent to Bluetooth earphones

[1107] Output: User receives audio information

[1108] Through these steps, visually impaired people can accurately grasp their surroundings in real time and move around safely.

[1109] (Application example 1)

[1110] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1111] Traditionally, visually impaired people have relied primarily on white canes and guide dogs to understand their surroundings when traveling. However, these methods have limited ability to detect specific dangers, making it difficult to obtain real-time information about new obstacles and moving objects and take immediate action. Particularly at intersections and in complex road environments, there has been a lack of information provided to enable safe travel.

[1112] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1113] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for acquiring surrounding situation data of the mobile object and generating safety information based on the analysis results, thereby enabling visually impaired people to grasp the surrounding situation in real time and move around safely.

[1114] "Mobility aids for the disabled" are aids used by visually impaired people to move around safely.

[1115] A "camera" is a photographing device for capturing images of the surroundings.

[1116] The "means for acquiring images" refers to a means for collecting image data of the surroundings in real time using a camera.

[1117] The "means for analyzing video" is a means for analyzing acquired video data and identifying the position and distance of an object.

[1118] The "means for identifying the position and distance of an object" is a means for calculating the spatial position and distance of a specific object based on the results of video analysis.

[1119] The "means for providing safety information by voice" is a means for notifying the user by voice of information generated based on the analysis results.

[1120] "Means for wirelessly communicating to a user" refers to means for transmitting audio information to a user's device using wireless communication technology such as Bluetooth or Wi-Fi.

[1121] "Means for acquiring surrounding situation data of a moving body" refers to means for sensing the surrounding environment of an autonomous vehicle or the like and collecting that data.

[1122] The "means for generating safety information based on the analysis results" is a means for analyzing data acquired from a mobile object and generating safety information that is useful to the user.

[1123] The system embodying this invention is used as an aid for visually impaired people to move around safely. Specifically, the system includes a camera mounted on the walking aid, an AI module for video analysis, a module for natural language processing, a speech synthesis engine, a Bluetooth module, and Bluetooth earphones.

[1124] The device's built-in camera captures real-time video of the surrounding area. This video data is then sent to an AI module after pre-processing, such as adjusting the resolution and removing noise, and analyzed to determine the location and distance of objects. Specifically, it identifies people, obstacles, traffic light status, and other objects in the video.

[1125] The information analyzed by the AI ​​module is passed to the natural language processing module, which generates a verbal message to notify the user, such as "There is a person 3 meters ahead" or "The light is red, do not cross."

[1126] The generated linguistic message is sent to a speech synthesis engine and converted into voice data, which is then transmitted to a Bluetooth earphone via a Bluetooth module, providing real-time notification to the user.

[1127] Furthermore, this system also has the ability to acquire data on the surrounding environment of the moving object and generate safety information based on that data. For example, the system can analyze the surrounding environment data acquired by an autonomous vehicle and notify the user, "A vehicle is approaching, please be careful."

[1128] The hardware used includes a camera, an AI module, a natural language processing module, a speech synthesis engine, and a Bluetooth module, while the software required is an ObjectDetectionModel (object detection algorithm), an NLPModel (natural language generation algorithm), and a TTSEngine (speech synthesis engine).

[1129] Specific examples

[1130] For example, when a user is walking on a train platform, the AI ​​module detects the edge of the platform or people ahead from the images captured by the device's camera, and generates messages based on that information, such as "The edge of the platform is two meters ahead" or "There is a person three meters ahead." This allows the user to move safely. In addition, the camera recognizes the status of traffic lights at intersections, and when the light is red, a message is displayed saying, "The light is red. Please wait until the light changes." This system allows visually impaired people to understand their surroundings in real time and move safely.

[1131] Prompt Sentence Examples

[1132] It visually monitors the status of traffic lights and the surrounding environment and provides real-time voice guidance to help drivers cross the crosswalk safely. Notifications include, "The light is green. Please cross safely." For example, a vehicle camera captures the traffic light, and an AI model identifies that the light is green. This information is sent to a natural language processing module, which generates a voice message saying, "The light is green," and sends it to a Bluetooth earphone.

[1133] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1134] Step 1:

[1135] The device uses a camera to capture images of the surroundings. The input is real-time image data, and the output is the raw image data.

[1136] Step 2:

[1137] Preprocesses the video data acquired by the device. Specifically, it removes noise and adjusts the resolution. The input is raw video data, and the output is preprocessed video data.

[1138] Step 3:

[1139] The preprocessed video data is sent to an AI module to detect objects and determine their location and distance. The input is the preprocessed video data, and the output is object location and distance information. Specifically, the AI ​​model recognizes people and obstacles in the video and calculates their respective positions and distances.

[1140] Step 4:

[1141] The device sends the object's position and distance information to the natural language processing module, which then generates a linguistic message to notify the user. The input is the object's position and distance information, and the output is a natural language message. Specifically, messages such as "There is a person 3 meters ahead" or "The light is red, do not cross" are generated.

[1142] Step 5:

[1143] The generated natural language message is passed to a speech synthesis engine and converted into audio data. The input is a natural language message, and the output is audio data. Specifically, the speech synthesis engine converts the text into an audio file.

[1144] Step 6:

[1145] The terminal transmits audio data to the user's Bluetooth earphone via the Bluetooth module. The input is audio data, and the output is audio notification to the Bluetooth earphone. Specifically, the Bluetooth module transmits audio data wirelessly to the earphone.

[1146] Step 7:

[1147] The user receives real-time audio information through a Bluetooth earphone. The input is the audio data sent to the Bluetooth earphone, and the output is audio notifications to the user. Specifically, the information conveyed to the user is "There is a person 3 meters ahead" or "The traffic light is red, do not cross."

[1148] Step 8:

[1149] The system acquires data on the surrounding situation of a moving object and generates safety information based on the analysis results. The input is the surrounding situation data from the moving object, and the output is safety information based on the analysis results. Specifically, it analyzes data acquired from the moving object's sensors and generates messages such as "A vehicle is approaching, please be careful."

[1150] Step 9:

[1151] The generated safety information is notified to the user as voice data. The input is a natural language message based on the analysis results, and the output is a safety information notification as voice data. Specifically, a voice synthesis engine converts the text into voice data and sends it to earphones via Bluetooth.

[1152] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1153] A specific embodiment of the present invention is described below. The system includes a camera mounted on a walking aid for the disabled (hereinafter referred to as a white cane), an AI module for video analysis, a module for natural language processing, a voice synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[1154] System configuration

[1155] The white cane is equipped with a small camera that captures images of the surroundings in real time. This image data is sent to the built-in AI module for video analysis. The AI ​​module identifies the location and distance of people and obstacles and sends the analysis results to the natural language processing module, which generates an appropriate verbal message. The emotion engine then recognizes the user's emotions and appropriately adjusts the voice information provided. Finally, the speech synthesis engine converts the generated verbal message into voice data, which is then sent to the user's Bluetooth earphones via the Bluetooth module.

[1156] Program processing flow

[1157] Video capture and pre-processing

[1158] The device (a camera mounted on a white cane) captures images of the surroundings in real time. This image data undergoes pre-processing such as noise reduction and resolution adjustment. The pre-processed image data is immediately sent to the AI ​​module.

[1159] Video data analysis

[1160] The device inputs the pre-processed video data into the AI ​​module for analysis. Specifically, it uses person detection and obstacle detection algorithms to recognize objects in the video and determine their location and distance. The AI ​​module can also identify the status of traffic lights and other specific objects.

[1161] Verbalizing information

[1162] The device passes the analysis results to a natural language processing module, which translates the information into words. The generated messages are specific, such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[1163] User Emotion Recognition

[1164] Along with verbalized information, the device recognizes the user's emotional state through an emotion engine, which uses voice tone, speed, skin conductance, and other physiological data to determine the user's emotional state (e.g., stress level or relief).

[1165] Adjusting audio information

[1166] The device adjusts the voice information as needed based on the output of the emotion engine. If the user is under high stress, the system will provide information in a calm and reassuring tone. The frequency and intensity of voice warnings will also be adjusted according to the stress level.

[1167] Generating and transmitting audio data

[1168] The device passes the adjusted language message to a speech synthesis engine to generate voice data, which is then sent to the user's earphones via a Bluetooth module.

[1169] User Notification

[1170] The user receives real-time voice information through a Bluetooth earphone. For example, specific voice messages such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" are conveyed to the user. Based on this voice information, the user can move safely.

[1171] Specific examples

[1172] When walking on the station platform

[1173] Consider a case where a user is walking on a train platform. The device's camera captures images of the surrounding area, and the AI ​​module detects that the edge of the platform is approaching or that a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing the user to travel with greater peace of mind.

[1174] When crossing an intersection

[1175] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[1176] This system allows visually impaired people to know what is happening around them in real time, enabling them to move around safely and with peace of mind.

[1177] The processing flow will be explained below.

[1178] Step 1:

[1179] The device (a camera mounted on a white cane) captures the surroundings. The camera continuously collects video data and passes it on to the next process in real time.

[1180] Step 2:

[1181] The device pre-processes the captured video data, which includes applying noise reduction filters and adjusting the resolution, converting the video data into a format suitable for the AI ​​analysis module.

[1182] Step 3:

[1183] The device provides preprocessed video data to the AI ​​module, which then uses the input data to run an object detection algorithm to identify the location and distance of people and obstacles. For example, it might detect a person standing three meters ahead and output that information.

[1184] Step 4:

[1185] The AI ​​module on the device identifies the state of a specific object (e.g., a traffic light). It detects that the traffic light is red and obtains that information.

[1186] Step 5:

[1187] The device passes the analysis results obtained from the AI ​​module to the natural language processing module, which uses this data to generate an appropriate verbal message to convey to the user, such as "There is a person 3 meters ahead" or "The traffic light is red, please wait."

[1188] Step 6:

[1189] The terminal temporarily stores the generated linguistic message, which is then sent to the emotion engine.

[1190] Step 7:

[1191] The device operates an emotion engine to recognize the user's emotions. The emotion engine analyzes physiological data such as the user's voice tone and skin conductance to determine the user's emotional state (e.g., high stress state).

[1192] Step 8:

[1193] The device adjusts the verbal messages as needed based on the user's recognized emotional state, for example, making the tone of the voice information gentler and adjusting the frequency of the messages if the user is in a high-stress state.

[1194] Step 9:

[1195] The device passes the adjusted language message to a speech synthesis engine to generate speech data, which is stored in temporary memory.

[1196] Step 10:

[1197] The device uses the Bluetooth module to transmit the generated audio data to the user's Bluetooth earphone, and transmits the data only after verifying that the Bluetooth connection is established.

[1198] Step 11:

[1199] The user receives real-time audio information through a Bluetooth earphone, such as "There is a person 3 meters ahead," "The traffic light is red, please wait," or "It's okay, please proceed calmly."

[1200] This series of processes allows visually impaired people to know the situation around them in real time, allowing them to move around safely and with peace of mind.

[1201] Example 2

[1202] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1203] For people with disabilities, especially those with visual impairments, to walk safely, it is important to quickly and accurately grasp information about their surroundings. However, current walking aids lack a system that can efficiently analyze video data acquired by a camera and provide the user with appropriate audio information. They also lack the ability to adjust the audio according to the user's psychological state and stress level. This makes it difficult for users to move around with a sufficient sense of security.

[1204] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for acquiring surrounding video using a camera mounted on a walking aid for disabled persons; means for preprocessing the acquired video; means for analyzing the preprocessed video and identifying the position and distance of an object; means for converting information on the position and distance of the identified object into natural language; means for recognizing the user's emotional state based on the converted natural language information; means for adjusting voice information based on the recognized emotional state; means for converting the adjusted voice information into voice data; and means for wirelessly transmitting the voice data to the user. This allows the user to grasp information about the surrounding environment in real time and receive voice guidance according to the user's psychological state at that time, allowing for safer and more secure travel.

[1205] A "walking aid for the disabled" is a tool used by disabled people, such as those with visual impairments, to assist them in walking.

[1206] A "camera" is a device that captures images and acquires them as digital data.

[1207] "Video" is data that represents a series of still images linked together over time to create a motion image.

[1208] "Preprocessing" refers to the process of performing initial processing such as noise removal and resolution adjustment on the acquired raw data.

[1209] An "AI module" is a software or hardware unit that uses artificial intelligence technology to analyze data and make inferences.

[1210] The "position and distance of an object" refers to the spatial coordinates of an object detected by the camera and the linear distance from the camera to the object.

[1211] "Natural language processing" refers to converting analyzed data into natural language that humans can understand.

[1212] A "natural language processing module" is a software module that performs text analysis, generation, etc.

[1213] "Emotional state" refers to the user's psychological and physiological state, including stress levels, feelings of security, etc.

[1214] An "emotion engine" is software that has the ability to recognize a user's emotional state and adjust the system's response based on that.

[1215] "Adjusting audio information" refers to changing the tone, speed, and frequency of audio depending on the user's emotional state.

[1216] "Speech data" refers to a speech signal generated based on text information in a natural language.

[1217] A "speech synthesis engine" is a software or hardware unit that converts text data into speech.

[1218] A "wireless system" is a method of transmitting and receiving data using wireless communication technology.

[1219] A "Bluetooth earphone" is a device worn in the ear that receives audio data wirelessly using Bluetooth technology.

[1220] The system of the present invention includes a camera mounted on a walking aid for the disabled, an artificial intelligence (AI) module for video analysis, a module for natural language processing, a speech synthesis engine, an emotion engine, a Bluetooth module, and a Bluetooth earphone.

[1221] The device (a camera mounted on a white cane) captures the surroundings in real time. This video data is first preprocessed to remove noise and adjust the resolution. Specifically, a Gaussian filter is applied using OpenCV to remove noise, and the resolution is adjusted to 640x480 using a resize method.

[1222] After preprocessing, the video data is immediately sent to the device's AI module. The AI ​​module uses deep learning libraries such as TensorFlow to recognize objects in the video. Specifically, it uses SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) to detect people and obstacles and determine the location and distance of each object. The AI ​​module can also identify the color of traffic lights (red or green).

[1223] The analysis results are sent to the device's natural language processing (NLP) module, which translates the information into natural language. This module uses libraries such as SpaCy to generate messages like "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead."

[1224] The device sends the generated linguistic message to the emotion engine, which collects physiological data from skin conductance sensors and heart rate sensors to determine the user's stress level and sense of security. For example, high skin conductance is recognized as a state of high stress. This emotional state information is used to adjust the voice information.

[1225] The device adjusts the tone and intensity of messages generated by the natural language processing module based on the output of the emotion engine. Specifically, if the user is under high stress, the tone of the voice will be adjusted to a calmer tone and the frequency of messages will be reduced.

[1226] The adjusted language message is passed to a speech synthesis engine, which uses APIs such as Google Text-to-Speech to convert the text data into speech data, which is then sent via a Bluetooth module to the user's Bluetooth earphones.

[1227] In this way, users can receive real-time voice information through Bluetooth earphones. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" can be conveyed to the user as a voice message, allowing the user to move safely.

[1228] Specific examples

[1229] When walking on the station platform

[1230] As a user walks along a station platform, the device's camera captures images of the surrounding area, and the AI ​​module identifies when the edge of the platform is approaching or when a person is standing ahead. The analyzed information is passed to a natural language processing module, which verbalizes it, saying things like, "The edge of the platform is two meters away," or "There is a person three meters ahead." This information is converted into audio data and transmitted to the user via Bluetooth earphones. The emotion engine also detects the user's stress level and adjusts the tone and intensity of audio warnings, allowing users to travel with greater peace of mind.

[1231] When crossing an intersection

[1232] When a user approaches an intersection, the device's camera captures the traffic light and the AI ​​module determines that the light is red. Messages such as "The light is red" and "Please wait until the light changes" are generated, converted into audio data, and sent to the user's earphones. The emotion engine evaluates the user's emotional state and adjusts the audio information as needed. For example, if the user is in a high-stress state, a reassuring tone may be used, such as "It's okay, just wait until the light turns green."

[1233] Example prompt sentence:

[1234] 1. Describe a scenario in which a user is walking along a train platform, and then describe how the system captures information about their surroundings and communicates it to the user.

[1235] 2. Describe a situation where a user is about to cross an intersection, and specify how the system captures the status of the traffic lights and notifies the user.

[1236] In this way, visually impaired people can learn about their surroundings in real time through the system, enabling them to move around safely and with peace of mind.

[1237] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1238] Specific processing flow of the system

[1239] Step 1:

[1240] The device (a camera mounted on a white cane) captures the surroundings in real time. The input is the surroundings video data, and the output is the captured raw video data. For example, the camera captures video at a rate of 30 frames per second with a resolution of 640x480 pixels. This video data is stored in temporary memory and goes to the next pre-processing step.

[1241] Step 2:

[1242] We perform preprocessing on raw video data acquired by the device camera. The input is raw video data, and the output is preprocessed video data. Specifically, we use OpenCV to remove noise using a Gaussian filter and apply it with the cv2.GaussianBlur function. We also use the cv2.resize function to convert it to 640x480 pixels to adjust the resolution.

[1243] Step 3:

[1244] Preprocessed video data is input into the device's AI module to identify the location and distance of objects. The input is preprocessed video data, and the output is object information (location and distance). Specifically, object detection is performed using TensorFlow's SSD or YOLO model, and the location and distance of each object is calculated. For example, the yolov3.detect method is used to identify people and obstacles and obtain their coordinates and distance.

[1245] Step 4:

[1246] Object information is input into the device's natural language processing module, and a linguistic message is generated. The input is object information, and the output is a natural language message. For example, SpaCy can be used to generate linguistic messages such as "There is a person 3 meters ahead" or "The edge of the platform is 2 meters ahead." In this case, grammatically correct sentences are constructed based on the object's position and distance.

[1247] Step 5:

[1248] The device's emotion engine collects physiological data to recognize the user's emotional state. The input is the user's physiological data (skin conductance, heart rate, etc.), and the output is the user's emotional state (stress level and relief). Specifically, it receives data from the skin conductance sensor in real time and analyzes it to determine the stress level. For example, a high skin conductance value is recognized as a high stress state.

[1249] Step 6:

[1250] The device adjusts the voice information based on the output of the emotion engine and the message from the natural language processing module. The input is the emotional state and the natural language message, and the output is the adjusted voice message. Specifically, the tone and speed of the voice are adjusted, and information is provided in a calmer tone to users in a high-stress state. For example, the tone can be set lower using the voice.adjust_tone method.

[1251] Step 7:

[1252] The device's speech synthesis engine inputs the adjusted language message and generates voice data. The input is the adjusted language message, and the output is voice data. Specifically, the Google Text-to-Speech API is used to convert text to voice. For example, the gTTS(text) method of the gTTS library is used to generate the voice file.

[1253] Step 8:

[1254] The device sends the generated audio data to the user's earphone via the Bluetooth module. The input is the audio data, and the output is the audio sent to the user's earphone. Specifically, the audio data is sent using the send method of the Bluetooth module. For example, the bluetooth.send_audio(data) method is used.

[1255] Step 9:

[1256] The user receives real-time audio information from the system via a Bluetooth earphone. The input is the audio transmitted to the Bluetooth earphone, and the output is the audio information recognized by the user. For example, specific information such as "There is a person 3 meters ahead" or "The end of the platform is 2 meters ahead" is conveyed by voice. This allows the user to move safely.

[1257] (Application example 2)

[1258] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1259] When visually impaired people move around public spaces, the many obstacles and dangers they encounter make it difficult for them to navigate safely. Furthermore, existing walking aids provide information without taking into account the user's emotional state, which can sometimes prevent them from reducing stress and anxiety. Furthermore, even in self-driving vehicles, visually impaired people have limited means of understanding their surroundings, reducing safety. There is a need for a system that can solve these issues and enable users to travel safely and with peace of mind.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1261] In this invention, the server includes means for acquiring surrounding images using a camera mounted on the walking aid for disabled people, means for analyzing the acquired images and identifying the position and distance of an object, means for providing audio safety information to the user based on the identified position and distance of the object, means for wirelessly transmitting the audio to the user, and means for recognizing the emotional state of the user and adjusting the audio information. This enables visually impaired people to move around safely and with peace of mind by receiving safety information in real time and providing appropriate audio information according to their emotional state.

[1262] A "walking aid for the disabled" is an auxiliary tool used by visually impaired people to move around safely, and is a device that has the function of detecting obstacles and surrounding conditions.

[1263] A "camera" is a photographing device that captures images of the surroundings and provides image data in real time.

[1264] "Video analysis" is a process that identifies information such as the position and distance of an object based on acquired video data, and typically uses AI technology.

[1265] "Determining the position and distance of an object" means measuring and understanding the position of surrounding objects and the distance to them through video analysis.

[1266] "Safety information" is information about the surrounding situation provided to the user, including the position and distance of obstacles, the status of traffic lights, and the like.

[1267] "Audio provision" is a means of transmitting analyzed information to the user by voice, allowing visually impaired people to receive information aurally.

[1268] "Wireless" refers to a method of transmitting data using wireless communication, such as transmitting audio to earphones using technology such as Bluetooth.

[1269] "Emotional state recognition" refers to monitoring a user's psychological and physiological state to identify emotions such as stress or tension.

[1270] "Adjusting voice information" refers to changing the tone and content of a voice message based on the user's emotional state so that the user receives appropriate information.

[1271] A "system" is a collection of devices and programs that combine the above elements and function to achieve a specific purpose.

[1272] This invention applies a system installed in a walking aid for the disabled to an autonomous vehicle, enabling visually impaired people to travel safely and with peace of mind inside the vehicle. Specific embodiments for carrying out this invention are described below.

[1273] System configuration

[1274] The autonomous vehicle is equipped with a camera to help visually impaired people understand their surroundings, a video analysis AI, a natural language processing module, a speech synthesis engine, an emotion engine, and Bluetooth earphones.

[1275] Hardware and Software

[1276] Camera: The onboard camera captures real-time footage of the surroundings.

[1277] Video analysis AI: Uses TensorFlow and OpenCV to analyze captured video data and identify the position and distance of objects.

[1278] Natural Language Processing module: Uses NLTK and spaCy to generate appropriate messages based on the analysis results.

[1279] Speech synthesis engine: Uses the Google Text-to-Speech API and Amazon Polly to convert text messages into audio data.

[1280] Emotion Engine: Uses the Affectiva SDK to recognize the user's emotional state and adjusts the tone and content of the voice based on that information.

[1281] Bluetooth module and earphones: A commercially available Bluetooth module is used to wirelessly transmit audio data to the user's earphones.

[1282] Processing flow

[1283] The server captures real-time video data captured by the camera and pre-processes the data. The pre-processed video data is analyzed using TensorFlow and OpenCV to determine the location and distance of objects. The analysis results are sent to a natural language processing module using NLTK and spaCy, which generates appropriate messages to be presented to the user.

[1284] The generated messages are adjusted by an emotion engine (Affectiva SDK) based on the user's emotional state, for example, if the user is in a high stress state, the tone of the voice will be changed to a calmer one.

[1285] Finally, a speech synthesis engine (Google Text-to-Speech API or Amazon Polly) is used to convert the text message into audio data, which is then sent to the user's earphones via a Bluetooth module.

[1286] Specific examples

[1287] Example of approaching an intersection

[1288] The camera captures the intersection.

[1289] Video analysis AI identifies traffic light conditions and approaching pedestrians.

[1290] The natural language processing module generates messages such as "The traffic light is red" and "There is a pedestrian ahead."

[1291] The emotion engine recognizes when the user is in a high stress state.

[1292] Generates a tone-adjusted message: "Please wait calmly until the traffic light changes."

[1293] The generated audio data is transmitted to the user through a Bluetooth earphone.

[1294] Prompt Sentence Examples

[1295] "Identify the traffic light status from the video of the intersection captured by the in-car camera, and if the light is red and there are pedestrians, generate a message saying, 'The light is red. There are pedestrians ahead.' Also, if the user is in a high-stress state, adjust the message to a more reassuring tone."

[1296] This invention allows visually impaired people to grasp information about their surroundings in real time even while inside an autonomous vehicle, enabling them to travel with peace of mind.

[1297] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1298] Step 1:

[1299] The camera captures the surrounding image data in real time. The scene and objects captured by the camera are used as input. The captured image data is then output.

[1300] Step 2:

[1301] The server receives the captured video data and performs pre-processing, which includes noise removal, resolution adjustment, and color correction. The raw captured video data is used as input, and the pre-processed, clean video data is obtained as output.

[1302] Step 3:

[1303] The server inputs the preprocessed video data into a video analysis AI (TensorFlow and OpenCV) for object recognition. The AI ​​identifies the location and distance of objects such as pedestrians, vehicles, and traffic lights. The preprocessed video data is used as input, and data on the location and distance of objects is obtained as output.

[1304] Step 4:

[1305] The server inputs the results of the video analysis AI into a natural language processing module (NLTK and spaCy), which converts the analysis results into a text message. For example, messages such as "There is a pedestrian ahead" or "The traffic light is red" are generated. Data on the object's position and distance is used as input, and a text message in natural language is obtained as output.

[1306] Step 5:

[1307] The server inputs the generated text message into the emotion engine (Affectiva SDK) to monitor the user's emotional state. The emotion engine evaluates the user's stress level, comfort level, etc., and adjusts the tone and speed of the voice message. The text message and the user's emotion data are used as input, and the adjusted text message is obtained as output.

[1308] Step 6:

[1309] The server inputs the adjusted text message into a speech synthesis engine (such as Google Text-to-Speech API or Amazon Polly) and converts it into voice data. For example, a voice message such as "The traffic light is red. There is a pedestrian ahead" is generated. The adjusted text message is used as input, and voice data is obtained as output.

[1310] Step 7:

[1311] The server sends the generated voice data to the terminal (earphone) via the Bluetooth module, allowing the user to receive information as voice in real time. Voice data is used as input, and as output, a voice message is transmitted to the user through the Bluetooth earphone.

[1312] In this way, visually impaired people will be able to receive real-time safety information and understand their surroundings even while inside an autonomous vehicle.

[1313] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1314] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1315] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1316] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1317] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1318] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1319] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1320] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1321] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1322] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1323] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1324] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1325] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1326] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1327] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1328] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1329] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1330] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1331] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1332] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1333] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1334] The following is further disclosed regarding the above embodiment.

[1335] (Claim 1)

[1336] A means for acquiring an image of the surroundings by a camera mounted on the walking aid for a disabled person;

[1337] means for analyzing the captured image and determining the position and distance of an object;

[1338] a means for providing a user with audio safety information based on the position and distance of the identified object;

[1339] means for wirelessly transmitting said audio to a user;

[1340] A system including:

[1341] (Claim 2)

[1342] 10. The system of claim 1, wherein the audio safety information includes the distance to a particular object and its condition.

[1343] (Claim 3)

[1344] 10. The system of claim 1, further comprising means for pre-processing video data acquired by the camera.

[1345] "Example 1"

[1346] (Claim 1)

[1347] a means for acquiring an image of the surroundings by an imaging device mounted on the portable measuring device;

[1348] means for preprocessing the acquired image to remove noise and adjust resolution;

[1349] means for analyzing the preprocessed image and determining the location and distance of an object;

[1350] means for converting the analysis result of the position and distance of the identified object into natural language;

[1351] means for synthesizing the converted natural language information into voice data;

[1352] means for wirelessly transmitting the generated voice data to a user;

[1353] A system including:

[1354] (Claim 2)

[1355] 10. The system of claim 1, wherein the audio safety information includes the distance to a particular object and its condition.

[1356] (Claim 3)

[1357] 10. The system of claim 1, further comprising means for pre-processing video data acquired by the camera.

[1358] "Application Example 1"

[1359] (Claim 1)

[1360] A means for acquiring an image of the surroundings by a camera mounted on the walking aid for a disabled person;

[1361] means for analyzing the captured image and determining the position and distance of an object;

[1362] a means for providing a user with audio safety information based on the position and distance of the identified object;

[1363] means for wirelessly transmitting said audio to a user;

[1364] means for acquiring surrounding situation data of the moving object and generating safety information based on the analysis results;

[1365] A system including:

[1366] (Claim 2)

[1367] 10. The system of claim 1, wherein the audio safety information includes the distance to a particular object and its condition.

[1368] (Claim 3)

[1369] 10. The system of claim 1, further comprising means for pre-processing video data acquired by the camera.

[1370] "Example 2: Combining Emotion Engines"

[1371] (Claim 1)

[1372] A means for acquiring an image of the surroundings by a camera mounted on the walking aid for a disabled person;

[1373] means for pre-processing the captured images;

[1374] means for analyzing the preprocessed image and determining the location and distance of an object;

[1375] a means for converting the information about the position and distance of the identified object into natural language;

[1376] means for recognizing the emotional state of the user based on the natural language information;

[1377] means for adjusting audio information based on the recognized emotional state;

[1378] means for converting the adjusted audio information into audio data;

[1379] means for wirelessly transmitting said audio data to a user;

[1380] A system including:

[1381] (Claim 2)

[1382] 10. The system of claim 1, wherein the audio safety information includes the distance to a particular object and its condition.

[1383] (Claim 3)

[1384] 10. The system of claim 1, further comprising means for pre-processing video data acquired by the camera.

[1385] "Application example 2 when combining emotion engines"

[1386] (Claim 1)

[1387] A means for acquiring an image of the surroundings by a camera mounted on the walking aid for a disabled person;

[1388] means for analyzing the captured image and determining the position and distance of an object;

[1389] a means for providing a user with audio safety information based on the position and distance of the identified object;

[1390] means for wirelessly transmitting said audio to a user;

[1391] means for recognizing the emotional state of the user and adjusting audio information;

[1392] A system including:

[1393] (Claim 2)

[1394] 10. The system of claim 1, wherein the audio safety information includes the distance to a particular object and its condition.

[1395] (Claim 3)

[1396] 10. The system of claim 1, further comprising means for pre-processing video data acquired by the camera. [Explanation of symbols]

[1397] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring an image of the surroundings by a camera mounted on the walking aid for a disabled person; means for analyzing the captured image and determining the position and distance of an object; a means for providing a user with audio safety information based on the position and distance of the identified object; means for wirelessly transmitting said audio to a user; A system including:

2. 2. The system of claim 1, wherein the audio safety information includes the distance to a particular object and its condition.

3. The system of claim 1 further comprising means for pre-processing video data acquired by said camera.

Images