System

Abstract

A system is provided.SOLUTION: A system comprising: sensor means for obtaining environmental information; preprocessing means for preprocessing the obtained environmental information; communication means for communicating the preprocessed environmental information; analysis means for analyzing the communicated environmental information; determination means for determining a risk factor based on an analysis result; feedback generation means for generating a feedback based on the determined risk factor; and presentation means for presenting the generated feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is extremely difficult for visually impaired people to walk safely, especially in urban areas where there are many obstacles and dangers. Conventional visual support systems have difficulty accurately grasping the surrounding environment and conveying information to the user in real time. This has resulted in insufficient support for visually impaired people's ability to move safely and independently in their daily lives. Therefore, there is a need for a system that enables visually impaired people to walk safely. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired data, a communication means for communicating the preprocessed data, an analysis means for analyzing the communicated data, a determination means for determining risk factors based on the analysis results, a feedback generation means for generating feedback based on the determined risk factors, and a presentation means for presenting the generated feedback.

[0006] Specifically, detailed information about the surroundings is collected using a camera, distance sensor, and audio capture means as sensor means. This data is compressed and noise-removed by a preprocessing means, and then sent to a server using a communication means. The server analyzes the data using an AI model as an analysis means and determines risk factors. Based on the determination means, a feedback generation means provides a warning to the user using means such as audio guidance, haptic signals, or vibration. This enables visually impaired people to grasp their surrounding environment in real time and move around safely.

[0007] "Surrounding environment information" is information such as objects, sounds, and distances around the user, which is acquired using a camera, distance sensor, sound acquisition means, and the like.

[0008] "Sensor means" refers to devices such as cameras, distance sensors, and audio acquisition means, and is a means for acquiring information about the surrounding environment.

[0009] The "preprocessing means" refers to a means for compressing the acquired environmental information and removing noise, thereby reducing the amount of data and enabling subsequent analysis to be carried out efficiently.

[0010] "Communication means" refers to the means for transmitting preprocessed data to the server, and primarily includes Wi-Fi and mobile data communications.

[0011] "Analysis means" refers to a means for receiving data transmitted by a communication means and analyzing the data using an AI model. The analysis means includes functions such as object recognition, voice recognition, and distance calculation.

[0012] The "determination means" is a means for detecting risk factors for the user and determining their importance based on the analysis results obtained by the analysis means.

[0013] The "feedback generating means" refers to a means for generating feedback in a format that is easy for the user to understand, based on the risk factors determined by the determining means.

[0014] "Presentation means" refers to means for conveying the generated feedback to the user, and includes audio guidance, tactile signals, vibrations, etc. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[0037] System Configuration

[0038] The system mainly consists of the following components:

[0039] 1. Terminal

[0040] Sensor means (camera, distance sensor, audio acquisition means)

[0041] Pretreatment means

[0042] communication means

[0043] Presentation means (audio guidance, tactile signals, vibration)

[0044] 2. Server

[0045] Analysis method (AI model)

[0046] means of judgment

[0047] Feedback Generation Method

[0048] System Operation Overview

[0049] 1. Obtaining information about the surrounding environment

[0050] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[0051] 2. Pretreatment

[0052] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[0053] 3. Data communication

[0054] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[0055] 4. Data Analysis and Judgment

[0056] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[0057] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[0058] 5. Feedback Generation and Submission

[0059] Server: Generates feedback in a format that is easy for the user to understand. Feedback data is sent to the device using voice guidance, haptic signals, vibrations, etc.

[0060] 6. Feedback Presentation

[0061] Device: Presents the received feedback to the user, such as audio guidance or vibration alerts.

[0062] Specific examples

[0063] Example 1: How to cross the crosswalk

[0064] 1. Obtaining information about the surrounding environment

[0065] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[0066] 2. Pretreatment

[0067] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[0068] 3. Data communication

[0069] The terminal transmits the pre-processed data to the server in real time.

[0070] 4. Data Analysis and Judgment

[0071] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[0072] The server determines that these elements are dangerous and concludes that the user needs to be instructed to stop them.

[0073] 5. Feedback Generation and Submission

[0074] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[0075] 6. Feedback Presentation

[0076] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[0077] This allows visually impaired users to cross the street safely. This system uses a similar process to adapt to various scenarios, providing powerful support for independent mobility for the visually impaired.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[0081] Step 2:

[0082] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[0083] Step 3:

[0084] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[0085] Step 4:

[0086] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[0087] Step 5:

[0088] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[0089] Step 6:

[0090] The server generates feedback based on the risk factors it has determined. Specifically, it generates voice guidance based on text information such as "The traffic light is red. Please stop" or "A car is approaching from the right. Be careful," and also sets vibration patterns depending on the level of urgency.

[0091] Step 7:

[0092] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[0093] Step 8:

[0094] The device presents the received feedback to the user. Specifically, it plays a voice message such as "The traffic light is red. Please stop" through the device's audio speaker, and simultaneously vibrates to warn the user.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] When visually impaired people walk or move around in their daily lives, they are unable to sense information about their surroundings, which can result in accidents and danger. Systems that solve these issues and support the safe walking and movement of visually impaired people are needed. Current technology struggles to provide accurate feedback in real time, and efficient information processing and communication methods are lacking.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means using a generative AI model as the analysis means and determining risk factors based on the analysis results, a feedback generation means for generating feedback based on the determined risk factors, and a presentation means for presenting the generated feedback, thereby enabling visually impaired people to accurately grasp their surroundings and move safely.

[0100] "Sensor means" refers to devices or techniques used to obtain information about the surrounding environment.

[0101] The "preprocessing means" refers to a device or technology that compresses the acquired environmental information and performs processing to remove noise.

[0102] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to the server, including Wi-Fi and mobile data communications.

[0103] "Analysis means" refers to devices or techniques for analyzing the communicated environmental information. In this invention, a generative AI model is used.

[0104] A "generative AI model" is an artificial intelligence model that analyzes data and detects patterns and features.

[0105] The "judgment means" refers to a device or technology that judges risk factors based on the results of the analysis means.

[0106] The "feedback generating means" refers to a device or technology that generates feedback to the user based on the determined risk factors.

[0107] The "presentation means" refers to a device or technology that presents the generated feedback to the user through audio guidance, tactile signals, vibrations, etc.

[0108] This invention relates to a system that assists visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. The components and operation of the system are described in detail below.

[0109] System Configuration

[0110] The system consists of the following components:

[0111] 1. Terminal

[0112] Sensor means (camera, distance sensor, audio acquisition means)

[0113] Pretreatment means

[0114] communication means

[0115] Presentation means (audio guidance, tactile signals, vibration)

[0116] 2. Server

[0117] Analysis method (generative AI model)

[0118] means of judgment

[0119] Feedback Generation Method

[0120] Operation overview

[0121] 1. Sensor means: The device uses cameras, distance sensors, and audio capture means to collect information about the surrounding environment, including information about obstacles, traffic lights, pedestrians, etc.

[0122] 2. Pre-processing: The terminal compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[0123] 3. Communication method: The device sends the preprocessed data to the server using Wi-Fi or mobile data communication.

[0124] 4. Analysis method: The server analyzes the received data using the generative AI model to detect dangerous elements. The generative AI model uses deep learning algorithms to analyze traffic light colors, approaching vehicles, etc.

[0125] 5. Judgment method: The server evaluates the risk factors based on the analysis results and determines the necessary feedback.

[0126] 6. Feedback generation means: The server generates feedback in a form that is easy for the user to understand, such as voice guidance, haptic signals, vibrations, etc.

[0127] 7. Presentation: The device presents the received feedback to the user. Possible methods include voice guidance and vibration alerts.

[0128] Specific examples

[0129] Example 1: How to cross the street

[0130] 1. Obtaining information about the surrounding environment

[0131] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[0132] 2. Pretreatment

[0133] The device preprocesses the image and distance data it acquires to reduce the amount of data and remove noise.

[0134] 3. Data communication

[0135] The terminal transmits the pre-processed data to the server in real time.

[0136] 4. Data Analysis and Judgment

[0137] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby. It determines this to be a danger and concludes that the user needs to stop.

[0138] 5. Feedback Generation and Submission

[0139] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[0140] 6. Feedback Presentation

[0141] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[0142] Examples of prompt statements

[0143] "Please explain the detailed processing steps of a system that helps visually impaired people cross the street."

[0144] "Please explain the specific operation of a system that detects traffic lights and vehicle movements and provides feedback."

[0145] In this way, the present invention provides an environment in which visually impaired people can walk safely, and provides detailed instructions on how to actually use the system. The system aims to improve walking safety by understanding the surrounding environment in real time and providing accurate feedback to the user.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] Obtaining information about the surrounding environment

[0149] The device acquires information about the surrounding environment using a camera, distance sensor, and audio acquisition means. For example, it collects information about the road the user is walking on, surrounding obstacles, traffic light status, and surrounding audio. This allows the device to acquire information about the user's surrounding environment in real time.

[0150] Input: Surrounding environment information (images, distance data, audio data)

[0151] Output: Raw data obtained

[0152] Step 2:

[0153] Pretreatment

[0154] The device preprocesses the raw data acquired in step 1. Unnecessary noise is removed from the camera image and important objects and information are highlighted. Data from the distance sensor is similarly processed to reduce noise. Audio data is appropriately filtered to highlight specific sounds (e.g., the sound of a car engine). This improves data quality and makes communication more efficient.

[0155] Input: Acquired raw data (images, range data, audio data)

[0156] Output: Preprocessed data (denoised data)

[0157] Step 3:

[0158] data communication

[0159] The device sends the preprocessed data to the server via a communication method. The data is sent in real time using Wi-Fi or mobile data. The data is appropriately compressed to ensure reliable communication.

[0160] Input: Preprocessed data

[0161] Output: Data sent to the server

[0162] Step 4:

[0163] Data analysis

[0164] The server receives the data sent in step 3 and analyzes it using the generative AI model. Deep learning algorithms are used to recognize traffic light status from image data and to identify approaching vehicles and obstacles from distance data. Audio data is also analyzed to detect important environmental sounds. This provides a clear overall picture of the surroundings.

[0165] Input: Data sent to the server

[0166] Output: Analysis results (traffic light status, obstacle location, audio importance)

[0167] Step 5:

[0168] Judgment

[0169] The server makes a decision based on the analysis results. For example, if the traffic light is red and there is an approaching car, it determines that it is dangerous for the user to start crossing the street. These decisions are made to ensure the user's safety.

[0170] Input: Analysis results

[0171] Output: Judgment result (specific action)

[0172] Step 6:

[0173] Feedback Generation

[0174] The server generates feedback based on the results of its judgment. The feedback is generated in a format that is easy for the user to understand (audio guidance, haptic signals, vibration, etc.). For example, audio guidance such as "The traffic light is red. Please stop" may be generated. The intensity of the vibration may also be changed to indicate the urgency of the warning.

[0175] Input: Judgment result

[0176] Output: Feedback data (audio guidance, haptic signals, vibration)

[0177] Step 7:

[0178] Feedback presentation

[0179] The device then presents the received feedback to the user, using the audio speaker to provide audio guidance and the haptic feedback device to emit vibrations, allowing the user to take appropriate action based on the surrounding situation.

[0180] Input: Feedback data

[0181] Output: Presentation to the user (audio guidance, haptic signals, vibration)

[0182] (Application example 1)

[0183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] Safe and efficient mobility in daily life is always an important issue for people with visual impairments. It is particularly difficult for them to accurately assess their surroundings and take appropriate action in busy urban environments, complex intersections, and situations involving interference from other pedestrians. Conventional methods are unable to provide sufficient information or warnings to address this issue, and a fundamental solution to ensure the safety of visually impaired people is needed.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0186] In this invention, the server includes sensor means for acquiring surrounding environmental information, preprocessing means for preprocessing the acquired environmental information, communication means for communicating the preprocessed environmental information, analysis means for analyzing the communicated environmental information, determination means for determining risk factors based on the analysis results, feedback generation means for generating feedback based on the determined risk factors, presentation means for presenting the generated feedback to the user by audio guidance and tactile signals or vibrations, and adjustment means for the system to adjust the content of the generated feedback based on the individual characteristics of the user. This provides an environment in which visually impaired people can travel safely and efficiently, and enables more appropriate guidance with feedback tailored to individual characteristics.

[0187] "Surrounding environmental information" refers to external situational data necessary for visually impaired people to walk, including obstacles, traffic lights, pedestrians, vehicles, etc.

[0188] "Sensor means" refers to an apparatus or device for acquiring information about the surrounding environment, and specifically includes a camera, a distance sensor, an audio acquisition means, and the like.

[0189] "Preprocessing means" refers to a process for compressing data and removing noise to make the acquired environmental information suitable for analysis.

[0190] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to an analytical device such as a server, and includes Wi-Fi and mobile data communications.

[0191] "Analysis means" refers to a device or software for analyzing the environmental information received on the server side and detecting risk factors.

[0192] "Determination means" refers to a device or software for evaluating the analysis results and determining safety and risk factors based on them.

[0193] "Feedback generating means" refers to a device or software for generating feedback to be provided to a user in response to a determined risk factor.

[0194] "Presentation means" refers to devices or technologies for conveying the generated feedback to the user, and specifically includes audio guidance, tactile signals, vibrations, etc.

[0195] "Adjustment means" refers to a device or software that allows the system to optimize the content of the generated feedback based on the individual characteristics of the user.

[0196] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[0197] System Configuration

[0198] The system mainly consists of the following components:

[0199] 1. Device:

[0200] Sensor means: Cameras, distance sensors, and audio capture means are used.

[0201] Preprocessing means: Compress the acquired environmental information and remove noise.

[0202] Communication method: Send the preprocessed data to the server.

[0203] Presentation methods: Provide feedback to the user via audio guidance, haptic signals, and vibration.

[0204] 2. Server:

[0205] Analysis method: Analyze data using AI models to detect risk factors.

[0206] Decision-making method: Evaluate the analysis results and determine the feedback required.

[0207] Feedback generation method: Generate feedback in a format that is easy for the user to understand.

[0208] Tailoring: Tailoring feedback based on the user's individual characteristics.

[0209] Operation overview

[0210] 1. Obtaining information about the surrounding environment:

[0211] The device's camera, distance sensor, and audio capture are used to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[0212] 2. Pretreatment:

[0213] The acquired environmental information is compressed and noise is removed, thereby reducing the amount of data and enabling efficient communication.

[0214] 3. Data communication:

[0215] The preprocessed data is sent to the server via a communication means, such as Wi-Fi or mobile data.

[0216] 4. Data Analysis and Decision-Making:

[0217] The server analyzes the transmitted data using an AI model as an analytical means to detect dangerous elements, such as approaching vehicles or red lights, and evaluates the analysis results based on the judgment means to generate necessary feedback.

[0218] 5. Feedback Generation and Submission:

[0219] The server generates feedback in a format that is easy for the user to understand and transmits it to the terminal using voice guidance, haptic signals, vibration, etc. The server also optimizes the content of the feedback based on the individual characteristics of the user using an adjustment means.

[0220] 6. Feedback suggestions:

[0221] The terminal presents the received feedback to the user by voice guidance, vibration, or the like.

[0222] Specific examples

[0223] Example 1: How to cross the street

[0224] When a user approaches a crosswalk, the device's camera captures the traffic light and surrounding conditions, and the distance sensor measures the position of the car ahead. This data is preprocessed and sent to the server. The server analyzes the data and detects that the traffic light is red and that a car is approaching nearby. Having determined this, the server generates a voice message such as "The traffic light is red. Please stop" and sends it to the device. The device then uses its audio speaker to provide guidance to the user, and simultaneously vibrates to alert the user.

[0225] Prompt Sentence Examples

[0226] I would like to develop a system that uses a camera to capture the surrounding environment in real time, detects dangers, and notifies visually impaired users with voice and vibration. This system uses AI to analyze the status of approaching vehicles and traffic lights and provides feedback. Please tell me the specific prompts and steps.

[0227] This system will provide an environment in which visually impaired people can move around safely and efficiently, and will also enable more appropriate guidance through feedback tailored to individual characteristics.

[0228] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0229] Step 1:

[0230] Obtaining information about the surrounding environment

[0231] Input: Device camera, distance sensor, and audio capture method

[0232] Data processing / calculation: The device uses these sensors to obtain real-time environmental information (images, distance data, audio)

[0233] Specifically, the camera captures traffic lights, pedestrians, and vehicle movements, the distance sensor measures the distance to objects, and the audio capture means collects surrounding sounds. The captured data is temporarily stored in the device.

[0234] Output: The raw environmental data (image data, distance data, audio data) is passed to the next step.

[0235] Step 2:

[0236] Pretreatment

[0237] Input: Raw environmental data obtained in step 1

[0238] Data processing / calculation: The device compresses the data and removes noise

[0239] Specifically, the device compresses and noise filters image data, normalizes distance data, and preprocesses audio data using noise reduction techniques.

[0240] Output: Compressed and denoised preprocessed data (compressed image, normalised distance data, denoised audio data) is passed to the next step.

[0241] Step 3:

[0242] data communication

[0243] Input: Preprocessed data from step 2

[0244] Data processing / calculation: The device sends pre-processed data to the server

[0245] Specifically, the device sends the preprocessed data to the server via Wi-Fi or mobile data, where the data is also packetized and encrypted.

[0246] Output: Preprocessed data sent to the server

[0247] Step 4:

[0248] Data analysis

[0249] Input: Preprocessed data sent to the server

[0250] Data processing / calculation: The server uses an AI model to analyze the received data and detect risk factors.

[0251] Specifically, the server inputs the received image, distance, and audio data into the AI ​​model to perform object recognition, determine traffic light status, and confirm the location of obstacles, for example, analyzing approaching vehicles and the status of traffic lights.

[0252] Output: Analysis results (e.g. approaching vehicle, red light)

[0253] Step 5:

[0254] Judgment

[0255] Input: Analysis results obtained in step 4

[0256] Data processing / calculation: The server evaluates safety based on the analysis results and determines the necessary feedback

[0257] Specifically, the server evaluates the analysis results based on AI decision-making logic and determines what kind of feedback is needed for the user, such as whether an emergency stop command is necessary.

[0258] Output: Feedback decision result (e.g., stop instruction)

[0259] Step 6:

[0260] Feedback Generation

[0261] Input: Feedback decision result from step 5

[0262] Data processing / calculation: The server generates feedback in a format that is easy for the user to understand.

[0263] Specifically, the server generates text data for voice guidance and also designs vibration patterns. The voice guidance generates messages such as "Please stop."

[0264] Output: Feedback data (voice message, vibration pattern)

[0265] Step 7:

[0266] Feedback presentation

[0267] Input: Feedback data generated in step 6

[0268] Data processing / calculation: The device presents feedback data to the user

[0269] Specifically, the device will use a speaker to play voice guidance and vibrate in sync, allowing the user to receive feedback and take appropriate action.

[0270] Output: Provide feedback to the user (audio guidance, vibration)

[0271] The above is the specific processing flow of the system that realizes the application example.

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] This invention is a system that supports visually impaired people to walk safely and with peace of mind. In particular, it has the function of recognizing the user's emotional state and adjusting the content of feedback based on that state. Detailed embodiments of this system are described below.

[0274] System Configuration

[0275] The system mainly consists of the following components:

[0276] 1. Terminal

[0277] Sensor means (camera, distance sensor, audio acquisition means)

[0278] Pretreatment means

[0279] communication means

[0280] Presentation means (audio guidance, tactile signals, vibration)

[0281] Emotion Recognition Engine

[0282] 2. Server

[0283] Analysis method (AI model)

[0284] means of judgment

[0285] Feedback Generation Method

[0286] System Operation Overview

[0287] 1. Obtaining information about the surrounding environment

[0288] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[0289] 2. Pretreatment

[0290] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[0291] 3. Data communication

[0292] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[0293] 4. Data Analysis and Judgment

[0294] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[0295] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[0296] 5. Emotion recognition

[0297] Device: The emotion recognition engine analyzes voice data and the user's facial expression data to recognize the user's emotional state in real time, thereby determining whether the user is stressed or relaxed.

[0298] 6. Feedback Generation and Submission

[0299] Server: Adjusts the content of the feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates additional voice guidance to help them relax. This is also sent in combination with the regular danger avoidance feedback.

[0300] The server generates feedback and sends it to the device.

[0301] 7. Feedback Presentation

[0302] The device then presents the received feedback to the user. Specifically, it plays guidance such as "The traffic light is red. Please stop" or "Take a deep breath and relax" through the device's audio speaker, and simultaneously vibrates to warn the user.

[0303] Specific examples

[0304] Example 1: A user experiencing stress

[0305] 1. Obtaining information about the surrounding environment

[0306] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[0307] 2. Pretreatment

[0308] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[0309] 3. Data communication

[0310] The terminal transmits the pre-processed data to the server in real time.

[0311] 4. Data Analysis and Judgment

[0312] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[0313] The server determines that these elements are dangerous and that the user needs to be instructed to stop them.

[0314] 5. Emotion recognition

[0315] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect when they are in a state of tension.

[0316] 6. Feedback Generation and Submission

[0317] In addition to the instruction "The traffic light is red, please stop," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the device.

[0318] 7. Feedback Presentation

[0319] The device will provide a voice prompt saying, "The traffic light is red. Please stop," along with a vibration warning. It will also provide additional relaxation guidance, saying, "Take a deep breath and relax."

[0320] In this way, by taking into account the user's emotional state, the system can not only avoid danger but also provide psychological support, enabling visually impaired users to move about in their daily lives with greater peace of mind.

[0321] The processing flow will be explained below.

[0322] Step 1:

[0323] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[0324] Step 2:

[0325] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[0326] Step 3:

[0327] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[0328] Step 4:

[0329] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[0330] Step 5:

[0331] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[0332] Step 6:

[0333] The device uses an emotion recognition engine to assess the user's emotional state. Specifically, it analyzes voice data and facial expression data to recognize stress, tension, relaxation, etc.

[0334] Step 7:

[0335] The server generates feedback based on the user's emotional state. Specifically, it generates instructions for avoiding danger (e.g., "The traffic light is red. Please stop") as well as feedback encouraging relaxation (e.g., "Take a deep breath and relax")

[0336] Step 8:

[0337] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[0338] Step 9:

[0339] The device then presents the received feedback to the user. Specifically, the device will announce "The traffic light is red. Please stop" through its audio speaker, and will also vibrate to warn the user. The device will also provide additional relaxation guidance, such as "Take a deep breath and relax."

[0340] Example 2

[0341] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0342] Conventional support systems for the visually impaired do not take into account the user's emotional state, and therefore are insufficient in providing support in stressful situations. In addition to acquiring environmental information, there is a need for feedback that reflects the user's psychological state.

[0343] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means for determining risk factors based on the analysis results, an emotion recognition means for recognizing the emotional state of the user based on the risk factors, a feedback generation means for generating feedback based on the determined risk factors and the emotional state, and a presentation means for presenting the generated feedback. This enables safe and secure walking assistance that takes the emotional state of the user into consideration.

[0344] "Surrounding environment information" refers to information about objects and situations that exist around the user, including obstacles, traffic lights, pedestrians, vehicles, and the like.

[0345] "Sensor means" refers to a hardware device for acquiring information about the surrounding environment, and includes a camera, a distance sensor, an audio acquisition means, and the like.

[0346] "Preprocessing means" refers to a processing mechanism for compressing acquired raw data and removing noise to reduce the amount of data and enable efficient communication.

[0347] "Communication means" refers to the mechanism for transmitting preprocessed data to the server, and Wi-Fi or mobile data communication is used.

[0348] "Analysis means" refers to the means for analyzing the data transmitted within the server, and uses AI models and machine learning algorithms.

[0349] The "judging means" refers to a means for evaluating and judging risk factors based on the analysis results obtained by the analyzing means.

[0350] "Emotion recognition means" refers to means for recognizing the user's emotional state, and evaluates the user's stress and relaxation state from voice data and facial expression data.

[0351] "Feedback generating means" refers to means for generating feedback to provide to a user based on the determined risk factors and emotional state.

[0352] "Presentation means" refers to means for conveying the generated feedback to the user, and includes audio guidance, tactile signals, and vibrations.

[0353] This invention relates to a system that supports visually impaired people to walk safely and with peace of mind. It is particularly characterized by its ability to recognize the user's emotional state and adjust feedback based on that state. This system is comprised of a terminal and a server, and specific embodiments of the system are described in detail below.

[0354] System Configuration

[0355] 1. Terminal

[0356] Sensor means: Cameras, distance sensors, and audio capture means are used to acquire information about the surrounding environment, such as obstacles, traffic lights, pedestrians, and vehicles.

[0357] Preprocessing means: Compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[0358] Communication method: Preprocessed data is sent to the server via Wi-Fi or mobile data communication.

[0359] Emotion recognition means: Analyzes voice data and facial expression data to recognize the user's emotional state in real time.

[0360] Presentation methods: Use audio prompts, haptic signals, and vibrations to provide feedback to the user.

[0361] 2. Server

[0362] Analysis method: Using AI models and machine learning algorithms, data sent from the device is analyzed to detect risk factors.

[0363] Judgment method: Based on the analysis results, evaluate risk factors and determine the need for feedback.

[0364] Feedback generation means: Generates appropriate feedback based on the determined risk factors and the user's emotional state.

[0365] Operation overview

[0366] 1. Obtaining information about the surrounding environment

[0367] The device uses a camera, distance sensors, and audio capture means to collect information about the user's surrounding environment, such as the status of traffic lights, approaching vehicles, and ambient noise.

[0368] 2. Pretreatment

[0369] The raw data acquired by the terminal is compressed and noise is removed, thereby improving communication efficiency.

[0370] 3. Data communication

[0371] The terminal transmits the preprocessed data to the server in real time.

[0372] 4. Data Analysis

[0373] The server analyzes the received data using an analysis means to detect dangerous elements, specifically by checking the color of traffic lights and approaching vehicles.

[0374] 5. Emotion recognition

[0375] The device analyzes voice and facial expression data to recognize the user's emotional state in real time, making it possible to determine whether the user is feeling stressed.

[0376] 6. Generate feedback

[0377] The server generates appropriate feedback for the user based on the analysis results and the user's emotional state. For example, if the traffic light is red, the server will provide a message saying, "The light is red. Please stop," and if the user is feeling stressed, the server will provide an additional message saying, "Take a deep breath and relax."

[0378] 7. Providing Feedback

[0379] The device will provide the generated feedback to the user via voice guidance or vibration, allowing the user to know the exact situation and take appropriate action, allowing them to act with peace of mind.

[0380] Specific examples

[0381] Text format

[0382] Please explain the detailed processing steps of a system that acquires information about the surrounding environment, recognizes the user's emotional state, and generates appropriate feedback when a visually impaired person approaches a crosswalk.

[0383] This system not only avoids danger but also provides comprehensive support that takes into account the user's emotional state, making it a technology that will enable visually impaired people to live their daily lives more safely and with peace of mind.

[0384] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0385] Step 1:

[0386] Obtaining information about the surrounding environment

[0387] The device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. As input, it acquires the current surrounding image, distance to objects, and environmental sound.

[0388] Specific operation: The device's camera captures the road conditions ahead, the distance sensor measures the distance to objects ahead, and the audio capture means collects surrounding sounds.

[0389] Output: Collected video data, distance data, and audio data are generated.

[0390] Step 2:

[0391] Data Preprocessing

[0392] The device processes the acquired raw data, using as input the video, distance, and audio data collected in step 1.

[0393] Specific operations: Compresses video data, cuts unnecessary data, and removes noise from audio data. Distance data is also simplified.

[0394] Output: Compressed video data, noise-removed audio data, and simplified distance data are generated.

[0395] Step 3:

[0396] Sending data

[0397] The device sends the preprocessed data to the server, using the compressed, denoised and simplified data generated in step 2 as input.

[0398] Specific operation: Preprocessed data is sent to the server in real time using Wi-Fi or mobile data communication.

[0399] Output: Generates data for analysis that is sent to the server.

[0400] Step 4:

[0401] Data analysis

[0402] The server parses the received data, using as input the parsing data sent in step 3.

[0403] Specific operation: Using an AI model, the color of the traffic light is recognized from the transmitted video data, and the presence or absence of approaching vehicles is analyzed from the distance data.

[0404] Output: The analysis results include the traffic light status (red / green) and information on approaching vehicles.

[0405] Step 5:

[0406] Determining risk factors

[0407] The server determines risk factors based on the analysis results, using the analysis results generated in step 4 as input.

[0408] Specific operation: Based on the analysis results, nearby obstacles and dangerous elements are evaluated and the level of danger is determined.

[0409] Output: The presence or absence of risk factors and detailed information about them is generated.

[0410] Step 6:

[0411] Recognition of emotional states

[0412] The device recognizes the user's emotional state using voice and facial expression data as input.

[0413] Specific operation: Using an AI model, the system analyzes the user's stress level from their voice and assesses their emotional state from facial expression data.

[0414] Output: The user's emotional state (stress level and relaxation level) is generated.

[0415] Step 7:

[0416] Generate feedback

[0417] The server generates feedback based on the risk factors and emotional state, using the risk factor information from step 5 and the emotional state from step 6 as input.

[0418] Specific behavior: In addition to basic instructions for dangerous elements, the system generates additional feedback depending on the user's emotional state. For example, when the traffic light is red, the system generates the instruction "The light is red. Please stop" along with the guidance "Take a deep breath and relax."

[0419] Output: A generated feedback message is generated.

[0420] Step 8:

[0421] Providing feedback

[0422] The device presents the generated feedback to the user, using as input the feedback message generated in step 7.

[0423] Specific operation: A feedback message is played using voice guidance means, and a warning is given to the user using vibration means.

[0424] Output: User received feedback, improved safety and security provided.

[0425] (Application example 2)

[0426] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0427] When visually impaired people shop in a physical store, it is difficult for them to accurately and quickly obtain information such as obstacles, product information, and store maps. In particular, when users are in a tense or stressed emotional state, there is a need for support for safer and more secure movement. To address these challenges, there is a need to provide a system that enables visually impaired people to move around a store safely, securely, and efficiently and obtain the information they need.

[0428] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0429] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, feedback adjustment means for adjusting the feedback content based on the emotional state, and presentation means for presenting the generated feedback, thereby enabling feedback according to the emotional state to be provided, allowing the user to move around the store with peace of mind and obtain the information they need.

[0430] "Surrounding environmental information" is a general term for visual and audio information including obstacles, traffic lights, pedestrians, store information, etc. in the area where a visually impaired person is located.

[0431] "Sensor means" refers to a device such as a camera, distance sensor, or audio capture device that is used to acquire information about the surrounding environment.

[0432] The "preprocessing means" is a means for compressing the acquired environmental information and removing noise to reduce the amount of data and enable efficient communication.

[0433] "Communication means" refers to a means for transmitting preprocessed environmental information to an analysis device such as a server, and mainly includes Wi-Fi and mobile data communication.

[0434] "Analysis means" refers to a means for analyzing the communicated environmental information, and is usually a device or method that uses an AI model to analyze the environmental information and extract risk factors and necessary information.

[0435] "Determination means" refers to a means for determining risk factors and other important factors based on the analysis results.

[0436] The "feedback generating means" is a means for generating feedback to be provided to the user based on the determined risk factors and the like.

[0437] The "presentation means" refers to a means for presenting the generated feedback to the user, and mainly includes voice guidance, tactile signals, vibrations, etc.

[0438] The "emotion recognition means" is a means for analyzing the user's voice, facial expressions, etc., and recognizing the user's emotional state in real time.

[0439] The "feedback adjustment means" is a means for appropriately adjusting the content of the feedback based on the recognized emotional state of the user.

[0440] The present invention provides a system that supports visually impaired people to move around and shop in a physical store safely and with peace of mind. Detailed embodiments are described below.

[0441] System Configuration

[0442] The system mainly consists of the following components:

[0443] 1. Terminal

[0444] Sensor means: camera, distance sensor, audio acquisition means

[0445] Preprocessing means: A device that compresses the acquired environmental information and removes noise

[0446] Communication methods: Wi-Fi and mobile data communication devices

[0447] Emotion recognition means: An engine that analyzes voice data and user facial expression data to recognize emotional states.

[0448] Presentation methods: audio guidance, tactile signals, vibration

[0449] 2. Server

[0450] Analysis method: A method for analyzing environmental information using an AI model and extracting necessary information

[0451] Judgment method: A method for determining risk factors and important information from the analysis results

[0452] Feedback generation method: A method for generating feedback based on the judged content

[0453] Feedback adjustment means: A means for adjusting the feedback content based on the user's emotional state

[0454] System Operation Overview

[0455] 1. Obtaining information about the surrounding environment

[0456] The device's camera, distance sensors, and audio capture gather information about the surrounding environment, including product location, obstacles in the store, and the movements of other customers.

[0457] 2. Pretreatment

[0458] The acquired environmental information is compressed and noise is removed at the device, reducing the amount of data and enabling more efficient communication and analysis.

[0459] 3. Data communication

[0460] The pre-processed data is sent to a server using Wi-Fi or mobile data.

[0461] 4. Data Analysis and Judgment

[0462] The server uses an AI model to analyze the received data, detect the location of products in the store and the presence of obstacles, and determine which products the user is approaching and what action they should take.

[0463] 5. Emotion recognition

[0464] The device's emotion recognition engine analyzes the user's voice and facial expression data to recognize their emotional state in real time, thereby determining whether they are stressed or relaxed.

[0465] 6. Feedback Generation and Submission

[0466] The server generates feedback and sends it to the device, including additional voice prompts to help the user relax if they are feeling stressed.

[0467] 7. Feedback Presentation

[0468] The device then presents the received feedback to the user through a voice speaker or vibration, for example, "There is a sweets section on this shelf."

[0469] Example

[0470] Example 1: User in a stressed state

[0471] 1. Obtaining information about the surrounding environment

[0472] When a user approaches a product shelf, the device's camera captures the entire shelf and product information, and a distance sensor measures the distance to the shelf.

[0473] 2. Pretreatment

[0474] The acquired image and distance data are preprocessed to remove noise and improve communication efficiency.

[0475] 3. Data communication

[0476] The preprocessed data is sent to the server.

[0477] 4. Data Analysis and Judgment

[0478] The server analyzes the received data and determines that it is necessary to provide the user with information about products on the shelves.

[0479] 5. Emotion recognition

[0480] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect their level of tension.

[0481] 6. Feedback Generation and Submission

[0482] In addition to the instruction "There is a sweets section on this shelf," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the terminal.

[0483] 7. Feedback Presentation

[0484] The device will provide a voice prompt saying, "There is a sweets section on this shelf," and will also vibrate to warn you. It will also provide additional relaxation guidance such as, "Take a deep breath and relax."

[0485] Examples of prompt statements

[0486] Example prompt sentence:

[0487] "Environmental information: product shelves. User is in a state of tension. Analyze the data and generate appropriate feedback."

[0488] "Feedback request: User is approaching a product shelf and is in a stressed state. Please analyze data and generate appropriate feedback."

[0489] In this way, by adjusting the feedback provided depending on the user's emotional state, it is possible to create a system that allows visually impaired users to move around and shop in a physical store with greater confidence.

[0490] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0491] Step 1:

[0492] Obtaining information about the surrounding environment

[0493] As a user moves around a physical store, the device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. Specifically, the camera captures product location and shelf information, the distance sensor measures the distance to the product shelf, and the audio capture means records surrounding audio information. The input is visual information, distance information, and audio information, and the output is integrated data of this information.

[0494] Step 2:

[0495] Pretreatment

[0496] The device preprocesses the acquired environmental information. Specifically, it resizes the acquired video data and removes noise. Similarly, it performs noise filtering on the distance data to improve its accuracy. It also performs noise reduction on the audio data. The input is the integrated data acquired in step 1, and the output is the compressed and noise-removed data.

[0497] Step 3:

[0498] data communication

[0499] The preprocessed data is sent from the device to a server. Specifically, the compressed and denoised data is sent to the server in real time using Wi-Fi or mobile data. The input is the preprocessed data, and the output is the data sent to the server.

[0500] Step 4:

[0501] Data analysis and judgment

[0502] The server analyzes the received data. Specifically, it uses an AI model to analyze the visual, distance, and audio information just sent to identify the presence of obstacles and product information. It also determines which product information in the store should be provided based on the user's situation. The input is the data sent from the device, and the output is the analysis and judgment results.

[0503] Step 5:

[0504] emotion recognition

[0505] The device acquires the user's voice and facial expression data and analyzes it using an emotion recognition engine. Specifically, it identifies the user's emotional state (e.g., stress, tension, relaxation) in real time. The input is the user's voice and facial expression data, and the output is the recognized emotional state.

[0506] Step 6:

[0507] Feedback generation and submission

[0508] The server generates feedback based on the results of data analysis and emotion recognition. Specifically, it generates additional guidance, such as product information and warning information required by the user, as well as relaxation guidance according to the user's emotional state. The generated feedback is sent to the device in real time. The input is the analysis results and emotion recognition results, and the output is the generated feedback.

[0509] Step 7:

[0510] Feedback presentation

[0511] The device presents the received feedback to the user. Specifically, it conveys information to the user using voice guidance or haptic signals (vibrations). For example, it provides voice guidance such as "There is a sweets corner on this shelf" or relaxation guidance such as "Take a deep breath and relax." The input is feedback from the server, and the output is information presented to the user.

[0512] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0513] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0514] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0515] [Second embodiment]

[0516] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0517] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0518] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0519] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0520] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0521] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0522] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0523] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0524] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0525] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0526] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0527] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0528] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[0529] System Configuration

[0530] The system mainly consists of the following components:

[0531] 1. Terminal

[0532] Sensor means (camera, distance sensor, audio acquisition means)

[0533] Pretreatment means

[0534] communication means

[0535] Presentation means (audio guidance, tactile signals, vibration)

[0536] 2. Server

[0537] Analysis method (AI model)

[0538] means of judgment

[0539] Feedback Generation Method

[0540] System Operation Overview

[0541] 1. Obtaining information about the surrounding environment

[0542] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[0543] 2. Pretreatment

[0544] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[0545] 3. Data communication

[0546] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[0547] 4. Data Analysis and Judgment

[0548] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[0549] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[0550] 5. Feedback Generation and Submission

[0551] Server: Generates feedback in a format that is easy for the user to understand. Feedback data is sent to the device using voice guidance, haptic signals, vibrations, etc.

[0552] 6. Feedback Presentation

[0553] Device: Presents the received feedback to the user, such as audio guidance or vibration alerts.

[0554] Specific examples

[0555] Example 1: How to cross the crosswalk

[0556] 1. Obtaining information about the surrounding environment

[0557] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[0558] 2. Pretreatment

[0559] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[0560] 3. Data communication

[0561] The terminal transmits the pre-processed data to the server in real time.

[0562] 4. Data Analysis and Judgment

[0563] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[0564] The server determines that these elements are dangerous and concludes that the user needs to be instructed to stop them.

[0565] 5. Feedback Generation and Submission

[0566] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[0567] 6. Feedback Presentation

[0568] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[0569] This allows visually impaired users to cross the street safely. This system uses a similar process to adapt to various scenarios, providing powerful support for independent mobility for the visually impaired.

[0570] The processing flow will be explained below.

[0571] Step 1:

[0572] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[0573] Step 2:

[0574] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[0575] Step 3:

[0576] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[0577] Step 4:

[0578] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[0579] Step 5:

[0580] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[0581] Step 6:

[0582] The server generates feedback based on the risk factors it has determined. Specifically, it generates voice guidance based on text information such as "The traffic light is red. Please stop" or "A car is approaching from the right. Be careful," and also sets vibration patterns depending on the level of urgency.

[0583] Step 7:

[0584] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[0585] Step 8:

[0586] The device presents the received feedback to the user. Specifically, it plays a voice message such as "The traffic light is red. Please stop" through the device's audio speaker, and simultaneously vibrates to warn the user.

[0587] Example 1

[0588] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0589] When visually impaired people walk or move around in their daily lives, they are unable to sense information about their surroundings, which can result in accidents and danger. Systems that solve these issues and support the safe walking and movement of visually impaired people are needed. Current technology struggles to provide accurate feedback in real time, and efficient information processing and communication methods are lacking.

[0590] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0591] In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means using a generative AI model as the analysis means and determining risk factors based on the analysis results, a feedback generation means for generating feedback based on the determined risk factors, and a presentation means for presenting the generated feedback, thereby enabling visually impaired people to accurately grasp their surroundings and move safely.

[0592] "Sensor means" refers to devices or techniques used to obtain information about the surrounding environment.

[0593] The "preprocessing means" refers to a device or technology that compresses the acquired environmental information and performs processing to remove noise.

[0594] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to the server, including Wi-Fi and mobile data communications.

[0595] "Analysis means" refers to devices or techniques for analyzing the communicated environmental information. In this invention, a generative AI model is used.

[0596] A "generative AI model" is an artificial intelligence model that analyzes data and detects patterns and features.

[0597] The "judgment means" refers to a device or technology that judges risk factors based on the results of the analysis means.

[0598] The "feedback generating means" refers to a device or technology that generates feedback to the user based on the determined risk factors.

[0599] The "presentation means" refers to a device or technology that presents the generated feedback to the user through audio guidance, tactile signals, vibrations, etc.

[0600] This invention relates to a system that assists visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. The components and operation of the system are described in detail below.

[0601] System Configuration

[0602] The system consists of the following components:

[0603] 1. Terminal

[0604] Sensor means (camera, distance sensor, audio acquisition means)

[0605] Pretreatment means

[0606] communication means

[0607] Presentation means (audio guidance, tactile signals, vibration)

[0608] 2. Server

[0609] Analysis method (generative AI model)

[0610] means of judgment

[0611] Feedback Generation Method

[0612] Operation overview

[0613] 1. Sensor means: The device uses cameras, distance sensors, and audio capture means to collect information about the surrounding environment, including information about obstacles, traffic lights, pedestrians, etc.

[0614] 2. Pre-processing: The terminal compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[0615] 3. Communication method: The device sends the preprocessed data to the server using Wi-Fi or mobile data communication.

[0616] 4. Analysis method: The server analyzes the received data using the generative AI model to detect dangerous elements. The generative AI model uses deep learning algorithms to analyze traffic light colors, approaching vehicles, etc.

[0617] 5. Judgment method: The server evaluates the risk factors based on the analysis results and determines the necessary feedback.

[0618] 6. Feedback generation means: The server generates feedback in a form that is easy for the user to understand, such as voice guidance, haptic signals, vibrations, etc.

[0619] 7. Presentation: The device presents the received feedback to the user. Possible methods include voice guidance and vibration alerts.

[0620] Specific examples

[0621] Example 1: How to cross the street

[0622] 1. Obtaining information about the surrounding environment

[0623] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[0624] 2. Pretreatment

[0625] The device preprocesses the image and distance data it acquires to reduce the amount of data and remove noise.

[0626] 3. Data communication

[0627] The terminal transmits the pre-processed data to the server in real time.

[0628] 4. Data Analysis and Judgment

[0629] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby. It determines this to be a danger and concludes that the user needs to stop.

[0630] 5. Feedback Generation and Submission

[0631] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[0632] 6. Feedback Presentation

[0633] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[0634] Examples of prompt statements

[0635] "Please explain the detailed processing steps of a system that helps visually impaired people cross the street."

[0636] "Please explain the specific operation of a system that detects traffic lights and vehicle movements and provides feedback."

[0637] In this way, the present invention provides an environment in which visually impaired people can walk safely, and provides detailed instructions on how to actually use the system. The system aims to improve walking safety by understanding the surrounding environment in real time and providing accurate feedback to the user.

[0638] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0639] Step 1:

[0640] Obtaining information about the surrounding environment

[0641] The device acquires information about the surrounding environment using a camera, distance sensor, and audio acquisition means. For example, it collects information about the road the user is walking on, surrounding obstacles, traffic light status, and surrounding audio. This allows the device to acquire information about the user's surrounding environment in real time.

[0642] Input: Surrounding environment information (images, distance data, audio data)

[0643] Output: Raw data obtained

[0644] Step 2:

[0645] Pretreatment

[0646] The device preprocesses the raw data acquired in step 1. Unnecessary noise is removed from the camera image and important objects and information are highlighted. Data from the distance sensor is similarly processed to reduce noise. Audio data is appropriately filtered to highlight specific sounds (e.g., the sound of a car engine). This improves data quality and makes communication more efficient.

[0647] Input: Acquired raw data (images, range data, audio data)

[0648] Output: Preprocessed data (denoised data)

[0649] Step 3:

[0650] data communication

[0651] The device sends the preprocessed data to the server via a communication method. The data is sent in real time using Wi-Fi or mobile data. The data is appropriately compressed to ensure reliable communication.

[0652] Input: Preprocessed data

[0653] Output: Data sent to the server

[0654] Step 4:

[0655] Data analysis

[0656] The server receives the data sent in step 3 and analyzes it using the generative AI model. Deep learning algorithms are used to recognize traffic light status from image data and to identify approaching vehicles and obstacles from distance data. Audio data is also analyzed to detect important environmental sounds. This provides a clear overall picture of the surroundings.

[0657] Input: Data sent to the server

[0658] Output: Analysis results (traffic light status, obstacle location, audio importance)

[0659] Step 5:

[0660] Judgment

[0661] The server makes a decision based on the analysis results. For example, if the traffic light is red and there is an approaching car, it determines that it is dangerous for the user to start crossing the street. These decisions are made to ensure the user's safety.

[0662] Input: Analysis results

[0663] Output: Judgment result (specific action)

[0664] Step 6:

[0665] Feedback Generation

[0666] The server generates feedback based on the results of its judgment. The feedback is generated in a format that is easy for the user to understand (audio guidance, haptic signals, vibration, etc.). For example, audio guidance such as "The traffic light is red. Please stop" may be generated. The intensity of the vibration may also be changed to indicate the urgency of the warning.

[0667] Input: Judgment result

[0668] Output: Feedback data (audio guidance, haptic signals, vibration)

[0669] Step 7:

[0670] Feedback presentation

[0671] The device then presents the received feedback to the user, using the audio speaker to provide audio guidance and the haptic feedback device to emit vibrations, allowing the user to take appropriate action based on the surrounding situation.

[0672] Input: Feedback data

[0673] Output: Presentation to the user (audio guidance, haptic signals, vibration)

[0674] (Application example 1)

[0675] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0676] Safe and efficient mobility in daily life is always an important issue for people with visual impairments. It is particularly difficult for them to accurately assess their surroundings and take appropriate action in busy urban environments, complex intersections, and situations involving interference from other pedestrians. Conventional methods are unable to provide sufficient information or warnings to address this issue, and a fundamental solution to ensure the safety of visually impaired people is needed.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0678] In this invention, the server includes sensor means for acquiring surrounding environmental information, preprocessing means for preprocessing the acquired environmental information, communication means for communicating the preprocessed environmental information, analysis means for analyzing the communicated environmental information, determination means for determining risk factors based on the analysis results, feedback generation means for generating feedback based on the determined risk factors, presentation means for presenting the generated feedback to the user by audio guidance and tactile signals or vibrations, and adjustment means for the system to adjust the content of the generated feedback based on the individual characteristics of the user. This provides an environment in which visually impaired people can travel safely and efficiently, and enables more appropriate guidance with feedback tailored to individual characteristics.

[0679] "Surrounding environmental information" refers to external situational data necessary for visually impaired people to walk, including obstacles, traffic lights, pedestrians, vehicles, etc.

[0680] "Sensor means" refers to an apparatus or device for acquiring information about the surrounding environment, and specifically includes a camera, a distance sensor, an audio acquisition means, and the like.

[0681] "Preprocessing means" refers to a process for compressing data and removing noise to make the acquired environmental information suitable for analysis.

[0682] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to an analytical device such as a server, and includes Wi-Fi and mobile data communications.

[0683] "Analysis means" refers to a device or software for analyzing the environmental information received on the server side and detecting risk factors.

[0684] "Determination means" refers to a device or software for evaluating the analysis results and determining safety and risk factors based on them.

[0685] "Feedback generating means" refers to a device or software for generating feedback to be provided to a user in response to a determined risk factor.

[0686] "Presentation means" refers to devices or technologies for conveying the generated feedback to the user, and specifically includes audio guidance, tactile signals, vibrations, etc.

[0687] "Adjustment means" refers to a device or software that allows the system to optimize the content of the generated feedback based on the individual characteristics of the user.

[0688] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[0689] System Configuration

[0690] The system mainly consists of the following components:

[0691] 1. Device:

[0692] Sensor means: Cameras, distance sensors, and audio capture means are used.

[0693] Preprocessing means: Compress the acquired environmental information and remove noise.

[0694] Communication method: Send the preprocessed data to the server.

[0695] Presentation methods: Provide feedback to the user via audio guidance, haptic signals, and vibration.

[0696] 2. Server:

[0697] Analysis method: Analyze data using AI models to detect risk factors.

[0698] Decision-making method: Evaluate the analysis results and determine the feedback required.

[0699] Feedback generation method: Generate feedback in a format that is easy for the user to understand.

[0700] Tailoring: Tailoring feedback based on the user's individual characteristics.

[0701] Operation overview

[0702] 1. Obtaining information about the surrounding environment:

[0703] The device's camera, distance sensor, and audio capture are used to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[0704] 2. Pretreatment:

[0705] The acquired environmental information is compressed and noise is removed, thereby reducing the amount of data and enabling efficient communication.

[0706] 3. Data communication:

[0707] The preprocessed data is sent to the server via a communication means, such as Wi-Fi or mobile data.

[0708] 4. Data Analysis and Decision-Making:

[0709] The server analyzes the transmitted data using an AI model as an analytical means to detect dangerous elements, such as approaching vehicles or red lights, and evaluates the analysis results based on the judgment means to generate necessary feedback.

[0710] 5. Feedback Generation and Submission:

[0711] The server generates feedback in a format that is easy for the user to understand and transmits it to the terminal using voice guidance, haptic signals, vibration, etc. The server also optimizes the content of the feedback based on the individual characteristics of the user using an adjustment means.

[0712] 6. Feedback suggestions:

[0713] The terminal presents the received feedback to the user by voice guidance, vibration, or the like.

[0714] Specific examples

[0715] Example 1: How to cross the street

[0716] When a user approaches a crosswalk, the device's camera captures the traffic light and surrounding conditions, and the distance sensor measures the position of the car ahead. This data is preprocessed and sent to the server. The server analyzes the data and detects that the traffic light is red and that a car is approaching nearby. Having determined this, the server generates a voice message such as "The traffic light is red. Please stop" and sends it to the device. The device then uses its audio speaker to provide guidance to the user, and simultaneously vibrates to alert the user.

[0717] Prompt Sentence Examples

[0718] I would like to develop a system that uses a camera to capture the surrounding environment in real time, detects dangers, and notifies visually impaired users with voice and vibration. This system uses AI to analyze the status of approaching vehicles and traffic lights and provides feedback. Please tell me the specific prompts and steps.

[0719] This system will provide an environment in which visually impaired people can move around safely and efficiently, and will also enable more appropriate guidance through feedback tailored to individual characteristics.

[0720] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0721] Step 1:

[0722] Obtaining information about the surrounding environment

[0723] Input: Device camera, distance sensor, and audio capture method

[0724] Data processing / calculation: The device uses these sensors to obtain real-time environmental information (images, distance data, audio)

[0725] Specifically, the camera captures traffic lights, pedestrians, and vehicle movements, the distance sensor measures the distance to objects, and the audio capture means collects surrounding sounds. The captured data is temporarily stored in the device.

[0726] Output: The raw environmental data (image data, distance data, audio data) is passed to the next step.

[0727] Step 2:

[0728] Pretreatment

[0729] Input: Raw environmental data obtained in step 1

[0730] Data processing / calculation: The device compresses the data and removes noise

[0731] Specifically, the device compresses and noise filters image data, normalizes distance data, and preprocesses audio data using noise reduction techniques.

[0732] Output: Compressed and denoised preprocessed data (compressed image, normalised distance data, denoised audio data) is passed to the next step.

[0733] Step 3:

[0734] data communication

[0735] Input: Preprocessed data from step 2

[0736] Data processing / calculation: The device sends pre-processed data to the server

[0737] Specifically, the device sends the preprocessed data to the server via Wi-Fi or mobile data, where the data is also packetized and encrypted.

[0738] Output: Preprocessed data sent to the server

[0739] Step 4:

[0740] Data analysis

[0741] Input: Preprocessed data sent to the server

[0742] Data processing / calculation: The server uses an AI model to analyze the received data and detect risk factors.

[0743] Specifically, the server inputs the received image, distance, and audio data into the AI ​​model to perform object recognition, determine traffic light status, and confirm the location of obstacles, for example, analyzing approaching vehicles and the status of traffic lights.

[0744] Output: Analysis results (e.g. approaching vehicle, red light)

[0745] Step 5:

[0746] Judgment

[0747] Input: Analysis results obtained in step 4

[0748] Data processing / calculation: The server evaluates safety based on the analysis results and determines the necessary feedback

[0749] Specifically, the server evaluates the analysis results based on AI decision-making logic and determines what kind of feedback is needed for the user, such as whether an emergency stop command is necessary.

[0750] Output: Feedback decision result (e.g., stop instruction)

[0751] Step 6:

[0752] Feedback Generation

[0753] Input: Feedback decision result from step 5

[0754] Data processing / calculation: The server generates feedback in a format that is easy for the user to understand.

[0755] Specifically, the server generates text data for voice guidance and also designs vibration patterns. The voice guidance generates messages such as "Please stop."

[0756] Output: Feedback data (voice message, vibration pattern)

[0757] Step 7:

[0758] Feedback presentation

[0759] Input: Feedback data generated in step 6

[0760] Data processing / calculation: The device presents feedback data to the user

[0761] Specifically, the device will use a speaker to play voice guidance and vibrate in sync, allowing the user to receive feedback and take appropriate action.

[0762] Output: Provide feedback to the user (audio guidance, vibration)

[0763] The above is the specific processing flow of the system that realizes the application example.

[0764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0765] This invention is a system that supports visually impaired people to walk safely and with peace of mind. In particular, it has the function of recognizing the user's emotional state and adjusting the content of feedback based on that state. Detailed embodiments of this system are described below.

[0766] System Configuration

[0767] The system mainly consists of the following components:

[0768] 1. Terminal

[0769] Sensor means (camera, distance sensor, audio acquisition means)

[0770] Pretreatment means

[0771] communication means

[0772] Presentation means (audio guidance, tactile signals, vibration)

[0773] Emotion Recognition Engine

[0774] 2. Server

[0775] Analysis method (AI model)

[0776] means of judgment

[0777] Feedback Generation Method

[0778] System Operation Overview

[0779] 1. Obtaining information about the surrounding environment

[0780] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[0781] 2. Pretreatment

[0782] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[0783] 3. Data communication

[0784] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[0785] 4. Data Analysis and Judgment

[0786] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[0787] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[0788] 5. Emotion recognition

[0789] Device: The emotion recognition engine analyzes voice data and the user's facial expression data to recognize the user's emotional state in real time, thereby determining whether the user is stressed or relaxed.

[0790] 6. Feedback Generation and Submission

[0791] Server: Adjusts the content of the feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates additional voice guidance to help them relax. This is also sent in combination with the regular danger avoidance feedback.

[0792] The server generates feedback and sends it to the device.

[0793] 7. Feedback Presentation

[0794] The device then presents the received feedback to the user. Specifically, it plays guidance such as "The traffic light is red. Please stop" or "Take a deep breath and relax" through the device's audio speaker, and simultaneously vibrates to warn the user.

[0795] Specific examples

[0796] Example 1: A user experiencing stress

[0797] 1. Obtaining information about the surrounding environment

[0798] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[0799] 2. Pretreatment

[0800] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[0801] 3. Data communication

[0802] The terminal transmits the pre-processed data to the server in real time.

[0803] 4. Data Analysis and Judgment

[0804] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[0805] The server determines that these elements are dangerous and that the user needs to be instructed to stop them.

[0806] 5. Emotion recognition

[0807] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect when they are in a state of tension.

[0808] 6. Feedback Generation and Submission

[0809] In addition to the instruction "The traffic light is red, please stop," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the device.

[0810] 7. Feedback Presentation

[0811] The device will provide a voice prompt saying, "The traffic light is red. Please stop," along with a vibration warning. It will also provide additional relaxation guidance, saying, "Take a deep breath and relax."

[0812] In this way, by taking into account the user's emotional state, the system can not only avoid danger but also provide psychological support, enabling visually impaired users to move about in their daily lives with greater peace of mind.

[0813] The processing flow will be explained below.

[0814] Step 1:

[0815] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[0816] Step 2:

[0817] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[0818] Step 3:

[0819] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[0820] Step 4:

[0821] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[0822] Step 5:

[0823] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[0824] Step 6:

[0825] The device uses an emotion recognition engine to assess the user's emotional state. Specifically, it analyzes voice data and facial expression data to recognize stress, tension, relaxation, etc.

[0826] Step 7:

[0827] The server generates feedback based on the user's emotional state. Specifically, it generates instructions for avoiding danger (e.g., "The traffic light is red. Please stop") as well as feedback encouraging relaxation (e.g., "Take a deep breath and relax")

[0828] Step 8:

[0829] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[0830] Step 9:

[0831] The device then presents the received feedback to the user. Specifically, the device will announce "The traffic light is red. Please stop" through its audio speaker, and will also vibrate to warn the user. The device will also provide additional relaxation guidance, such as "Take a deep breath and relax."

[0832] Example 2

[0833] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0834] Conventional support systems for the visually impaired do not take into account the user's emotional state, and therefore are insufficient in providing support in stressful situations. In addition to acquiring environmental information, there is a need for feedback that reflects the user's psychological state.

[0835] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means for determining risk factors based on the analysis results, an emotion recognition means for recognizing the emotional state of the user based on the risk factors, a feedback generation means for generating feedback based on the determined risk factors and the emotional state, and a presentation means for presenting the generated feedback. This enables safe and secure walking assistance that takes the emotional state of the user into consideration.

[0836] "Surrounding environment information" refers to information about objects and situations that exist around the user, including obstacles, traffic lights, pedestrians, vehicles, and the like.

[0837] "Sensor means" refers to a hardware device for acquiring information about the surrounding environment, and includes a camera, a distance sensor, an audio acquisition means, and the like.

[0838] "Preprocessing means" refers to a processing mechanism for compressing acquired raw data and removing noise to reduce the amount of data and enable efficient communication.

[0839] "Communication means" refers to the mechanism for transmitting preprocessed data to the server, and Wi-Fi or mobile data communication is used.

[0840] "Analysis means" refers to the means for analyzing the data transmitted within the server, and uses AI models and machine learning algorithms.

[0841] The "judging means" refers to a means for evaluating and judging risk factors based on the analysis results obtained by the analyzing means.

[0842] "Emotion recognition means" refers to means for recognizing the user's emotional state, and evaluates the user's stress and relaxation state from voice data and facial expression data.

[0843] "Feedback generating means" refers to means for generating feedback to provide to a user based on the determined risk factors and emotional state.

[0844] "Presentation means" refers to means for conveying the generated feedback to the user, and includes audio guidance, tactile signals, and vibrations.

[0845] This invention relates to a system that supports visually impaired people to walk safely and with peace of mind. It is particularly characterized by its ability to recognize the user's emotional state and adjust feedback based on that state. This system is comprised of a terminal and a server, and specific embodiments of the system are described in detail below.

[0846] System Configuration

[0847] 1. Terminal

[0848] Sensor means: Cameras, distance sensors, and audio capture means are used to acquire information about the surrounding environment, such as obstacles, traffic lights, pedestrians, and vehicles.

[0849] Preprocessing means: Compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[0850] Communication method: Preprocessed data is sent to the server via Wi-Fi or mobile data communication.

[0851] Emotion recognition means: Analyzes voice data and facial expression data to recognize the user's emotional state in real time.

[0852] Presentation methods: Use audio prompts, haptic signals, and vibrations to provide feedback to the user.

[0853] 2. Server

[0854] Analysis method: Using AI models and machine learning algorithms, data sent from the device is analyzed to detect risk factors.

[0855] Judgment method: Based on the analysis results, evaluate risk factors and determine the need for feedback.

[0856] Feedback generation means: Generates appropriate feedback based on the determined risk factors and the user's emotional state.

[0857] Operation overview

[0858] 1. Obtaining information about the surrounding environment

[0859] The device uses a camera, distance sensors, and audio capture means to collect information about the user's surrounding environment, such as the status of traffic lights, approaching vehicles, and ambient noise.

[0860] 2. Pretreatment

[0861] The raw data acquired by the terminal is compressed and noise is removed, thereby improving communication efficiency.

[0862] 3. Data communication

[0863] The terminal transmits the preprocessed data to the server in real time.

[0864] 4. Data Analysis

[0865] The server analyzes the received data using an analysis means to detect dangerous elements, specifically by checking the color of traffic lights and approaching vehicles.

[0866] 5. Emotion recognition

[0867] The device analyzes voice and facial expression data to recognize the user's emotional state in real time, making it possible to determine whether the user is feeling stressed.

[0868] 6. Generate feedback

[0869] The server generates appropriate feedback for the user based on the analysis results and the user's emotional state. For example, if the traffic light is red, the server will provide a message saying, "The light is red. Please stop," and if the user is feeling stressed, the server will provide an additional message saying, "Take a deep breath and relax."

[0870] 7. Providing Feedback

[0871] The device will provide the generated feedback to the user via voice guidance or vibration, allowing the user to know the exact situation and take appropriate action, allowing them to act with peace of mind.

[0872] Specific examples

[0873] Text format

[0874] Please explain the detailed processing steps of a system that acquires information about the surrounding environment, recognizes the user's emotional state, and generates appropriate feedback when a visually impaired person approaches a crosswalk.

[0875] This system not only avoids danger but also provides comprehensive support that takes into account the user's emotional state, making it a technology that will enable visually impaired people to live their daily lives more safely and with peace of mind.

[0876] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0877] Step 1:

[0878] Obtaining information about the surrounding environment

[0879] The device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. As input, it acquires the current surrounding image, distance to objects, and environmental sound.

[0880] Specific operation: The device's camera captures the road conditions ahead, the distance sensor measures the distance to objects ahead, and the audio capture means collects surrounding sounds.

[0881] Output: Collected video data, distance data, and audio data are generated.

[0882] Step 2:

[0883] Data Preprocessing

[0884] The device processes the acquired raw data, using as input the video, distance, and audio data collected in step 1.

[0885] Specific operations: Compresses video data, cuts unnecessary data, and removes noise from audio data. Distance data is also simplified.

[0886] Output: Compressed video data, noise-removed audio data, and simplified distance data are generated.

[0887] Step 3:

[0888] Sending data

[0889] The device sends the preprocessed data to the server, using the compressed, denoised and simplified data generated in step 2 as input.

[0890] Specific operation: Preprocessed data is sent to the server in real time using Wi-Fi or mobile data communication.

[0891] Output: Generates data for analysis that is sent to the server.

[0892] Step 4:

[0893] Data analysis

[0894] The server parses the received data, using as input the parsing data sent in step 3.

[0895] Specific operation: Using an AI model, the color of the traffic light is recognized from the transmitted video data, and the presence or absence of approaching vehicles is analyzed from the distance data.

[0896] Output: The analysis results include the traffic light status (red / green) and information on approaching vehicles.

[0897] Step 5:

[0898] Determining risk factors

[0899] The server determines risk factors based on the analysis results, using the analysis results generated in step 4 as input.

[0900] Specific operation: Based on the analysis results, nearby obstacles and dangerous elements are evaluated and the level of danger is determined.

[0901] Output: The presence or absence of risk factors and detailed information about them is generated.

[0902] Step 6:

[0903] Recognition of emotional states

[0904] The device recognizes the user's emotional state using voice and facial expression data as input.

[0905] Specific operation: Using an AI model, the system analyzes the user's stress level from their voice and assesses their emotional state from facial expression data.

[0906] Output: The user's emotional state (stress level and relaxation level) is generated.

[0907] Step 7:

[0908] Generate feedback

[0909] The server generates feedback based on the risk factors and emotional state, using the risk factor information from step 5 and the emotional state from step 6 as input.

[0910] Specific behavior: In addition to basic instructions for dangerous elements, the system generates additional feedback depending on the user's emotional state. For example, when the traffic light is red, the system generates the instruction "The light is red. Please stop" along with the guidance "Take a deep breath and relax."

[0911] Output: A generated feedback message is generated.

[0912] Step 8:

[0913] Providing feedback

[0914] The device presents the generated feedback to the user, using as input the feedback message generated in step 7.

[0915] Specific operation: A feedback message is played using voice guidance means, and a warning is given to the user using vibration means.

[0916] Output: User received feedback, improved safety and security provided.

[0917] (Application example 2)

[0918] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0919] When visually impaired people shop in a physical store, it is difficult for them to accurately and quickly obtain information such as obstacles, product information, and store maps. In particular, when users are in a tense or stressed emotional state, there is a need for support for safer and more secure movement. To address these challenges, there is a need to provide a system that enables visually impaired people to move around a store safely, securely, and efficiently and obtain the information they need.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0921] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, feedback adjustment means for adjusting the feedback content based on the emotional state, and presentation means for presenting the generated feedback, thereby enabling feedback according to the emotional state to be provided, allowing the user to move around the store with peace of mind and obtain the information they need.

[0922] "Surrounding environmental information" is a general term for visual and audio information including obstacles, traffic lights, pedestrians, store information, etc. in the area where a visually impaired person is located.

[0923] "Sensor means" refers to a device such as a camera, distance sensor, or audio capture device that is used to acquire information about the surrounding environment.

[0924] The "preprocessing means" is a means for compressing the acquired environmental information and removing noise to reduce the amount of data and enable efficient communication.

[0925] "Communication means" refers to a means for transmitting preprocessed environmental information to an analysis device such as a server, and mainly includes Wi-Fi and mobile data communication.

[0926] "Analysis means" refers to a means for analyzing the communicated environmental information, and is usually a device or method that uses an AI model to analyze the environmental information and extract risk factors and necessary information.

[0927] "Determination means" refers to a means for determining risk factors and other important factors based on the analysis results.

[0928] The "feedback generating means" is a means for generating feedback to be provided to the user based on the determined risk factors and the like.

[0929] The "presentation means" refers to a means for presenting the generated feedback to the user, and mainly includes voice guidance, tactile signals, vibrations, etc.

[0930] The "emotion recognition means" is a means for analyzing the user's voice, facial expressions, etc., and recognizing the user's emotional state in real time.

[0931] The "feedback adjustment means" is a means for appropriately adjusting the content of the feedback based on the recognized emotional state of the user.

[0932] The present invention provides a system that supports visually impaired people to move around and shop in a physical store safely and with peace of mind. Detailed embodiments are described below.

[0933] System Configuration

[0934] The system mainly consists of the following components:

[0935] 1. Terminal

[0936] Sensor means: camera, distance sensor, audio acquisition means

[0937] Preprocessing means: A device that compresses the acquired environmental information and removes noise

[0938] Communication methods: Wi-Fi and mobile data communication devices

[0939] Emotion recognition means: An engine that analyzes voice data and user facial expression data to recognize emotional states.

[0940] Presentation methods: audio guidance, tactile signals, vibration

[0941] 2. Server

[0942] Analysis method: A method for analyzing environmental information using an AI model and extracting necessary information

[0943] Judgment method: A method for determining risk factors and important information from the analysis results

[0944] Feedback generation method: A method for generating feedback based on the judged content

[0945] Feedback adjustment means: A means for adjusting the feedback content based on the user's emotional state

[0946] System Operation Overview

[0947] 1. Obtaining information about the surrounding environment

[0948] The device's camera, distance sensors, and audio capture gather information about the surrounding environment, including product location, obstacles in the store, and the movements of other customers.

[0949] 2. Pretreatment

[0950] The acquired environmental information is compressed and noise is removed at the device, reducing the amount of data and enabling more efficient communication and analysis.

[0951] 3. Data communication

[0952] The pre-processed data is sent to a server using Wi-Fi or mobile data.

[0953] 4. Data Analysis and Judgment

[0954] The server uses an AI model to analyze the received data, detect the location of products in the store and the presence of obstacles, and determine which products the user is approaching and what action they should take.

[0955] 5. Emotion recognition

[0956] The device's emotion recognition engine analyzes the user's voice and facial expression data to recognize their emotional state in real time, thereby determining whether they are stressed or relaxed.

[0957] 6. Feedback Generation and Submission

[0958] The server generates feedback and sends it to the device, including additional voice prompts to help the user relax if they are feeling stressed.

[0959] 7. Feedback Presentation

[0960] The device then presents the received feedback to the user through a voice speaker or vibration, for example, "There is a sweets section on this shelf."

[0961] Example

[0962] Example 1: User in a stressed state

[0963] 1. Obtaining information about the surrounding environment

[0964] When a user approaches a product shelf, the device's camera captures the entire shelf and product information, and a distance sensor measures the distance to the shelf.

[0965] 2. Pretreatment

[0966] The acquired image and distance data are preprocessed to remove noise and improve communication efficiency.

[0967] 3. Data communication

[0968] The preprocessed data is sent to the server.

[0969] 4. Data Analysis and Judgment

[0970] The server analyzes the received data and determines that it is necessary to provide the user with information about products on the shelves.

[0971] 5. Emotion recognition

[0972] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect their level of tension.

[0973] 6. Feedback Generation and Submission

[0974] In addition to the instruction "There is a sweets section on this shelf," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the terminal.

[0975] 7. Feedback Presentation

[0976] The device will provide a voice prompt saying, "There is a sweets section on this shelf," and will also vibrate to warn you. It will also provide additional relaxation guidance such as, "Take a deep breath and relax."

[0977] Examples of prompt statements

[0978] Example prompt sentence:

[0979] "Environmental information: product shelves. User is in a state of tension. Analyze the data and generate appropriate feedback."

[0980] "Feedback request: User is approaching a product shelf and is in a stressed state. Please analyze data and generate appropriate feedback."

[0981] In this way, by adjusting the feedback provided depending on the user's emotional state, it is possible to create a system that allows visually impaired users to move around and shop in a physical store with greater confidence.

[0982] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0983] Step 1:

[0984] Obtaining information about the surrounding environment

[0985] As a user moves around a physical store, the device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. Specifically, the camera captures product location and shelf information, the distance sensor measures the distance to the product shelf, and the audio capture means records surrounding audio information. The input is visual information, distance information, and audio information, and the output is integrated data of this information.

[0986] Step 2:

[0987] Pretreatment

[0988] The device preprocesses the acquired environmental information. Specifically, it resizes the acquired video data and removes noise. Similarly, it performs noise filtering on the distance data to improve its accuracy. It also performs noise reduction on the audio data. The input is the integrated data acquired in step 1, and the output is the compressed and noise-removed data.

[0989] Step 3:

[0990] data communication

[0991] The preprocessed data is sent from the device to a server. Specifically, the compressed and denoised data is sent to the server in real time using Wi-Fi or mobile data. The input is the preprocessed data, and the output is the data sent to the server.

[0992] Step 4:

[0993] Data analysis and judgment

[0994] The server analyzes the received data. Specifically, it uses an AI model to analyze the visual, distance, and audio information just sent to identify the presence of obstacles and product information. It also determines which product information in the store should be provided based on the user's situation. The input is the data sent from the device, and the output is the analysis and judgment results.

[0995] Step 5:

[0996] emotion recognition

[0997] The device acquires the user's voice and facial expression data and analyzes it using an emotion recognition engine. Specifically, it identifies the user's emotional state (e.g., stress, tension, relaxation) in real time. The input is the user's voice and facial expression data, and the output is the recognized emotional state.

[0998] Step 6:

[0999] Feedback generation and submission

[1000] The server generates feedback based on the results of data analysis and emotion recognition. Specifically, it generates additional guidance, such as product information and warning information required by the user, as well as relaxation guidance according to the user's emotional state. The generated feedback is sent to the device in real time. The input is the analysis results and emotion recognition results, and the output is the generated feedback.

[1001] Step 7:

[1002] Feedback presentation

[1003] The device presents the received feedback to the user. Specifically, it conveys information to the user using voice guidance or haptic signals (vibrations). For example, it provides voice guidance such as "There is a sweets corner on this shelf" or relaxation guidance such as "Take a deep breath and relax." The input is feedback from the server, and the output is information presented to the user.

[1004] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1005] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1006] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1007] [Third embodiment]

[1008] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1009] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1010] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1011] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1012] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1013] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1014] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1015] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1016] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1017] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1018] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1019] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1020] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[1021] System Configuration

[1022] The system mainly consists of the following components:

[1023] 1. Terminal

[1024] Sensor means (camera, distance sensor, audio acquisition means)

[1025] Pretreatment means

[1026] communication means

[1027] Presentation means (audio guidance, tactile signals, vibration)

[1028] 2. Server

[1029] Analysis method (AI model)

[1030] means of judgment

[1031] Feedback Generation Method

[1032] System Operation Overview

[1033] 1. Obtaining information about the surrounding environment

[1034] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[1035] 2. Pretreatment

[1036] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[1037] 3. Data communication

[1038] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[1039] 4. Data Analysis and Judgment

[1040] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[1041] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[1042] 5. Feedback Generation and Submission

[1043] Server: Generates feedback in a format that is easy for the user to understand. Feedback data is sent to the device using voice guidance, haptic signals, vibrations, etc.

[1044] 6. Feedback Presentation

[1045] Device: Presents the received feedback to the user, such as audio guidance or vibration alerts.

[1046] Specific examples

[1047] Example 1: How to cross the crosswalk

[1048] 1. Obtaining information about the surrounding environment

[1049] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[1050] 2. Pretreatment

[1051] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[1052] 3. Data communication

[1053] The terminal transmits the pre-processed data to the server in real time.

[1054] 4. Data Analysis and Judgment

[1055] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[1056] The server determines that these elements are dangerous and concludes that the user needs to be instructed to stop them.

[1057] 5. Feedback Generation and Submission

[1058] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[1059] 6. Feedback Presentation

[1060] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[1061] This allows visually impaired users to cross the street safely. This system uses a similar process to adapt to various scenarios, providing powerful support for independent mobility for the visually impaired.

[1062] The processing flow will be explained below.

[1063] Step 1:

[1064] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[1065] Step 2:

[1066] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[1067] Step 3:

[1068] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[1069] Step 4:

[1070] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[1071] Step 5:

[1072] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[1073] Step 6:

[1074] The server generates feedback based on the risk factors it has determined. Specifically, it generates voice guidance based on text information such as "The traffic light is red. Please stop" or "A car is approaching from the right. Be careful," and also sets vibration patterns depending on the level of urgency.

[1075] Step 7:

[1076] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[1077] Step 8:

[1078] The device presents the received feedback to the user. Specifically, it plays a voice message such as "The traffic light is red. Please stop" through the device's audio speaker, and simultaneously vibrates to warn the user.

[1079] Example 1

[1080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1081] When visually impaired people walk or move around in their daily lives, they are unable to sense information about their surroundings, which can result in accidents and danger. Systems that solve these issues and support the safe walking and movement of visually impaired people are needed. Current technology struggles to provide accurate feedback in real time, and efficient information processing and communication methods are lacking.

[1082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1083] In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means using a generative AI model as the analysis means and determining risk factors based on the analysis results, a feedback generation means for generating feedback based on the determined risk factors, and a presentation means for presenting the generated feedback, thereby enabling visually impaired people to accurately grasp their surroundings and move safely.

[1084] "Sensor means" refers to devices or techniques used to obtain information about the surrounding environment.

[1085] The "preprocessing means" refers to a device or technology that compresses the acquired environmental information and performs processing to remove noise.

[1086] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to the server, including Wi-Fi and mobile data communications.

[1087] "Analysis means" refers to devices or techniques for analyzing the communicated environmental information. In this invention, a generative AI model is used.

[1088] A "generative AI model" is an artificial intelligence model that analyzes data and detects patterns and features.

[1089] The "judgment means" refers to a device or technology that judges risk factors based on the results of the analysis means.

[1090] The "feedback generating means" refers to a device or technology that generates feedback to the user based on the determined risk factors.

[1091] The "presentation means" refers to a device or technology that presents the generated feedback to the user through audio guidance, tactile signals, vibrations, etc.

[1092] This invention relates to a system that assists visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. The components and operation of the system are described in detail below.

[1093] System Configuration

[1094] The system consists of the following components:

[1095] 1. Terminal

[1096] Sensor means (camera, distance sensor, audio acquisition means)

[1097] Pretreatment means

[1098] communication means

[1099] Presentation means (audio guidance, tactile signals, vibration)

[1100] 2. Server

[1101] Analysis method (generative AI model)

[1102] means of judgment

[1103] Feedback Generation Method

[1104] Operation overview

[1105] 1. Sensor means: The device uses cameras, distance sensors, and audio capture means to collect information about the surrounding environment, including information about obstacles, traffic lights, pedestrians, etc.

[1106] 2. Pre-processing: The terminal compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[1107] 3. Communication method: The device sends the preprocessed data to the server using Wi-Fi or mobile data communication.

[1108] 4. Analysis method: The server analyzes the received data using the generative AI model to detect dangerous elements. The generative AI model uses deep learning algorithms to analyze traffic light colors, approaching vehicles, etc.

[1109] 5. Judgment method: The server evaluates the risk factors based on the analysis results and determines the necessary feedback.

[1110] 6. Feedback generation means: The server generates feedback in a form that is easy for the user to understand, such as voice guidance, haptic signals, vibrations, etc.

[1111] 7. Presentation: The device presents the received feedback to the user. Possible methods include voice guidance and vibration alerts.

[1112] Specific examples

[1113] Example 1: How to cross the street

[1114] 1. Obtaining information about the surrounding environment

[1115] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[1116] 2. Pretreatment

[1117] The device preprocesses the image and distance data it acquires to reduce the amount of data and remove noise.

[1118] 3. Data communication

[1119] The terminal transmits the pre-processed data to the server in real time.

[1120] 4. Data Analysis and Judgment

[1121] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby. It determines this to be a danger and concludes that the user needs to stop.

[1122] 5. Feedback Generation and Submission

[1123] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[1124] 6. Feedback Presentation

[1125] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[1126] Examples of prompt statements

[1127] "Please explain the detailed processing steps of a system that helps visually impaired people cross the street."

[1128] "Please explain the specific operation of a system that detects traffic lights and vehicle movements and provides feedback."

[1129] In this way, the present invention provides an environment in which visually impaired people can walk safely, and provides detailed instructions on how to actually use the system. The system aims to improve walking safety by understanding the surrounding environment in real time and providing accurate feedback to the user.

[1130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1131] Step 1:

[1132] Obtaining information about the surrounding environment

[1133] The device acquires information about the surrounding environment using a camera, distance sensor, and audio acquisition means. For example, it collects information about the road the user is walking on, surrounding obstacles, traffic light status, and surrounding audio. This allows the device to acquire information about the user's surrounding environment in real time.

[1134] Input: Surrounding environment information (images, distance data, audio data)

[1135] Output: Raw data obtained

[1136] Step 2:

[1137] Pretreatment

[1138] The device preprocesses the raw data acquired in step 1. Unnecessary noise is removed from the camera image and important objects and information are highlighted. Data from the distance sensor is similarly processed to reduce noise. Audio data is appropriately filtered to highlight specific sounds (e.g., the sound of a car engine). This improves data quality and makes communication more efficient.

[1139] Input: Acquired raw data (images, range data, audio data)

[1140] Output: Preprocessed data (denoised data)

[1141] Step 3:

[1142] data communication

[1143] The device sends the preprocessed data to the server via a communication method. The data is sent in real time using Wi-Fi or mobile data. The data is appropriately compressed to ensure reliable communication.

[1144] Input: Preprocessed data

[1145] Output: Data sent to the server

[1146] Step 4:

[1147] Data analysis

[1148] The server receives the data sent in step 3 and analyzes it using the generative AI model. Deep learning algorithms are used to recognize traffic light status from image data and to identify approaching vehicles and obstacles from distance data. Audio data is also analyzed to detect important environmental sounds. This provides a clear overall picture of the surroundings.

[1149] Input: Data sent to the server

[1150] Output: Analysis results (traffic light status, obstacle location, audio importance)

[1151] Step 5:

[1152] Judgment

[1153] The server makes a decision based on the analysis results. For example, if the traffic light is red and there is an approaching car, it determines that it is dangerous for the user to start crossing the street. These decisions are made to ensure the user's safety.

[1154] Input: Analysis results

[1155] Output: Judgment result (specific action)

[1156] Step 6:

[1157] Feedback Generation

[1158] The server generates feedback based on the results of its judgment. The feedback is generated in a format that is easy for the user to understand (audio guidance, haptic signals, vibration, etc.). For example, audio guidance such as "The traffic light is red. Please stop" may be generated. The intensity of the vibration may also be changed to indicate the urgency of the warning.

[1159] Input: Judgment result

[1160] Output: Feedback data (audio guidance, haptic signals, vibration)

[1161] Step 7:

[1162] Feedback presentation

[1163] The device then presents the received feedback to the user, using the audio speaker to provide audio guidance and the haptic feedback device to emit vibrations, allowing the user to take appropriate action based on the surrounding situation.

[1164] Input: Feedback data

[1165] Output: Presentation to the user (audio guidance, haptic signals, vibration)

[1166] (Application example 1)

[1167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1168] Safe and efficient mobility in daily life is always an important issue for people with visual impairments. It is particularly difficult for them to accurately assess their surroundings and take appropriate action in busy urban environments, complex intersections, and situations involving interference from other pedestrians. Conventional methods are unable to provide sufficient information or warnings to address this issue, and a fundamental solution to ensure the safety of visually impaired people is needed.

[1169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1170] In this invention, the server includes sensor means for acquiring surrounding environmental information, preprocessing means for preprocessing the acquired environmental information, communication means for communicating the preprocessed environmental information, analysis means for analyzing the communicated environmental information, determination means for determining risk factors based on the analysis results, feedback generation means for generating feedback based on the determined risk factors, presentation means for presenting the generated feedback to the user by audio guidance and tactile signals or vibrations, and adjustment means for the system to adjust the content of the generated feedback based on the individual characteristics of the user. This provides an environment in which visually impaired people can travel safely and efficiently, and enables more appropriate guidance with feedback tailored to individual characteristics.

[1171] "Surrounding environmental information" refers to external situational data necessary for visually impaired people to walk, including obstacles, traffic lights, pedestrians, vehicles, etc.

[1172] "Sensor means" refers to an apparatus or device for acquiring information about the surrounding environment, and specifically includes a camera, a distance sensor, an audio acquisition means, and the like.

[1173] "Preprocessing means" refers to a process for compressing data and removing noise to make the acquired environmental information suitable for analysis.

[1174] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to an analytical device such as a server, and includes Wi-Fi and mobile data communications.

[1175] "Analysis means" refers to a device or software for analyzing the environmental information received on the server side and detecting risk factors.

[1176] "Determination means" refers to a device or software for evaluating the analysis results and determining safety and risk factors based on them.

[1177] "Feedback generating means" refers to a device or software for generating feedback to be provided to a user in response to a determined risk factor.

[1178] "Presentation means" refers to devices or technologies for conveying the generated feedback to the user, and specifically includes audio guidance, tactile signals, vibrations, etc.

[1179] "Adjustment means" refers to a device or software that allows the system to optimize the content of the generated feedback based on the individual characteristics of the user.

[1180] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[1181] System Configuration

[1182] The system mainly consists of the following components:

[1183] 1. Device:

[1184] Sensor means: Cameras, distance sensors, and audio capture means are used.

[1185] Preprocessing means: Compress the acquired environmental information and remove noise.

[1186] Communication method: Send the preprocessed data to the server.

[1187] Presentation methods: Provide feedback to the user via audio guidance, haptic signals, and vibration.

[1188] 2. Server:

[1189] Analysis method: Analyze data using AI models to detect risk factors.

[1190] Decision-making method: Evaluate the analysis results and determine the feedback required.

[1191] Feedback generation method: Generate feedback in a format that is easy for the user to understand.

[1192] Tailoring: Tailoring feedback based on the user's individual characteristics.

[1193] Operation overview

[1194] 1. Obtaining information about the surrounding environment:

[1195] The device's camera, distance sensor, and audio capture are used to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[1196] 2. Pretreatment:

[1197] The acquired environmental information is compressed and noise is removed, thereby reducing the amount of data and enabling efficient communication.

[1198] 3. Data communication:

[1199] The preprocessed data is sent to the server via a communication means, such as Wi-Fi or mobile data.

[1200] 4. Data Analysis and Decision-Making:

[1201] The server analyzes the transmitted data using an AI model as an analytical means to detect dangerous elements, such as approaching vehicles or red lights, and evaluates the analysis results based on the judgment means to generate necessary feedback.

[1202] 5. Feedback Generation and Submission:

[1203] The server generates feedback in a format that is easy for the user to understand and transmits it to the terminal using voice guidance, haptic signals, vibration, etc. The server also optimizes the content of the feedback based on the individual characteristics of the user using an adjustment means.

[1204] 6. Feedback suggestions:

[1205] The terminal presents the received feedback to the user by voice guidance, vibration, or the like.

[1206] Specific examples

[1207] Example 1: How to cross the street

[1208] When a user approaches a crosswalk, the device's camera captures the traffic light and surrounding conditions, and the distance sensor measures the position of the car ahead. This data is preprocessed and sent to the server. The server analyzes the data and detects that the traffic light is red and that a car is approaching nearby. Having determined this, the server generates a voice message such as "The traffic light is red. Please stop" and sends it to the device. The device then uses its audio speaker to provide guidance to the user, and simultaneously vibrates to alert the user.

[1209] Prompt Sentence Examples

[1210] I would like to develop a system that uses a camera to capture the surrounding environment in real time, detects dangers, and notifies visually impaired users with voice and vibration. This system uses AI to analyze the status of approaching vehicles and traffic lights and provides feedback. Please tell me the specific prompts and steps.

[1211] This system will provide an environment in which visually impaired people can move around safely and efficiently, and will also enable more appropriate guidance through feedback tailored to individual characteristics.

[1212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1213] Step 1:

[1214] Obtaining information about the surrounding environment

[1215] Input: Device camera, distance sensor, and audio capture method

[1216] Data processing / calculation: The device uses these sensors to obtain real-time environmental information (images, distance data, audio)

[1217] Specifically, the camera captures traffic lights, pedestrians, and vehicle movements, the distance sensor measures the distance to objects, and the audio capture means collects surrounding sounds. The captured data is temporarily stored in the device.

[1218] Output: The raw environmental data (image data, distance data, audio data) is passed to the next step.

[1219] Step 2:

[1220] Pretreatment

[1221] Input: Raw environmental data obtained in step 1

[1222] Data processing / calculation: The device compresses the data and removes noise

[1223] Specifically, the device compresses and noise filters image data, normalizes distance data, and preprocesses audio data using noise reduction techniques.

[1224] Output: Compressed and denoised preprocessed data (compressed image, normalised distance data, denoised audio data) is passed to the next step.

[1225] Step 3:

[1226] data communication

[1227] Input: Preprocessed data from step 2

[1228] Data processing / calculation: The device sends pre-processed data to the server

[1229] Specifically, the device sends the preprocessed data to the server via Wi-Fi or mobile data, where the data is also packetized and encrypted.

[1230] Output: Preprocessed data sent to the server

[1231] Step 4:

[1232] Data analysis

[1233] Input: Preprocessed data sent to the server

[1234] Data processing / calculation: The server uses an AI model to analyze the received data and detect risk factors.

[1235] Specifically, the server inputs the received image, distance, and audio data into the AI ​​model to perform object recognition, determine traffic light status, and confirm the location of obstacles, for example, analyzing approaching vehicles and the status of traffic lights.

[1236] Output: Analysis results (e.g. approaching vehicle, red light)

[1237] Step 5:

[1238] Judgment

[1239] Input: Analysis results obtained in step 4

[1240] Data processing / calculation: The server evaluates safety based on the analysis results and determines the necessary feedback

[1241] Specifically, the server evaluates the analysis results based on AI decision-making logic and determines what kind of feedback is needed for the user, such as whether an emergency stop command is necessary.

[1242] Output: Feedback decision result (e.g., stop instruction)

[1243] Step 6:

[1244] Feedback Generation

[1245] Input: Feedback decision result from step 5

[1246] Data processing / calculation: The server generates feedback in a format that is easy for the user to understand.

[1247] Specifically, the server generates text data for voice guidance and also designs vibration patterns. The voice guidance generates messages such as "Please stop."

[1248] Output: Feedback data (voice message, vibration pattern)

[1249] Step 7:

[1250] Feedback presentation

[1251] Input: Feedback data generated in step 6

[1252] Data processing / calculation: The device presents feedback data to the user

[1253] Specifically, the device will use a speaker to play voice guidance and vibrate in sync, allowing the user to receive feedback and take appropriate action.

[1254] Output: Provide feedback to the user (audio guidance, vibration)

[1255] The above is the specific processing flow of the system that realizes the application example.

[1256] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1257] This invention is a system that supports visually impaired people to walk safely and with peace of mind. In particular, it has the function of recognizing the user's emotional state and adjusting the content of feedback based on that state. Detailed embodiments of this system are described below.

[1258] System Configuration

[1259] The system mainly consists of the following components:

[1260] 1. Terminal

[1261] Sensor means (camera, distance sensor, audio acquisition means)

[1262] Pretreatment means

[1263] communication means

[1264] Presentation means (audio guidance, tactile signals, vibration)

[1265] Emotion Recognition Engine

[1266] 2. Server

[1267] Analysis method (AI model)

[1268] means of judgment

[1269] Feedback Generation Method

[1270] System Operation Overview

[1271] 1. Obtaining information about the surrounding environment

[1272] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[1273] 2. Pretreatment

[1274] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[1275] 3. Data communication

[1276] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[1277] 4. Data Analysis and Judgment

[1278] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[1279] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[1280] 5. Emotion recognition

[1281] Device: The emotion recognition engine analyzes voice data and the user's facial expression data to recognize the user's emotional state in real time, thereby determining whether the user is stressed or relaxed.

[1282] 6. Feedback Generation and Submission

[1283] Server: Adjusts the content of the feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates additional voice guidance to help them relax. This is also sent in combination with the regular danger avoidance feedback.

[1284] The server generates feedback and sends it to the device.

[1285] 7. Feedback Presentation

[1286] The device then presents the received feedback to the user. Specifically, it plays guidance such as "The traffic light is red. Please stop" or "Take a deep breath and relax" through the device's audio speaker, and simultaneously vibrates to warn the user.

[1287] Specific examples

[1288] Example 1: A user experiencing stress

[1289] 1. Obtaining information about the surrounding environment

[1290] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[1291] 2. Pretreatment

[1292] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[1293] 3. Data communication

[1294] The terminal transmits the pre-processed data to the server in real time.

[1295] 4. Data Analysis and Judgment

[1296] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[1297] The server determines that these elements are dangerous and that the user needs to be instructed to stop them.

[1298] 5. Emotion recognition

[1299] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect when they are in a state of tension.

[1300] 6. Feedback Generation and Submission

[1301] In addition to the instruction "The traffic light is red, please stop," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the device.

[1302] 7. Feedback Presentation

[1303] The device will provide a voice prompt saying, "The traffic light is red. Please stop," along with a vibration warning. It will also provide additional relaxation guidance, saying, "Take a deep breath and relax."

[1304] In this way, by taking into account the user's emotional state, the system can not only avoid danger but also provide psychological support, enabling visually impaired users to move about in their daily lives with greater peace of mind.

[1305] The processing flow will be explained below.

[1306] Step 1:

[1307] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[1308] Step 2:

[1309] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[1310] Step 3:

[1311] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[1312] Step 4:

[1313] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[1314] Step 5:

[1315] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[1316] Step 6:

[1317] The device uses an emotion recognition engine to assess the user's emotional state. Specifically, it analyzes voice data and facial expression data to recognize stress, tension, relaxation, etc.

[1318] Step 7:

[1319] The server generates feedback based on the user's emotional state. Specifically, it generates instructions for avoiding danger (e.g., "The traffic light is red. Please stop") as well as feedback encouraging relaxation (e.g., "Take a deep breath and relax")

[1320] Step 8:

[1321] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[1322] Step 9:

[1323] The device then presents the received feedback to the user. Specifically, the device will announce "The traffic light is red. Please stop" through its audio speaker, and will also vibrate to warn the user. The device will also provide additional relaxation guidance, such as "Take a deep breath and relax."

[1324] Example 2

[1325] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1326] Conventional support systems for the visually impaired do not take into account the user's emotional state, and therefore are insufficient in providing support in stressful situations. In addition to acquiring environmental information, there is a need for feedback that reflects the user's psychological state.

[1327] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means for determining risk factors based on the analysis results, an emotion recognition means for recognizing the emotional state of the user based on the risk factors, a feedback generation means for generating feedback based on the determined risk factors and the emotional state, and a presentation means for presenting the generated feedback. This enables safe and secure walking assistance that takes the emotional state of the user into consideration.

[1328] "Surrounding environment information" refers to information about objects and situations that exist around the user, including obstacles, traffic lights, pedestrians, vehicles, and the like.

[1329] "Sensor means" refers to a hardware device for acquiring information about the surrounding environment, and includes a camera, a distance sensor, an audio acquisition means, and the like.

[1330] "Preprocessing means" refers to a processing mechanism for compressing acquired raw data and removing noise to reduce the amount of data and enable efficient communication.

[1331] "Communication means" refers to the mechanism for transmitting preprocessed data to the server, and Wi-Fi or mobile data communication is used.

[1332] "Analysis means" refers to the means for analyzing the data transmitted within the server, and uses AI models and machine learning algorithms.

[1333] The "judging means" refers to a means for evaluating and judging risk factors based on the analysis results obtained by the analyzing means.

[1334] "Emotion recognition means" refers to means for recognizing the user's emotional state, and evaluates the user's stress and relaxation state from voice data and facial expression data.

[1335] "Feedback generating means" refers to means for generating feedback to provide to a user based on the determined risk factors and emotional state.

[1336] "Presentation means" refers to means for conveying the generated feedback to the user, and includes audio guidance, tactile signals, and vibrations.

[1337] This invention relates to a system that supports visually impaired people to walk safely and with peace of mind. It is particularly characterized by its ability to recognize the user's emotional state and adjust feedback based on that state. This system is comprised of a terminal and a server, and specific embodiments of the system are described in detail below.

[1338] System Configuration

[1339] 1. Terminal

[1340] Sensor means: Cameras, distance sensors, and audio capture means are used to acquire information about the surrounding environment, such as obstacles, traffic lights, pedestrians, and vehicles.

[1341] Preprocessing means: Compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[1342] Communication method: Preprocessed data is sent to the server via Wi-Fi or mobile data communication.

[1343] Emotion recognition means: Analyzes voice data and facial expression data to recognize the user's emotional state in real time.

[1344] Presentation methods: Use audio prompts, haptic signals, and vibrations to provide feedback to the user.

[1345] 2. Server

[1346] Analysis method: Using AI models and machine learning algorithms, data sent from the device is analyzed to detect risk factors.

[1347] Judgment method: Based on the analysis results, evaluate risk factors and determine the need for feedback.

[1348] Feedback generation means: Generates appropriate feedback based on the determined risk factors and the user's emotional state.

[1349] Operation overview

[1350] 1. Obtaining information about the surrounding environment

[1351] The device uses a camera, distance sensors, and audio capture means to collect information about the user's surrounding environment, such as the status of traffic lights, approaching vehicles, and ambient noise.

[1352] 2. Pretreatment

[1353] The raw data acquired by the terminal is compressed and noise is removed, thereby improving communication efficiency.

[1354] 3. Data communication

[1355] The terminal transmits the preprocessed data to the server in real time.

[1356] 4. Data Analysis

[1357] The server analyzes the received data using an analysis means to detect dangerous elements, specifically by checking the color of traffic lights and approaching vehicles.

[1358] 5. Emotion recognition

[1359] The device analyzes voice and facial expression data to recognize the user's emotional state in real time, making it possible to determine whether the user is feeling stressed.

[1360] 6. Generate feedback

[1361] The server generates appropriate feedback for the user based on the analysis results and the user's emotional state. For example, if the traffic light is red, the server will provide a message saying, "The light is red. Please stop," and if the user is feeling stressed, the server will provide an additional message saying, "Take a deep breath and relax."

[1362] 7. Providing Feedback

[1363] The device will provide the generated feedback to the user via voice guidance or vibration, allowing the user to know the exact situation and take appropriate action, allowing them to act with peace of mind.

[1364] Specific examples

[1365] Text format

[1366] Please explain the detailed processing steps of a system that acquires information about the surrounding environment, recognizes the user's emotional state, and generates appropriate feedback when a visually impaired person approaches a crosswalk.

[1367] This system not only avoids danger but also provides comprehensive support that takes into account the user's emotional state, making it a technology that will enable visually impaired people to live their daily lives more safely and with peace of mind.

[1368] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1369] Step 1:

[1370] Obtaining information about the surrounding environment

[1371] The device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. As input, it acquires the current surrounding image, distance to objects, and environmental sound.

[1372] Specific operation: The device's camera captures the road conditions ahead, the distance sensor measures the distance to objects ahead, and the audio capture means collects surrounding sounds.

[1373] Output: Collected video data, distance data, and audio data are generated.

[1374] Step 2:

[1375] Data Preprocessing

[1376] The device processes the acquired raw data, using as input the video, distance, and audio data collected in step 1.

[1377] Specific operations: Compresses video data, cuts unnecessary data, and removes noise from audio data. Distance data is also simplified.

[1378] Output: Compressed video data, noise-removed audio data, and simplified distance data are generated.

[1379] Step 3:

[1380] Sending data

[1381] The device sends the preprocessed data to the server, using the compressed, denoised and simplified data generated in step 2 as input.

[1382] Specific operation: Preprocessed data is sent to the server in real time using Wi-Fi or mobile data communication.

[1383] Output: Generates data for analysis that is sent to the server.

[1384] Step 4:

[1385] Data analysis

[1386] The server parses the received data, using as input the parsing data sent in step 3.

[1387] Specific operation: Using an AI model, the color of the traffic light is recognized from the transmitted video data, and the presence or absence of approaching vehicles is analyzed from the distance data.

[1388] Output: The analysis results include the traffic light status (red / green) and information on approaching vehicles.

[1389] Step 5:

[1390] Determining risk factors

[1391] The server determines risk factors based on the analysis results, using the analysis results generated in step 4 as input.

[1392] Specific operation: Based on the analysis results, nearby obstacles and dangerous elements are evaluated and the level of danger is determined.

[1393] Output: The presence or absence of risk factors and detailed information about them is generated.

[1394] Step 6:

[1395] Recognition of emotional states

[1396] The device recognizes the user's emotional state using voice and facial expression data as input.

[1397] Specific operation: Using an AI model, the system analyzes the user's stress level from their voice and assesses their emotional state from facial expression data.

[1398] Output: The user's emotional state (stress level and relaxation level) is generated.

[1399] Step 7:

[1400] Generate feedback

[1401] The server generates feedback based on the risk factors and emotional state, using the risk factor information from step 5 and the emotional state from step 6 as input.

[1402] Specific behavior: In addition to basic instructions for dangerous elements, the system generates additional feedback depending on the user's emotional state. For example, when the traffic light is red, the system generates the instruction "The light is red. Please stop" along with the guidance "Take a deep breath and relax."

[1403] Output: A generated feedback message is generated.

[1404] Step 8:

[1405] Providing feedback

[1406] The device presents the generated feedback to the user, using as input the feedback message generated in step 7.

[1407] Specific operation: A feedback message is played using voice guidance means, and a warning is given to the user using vibration means.

[1408] Output: User received feedback, improved safety and security provided.

[1409] (Application example 2)

[1410] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] When visually impaired people shop in a physical store, it is difficult for them to accurately and quickly obtain information such as obstacles, product information, and store maps. In particular, when users are in a tense or stressed emotional state, there is a need for support for safer and more secure movement. To address these challenges, there is a need to provide a system that enables visually impaired people to move around a store safely, securely, and efficiently and obtain the information they need.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1413] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, feedback adjustment means for adjusting the feedback content based on the emotional state, and presentation means for presenting the generated feedback, thereby enabling feedback according to the emotional state to be provided, allowing the user to move around the store with peace of mind and obtain the information they need.

[1414] "Surrounding environmental information" is a general term for visual and audio information including obstacles, traffic lights, pedestrians, store information, etc. in the area where a visually impaired person is located.

[1415] "Sensor means" refers to a device such as a camera, distance sensor, or audio capture device that is used to acquire information about the surrounding environment.

[1416] The "preprocessing means" is a means for compressing the acquired environmental information and removing noise to reduce the amount of data and enable efficient communication.

[1417] "Communication means" refers to a means for transmitting preprocessed environmental information to an analysis device such as a server, and mainly includes Wi-Fi and mobile data communication.

[1418] "Analysis means" refers to a means for analyzing the communicated environmental information, and is usually a device or method that uses an AI model to analyze the environmental information and extract risk factors and necessary information.

[1419] "Determination means" refers to a means for determining risk factors and other important factors based on the analysis results.

[1420] The "feedback generating means" is a means for generating feedback to be provided to the user based on the determined risk factors and the like.

[1421] The "presentation means" refers to a means for presenting the generated feedback to the user, and mainly includes voice guidance, tactile signals, vibrations, etc.

[1422] The "emotion recognition means" is a means for analyzing the user's voice, facial expressions, etc., and recognizing the user's emotional state in real time.

[1423] The "feedback adjustment means" is a means for appropriately adjusting the content of the feedback based on the recognized emotional state of the user.

[1424] The present invention provides a system that supports visually impaired people to move around and shop in a physical store safely and with peace of mind. Detailed embodiments are described below.

[1425] System Configuration

[1426] The system mainly consists of the following components:

[1427] 1. Terminal

[1428] Sensor means: camera, distance sensor, audio acquisition means

[1429] Preprocessing means: A device that compresses the acquired environmental information and removes noise

[1430] Communication methods: Wi-Fi and mobile data communication devices

[1431] Emotion recognition means: An engine that analyzes voice data and user facial expression data to recognize emotional states.

[1432] Presentation methods: audio guidance, tactile signals, vibration

[1433] 2. Server

[1434] Analysis method: A method for analyzing environmental information using an AI model and extracting necessary information

[1435] Judgment method: A method for determining risk factors and important information from the analysis results

[1436] Feedback generation method: A method for generating feedback based on the judged content

[1437] Feedback adjustment means: A means for adjusting the feedback content based on the user's emotional state

[1438] System Operation Overview

[1439] 1. Obtaining information about the surrounding environment

[1440] The device's camera, distance sensors, and audio capture gather information about the surrounding environment, including product location, obstacles in the store, and the movements of other customers.

[1441] 2. Pretreatment

[1442] The acquired environmental information is compressed and noise is removed at the device, reducing the amount of data and enabling more efficient communication and analysis.

[1443] 3. Data communication

[1444] The pre-processed data is sent to a server using Wi-Fi or mobile data.

[1445] 4. Data Analysis and Judgment

[1446] The server uses an AI model to analyze the received data, detect the location of products in the store and the presence of obstacles, and determine which products the user is approaching and what action they should take.

[1447] 5. Emotion recognition

[1448] The device's emotion recognition engine analyzes the user's voice and facial expression data to recognize their emotional state in real time, thereby determining whether they are stressed or relaxed.

[1449] 6. Feedback Generation and Submission

[1450] The server generates feedback and sends it to the device, including additional voice prompts to help the user relax if they are feeling stressed.

[1451] 7. Feedback Presentation

[1452] The device then presents the received feedback to the user through a voice speaker or vibration, for example, "There is a sweets section on this shelf."

[1453] Example

[1454] Example 1: User in a stressed state

[1455] 1. Obtaining information about the surrounding environment

[1456] When a user approaches a product shelf, the device's camera captures the entire shelf and product information, and a distance sensor measures the distance to the shelf.

[1457] 2. Pretreatment

[1458] The acquired image and distance data are preprocessed to remove noise and improve communication efficiency.

[1459] 3. Data communication

[1460] The preprocessed data is sent to the server.

[1461] 4. Data Analysis and Judgment

[1462] The server analyzes the received data and determines that it is necessary to provide the user with information about products on the shelves.

[1463] 5. Emotion recognition

[1464] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect their level of tension.

[1465] 6. Feedback Generation and Submission

[1466] In addition to the instruction "There is a sweets section on this shelf," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the terminal.

[1467] 7. Feedback Presentation

[1468] The device will provide a voice prompt saying, "There is a sweets section on this shelf," and will also vibrate to warn you. It will also provide additional relaxation guidance such as, "Take a deep breath and relax."

[1469] Examples of prompt statements

[1470] Example prompt sentence:

[1471] "Environmental information: product shelves. User is in a state of tension. Analyze the data and generate appropriate feedback."

[1472] "Feedback request: User is approaching a product shelf and is in a stressed state. Please analyze data and generate appropriate feedback."

[1473] In this way, by adjusting the feedback provided depending on the user's emotional state, it is possible to create a system that allows visually impaired users to move around and shop in a physical store with greater confidence.

[1474] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1475] Step 1:

[1476] Obtaining information about the surrounding environment

[1477] As a user moves around a physical store, the device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. Specifically, the camera captures product location and shelf information, the distance sensor measures the distance to the product shelf, and the audio capture means records surrounding audio information. The input is visual information, distance information, and audio information, and the output is integrated data of this information.

[1478] Step 2:

[1479] Pretreatment

[1480] The device preprocesses the acquired environmental information. Specifically, it resizes the acquired video data and removes noise. Similarly, it performs noise filtering on the distance data to improve its accuracy. It also performs noise reduction on the audio data. The input is the integrated data acquired in step 1, and the output is the compressed and noise-removed data.

[1481] Step 3:

[1482] data communication

[1483] The preprocessed data is sent from the device to a server. Specifically, the compressed and denoised data is sent to the server in real time using Wi-Fi or mobile data. The input is the preprocessed data, and the output is the data sent to the server.

[1484] Step 4:

[1485] Data analysis and judgment

[1486] The server analyzes the received data. Specifically, it uses an AI model to analyze the visual, distance, and audio information just sent to identify the presence of obstacles and product information. It also determines which product information in the store should be provided based on the user's situation. The input is the data sent from the device, and the output is the analysis and judgment results.

[1487] Step 5:

[1488] emotion recognition

[1489] The device acquires the user's voice and facial expression data and analyzes it using an emotion recognition engine. Specifically, it identifies the user's emotional state (e.g., stress, tension, relaxation) in real time. The input is the user's voice and facial expression data, and the output is the recognized emotional state.

[1490] Step 6:

[1491] Feedback generation and submission

[1492] The server generates feedback based on the results of data analysis and emotion recognition. Specifically, it generates additional guidance, such as product information and warning information required by the user, as well as relaxation guidance according to the user's emotional state. The generated feedback is sent to the device in real time. The input is the analysis results and emotion recognition results, and the output is the generated feedback.

[1493] Step 7:

[1494] Feedback presentation

[1495] The device presents the received feedback to the user. Specifically, it conveys information to the user using voice guidance or haptic signals (vibrations). For example, it provides voice guidance such as "There is a sweets corner on this shelf" or relaxation guidance such as "Take a deep breath and relax." The input is feedback from the server, and the output is information presented to the user.

[1496] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1497] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1498] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1499] [Fourth embodiment]

[1500] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1501] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1502] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1503] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1504] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1505] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1506] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1507] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1508] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1509] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1510] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1511] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1512] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1513] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[1514] System Configuration

[1515] The system mainly consists of the following components:

[1516] 1. Terminal

[1517] Sensor means (camera, distance sensor, audio acquisition means)

[1518] Pretreatment means

[1519] communication means

[1520] Presentation means (audio guidance, tactile signals, vibration)

[1521] 2. Server

[1522] Analysis method (AI model)

[1523] means of judgment

[1524] Feedback Generation Method

[1525] System Operation Overview

[1526] 1. Obtaining information about the surrounding environment

[1527] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[1528] 2. Pretreatment

[1529] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[1530] 3. Data communication

[1531] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[1532] 4. Data Analysis and Judgment

[1533] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[1534] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[1535] 5. Feedback Generation and Submission

[1536] Server: Generates feedback in a format that is easy for the user to understand. Feedback data is sent to the device using voice guidance, haptic signals, vibrations, etc.

[1537] 6. Feedback Presentation

[1538] Device: Presents the received feedback to the user, such as audio guidance or vibration alerts.

[1539] Specific examples

[1540] Example 1: How to cross the crosswalk

[1541] 1. Obtaining information about the surrounding environment

[1542] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[1543] 2. Pretreatment

[1544] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[1545] 3. Data communication

[1546] The terminal transmits the pre-processed data to the server in real time.

[1547] 4. Data Analysis and Judgment

[1548] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[1549] The server determines that these elements are dangerous and concludes that the user needs to be instructed to stop them.

[1550] 5. Feedback Generation and Submission

[1551] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[1552] 6. Feedback Presentation

[1553] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[1554] This allows visually impaired users to cross the street safely. This system uses a similar process to adapt to various scenarios, providing powerful support for independent mobility for the visually impaired.

[1555] The processing flow will be explained below.

[1556] Step 1:

[1557] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[1558] Step 2:

[1559] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[1560] Step 3:

[1561] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[1562] Step 4:

[1563] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[1564] Step 5:

[1565] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[1566] Step 6:

[1567] The server generates feedback based on the risk factors it has determined. Specifically, it generates voice guidance based on text information such as "The traffic light is red. Please stop" or "A car is approaching from the right. Be careful," and also sets vibration patterns depending on the level of urgency.

[1568] Step 7:

[1569] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[1570] Step 8:

[1571] The device presents the received feedback to the user. Specifically, it plays a voice message such as "The traffic light is red. Please stop" through the device's audio speaker, and simultaneously vibrates to warn the user.

[1572] Example 1

[1573] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1574] When visually impaired people walk or move around in their daily lives, they are unable to sense information about their surroundings, which can result in accidents and danger. Systems that solve these issues and support the safe walking and movement of visually impaired people are needed. Current technology struggles to provide accurate feedback in real time, and efficient information processing and communication methods are lacking.

[1575] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1576] In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means using a generative AI model as the analysis means and determining risk factors based on the analysis results, a feedback generation means for generating feedback based on the determined risk factors, and a presentation means for presenting the generated feedback, thereby enabling visually impaired people to accurately grasp their surroundings and move safely.

[1577] "Sensor means" refers to devices or techniques used to obtain information about the surrounding environment.

[1578] The "preprocessing means" refers to a device or technology that compresses the acquired environmental information and performs processing to remove noise.

[1579] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to the server, including Wi-Fi and mobile data communications.

[1580] "Analysis means" refers to devices or techniques for analyzing the communicated environmental information. In this invention, a generative AI model is used.

[1581] A "generative AI model" is an artificial intelligence model that analyzes data and detects patterns and features.

[1582] The "judgment means" refers to a device or technology that judges risk factors based on the results of the analysis means.

[1583] The "feedback generating means" refers to a device or technology that generates feedback to the user based on the determined risk factors.

[1584] The "presentation means" refers to a device or technology that presents the generated feedback to the user through audio guidance, tactile signals, vibrations, etc.

[1585] This invention relates to a system that assists visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. The components and operation of the system are described in detail below.

[1586] System Configuration

[1587] The system consists of the following components:

[1588] 1. Terminal

[1589] Sensor means (camera, distance sensor, audio acquisition means)

[1590] Pretreatment means

[1591] communication means

[1592] Presentation means (audio guidance, tactile signals, vibration)

[1593] 2. Server

[1594] Analysis method (generative AI model)

[1595] means of judgment

[1596] Feedback Generation Method

[1597] Operation overview

[1598] 1. Sensor means: The device uses cameras, distance sensors, and audio capture means to collect information about the surrounding environment, including information about obstacles, traffic lights, pedestrians, etc.

[1599] 2. Pre-processing: The terminal compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[1600] 3. Communication method: The device sends the preprocessed data to the server using Wi-Fi or mobile data communication.

[1601] 4. Analysis method: The server analyzes the received data using the generative AI model to detect dangerous elements. The generative AI model uses deep learning algorithms to analyze traffic light colors, approaching vehicles, etc.

[1602] 5. Judgment method: The server evaluates the risk factors based on the analysis results and determines the necessary feedback.

[1603] 6. Feedback generation means: The server generates feedback in a form that is easy for the user to understand, such as voice guidance, haptic signals, vibrations, etc.

[1604] 7. Presentation: The device presents the received feedback to the user. Possible methods include voice guidance and vibration alerts.

[1605] Specific examples

[1606] Example 1: How to cross the street

[1607] 1. Obtaining information about the surrounding environment

[1608] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[1609] 2. Pretreatment

[1610] The device preprocesses the image and distance data it acquires to reduce the amount of data and remove noise.

[1611] 3. Data communication

[1612] The terminal transmits the pre-processed data to the server in real time.

[1613] 4. Data Analysis and Judgment

[1614] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby. It determines this to be a danger and concludes that the user needs to stop.

[1615] 5. Feedback Generation and Submission

[1616] The server generates voice guidance such as "The traffic light is red. Please stop" and sends it to the terminal.

[1617] 6. Feedback Presentation

[1618] The device uses a voice speaker to inform the user, "The traffic light is red. Please stop," and at the same time vibrates to alert them.

[1619] Examples of prompt statements

[1620] "Please explain the detailed processing steps of a system that helps visually impaired people cross the street."

[1621] "Please explain the specific operation of a system that detects traffic lights and vehicle movements and provides feedback."

[1622] In this way, the present invention provides an environment in which visually impaired people can walk safely, and provides detailed instructions on how to actually use the system. The system aims to improve walking safety by understanding the surrounding environment in real time and providing accurate feedback to the user.

[1623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1624] Step 1:

[1625] Obtaining information about the surrounding environment

[1626] The device acquires information about the surrounding environment using a camera, distance sensor, and audio acquisition means. For example, it collects information about the road the user is walking on, surrounding obstacles, traffic light status, and surrounding audio. This allows the device to acquire information about the user's surrounding environment in real time.

[1627] Input: Surrounding environment information (images, distance data, audio data)

[1628] Output: Raw data obtained

[1629] Step 2:

[1630] Pretreatment

[1631] The device preprocesses the raw data acquired in step 1. Unnecessary noise is removed from the camera image and important objects and information are highlighted. Data from the distance sensor is similarly processed to reduce noise. Audio data is appropriately filtered to highlight specific sounds (e.g., the sound of a car engine). This improves data quality and makes communication more efficient.

[1632] Input: Acquired raw data (images, range data, audio data)

[1633] Output: Preprocessed data (denoised data)

[1634] Step 3:

[1635] data communication

[1636] The device sends the preprocessed data to the server via a communication method. The data is sent in real time using Wi-Fi or mobile data. The data is appropriately compressed to ensure reliable communication.

[1637] Input: Preprocessed data

[1638] Output: Data sent to the server

[1639] Step 4:

[1640] Data analysis

[1641] The server receives the data sent in step 3 and analyzes it using the generative AI model. Deep learning algorithms are used to recognize traffic light status from image data and to identify approaching vehicles and obstacles from distance data. Audio data is also analyzed to detect important environmental sounds. This provides a clear overall picture of the surroundings.

[1642] Input: Data sent to the server

[1643] Output: Analysis results (traffic light status, obstacle location, audio importance)

[1644] Step 5:

[1645] Judgment

[1646] The server makes a decision based on the analysis results. For example, if the traffic light is red and there is an approaching car, it determines that it is dangerous for the user to start crossing the street. These decisions are made to ensure the user's safety.

[1647] Input: Analysis results

[1648] Output: Judgment result (specific action)

[1649] Step 6:

[1650] Feedback Generation

[1651] The server generates feedback based on the results of its judgment. The feedback is generated in a format that is easy for the user to understand (audio guidance, haptic signals, vibration, etc.). For example, audio guidance such as "The traffic light is red. Please stop" may be generated. The intensity of the vibration may also be changed to indicate the urgency of the warning.

[1652] Input: Judgment result

[1653] Output: Feedback data (audio guidance, haptic signals, vibration)

[1654] Step 7:

[1655] Feedback presentation

[1656] The device then presents the received feedback to the user, using the audio speaker to provide audio guidance and the haptic feedback device to emit vibrations, allowing the user to take appropriate action based on the surrounding situation.

[1657] Input: Feedback data

[1658] Output: Presentation to the user (audio guidance, haptic signals, vibration)

[1659] (Application example 1)

[1660] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1661] Safe and efficient mobility in daily life is always an important issue for people with visual impairments. It is particularly difficult for them to accurately assess their surroundings and take appropriate action in busy urban environments, complex intersections, and situations involving interference from other pedestrians. Conventional methods are unable to provide sufficient information or warnings to address this issue, and a fundamental solution to ensure the safety of visually impaired people is needed.

[1662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1663] In this invention, the server includes sensor means for acquiring surrounding environmental information, preprocessing means for preprocessing the acquired environmental information, communication means for communicating the preprocessed environmental information, analysis means for analyzing the communicated environmental information, determination means for determining risk factors based on the analysis results, feedback generation means for generating feedback based on the determined risk factors, presentation means for presenting the generated feedback to the user by audio guidance and tactile signals or vibrations, and adjustment means for the system to adjust the content of the generated feedback based on the individual characteristics of the user. This provides an environment in which visually impaired people can travel safely and efficiently, and enables more appropriate guidance with feedback tailored to individual characteristics.

[1664] "Surrounding environmental information" refers to external situational data necessary for visually impaired people to walk, including obstacles, traffic lights, pedestrians, vehicles, etc.

[1665] "Sensor means" refers to an apparatus or device for acquiring information about the surrounding environment, and specifically includes a camera, a distance sensor, an audio acquisition means, and the like.

[1666] "Preprocessing means" refers to a process for compressing data and removing noise to make the acquired environmental information suitable for analysis.

[1667] "Communication means" refers to the devices and technologies used to transmit preprocessed environmental information to an analytical device such as a server, and includes Wi-Fi and mobile data communications.

[1668] "Analysis means" refers to a device or software for analyzing the environmental information received on the server side and detecting risk factors.

[1669] "Determination means" refers to a device or software for evaluating the analysis results and determining safety and risk factors based on them.

[1670] "Feedback generating means" refers to a device or software for generating feedback to be provided to a user in response to a determined risk factor.

[1671] "Presentation means" refers to devices or technologies for conveying the generated feedback to the user, and specifically includes audio guidance, tactile signals, vibrations, etc.

[1672] "Adjustment means" refers to a device or software that allows the system to optimize the content of the generated feedback based on the individual characteristics of the user.

[1673] The present invention provides a system for enabling visually impaired people to walk safely. The system acquires information about the surrounding environment, analyzes the information, and provides feedback to the user. Detailed embodiments of the system are described below.

[1674] System Configuration

[1675] The system mainly consists of the following components:

[1676] 1. Device:

[1677] Sensor means: Cameras, distance sensors, and audio capture means are used.

[1678] Preprocessing means: Compress the acquired environmental information and remove noise.

[1679] Communication method: Send the preprocessed data to the server.

[1680] Presentation methods: Provide feedback to the user via audio guidance, haptic signals, and vibration.

[1681] 2. Server:

[1682] Analysis method: Analyze data using AI models to detect risk factors.

[1683] Decision-making method: Evaluate the analysis results and determine the feedback required.

[1684] Feedback generation method: Generate feedback in a format that is easy for the user to understand.

[1685] Tailoring: Tailoring feedback based on the user's individual characteristics.

[1686] Operation overview

[1687] 1. Obtaining information about the surrounding environment:

[1688] The device's camera, distance sensor, and audio capture are used to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[1689] 2. Pretreatment:

[1690] The acquired environmental information is compressed and noise is removed, thereby reducing the amount of data and enabling efficient communication.

[1691] 3. Data communication:

[1692] The preprocessed data is sent to the server via a communication means, such as Wi-Fi or mobile data.

[1693] 4. Data Analysis and Decision-Making:

[1694] The server analyzes the transmitted data using an AI model as an analytical means to detect dangerous elements, such as approaching vehicles or red lights, and evaluates the analysis results based on the judgment means to generate necessary feedback.

[1695] 5. Feedback Generation and Submission:

[1696] The server generates feedback in a format that is easy for the user to understand and transmits it to the terminal using voice guidance, haptic signals, vibration, etc. The server also optimizes the content of the feedback based on the individual characteristics of the user using an adjustment means.

[1697] 6. Feedback suggestions:

[1698] The terminal presents the received feedback to the user by voice guidance, vibration, or the like.

[1699] Specific examples

[1700] Example 1: How to cross the street

[1701] When a user approaches a crosswalk, the device's camera captures the traffic light and surrounding conditions, and the distance sensor measures the position of the car ahead. This data is preprocessed and sent to the server. The server analyzes the data and detects that the traffic light is red and that a car is approaching nearby. Having determined this, the server generates a voice message such as "The traffic light is red. Please stop" and sends it to the device. The device then uses its audio speaker to provide guidance to the user, and simultaneously vibrates to alert the user.

[1702] Prompt Sentence Examples

[1703] I would like to develop a system that uses a camera to capture the surrounding environment in real time, detects dangers, and notifies visually impaired users with voice and vibration. This system uses AI to analyze the status of approaching vehicles and traffic lights and provides feedback. Please tell me the specific prompts and steps.

[1704] This system will provide an environment in which visually impaired people can move around safely and efficiently, and will also enable more appropriate guidance through feedback tailored to individual characteristics.

[1705] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1706] Step 1:

[1707] Obtaining information about the surrounding environment

[1708] Input: Device camera, distance sensor, and audio capture method

[1709] Data processing / calculation: The device uses these sensors to obtain real-time environmental information (images, distance data, audio)

[1710] Specifically, the camera captures traffic lights, pedestrians, and vehicle movements, the distance sensor measures the distance to objects, and the audio capture means collects surrounding sounds. The captured data is temporarily stored in the device.

[1711] Output: The raw environmental data (image data, distance data, audio data) is passed to the next step.

[1712] Step 2:

[1713] Pretreatment

[1714] Input: Raw environmental data obtained in step 1

[1715] Data processing / calculation: The device compresses the data and removes noise

[1716] Specifically, the device compresses and noise filters image data, normalizes distance data, and preprocesses audio data using noise reduction techniques.

[1717] Output: Compressed and denoised preprocessed data (compressed image, normalised distance data, denoised audio data) is passed to the next step.

[1718] Step 3:

[1719] data communication

[1720] Input: Preprocessed data from step 2

[1721] Data processing / calculation: The device sends pre-processed data to the server

[1722] Specifically, the device sends the preprocessed data to the server via Wi-Fi or mobile data, where the data is also packetized and encrypted.

[1723] Output: Preprocessed data sent to the server

[1724] Step 4:

[1725] Data analysis

[1726] Input: Preprocessed data sent to the server

[1727] Data processing / calculation: The server uses an AI model to analyze the received data and detect risk factors.

[1728] Specifically, the server inputs the received image, distance, and audio data into the AI ​​model to perform object recognition, determine traffic light status, and confirm the location of obstacles, for example, analyzing approaching vehicles and the status of traffic lights.

[1729] Output: Analysis results (e.g. approaching vehicle, red light)

[1730] Step 5:

[1731] Judgment

[1732] Input: Analysis results obtained in step 4

[1733] Data processing / calculation: The server evaluates safety based on the analysis results and determines the necessary feedback

[1734] Specifically, the server evaluates the analysis results based on AI decision-making logic and determines what kind of feedback is needed for the user, such as whether an emergency stop command is necessary.

[1735] Output: Feedback decision result (e.g., stop instruction)

[1736] Step 6:

[1737] Feedback Generation

[1738] Input: Feedback decision result from step 5

[1739] Data processing / calculation: The server generates feedback in a format that is easy for the user to understand.

[1740] Specifically, the server generates text data for voice guidance and also designs vibration patterns. The voice guidance generates messages such as "Please stop."

[1741] Output: Feedback data (voice message, vibration pattern)

[1742] Step 7:

[1743] Feedback presentation

[1744] Input: Feedback data generated in step 6

[1745] Data processing / calculation: The device presents feedback data to the user

[1746] Specifically, the device will use a speaker to play voice guidance and vibrate in sync, allowing the user to receive feedback and take appropriate action.

[1747] Output: Provide feedback to the user (audio guidance, vibration)

[1748] The above is the specific processing flow of the system that realizes the application example.

[1749] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1750] This invention is a system that supports visually impaired people to walk safely and with peace of mind. In particular, it has the function of recognizing the user's emotional state and adjusting the content of feedback based on that state. Detailed embodiments of this system are described below.

[1751] System Configuration

[1752] The system mainly consists of the following components:

[1753] 1. Terminal

[1754] Sensor means (camera, distance sensor, audio acquisition means)

[1755] Pretreatment means

[1756] communication means

[1757] Presentation means (audio guidance, tactile signals, vibration)

[1758] Emotion Recognition Engine

[1759] 2. Server

[1760] Analysis method (AI model)

[1761] means of judgment

[1762] Feedback Generation Method

[1763] System Operation Overview

[1764] 1. Obtaining information about the surrounding environment

[1765] Device: Uses cameras, distance sensors, and audio capture to collect information about the surrounding environment, including obstacles, traffic lights, pedestrians, etc.

[1766] 2. Pretreatment

[1767] Terminal: Compresses acquired environmental information and removes noise, thereby reducing data volume and enabling efficient communication.

[1768] 3. Data communication

[1769] Terminal: The terminal transmits the preprocessed data to the server via a communication means. Wi-Fi and mobile data communication are used.

[1770] 4. Data Analysis and Judgment

[1771] Server: Analyzes the transmitted data using an AI model to detect dangerous elements, such as approaching vehicles or red lights.

[1772] Server: Based on the decision means, evaluates the analysis results and generates the necessary feedback.

[1773] 5. Emotion recognition

[1774] Device: The emotion recognition engine analyzes voice data and the user's facial expression data to recognize the user's emotional state in real time, thereby determining whether the user is stressed or relaxed.

[1775] 6. Feedback Generation and Submission

[1776] Server: Adjusts the content of the feedback based on the user's emotional state. For example, if the user is feeling stressed, it generates additional voice guidance to help them relax. This is also sent in combination with the regular danger avoidance feedback.

[1777] The server generates feedback and sends it to the device.

[1778] 7. Feedback Presentation

[1779] The device then presents the received feedback to the user. Specifically, it plays guidance such as "The traffic light is red. Please stop" or "Take a deep breath and relax" through the device's audio speaker, and simultaneously vibrates to warn the user.

[1780] Specific examples

[1781] Example 1: A user experiencing stress

[1782] 1. Obtaining information about the surrounding environment

[1783] As the user approaches a crosswalk, the device's camera captures the traffic lights and surrounding area, while the distance sensor measures the position of the car ahead.

[1784] 2. Pretreatment

[1785] The video and distance data acquired by the device is preprocessed to reduce the amount of data and remove noise.

[1786] 3. Data communication

[1787] The terminal transmits the pre-processed data to the server in real time.

[1788] 4. Data Analysis and Judgment

[1789] The server analyzes the received data and detects that the traffic light is red and that a car is approaching nearby.

[1790] The server determines that these elements are dangerous and that the user needs to be instructed to stop them.

[1791] 5. Emotion recognition

[1792] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect when they are in a state of tension.

[1793] 6. Feedback Generation and Submission

[1794] In addition to the instruction "The traffic light is red, please stop," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the device.

[1795] 7. Feedback Presentation

[1796] The device will provide a voice prompt saying, "The traffic light is red. Please stop," along with a vibration warning. It will also provide additional relaxation guidance, saying, "Take a deep breath and relax."

[1797] In this way, by taking into account the user's emotional state, the system can not only avoid danger but also provide psychological support, enabling visually impaired users to move about in their daily lives with greater peace of mind.

[1798] The processing flow will be explained below.

[1799] Step 1:

[1800] The device acquires information about the surrounding environment. Specifically, the camera captures images of the front at 30 frames per second, the distance sensor measures the distance to surrounding obstacles 10 times per second, and the audio acquisition means captures environmental sounds in real time.

[1801] Step 2:

[1802] The device preprocesses the acquired environmental information. Specifically, it compresses image data into JPEG format, filters out noise from distance data, and performs noise reduction on audio data. The preprocessed data is then packaged into packets.

[1803] Step 3:

[1804] The device sends the pre-processed data to the server via a communication means, such as Wi-Fi or mobile data, transmitting compressed data packets via the TCP / IP protocol.

[1805] Step 4:

[1806] The data received by the server is analyzed using an AI model. Specifically, camera data is analyzed using a Convolutional Neural Network (CNN) to identify objects (traffic lights, people, cars, etc.). This is combined with distance data to create spatial information, and specific sounds (such as a car horn) can be identified from audio data.

[1807] Step 5:

[1808] The server determines potential dangers based on the analysis results. Specifically, it identifies potential dangers the user may face based on the type and location of objects detected by the AI ​​model. For example, it determines whether a traffic light is red or if a car is approaching.

[1809] Step 6:

[1810] The device uses an emotion recognition engine to assess the user's emotional state. Specifically, it analyzes voice data and facial expression data to recognize stress, tension, relaxation, etc.

[1811] Step 7:

[1812] The server generates feedback based on the user's emotional state. Specifically, it generates instructions for avoiding danger (e.g., "The traffic light is red. Please stop") as well as feedback encouraging relaxation (e.g., "Take a deep breath and relax")

[1813] Step 8:

[1814] The server transmits the generated feedback to the terminal via a communication means. Specifically, the server compresses the feedback data including the voice guide data and the vibration pattern and transmits the compressed data to the terminal.

[1815] Step 9:

[1816] The device then presents the received feedback to the user. Specifically, the device will announce "The traffic light is red. Please stop" through its audio speaker, and will also vibrate to warn the user. The device will also provide additional relaxation guidance, such as "Take a deep breath and relax."

[1817] Example 2

[1818] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1819] Conventional support systems for the visually impaired do not take into account the user's emotional state, and therefore are insufficient in providing support in stressful situations. In addition to acquiring environmental information, there is a need for feedback that reflects the user's psychological state.

[1820] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a sensor means for acquiring surrounding environmental information, a preprocessing means for preprocessing the acquired environmental information, a communication means for communicating the preprocessed environmental information, an analysis means for analyzing the communicated environmental information, a determination means for determining risk factors based on the analysis results, an emotion recognition means for recognizing the emotional state of the user based on the risk factors, a feedback generation means for generating feedback based on the determined risk factors and the emotional state, and a presentation means for presenting the generated feedback. This enables safe and secure walking assistance that takes the emotional state of the user into consideration.

[1821] "Surrounding environment information" refers to information about objects and situations that exist around the user, including obstacles, traffic lights, pedestrians, vehicles, and the like.

[1822] "Sensor means" refers to a hardware device for acquiring information about the surrounding environment, and includes a camera, a distance sensor, an audio acquisition means, and the like.

[1823] "Preprocessing means" refers to a processing mechanism for compressing acquired raw data and removing noise to reduce the amount of data and enable efficient communication.

[1824] "Communication means" refers to the mechanism for transmitting preprocessed data to the server, and Wi-Fi or mobile data communication is used.

[1825] "Analysis means" refers to the means for analyzing the data transmitted within the server, and uses AI models and machine learning algorithms.

[1826] The "judging means" refers to a means for evaluating and judging risk factors based on the analysis results obtained by the analyzing means.

[1827] "Emotion recognition means" refers to means for recognizing the user's emotional state, and evaluates the user's stress and relaxation state from voice data and facial expression data.

[1828] "Feedback generating means" refers to means for generating feedback to provide to a user based on the determined risk factors and emotional state.

[1829] "Presentation means" refers to means for conveying the generated feedback to the user, and includes audio guidance, tactile signals, and vibrations.

[1830] This invention relates to a system that supports visually impaired people to walk safely and with peace of mind. It is particularly characterized by its ability to recognize the user's emotional state and adjust feedback based on that state. This system is comprised of a terminal and a server, and specific embodiments of the system are described in detail below.

[1831] System Configuration

[1832] 1. Terminal

[1833] Sensor means: Cameras, distance sensors, and audio capture means are used to acquire information about the surrounding environment, such as obstacles, traffic lights, pedestrians, and vehicles.

[1834] Preprocessing means: Compresses the acquired environmental information and removes noise, thereby reducing the amount of data and enabling efficient communication.

[1835] Communication method: Preprocessed data is sent to the server via Wi-Fi or mobile data communication.

[1836] Emotion recognition means: Analyzes voice data and facial expression data to recognize the user's emotional state in real time.

[1837] Presentation methods: Use audio prompts, haptic signals, and vibrations to provide feedback to the user.

[1838] 2. Server

[1839] Analysis method: Using AI models and machine learning algorithms, data sent from the device is analyzed to detect risk factors.

[1840] Judgment method: Based on the analysis results, evaluate risk factors and determine the need for feedback.

[1841] Feedback generation means: Generates appropriate feedback based on the determined risk factors and the user's emotional state.

[1842] Operation overview

[1843] 1. Obtaining information about the surrounding environment

[1844] The device uses a camera, distance sensors, and audio capture means to collect information about the user's surrounding environment, such as the status of traffic lights, approaching vehicles, and ambient noise.

[1845] 2. Pretreatment

[1846] The raw data acquired by the terminal is compressed and noise is removed, thereby improving communication efficiency.

[1847] 3. Data communication

[1848] The terminal transmits the preprocessed data to the server in real time.

[1849] 4. Data Analysis

[1850] The server analyzes the received data using an analysis means to detect dangerous elements, specifically by checking the color of traffic lights and approaching vehicles.

[1851] 5. Emotion recognition

[1852] The device analyzes voice and facial expression data to recognize the user's emotional state in real time, making it possible to determine whether the user is feeling stressed.

[1853] 6. Generate feedback

[1854] The server generates appropriate feedback for the user based on the analysis results and the user's emotional state. For example, if the traffic light is red, the server will provide a message saying, "The light is red. Please stop," and if the user is feeling stressed, the server will provide an additional message saying, "Take a deep breath and relax."

[1855] 7. Providing Feedback

[1856] The device will provide the generated feedback to the user via voice guidance or vibration, allowing the user to know the exact situation and take appropriate action, allowing them to act with peace of mind.

[1857] Specific examples

[1858] Text format

[1859] Please explain the detailed processing steps of a system that acquires information about the surrounding environment, recognizes the user's emotional state, and generates appropriate feedback when a visually impaired person approaches a crosswalk.

[1860] This system not only avoids danger but also provides comprehensive support that takes into account the user's emotional state, making it a technology that will enable visually impaired people to live their daily lives more safely and with peace of mind.

[1861] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1862] Step 1:

[1863] Obtaining information about the surrounding environment

[1864] The device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. As input, it acquires the current surrounding image, distance to objects, and environmental sound.

[1865] Specific operation: The device's camera captures the road conditions ahead, the distance sensor measures the distance to objects ahead, and the audio capture means collects surrounding sounds.

[1866] Output: Collected video data, distance data, and audio data are generated.

[1867] Step 2:

[1868] Data Preprocessing

[1869] The device processes the acquired raw data, using as input the video, distance, and audio data collected in step 1.

[1870] Specific operations: Compresses video data, cuts unnecessary data, and removes noise from audio data. Distance data is also simplified.

[1871] Output: Compressed video data, noise-removed audio data, and simplified distance data are generated.

[1872] Step 3:

[1873] Sending data

[1874] The device sends the preprocessed data to the server, using the compressed, denoised and simplified data generated in step 2 as input.

[1875] Specific operation: Preprocessed data is sent to the server in real time using Wi-Fi or mobile data communication.

[1876] Output: Generates data for analysis that is sent to the server.

[1877] Step 4:

[1878] Data analysis

[1879] The server parses the received data, using as input the parsing data sent in step 3.

[1880] Specific operation: Using an AI model, the color of the traffic light is recognized from the transmitted video data, and the presence or absence of approaching vehicles is analyzed from the distance data.

[1881] Output: The analysis results include the traffic light status (red / green) and information on approaching vehicles.

[1882] Step 5:

[1883] Determining risk factors

[1884] The server determines risk factors based on the analysis results, using the analysis results generated in step 4 as input.

[1885] Specific operation: Based on the analysis results, nearby obstacles and dangerous elements are evaluated and the level of danger is determined.

[1886] Output: The presence or absence of risk factors and detailed information about them is generated.

[1887] Step 6:

[1888] Recognition of emotional states

[1889] The device recognizes the user's emotional state using voice and facial expression data as input.

[1890] Specific operation: Using an AI model, the system analyzes the user's stress level from their voice and assesses their emotional state from facial expression data.

[1891] Output: The user's emotional state (stress level and relaxation level) is generated.

[1892] Step 7:

[1893] Generate feedback

[1894] The server generates feedback based on the risk factors and emotional state, using the risk factor information from step 5 and the emotional state from step 6 as input.

[1895] Specific behavior: In addition to basic instructions for dangerous elements, the system generates additional feedback depending on the user's emotional state. For example, when the traffic light is red, the system generates the instruction "The light is red. Please stop" along with the guidance "Take a deep breath and relax."

[1896] Output: A generated feedback message is generated.

[1897] Step 8:

[1898] Providing feedback

[1899] The device presents the generated feedback to the user, using as input the feedback message generated in step 7.

[1900] Specific operation: A feedback message is played using voice guidance means, and a warning is given to the user using vibration means.

[1901] Output: User received feedback, improved safety and security provided.

[1902] (Application example 2)

[1903] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1904] When visually impaired people shop in a physical store, it is difficult for them to accurately and quickly obtain information such as obstacles, product information, and store maps. In particular, when users are in a tense or stressed emotional state, there is a need for support for safer and more secure movement. To address these challenges, there is a need to provide a system that enables visually impaired people to move around a store safely, securely, and efficiently and obtain the information they need.

[1905] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1906] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, feedback adjustment means for adjusting the feedback content based on the emotional state, and presentation means for presenting the generated feedback, thereby enabling feedback according to the emotional state to be provided, allowing the user to move around the store with peace of mind and obtain the information they need.

[1907] "Surrounding environmental information" is a general term for visual and audio information including obstacles, traffic lights, pedestrians, store information, etc. in the area where a visually impaired person is located.

[1908] "Sensor means" refers to a device such as a camera, distance sensor, or audio capture device that is used to acquire information about the surrounding environment.

[1909] The "preprocessing means" is a means for compressing the acquired environmental information and removing noise to reduce the amount of data and enable efficient communication.

[1910] "Communication means" refers to a means for transmitting preprocessed environmental information to an analysis device such as a server, and mainly includes Wi-Fi and mobile data communication.

[1911] "Analysis means" refers to a means for analyzing the communicated environmental information, and is usually a device or method that uses an AI model to analyze the environmental information and extract risk factors and necessary information.

[1912] "Determination means" refers to a means for determining risk factors and other important factors based on the analysis results.

[1913] The "feedback generating means" is a means for generating feedback to be provided to the user based on the determined risk factors and the like.

[1914] The "presentation means" refers to a means for presenting the generated feedback to the user, and mainly includes voice guidance, tactile signals, vibrations, etc.

[1915] The "emotion recognition means" is a means for analyzing the user's voice, facial expressions, etc., and recognizing the user's emotional state in real time.

[1916] The "feedback adjustment means" is a means for appropriately adjusting the content of the feedback based on the recognized emotional state of the user.

[1917] The present invention provides a system that supports visually impaired people to move around and shop in a physical store safely and with peace of mind. Detailed embodiments are described below.

[1918] System Configuration

[1919] The system mainly consists of the following components:

[1920] 1. Terminal

[1921] Sensor means: camera, distance sensor, audio acquisition means

[1922] Preprocessing means: A device that compresses the acquired environmental information and removes noise

[1923] Communication methods: Wi-Fi and mobile data communication devices

[1924] Emotion recognition means: An engine that analyzes voice data and user facial expression data to recognize emotional states.

[1925] Presentation methods: audio guidance, tactile signals, vibration

[1926] 2. Server

[1927] Analysis method: A method for analyzing environmental information using an AI model and extracting necessary information

[1928] Judgment method: A method for determining risk factors and important information from the analysis results

[1929] Feedback generation method: A method for generating feedback based on the judged content

[1930] Feedback adjustment means: A means for adjusting the feedback content based on the user's emotional state

[1931] System Operation Overview

[1932] 1. Obtaining information about the surrounding environment

[1933] The device's camera, distance sensors, and audio capture gather information about the surrounding environment, including product location, obstacles in the store, and the movements of other customers.

[1934] 2. Pretreatment

[1935] The acquired environmental information is compressed and noise is removed at the device, reducing the amount of data and enabling more efficient communication and analysis.

[1936] 3. Data communication

[1937] The pre-processed data is sent to a server using Wi-Fi or mobile data.

[1938] 4. Data Analysis and Judgment

[1939] The server uses an AI model to analyze the received data, detect the location of products in the store and the presence of obstacles, and determine which products the user is approaching and what action they should take.

[1940] 5. Emotion recognition

[1941] The device's emotion recognition engine analyzes the user's voice and facial expression data to recognize their emotional state in real time, thereby determining whether they are stressed or relaxed.

[1942] 6. Feedback Generation and Submission

[1943] The server generates feedback and sends it to the device, including additional voice prompts to help the user relax if they are feeling stressed.

[1944] 7. Feedback Presentation

[1945] The device then presents the received feedback to the user through a voice speaker or vibration, for example, "There is a sweets section on this shelf."

[1946] Example

[1947] Example 1: User in a stressed state

[1948] 1. Obtaining information about the surrounding environment

[1949] When a user approaches a product shelf, the device's camera captures the entire shelf and product information, and a distance sensor measures the distance to the shelf.

[1950] 2. Pretreatment

[1951] The acquired image and distance data are preprocessed to remove noise and improve communication efficiency.

[1952] 3. Data communication

[1953] The preprocessed data is sent to the server.

[1954] 4. Data Analysis and Judgment

[1955] The server analyzes the received data and determines that it is necessary to provide the user with information about products on the shelves.

[1956] 5. Emotion recognition

[1957] The device's emotion recognition engine analyzes the user's voice and facial expressions to detect their level of tension.

[1958] 6. Feedback Generation and Submission

[1959] In addition to the instruction "There is a sweets section on this shelf," the server generates additional voice guidance such as "Take a deep breath and relax" and sends it to the terminal.

[1960] 7. Feedback Presentation

[1961] The device will provide a voice prompt saying, "There is a sweets section on this shelf," and will also vibrate to warn you. It will also provide additional relaxation guidance such as, "Take a deep breath and relax."

[1962] Examples of prompt statements

[1963] Example prompt sentence:

[1964] "Environmental information: product shelves. User is in a state of tension. Analyze the data and generate appropriate feedback."

[1965] "Feedback request: User is approaching a product shelf and is in a stressed state. Please analyze data and generate appropriate feedback."

[1966] In this way, by adjusting the feedback provided depending on the user's emotional state, it is possible to create a system that allows visually impaired users to move around and shop in a physical store with greater confidence.

[1967] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1968] Step 1:

[1969] Obtaining information about the surrounding environment

[1970] As a user moves around a physical store, the device acquires information about the surrounding environment using a camera, distance sensor, and audio capture means. Specifically, the camera captures product location and shelf information, the distance sensor measures the distance to the product shelf, and the audio capture means records surrounding audio information. The input is visual information, distance information, and audio information, and the output is integrated data of this information.

[1971] Step 2:

[1972] Pretreatment

[1973] The device preprocesses the acquired environmental information. Specifically, it resizes the acquired video data and removes noise. Similarly, it performs noise filtering on the distance data to improve its accuracy. It also performs noise reduction on the audio data. The input is the integrated data acquired in step 1, and the output is the compressed and noise-removed data.

[1974] Step 3:

[1975] data communication

[1976] The preprocessed data is sent from the device to a server. Specifically, the compressed and denoised data is sent to the server in real time using Wi-Fi or mobile data. The input is the preprocessed data, and the output is the data sent to the server.

[1977] Step 4:

[1978] Data analysis and judgment

[1979] The server analyzes the received data. Specifically, it uses an AI model to analyze the visual, distance, and audio information just sent to identify the presence of obstacles and product information. It also determines which product information in the store should be provided based on the user's situation. The input is the data sent from the device, and the output is the analysis and judgment results.

[1980] Step 5:

[1981] emotion recognition

[1982] The device acquires the user's voice and facial expression data and analyzes it using an emotion recognition engine. Specifically, it identifies the user's emotional state (e.g., stress, tension, relaxation) in real time. The input is the user's voice and facial expression data, and the output is the recognized emotional state.

[1983] Step 6:

[1984] Feedback generation and submission

[1985] The server generates feedback based on the results of data analysis and emotion recognition. Specifically, it generates additional guidance, such as product information and warning information required by the user, as well as relaxation guidance according to the user's emotional state. The generated feedback is sent to the device in real time. The input is the analysis results and emotion recognition results, and the output is the generated feedback.

[1986] Step 7:

[1987] Feedback presentation

[1988] The device presents the received feedback to the user. Specifically, it conveys information to the user using voice guidance or haptic signals (vibrations). For example, it provides voice guidance such as "There is a sweets corner on this shelf" or relaxation guidance such as "Take a deep breath and relax." The input is feedback from the server, and the output is information presented to the user.

[1989] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1990] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1991] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1992] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1993] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1994] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1995] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1996] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1997] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1998] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1999] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2000] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2001] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2002] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2003] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2004] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2005] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2006] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2007] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2008] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2009] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2010] The following is further disclosed regarding the above embodiment.

[2011] (Claim 1)

[2012] a sensor means for acquiring surrounding environmental information;

[2013] a preprocessing means for preprocessing the acquired environmental information;

[2014] communication means for communicating the preprocessed environmental information;

[2015] analysis means for analyzing the communicated environmental information;

[2016] a determination means for determining risk factors based on the analysis results;

[2017] feedback generating means for generating feedback based on the determined risk factors;

[2018] presentation means for presenting the generated feedback;

[2019] A system including:

[2020] (Claim 2)

[2021] The system according to claim 1, wherein the sensor means comprises a camera, a distance sensor, and an audio capture means.

[2022] (Claim 3)

[2023] 2. The system of claim 1, wherein the feedback generating means uses audio guidance means, tactile signal means, or vibration means.

[2024] "Example 1"

[2025] (Claim 1)

[2026] a sensor means for acquiring surrounding environmental information;

[2027] a preprocessing means for preprocessing the acquired environmental information;

[2028] communication means for communicating the preprocessed environmental information;

[2029] analysis means for analyzing the communicated environmental information;

[2030] A generating AI model is used as an analysis means, and a judgment means is used to judge risk factors based on the analysis results;

[2031] feedback generating means for generating feedback based on the determined risk factors;

[2032] presentation means for presenting the generated feedback;

[2033] A system including:

[2034] (Claim 2)

[2035] The system according to claim 1, wherein the sensor means comprises a camera, a distance sensor, and an audio capture means.

[2036] (Claim 3)

[2037] 2. The system of claim 1, wherein the feedback generating means uses audio guidance means, tactile signal means, or vibration means.

[2038] "Application Example 1"

[2039] (Claim 1)

[2040] a sensor means for acquiring surrounding environmental information;

[2041] a preprocessing means for preprocessing the acquired environmental information;

[2042] communication means for communicating the preprocessed environmental information;

[2043] analysis means for analyzing the communicated environmental information;

[2044] a determination means for determining risk factors based on the analysis results;

[2045] feedback generating means for generating feedback based on the determined risk factors;

[2046] a presentation means for presenting the generated feedback to the user by audio guidance and haptic signals or vibrations;

[2047] an adjusting means for adjusting the content of the generated feedback based on the individual characteristics of the user;

[2048] A system including:

[2049] (Claim 2)

[2050] 2. The system according to claim 1, wherein the sensor means comprises a camera, a distance sensor, and an audio capture means, and further comprises a communication means for transmitting the preprocessed data to a server.

[2051] (Claim 3)

[2052] 2. The system according to claim 1, wherein the feedback generating means includes means for analyzing the status of approaching vehicles and traffic lights and providing real-time feedback to the user by voice guidance and vibration.

[2053] "Example 2: Combining Emotion Engines"

[2054] (Claim 1)

[2055] a sensor means for acquiring surrounding environmental information;

[2056] a preprocessing means for preprocessing the acquired environmental information;

[2057] communication means for communicating the preprocessed environmental information;

[2058] analysis means for analyzing the communicated environmental information;

[2059] a determination means for determining risk factors based on the analysis results;

[2060] emotion recognition means for recognizing an emotional state of a user based on risk factors;

[2061] feedback generating means for generating feedback based on the determined risk factors and emotional state;

[2062] presentation means for presenting the generated feedback;

[2063] A system including:

[2064] (Claim 2)

[2065] The system according to claim 1, wherein the sensor means comprises a camera, a distance sensor, and an audio capture means.

[2066] (Claim 3)

[2067] 2. The system of claim 1, wherein the feedback generating means uses audio guidance means, tactile signal means, or vibration means.

[2068] "Application example 2 when combining emotion engines"

[2069] (Claim 1)

[2070] a sensor means for acquiring surrounding environmental information;

[2071] a preprocessing means for preprocessing the acquired environmental information;

[2072] communication means for communicating the preprocessed environmental information;

[2073] analysis means for analyzing the communicated environmental information;

[2074] a determination means for determining risk factors based on the analysis results;

[2075] feedback generating means for generating feedback based on the determined risk factors;

[2076] a presentation means for presenting the generated feedback;

[2077] emotion recognition means for recognizing an emotional state of a user;

[2078] feedback adjustment means for adjusting the feedback content based on the emotional state;

[2079] A system including:

[2080] (Claim 2)

[2081] The system according to claim 1, wherein the sensor means comprises a camera, a distance sensor, and an audio capture means.

[2082] (Claim 3)

[2083] 2. The system of claim 1, wherein the feedback generating means uses audio guidance means, tactile signal means, or vibration means. [Explanation of symbols]

[2084] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a sensor means for acquiring surrounding environmental information; a preprocessing means for preprocessing the acquired environmental information; communication means for communicating the preprocessed environmental information; analysis means for analyzing the communicated environmental information; a determination means for determining risk factors based on the analysis results; feedback generating means for generating feedback based on the determined risk factors; presentation means for presenting the generated feedback; A system including:

2. 2. The system according to claim 1, wherein the sensor means comprises a camera, a distance sensor, and an audio capture means.

3. 2. The system of claim 1, wherein the feedback generating means uses audio guidance means, tactile signal means, or vibration means.

Images