System

Abstract

To provide a system.SOLUTION: A system includes: means for providing information to substitute a field of view by effectively utilizing generative AI; means for accessing AI via a smart device or a headset; means for acquiring information of an environment in real time; means for recognizing a peripheral object or person; and means for transferring information in the form of voice, audio message, or tactile feedback.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Understanding the surrounding environment and moving around safely are major challenges for the visually impaired in their daily lives. In order to promote their social participation, there is a demand for methods to provide information that complements visual information. [Means for solving the problem]

[0005] This invention utilizes generative AI to provide information that replaces vision. Users can access the AI ​​via a smart device or headset and obtain information about the environment in real time. The AI ​​recognizes surrounding objects and people and conveys information in the form of voice, audio messages, haptic feedback, etc. This supports safe movement and understanding of the environment for visually impaired people, improving their social participation. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In this embodiment of the present invention, a visually impaired person wears a smart device or headset and accesses the generative AI. The generative AI analyzes information obtained from cameras and sensors and provides the visually impaired person with information about the surrounding environment. Specifically, the AI ​​recognizes the position, shape, and movement of objects and people, and communicates this to the visually impaired person through voice messages and haptic feedback.

[0029] "Example 2"

[0030] As a concrete example, when a visually impaired person walks through the city, the generative AI recognizes obstacles, intersections, traffic light status, etc. ahead and conveys them to the visually impaired person through voice messages. It also infers people's presence and movements from their movements and voices, and provides haptic feedback to the visually impaired person. This enables the visually impaired person to understand the situation around them and move around safely.

[0031] "Example 3"

[0032] Furthermore, generative AI can help visually impaired people participate in society. For example, it can analyze people's reactions and the atmosphere at meetings and events, and convey that to visually impaired people. It can also read the text information in books and documents that visually impaired people want to read, and read it aloud. This allows visually impaired people to participate in social activities more richly and independently.

[0033] The process flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: A visually impaired person wears a smart device or headset and accesses the generative AI.

[0036] Step 2: The generative AI analyzes the information obtained from the camera and sensors.

[0037] Step 3: The AI ​​recognizes the position, shape, and movement of objects and people, and communicates this to the visually impaired through voice messages and haptic feedback.

[0038] "Example 2"

[0039] Step 1: When a visually impaired person walks around town, the generative AI recognizes obstacles, intersections, traffic light status, etc. ahead.

[0040] Step 2: The AI ​​infers people's presence and movements from their movements and voices.

[0041] Step 3: The AI ​​will communicate information to the visually impaired through voice messages and provide haptic feedback, allowing them to understand their surroundings and navigate safely.

[0042] "Example 3"

[0043] Step 1: Generative AI helps visually impaired people participate in society, for example by analyzing people's reactions and the atmosphere at meetings and events.

[0044] Step 2: The AI ​​reads the text information in the books and materials that the visually impaired person wants to read.

[0045] Step 3: The AI ​​will read out the information aloud and convey it to the visually impaired, enabling them to participate in social activities more richly and independently.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0048] With current technology, visually impaired people face many challenges when moving independently, and there are limited means to safely and effectively obtain information about their surroundings. In particular, the lack of a system that provides detailed information on the position, shape, and movement of people and objects in real time is a major factor that limits the freedom of movement of visually impaired people and reduces their opportunities for social participation. For this reason, new technological solutions are needed to help visually impaired people better understand their surrounding environment and move safely.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0050] In this invention, the server includes a means for analyzing environmental data using a generative AI model, a means for converting the analysis results into voice messages and haptic feedback, and a means for processing the collected environmental data in real time. This allows visually impaired people to receive detailed information about the positions, shapes, and movements of people and objects around them in real time through voice and haptic feedback. As a result, it is possible to support safe mobility for visually impaired people and expand opportunities for social participation.

[0051] "Generative AI" is a type of artificial intelligence that learns from data and generates models and algorithms to perform specific tasks.

[0052] A "smart device" is a portable electronic device that can connect to the Internet and provide multiple functions by running applications.

[0053] A "headset" is a device that generally has audio output (speaker) and audio input (microphone) functions and is worn on the head.

[0054] "Means of acquiring environmental information in real time" is a process or technology that instantly captures the current situation and changes and provides information without delay.

[0055] "Means for recognizing surrounding objects and people" refers to technology that analyzes visual, auditory, or other sensor information to determine the presence and location of objects or people within a particular environment.

[0056] "Means of conveying information in the form of voice, audio messages, tactile feedback, etc." are methods of providing information to users using sensory means such as sound or vibration.

[0057] The present invention relates to a system for assisting visually impaired people in understanding their surroundings and navigating safely, the system being implemented using specific hardware and software.

[0058] The servers host generative AI models that are designed to analyze data collected from the environment and identify information such as the location, shape, and movement of objects and people. The software used includes, for example, customized versions of OpenAI's GPT and Google's TensorFlow, which provide complex data analysis and learning capabilities that increase the accuracy and efficiency of the system.

[0059] The terminals are devices worn by the visually impaired, such as smart devices or headsets. These devices are equipped with cameras and sensors to collect environmental data. The collected data is sent to a server and analyzed by a generative AI model.

[0060] The analysis results are sent back to the device and provided to the visually impaired as audio messages and / or haptic feedback, helping users better understand their surroundings and providing them with information to navigate safely.

[0061] A specific example is the prompt, "Please identify the location and movement of people and objects around you and provide that information in a voice message," which dictates the type of data that will be analyzed by the generative AI model.

[0062] This embodiment of the invention focuses on leveraging advanced technology to solve everyday challenges faced by the visually impaired, with a combination of hardware and software to provide real-time environmental information to help them become more independent.

[0063] The flow of the identification process will be described with reference to FIG.

[0064] Step 1: The device collects surrounding environment data from the cameras and sensors installed in the smart device or headset worn by the visually impaired person. The input of this step is the video, audio, temperature, distance, and other data of the surrounding environment acquired by the device's camera and sensors. The output is the collected raw environmental data.

[0065] Step 2: The terminal transmits the collected environmental data to the server. In this step, data transmission is performed using wireless communication technology. The input is the environmental data collected in step 1, and the output is the data transmitted to the server.

[0066] Step 3: The server analyzes the received environmental data using the generative AI model. This analysis includes processes such as object recognition, person recognition, and motion detection. The input is the environmental data received in step 2, and the output is the analysis result. This analysis result includes information such as the position, shape, and movement of surrounding objects and people.

[0067] Step 4: The server generates information useful to visually impaired people based on the analysis results. This information is provided in the form of voice messages and haptic feedback. The input is the analysis results from step 3, and the output is information to be conveyed to visually impaired people.

[0068] Step 5: The terminal conveys the information received from the server to the visually impaired person using the terminal's built-in speaker and haptic feedback device. The input is the information generated in step 4, and the output is a voice message or haptic feedback that can be recognized by the visually impaired person.

[0069] An example of a specific operation is processing based on the prompt, "Please identify the positions and movements of people and objects around you and convey that information in a voice message." In this case, the device uses a camera to capture images of the surroundings and sends the data to the server. The server analyzes the data, identifies the positions and movements of people and objects, and sends this information back to the device as a voice message for the visually impaired person. The device then conveys this message to the visually impaired person through a speaker.

[0070] (Application example 1)

[0071] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] Currently, the main challenge that visually impaired people face when shopping in brick-and-mortar stores is the difficulty of understanding the layout of the store and the location of products. Another major challenge is moving safely while avoiding obstacles. These challenges prevent visually impaired people from enjoying shopping independently and limit their opportunities for social participation.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0074] In this invention, the server includes a means for accessing the generating AI via a smart device or a headset, a means for acquiring and analyzing information on the in-store environment in real time, a means for recognizing surrounding objects and people and identifying their positions and shapes, and a means for conveying information to visually impaired people through voice messages and haptic feedback, thereby enabling visually impaired people to grasp the positions of products and the presence of obstacles in a physical store and to shop efficiently while moving around safely.

[0075] "Generative AI" is a type of artificial intelligence that analyzes environmental data and generates and provides useful information to users based on the results of that analysis.

[0076] A "smart device" is a portable electronic device that can connect to the Internet and perform multiple functions.

[0077] A "headset" is a device that can be worn on the ears to receive voice communication and voice guidance.

[0078] "Real-time environmental information acquisition" is the process of collecting and updating data about the surrounding environment in close sync with the current time.

[0079] "Means of recognizing surrounding objects and people" is a technology that identifies and identifies the presence, position, and shape of objects and people from input from cameras, sensors, etc.

[0080] A "voice or audio message" is a method of transmitting information in the form of sound waves that can be heard by the human ear.

[0081] "Haptic feedback" is a technology that provides users with information through their sense of touch, conveying information through vibrations and changes in pressure.

[0082] An embodiment of the invention includes the use of generative AI via a smart device or headset. The server uses a generative AI model to analyze environmental data sent from the smart device or headset. This analysis identifies the position, shape, and movement of surrounding objects and people, and conveys information to the visually impaired person through audio messages and haptic feedback.

[0083] The hardware used includes smart devices and headsets. These devices acquire real-time environmental data through built-in cameras and sensors and transmit it to a server. The server uses software libraries such as OpenCV and TensorFlow to perform object detection and image analysis, and conveys the obtained information to the user through a voice synthesis system.

[0084] For example, when a user puts on the smart glasses and enters a store, voice guidance is provided such as, "There is a product shelf ahead. The safe route is to go straight." This information is generated by analyzing video data captured by the smart glasses' camera. Also, if a user is looking for a specific product, the generating AI provides specific instructions such as, "Turn right and you will find the product." This allows visually impaired people to efficiently find the products they need while moving around safely.

[0085] The flow of the identification process will be described with reference to FIG.

[0086] Step 1: The device's camera and sensors acquire surrounding environmental data. In this step, the cameras and sensors in the smart device or headset worn by the user collect information on surrounding objects, people, obstacles, etc. in real time. The input is data from optical images and distance sensors, and the output is sent to the server as digitized environmental data.

[0087] Step 2: The server analyzes the environmental data. The received environmental data is analyzed by the generative AI model to identify the position, shape, and movement of surrounding objects and people. The input of this step is the digitized environmental data from step 1, and the output is the object identification information and movement patterns as the analysis results.

[0088] Step 3: Based on the analysis results, the server generates voice messages and haptic feedback in a format that is easy for the user to understand based on the analysis results. Examples include product location guidance and obstacle warnings. The input is the object identification information and movement patterns from step 2, and the output is the voice messages and haptic feedback provided through the user's smart device or headset.

[0089] Step 4: The user adjusts their behavior based on the provided information. Based on the voice messages and haptic feedback they receive, the user may adjust their route through the store or search for a specific product. The input to this step is the information from step 3, and the output is a change in the user's behavior, such as moving in a particular direction or picking up a product.

[0090] This process allows visually impaired people to shop safely and efficiently in physical stores.

[0091] Example 2

[0092] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Currently, the biggest challenge that visually impaired people face when walking around town is that they cannot directly obtain the information they need to move safely, such as the presence of obstacles, traffic conditions, and the movements of people around them. This exposes them to great anxiety and danger when they move around. Another major problem is that it is difficult for visually impaired people to communicate and synchronize their activities with those around them when participating in society.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0095] In this invention, the server includes a means for collecting data from external sensors in real time, a means for analyzing the surrounding environment and the situation of people from the collected data using a generative AI model, and a means for providing appropriate instructions and information to visually impaired people in the form of voice messages and haptic feedback based on the analysis results, which enables visually impaired people to more accurately understand the surrounding environment, move safely, and easily communicate and synchronize their activities when participating in society.

[0096] "Means of collecting environmental data from external sensors in real time" refers to the process of instantly acquiring data about the current surrounding environment through sensors such as cameras, microphones, and GPS mounted on smart devices or dedicated devices, and sending it to a server for analysis.

[0097] "Means of analyzing the positions and movements of surrounding objects and people from collected environmental data using a generative AI model" refers to a method in which an artificial intelligence model trained using machine learning technology extracts and interprets information such as the positions of objects and the movements of people based on collected data.

[0098] "Means of providing instructions and information to visually impaired persons in the form of voice messages and haptic feedback based on analysis results" refers to a method of directly transmitting necessary information and appropriate instructions to users based on the results of analysis by AI, using formats such as voice and vibration that are easy for visually impaired persons to understand.

[0099] "Means of accessing AI via smart devices or dedicated devices and transmitting information in real time" refers to the process of communicating with AI on a server through internet-connected devices such as smartphones, tablets, and dedicated wearable devices, sending collected data, and receiving analysis results.

[0100] In the embodiment of the present invention, a system is provided to enable visually impaired people to understand their surrounding environment and navigate safely. The system mainly consists of three main components: data collection from external sensors, data analysis by generative AI models, and information provision through voice messages and haptic feedback.

[0101] The server uses cloud-based computing resources to host generative AI models that run on machine learning frameworks such as TensorFlow and PyTorch and are used to analyze data collected from external sensors, such as image data from cameras and audio data from microphones built into smart devices and dedicated devices.

[0102] The terminal is a smart device or dedicated device carried by the visually impaired person, which collects environmental data through external sensors such as a camera, microphone, and GPS, and transmits it to a server. In addition, it provides voice messages and haptic feedback to the visually impaired person based on instructions from the server. For example, if the server sends an instruction to the terminal saying, "There is an obstacle 3 meters ahead. Move left," the terminal outputs this message in voice and provides instructions to the visually impaired person to avoid the obstacle.

[0103] Based on the voice messages and haptic feedback provided, the user can understand the surrounding situation and move safely by avoiding obstacles. In addition, the haptic feedback indicating the movement and presence of people can help the user avoid collisions with other pedestrians. In this way, the present invention provides a concrete method for the visually impaired to move more independently and safely.

[0104] The flow of the identification process will be described with reference to FIG.

[0105] Step 1: Collect data from external sensors

[0106] The terminal collects environmental data from the visually impaired person's smart device or dedicated device through external sensors such as cameras, microphones, and GPS. The input of this step is the surrounding images, audio, and location information, and the output is a digitized form of these data. The terminal temporarily stores the data and converts it into a format that can be transmitted over the network in order to send it to the server.

[0107] Step 2: Analyze data and generate instructions

[0108] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding environment, the location of obstacles, people's movements, etc. from this data. The input for this step is image data, audio data, and location information, and the output is the analysis results such as the location of obstacles and people's movements. Based on the analysis results, the generative AI model generates specific instructions and information to be provided to the visually impaired person.

[0109] Step 3: Provide information

[0110] The terminal provides the instructions and information received from the server to the visually impaired person as voice messages and haptic feedback. The input of this step is the instructions and information sent from the server, and the output is the voice message and haptic feedback received directly by the visually impaired person. The terminal outputs the instructions as voice messages using a voice synthesis technology that converts text to speech, and provides the necessary information as a tactile sensation through a haptic feedback device.

[0111] Through these steps, visually impaired people can better understand their surroundings, receive specific instructions to avoid obstacles, and navigate safely with information about the movements and positions of people around them. This process happens in real time, significantly improving support for visually impaired people in participating in society.

[0112] (Application example 2)

[0113] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] In modern society, personal safety is a major concern. Accurately understanding the perception of the surrounding environment and people and their impact on safety is essential to ensure personal safety, especially when going out alone or at night. However, current technology has limited effective means to support personal safety in a given environment in real time and immediately warn of the approach of suspicious people or the occurrence of an emergency. Against this background, there is a need for the development of supportive technologies that allow individuals to actively manage their own safety and protect themselves from danger.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0116] In this invention, the server includes a means for acquiring environmental information in real time, a means for recognizing surrounding objects and people, and a means for transmitting information to the user via voice, voice message, and haptic feedback. This allows the user to grasp the surrounding situation in real time and increase safety awareness. In addition, by using this system, a warning can be received immediately when a suspicious person approaches or an emergency occurs, and appropriate measures can be taken. In this way, this invention provides important support for ensuring personal safety.

[0117] "Generative AI" is a type of artificial intelligence based on machine learning technology, a system that recognizes environmental information, objects, and the state of people based on input data, and generates and provides useful information to users.

[0118] "Means of acquiring environmental information in real time" refers to technologies and methods that use devices such as sensors, cameras, and microphones to instantly capture and analyze the surrounding situation and sounds at that moment.

[0119] "Means for recognizing surrounding objects and people" refers to techniques or methods that use image recognition technology or voice recognition technology to identify the characteristics of surrounding objects and people and understand that information.

[0120] "Voice or audio message" is a form of audible linguistic information used to convey system-generated information to the user.

[0121] "Haptic feedback" is a means of conveying information from a device to the user through physical sensations such as vibration or pressure.

[0122] In the embodiment of the present invention, a smartphone, smart glasses, or a head-mounted display is used as a terminal, and surrounding environmental information is acquired in real time through a camera or microphone built into these terminals. The acquired information is transmitted to a server equipped with a generative AI model. The server uses a deep learning framework to recognize surrounding objects and people through image recognition technology and voice recognition technology, and generates an appropriate response according to the situation. The generated information is communicated to the user in the form of voice, voice message, or haptic feedback.

[0123] The hardware used includes smartphones with Qualcomm Snapdragon processors and wearable devices such as Google Glass, while the software uses machine learning frameworks such as TensorFlow and PyTorch.

[0124] As a specific example, if a user is walking at night and the camera captures a suspicious person approaching, the server generates a voice message saying, "Caution: Someone is approaching from behind. Please move to a safe place immediately," and transmits this to the user through the terminal. In addition, in the event of an emergency, it is also possible to provide haptic feedback saying, "An emergency has occurred. Please evacuate to a safe place immediately." In this way, the present invention provides a concrete means for the user to grasp the situation around them in real time and ensure their safety.

[0125] The flow of the identification process will be described with reference to FIG.

[0126] Step 1: The device collects surrounding environment information. In this step, the device collects surrounding video and audio in real time using the camera and microphone built into the smartphone, smart glasses, or head-mounted display. At this time, the device takes video and audio data as input and prepares to send it to the server.

[0127] Step 2: Send the collected environmental information to the server. The terminal sends the video and audio data collected in step 1 to the server. In this process, the data is transferred to the server via the Internet.

[0128] Step 3: Server analyzes environmental information. The video and audio data sent to the server is analyzed by a generative AI model. This AI model uses a deep learning framework to recognize surrounding objects and people from the video, and analyze environmental sounds and conversations from the audio data. The output of this step is the analysis results, which include specific situational awareness and identification of potential hazards.

[0129] Step 4: Generate an appropriate response based on the analysis results. Based on the analysis results obtained in step 3, the server generates information to be communicated to the user. This information is about the appropriate action the user should take at that moment, such as safety warnings and instructions. The generated information can take the form of a voice message or haptic feedback.

[0130] Step 5: Communicate the generated information to the user. The server sends the information generated in step 4 to the terminal, which then communicates the information to the user. The information is communicated to the user through the speaker in the case of a voice message, or by using the device's vibration function in the case of haptic feedback. In this step, the user receives specific information to ensure their own safety.

[0131] Example 3

[0132] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0133] When visually impaired people participate in social activities such as meetings and events, they cannot visually grasp the reactions and atmosphere of those around them. In addition, it is difficult for them to independently read the text information in books and materials they want to read. These challenges are factors that limit the independent participation of visually impaired people in social activities.

[0134] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0135] In this invention, the server includes a means for analyzing reactions and atmosphere at a meeting or event, a means for reading out text information of documents or materials, and a means for interpreting prompts input by voice and generating appropriate responses, thereby enabling visually impaired people to understand the surrounding situation in social activities and obtain necessary information independently.

[0136] "Generative AI" is a type of artificial intelligence that can generate new information based on learned data.

[0137] "A means of analyzing reactions and atmosphere at meetings and events" refers to the ability to understand the emotions and mood of participants at a particular gathering or event and express them in verbal or other forms.

[0138] "A means for converting the textual information of a document or material into audible speech" refers to a process or device that converts printed or electronically provided textual information into audio output.

[0139] "Means of accessing AI via smart devices and headsets" refers to methods of using intelligent electronic devices and audio output devices to communicate with and interact with artificial intelligence systems.

[0140] "Means of acquiring environmental information in real time" refers to the function of instantly collecting and analyzing the surrounding situation and data at the current time.

[0141] "Means of recognizing surrounding objects and people" refers to technologies and methods that use cameras, sensors, etc. to identify and identify nearby objects and people.

[0142] "Means of conveying information in the form of voice, audio messages, haptic feedback, etc." refers to methods of directly transmitting information to the user using sound waveforms, vibrations, etc.

[0143] "Ability to interpret voice prompts and generate appropriate responses" refers to the artificial intelligence's ability to understand spoken instructions or questions from a user and provide information or a response to them.

[0144] The embodiment of the present invention is mainly composed of three parties: a server, a terminal, and a user. The server is equipped with a generative AI model, which is designed to solve the problems that visually impaired people face when participating in social activities. The terminal mainly takes the form of a smart device or a headset, and plays a role in transmitting input from the user to the server. The user is a visually impaired person, and can acquire and deepen his / her understanding of information in social activities through this system.

[0145] The generative AI model that runs on the server is implemented using advanced technologies such as OpenAI's GPT and Google's BERT. This model analyzes video and audio data from meetings and events, and extracts the atmosphere and reactions of the event as text information. It also has the ability to read text information from images of documents and materials and convert it into audio data.

[0146] The device receives the user's voice prompts through a microphone, converts them into text data, and sends them to the server. It also uses the device's camera to take images of documents and materials and send them to the server. The server processes the received data with a generative AI model to generate an appropriate response.

[0147] As a specific example of use, if a user voice-inputs a prompt such as "Tell me about the atmosphere of the current meeting," the device converts this input into text and sends it to the server. The generative AI model on the server analyzes and extracts information about the atmosphere of the meeting and sends the results as text to the device. The device converts this text into speech and reads it aloud to the user. In addition, if a user requests "Read this page," the device takes an image of the document and sends it to the server. The server extracts text from the image, converts it into speech, and sends it to the device. The device reads this speech aloud to the user.

[0148] This format allows visually impaired people to receive specific support to help them acquire and understand information in social activities.

[0149] The flow of the identification process will be described with reference to FIG.

[0150] Step 1: The user speaks to the device, including a prompt about the mood of the meeting or event and a request for a document to be read aloud. The device receives the speech and converts it into text data.

[0151] Step 2: The converted text data is sent over the internet to a server, which receives the data and prepares it for processing by the generative AI model.

[0152] Step 3: For requests related to the atmosphere of a meeting or event, the server uses the generative AI model to analyze the atmosphere and reactions from the provided audio and video data. For requests to read a document aloud, the server receives the image data of the document sent from the device and performs text extraction.

[0153] Step 4: The generative AI model generates an appropriate text response based on the analyzed or extracted information. For a request about the atmosphere of the meeting, it generates a response summarizing the emotions and key points of the meeting, and for a request to read a document, it generates the extracted text information.

[0154] Step 5: The generated text response is sent from the server to the device, which receives the text and converts it into audio data using its built-in text-to-speech conversion function.

[0155] Step 6: The converted voice data is output to the user, allowing the user to understand the atmosphere of the meeting or event and hear the contents of the document.

[0156] Through this processing flow, visually impaired people can acquire information and deepen their understanding in social activities, enabling them to participate more independently.

[0157] (Application example 3)

[0158] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] When visually impaired people shop at a store, they face the challenge of being unable to grasp the location and information of products, the atmosphere of the store, and the reactions of people. For this reason, visually impaired people may feel restricted in their ability to shop independently, limiting their opportunities to participate in society. Another problem is the lack of systems that efficiently provide the information necessary for safe movement and product selection at a store.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0161] In this invention, the server includes a means for providing audio guidance to visually impaired people based on analysis of product information and people's reactions in a sales area using a generation AI, a means for acquiring environmental information in real time, analyzing it, and generating audio guidance information, and a means for utilizing environmental data including product positions and sale information in the sales area. This enables visually impaired people to safely and efficiently select products and move around in sales areas, expanding opportunities for social participation.

[0162] "Generative AI" is an algorithm based on machine learning and deep learning techniques, and is a system that generates output from specific input in a format that is easy for humans to understand.

[0163] A "visually impaired person" refers to a person who has difficulty recognizing general visual information due to a visual impairment.

[0164] "Product information in the sales area" includes all information necessary for purchasing a product, such as the type of product, price, location, and sale information.

[0165] "Voice guidance based on analysis of people's reactions" refers to analyzing the facial expressions, words, actions, and other reactions of people around you and providing voice instructions and information based on that information.

[0166] "Means of acquiring real-time environmental information" refers to methods or technologies that continuously collect surrounding environmental data in accordance with the current time.

[0167] "Means for generating voice guidance information" refers to the process of analyzing collected information and data and using that information to provide useful instructions and information in voice format to visually impaired people.

[0168] "Environmental data including product location and sales information within the sales area" is general data about the state and condition of the sales area, including information indicating where products are located and which products are on sale.

[0169] A "server" is a computer system or software that collects, processes, stores data, and provides information to clients.

[0170] The embodiment of the present invention relates to a system that supports visually impaired people in shopping at a sales point. The system utilizes a generative AI model to provide voice guidance based on an analysis of product information and people's reactions in the sales point. The server has a function of acquiring environmental information in real time, analyzing it, and generating voice guidance information. The server also has a function of utilizing environmental data including product positions and sale information in the sales point. Terminals such as smart devices and headsets are used to implement the system, and visually impaired people access the generative AI through these terminals. The terminal has a function of conveying information to the user in the form of voice, audio messages, haptic feedback, etc.

[0171] The hardware used includes smart devices (e.g. smartphones), in-store cameras and sensors, and the software uses voice recognition and synthesis systems, location services, and generative AI models (e.g. OpenAI's GPT-4). This system allows visually impaired people to safely and efficiently select and navigate products at sales points, expanding their opportunities for social participation.

[0172] For example, when a user uses a smart device to ask, "What sweets are on sale nearby?", the server analyzes the in-store environmental data and uses a generative AI model to generate guidance information such as, "There is chocolate on sale near you. It's on the shelf about 3 meters to your right," and conveys this information to the user via a voice message on the device. This process combines the user's location information with in-store product information to provide optimal guidance.

[0173] The flow of the identification process will be described with reference to FIG.

[0174] Step 1: The user inputs a question into the terminal.

[0175] Users use their smart devices to voice-input questions about specific product information or in-store directions, which are then converted into text data by the device's on-board voice recognition system.

[0176] Step 2: The device sends the text data of the question to the server.

[0177] The converted text data is sent over the Internet to a server, which receives the text data and uses it as input for processing.

[0178] Step 3: The server parses the text data and sends prompts to the generative AI model to generate appropriate responses.

[0179] The server analyzes the received text data and forms prompts to generate specific responses based on the user's question, which are then sent to a generative AI model for use in generating responses.

[0180] Step 4: The generative AI model generates a response based on the prompt.

[0181] The generative AI model processes the prompts received from the server and generates appropriate responses to the user's questions, which may include in-store product locations, sales information, and descriptions of the in-store environment.

[0182] Step 5: Convert the server-generated response into audio data.

[0183] The server converts the response text received from the generative AI model into voice data using a voice synthesis system, which is then used to convey information to the user.

[0184] Step 6: The server sends the voice data to the terminal.

[0185] The voice data is again transmitted over the Internet to the user's terminal, which receives the voice data and outputs it to the user.

[0186] Step 7: The user listens to the audio output to obtain the information.

[0187] The user hears product information and in-store guidance information in response to questions through voice output from the terminal, allowing the user to obtain the necessary information and shop efficiently at the sales point.

[0188] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0189] "Example 1"

[0190] In one embodiment of the present invention, the generation AI has an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotions from the user's tone of voice, facial expressions, behavior, etc., and provides the information to the generation AI. The generation AI adjusts the method of providing information to the user based on the information provided by the emotion engine. For example, if the user feels angry, the generation AI will calm the way in which information is communicated, or communicate only the necessary information. This makes it possible to respond appropriately according to the user's emotional state.

[0191] "Example 2"

[0192] In another embodiment of the present invention, the emotion engine also analyzes the user's biometric information (e.g., heart rate and skin electrical conductivity). This allows the user's emotional state to be grasped more accurately. The generation AI infers the user's emotional state based on this biometric information and adjusts the method of providing information. For example, if the user feels tense, the generation AI takes measures to improve the user's emotional state, such as providing relaxing music.

[0193] "Example 3"

[0194] Furthermore, in another embodiment of the present invention, the generation AI adjusts the intensity and pattern of the haptic feedback according to the user's emotional state. For example, if the user feels surprised, the generation AI provides strong haptic feedback to attract the user's attention. On the other hand, if the user feels relaxed, the generation AI provides gentle haptic feedback. This makes it possible to provide haptic feedback appropriate to the user's emotional state.

[0195] The process flow of each embodiment will be described below.

[0196] "Example 1"

[0197] Step 1: The emotion engine starts working, inferring emotions from the user's tone of voice, facial expressions, behavior, etc.

[0198] Step 2: The emotion engine provides the inferred emotion information to the generative AI.

[0199] Step 3: The generative AI adjusts how it presents information to the user based on the information provided by the emotion engine.

[0200] "Example 2"

[0201] Step 1: The emotion engine analyzes the user's biometric information (e.g., heart rate and skin conductivity).

[0202] Step 2: The generative AI uses this biometric information to infer the user's emotional state and adjusts the way it provides information.

[0203] Step 3: The generative AI responds to improve the user's emotional state. For example, if the user feels tense, the generative AI provides relaxing music.

[0204] "Example 3"

[0205] Step 1: The generative AI obtains the user's emotional state from the emotion engine.

[0206] Step 2: The generative AI adjusts the intensity and pattern of haptic feedback depending on the user's emotional state.

[0207] Step 3: The generative AI provides calibrated haptic feedback to the user.

[0208] Example 1

[0209] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0210] Visually impaired people face the challenge of having difficulty accurately understanding the surrounding environment when engaging in daily life and social activities. Conventional assistive technologies only provide environmental information one-way, and there is also a problem in that they do not provide information that takes into account the user's emotional state. This can make it difficult for users to accept information or to respond appropriately in emergency situations.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0212] In this invention, the server includes a means for analyzing environmental data transmitted from a mobile terminal or a headset, a means for estimating the user's emotional state, and a means for adjusting the information provision method based on the analyzed environmental information and emotional information. This allows visually impaired people to understand the surrounding environment more accurately and easily, and provides appropriate information according to their emotional state. This adaptive information provision also enables safe mobility support and improved social participation for visually impaired people.

[0213] "Generative AI" is an algorithm trained using machine learning and deep learning techniques, and is a system capable of generating or analyzing new information from input data.

[0214] "Environmental data" refers to information about the physical situation and conditions surrounding the user, such as images, audio, and location information, collected through sensors and cameras.

[0215] A "mobile terminal" is an electronic device, such as a smartphone or tablet, that is portable and has wireless communication capabilities.

[0216] A "headset" is a device worn on the ears that enables voice communication and listening to music. In support systems for the visually impaired, a headset is used to transmit voice messages and accept voice input.

[0217] "User's emotional state" refers to the user's current psychological and emotional state, and is inferred from tone of voice, facial expressions, behavior, etc.

[0218] "Adjusting the way information is presented" refers to appropriately changing the way information is conveyed (e.g., the tone of a voice message or the intensity of haptic feedback) based on the received environmental data and the user's emotional state.

[0219] The embodiment of the present invention is centered around a mobile device or headset worn by a visually impaired person and a generative AI model that runs on a server that controls them. The mobile device or headset includes a camera and sensors for recognizing the environment, and a microphone for inferring the user's emotional state. Data collected through these hardware devices is transmitted to the server using wireless communication technology.

[0220] On the server, a generative AI model using machine learning frameworks such as TensorFlow and PyTorch analyzes the transmitted data. This analysis includes identifying the location, shape, and movement of objects and people using image recognition technology, and inferring the user's emotional state using voice analysis technology. Based on the analysis results, the generative AI model determines the optimal way to provide information to the user and conveys that information to the user via a mobile device or headset in the form of a voice message or haptic feedback.

[0221] For example, while a user is walking in a park, the camera on the mobile device captures a bench and people walking in front of them, and this data is sent to the server. The generative AI model on the server analyzes the position of the bench and the movement of the people walking from this image, and provides the user with a voice message saying, "There is a bench 3 meters to the right. There is a person walking ahead." At the same time, it infers the emotion from the user's tone of voice and adjusts the tone of the message as necessary.

[0222] In this way, this invention provides an information provision system that integrates both environmental recognition and emotion estimation. This allows visually impaired people to understand their surroundings in more detail and more appropriately, and to receive information customized according to their emotional state. This system is expected to contribute to improving the quality of daily life of visually impaired people and promoting their participation in society.

[0223] The flow of the identification process will be described with reference to FIG.

[0224] Step 1: Collect environmental data

[0225] The device collects data about the surrounding environment using cameras and sensors. This data includes images, audio, and location information. The input is raw data from the environment, and the output is a digitalized version of the raw data. Through this conversion process, the device transmits visible light images, audio data, and location information from GPS to a server.

[0226] Step 2: Data analysis and sentiment estimation

[0227] The server analyzes the received environmental data using a generative AI model and an emotion engine. The generative AI model performs image recognition to identify the location, shape, and movement of people and objects. The emotion engine infers the user's emotional state from voice data. The input is the environmental data sent from the device, and the output is object identification information and the user's emotional state. Through this process, the server generates detailed information about both the surrounding environment and the user's emotions.

[0228] Step 3: Decide how to provide information

[0229] The generative AI model on the server combines the analyzed environmental information and emotional information to determine how to provide information to the user. In this process, it is possible to adapt the information delivery method to the user's emotional state. The inputs are environmental information and the user's emotional state, and the output is an instruction for the adapted information delivery method. The server sends the information delivery method to the terminal based on this instruction.

[0230] Step 4: Communicate

[0231] The device provides information to the user through voice messages and haptic feedback according to the instructions received from the server. In this step, the input is an instruction from the server, and the output is a transmission of information in a form that the user can understand. Through this process, the device provides the user with a voice message such as "There is a pedestrian crossing ahead" or haptic feedback indicating that someone is nearby.

[0232] (Application example 1)

[0233] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] In modern society, visually impaired people face many difficulties in moving around and in their daily lives. In particular, when they go out, it is difficult for them to accurately grasp their surroundings, which is an obstacle to safe movement and participation in society. In addition, because the methods of transmitting information are limited, there is a lack of means to reduce the anxiety and stress felt by visually impaired people. Under these circumstances, there is a demand for the development of support methods that enable visually impaired people to participate in society more safely and with confidence.

[0235] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0236] In this invention, the server includes a means for analyzing environmental information using a generating AI and providing the user with a voice message or haptic feedback in place of visual information based on the analysis result, a means for accessing the AI ​​via a smart device or a headset and acquiring environmental data in real time, a means for recognizing surrounding objects and people, their movements and shapes using a camera or a sensor, and a means having an emotion engine for predicting the user's emotional state and adjusting the method of providing information based on that information. This enables visually impaired people to understand the surrounding environment more accurately and realize safe movement and social participation. Furthermore, by adopting an information transmission method according to the user's emotional state, it is possible to reduce the anxiety and stress felt by the user and support more comfortable social participation.

[0237] "Generative AI" is a system that uses artificial intelligence technology to analyze environmental data and provide information to users based on the results of that analysis.

[0238] A "voice message" is a means of verbal communication, a message that conveys information to the user by voice.

[0239] "Haptic feedback" is a means of transmitting information through the sense of touch, a method of providing information to the user through physical stimulation from a device.

[0240] A "smart device" is a portable electronic device that has the ability to connect to the Internet and integrates multiple smart functions.

[0241] A "headset" is a device worn on the head that allows for voice communication and audio output, and in this context is used to receive audio messages and haptic feedback.

[0242] "Cameras and sensors" are devices that capture the surrounding environment and objects as images and data, and this information is used by the generative AI to analyze it.

[0243] An "emotion engine" is a technology or system that infers a user's emotional state from their tone of voice, facial expressions, behavior, etc., and adjusts the way information is provided based on that information.

[0244] The embodiment of this invention is a system for visually impaired people to understand the surrounding environment more safely and effectively when going out or in daily life. The system includes generative AI, smart devices and headsets, cameras and sensors, and an emotion engine.

[0245] The server is equipped with a generative AI that analyzes the environmental data sent from the cameras and sensors. The analyzed data includes information such as the position, shape, and movement of objects, and this information is conveyed to the user through voice messages and haptic feedback. The smart device or headset acts as an interface for the user to receive this information and transmits the environmental data to the generative AI in real time.

[0246] The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and behavior, and provides that information to the generation AI. The generation AI then adjusts the way it provides information based on this emotional information. For example, if the user is feeling anxious or scared, it will provide the necessary information in a calmer tone of voice.

[0247] To realize this system, cameras, sensors, and an emotion engine must be built into smart devices or headsets, and algorithms must be developed on the server side for the generative AI to process this data and generate appropriate feedback.

[0248] For example, when a user is approaching an intersection, the camera and sensors capture the user's position and the movement of surrounding vehicles, and the generation AI uses this information to generate a voice message to the user through the headset saying, "A car is approaching from the right. Please cross when it is safe." Also, if the emotion engine detects that the user is nervous, the generation AI adjusts the way the information is conveyed to be more gentle.

[0249] This enables visually impaired people to understand their environment more safely and effectively, achieving greater independence in their daily lives and social participation.

[0250] The flow of the identification process will be described with reference to FIG.

[0251] Step 1: Collect environmental data

[0252] The camera and sensors installed on the device collect data about the user's surroundings, including the position, shape, and movement of objects, and the collected data is sent to a server in real time.

[0253] Step 2: Inferring emotional state

[0254] The device's on-board emotion engine infers the user's emotional state from their tone of voice, facial expressions, and behavior, and this inferred emotional state is sent to the server to tailor the way information is presented.

[0255] Step 3: Analyze data and generate information

[0256] The generative AI installed on the server analyzes the environmental data and emotional state data collected and sent in steps 1 and 2. Based on this analysis, useful information for the user is generated, such as how to cross an intersection safely or information about surrounding objects.

[0257] Step 4: Communicate

[0258] The generated information is sent from the server to the device, which then conveys it to the user in the form of voice messages and haptic feedback. The method of conveying the information is adjusted based on the user's emotional state. For example, if the user is feeling anxious, the information is conveyed in a calmer tone of voice.

[0259] Step 5: Receiving and responding to feedback

[0260] The user takes action based on the information provided, while the user's reactions and additional emotional state are fed back to the server via the device, and this feedback is used to improve the accuracy of the system.

[0261] Through this process, users will gain a deeper understanding of their surroundings, allowing them to safely and effectively participate in their daily lives and society. The system will also be more responsive to the user's emotional state, providing more personalized information.

[0262] Example 2

[0263] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0264] One of the main challenges faced by visually impaired people is the difficulty of safely and effectively perceiving the surrounding environment when going out. This is due to various factors, such as the presence of obstacles, people's movements, and changing traffic conditions. Furthermore, if the way information is presented is not adapted to the user's emotional state and individual needs, it can cause anxiety and stress. Conventional assistive technologies cannot provide sufficient solutions to these challenges.

[0265] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0266] In this invention, the server includes a means for providing information to substitute for the field of view using a generating AI, a means for acquiring environmental information in real time and recognizing surrounding objects and people, and a means for analyzing the user's biometric information and inferring the user's emotional state. This allows visually impaired people to safely recognize their surroundings and receive voice messages and haptic feedback to support their movement. In addition, the method of providing information is adjusted according to the user's emotional state, reducing the user's stress and allowing them to go out more comfortably.

[0267] "Generative AI" is a type of artificial intelligence, a technology that generates new information and data from input data.

[0268] "Alternatives to vision" are information based on senses other than vision (hearing, touch, etc.) that are used by people who cannot rely on vision to understand their surroundings and situations.

[0269] A "smart device" is an electronic device that can connect to the Internet and other devices and has multiple functions.

[0270] A "headset" is a device, typically worn on the head, that provides both audio input (microphone) and output (headphones).

[0271] "Means for acquiring environmental information in real time" refers to techniques and processes for instantly collecting and analyzing information from the user's surrounding environment.

[0272] "Means of recognizing surrounding objects and people" refers to technologies that use cameras, sensors, voice recognition technology, etc. to identify and understand the presence, location, and actions of objects and people.

[0273] "Voice, audio messages, and haptic feedback" are means of communicating information to the user through non-visual means such as sound or vibration.

[0274] "User's biological information" is data that indicates the user's physical condition, such as heart rate, electrical conductivity of the skin, etc.

[0275] "Means for inferring emotional state" is a technology that analyzes and understands a user's emotional and psychological state at any given time from their biometric information and behavioral patterns.

[0276] "Means for adjusting the method of information provision" are techniques or processes that optimize the method of information delivery (e.g., message content and presentation) according to the user's condition and needs.

[0277] The embodiment of the present invention is a system for supporting visually impaired people to understand the surrounding environment and move safely when they go out. This system is configured by utilizing a generative AI model, a smart device, a sensor, a headset, and a biometric information measuring device.

[0278] The generative AI model deployed on the server analyzes the location and movement of surrounding objects and people, as well as the user's biometric information, based on data sent from smart devices and sensors. This analysis involves advanced data processing and data calculation. The hardware used includes high-resolution cameras, microphones, heart rate sensors, and skin conductivity sensors, while the software includes image recognition algorithms, voice recognition algorithms, and emotional state analysis algorithms.

[0279] The terminal refers to a smart device (smartphone or tablet) that collects information about the surroundings using a built-in camera and microphone and transmits it to the server. The terminal also receives responses from the server and conveys them to the user as voice messages or haptic feedback. The headset is used to convey voice messages directly to the user.

[0280] As a concrete example, consider a situation where a user is walking down a street. The device's camera captures obstacles ahead, and the generative AI model generates a voice message based on the prompt sentence "There is a chair 3 meters ahead," which is conveyed to the user through the headset. At the same time, the device uses a microphone to capture surrounding sounds, analyzes people's movements, and provides haptic feedback through a haptic device worn on the wrist. Based on the emotional state indicated by the user's biometric information, if it determines that "the user's heart rate is high and therefore tense," the generative AI model processes the prompt sentence to provide relaxing music.

[0281] This format allows visually impaired people to obtain a wealth of information about their surroundings, and the provision of information that takes into account their emotional state increases their sense of security when out and about.

[0282] The flow of the identification process will be described with reference to FIG.

[0283] Step 1: Collect environmental data

[0284] The device uses its built-in camera, microphone, and biometric sensors to collect environmental data and biometric information about the user. The inputs for this step are the sights and sounds of the surroundings, the user's heart rate, and the electrical conductivity of the skin, and the output is a digital representation of the collected environmental data and biometric information. The device converts this data into digital information for processing.

[0285] Step 2: Send and analyze data

[0286] The collected data is transmitted to a server via wireless communication. The generative AI model on the server analyzes the transmitted environmental data and biometric information to identify the positions, movements, and emotional state of surrounding objects and people. The inputs of this step are environmental data and biometric information, and the outputs are the identification information of objects and people and the emotional state of the user. The server analyzes the data and infers the identification information and the emotional state.

[0287] Step 3: Coordinate input and generate feedback

[0288] The generative AI model adjusts the content and format of the information provided to the user based on the analysis results. If the user is tense, it selects music to relax them, provides the location of obstacles with voice messages, and conveys the movements of people around them with haptic feedback. The input for this step is the identification information of objects and people and the user's emotional state, and the output is the information provided, such as voice messages, haptic feedback, and music. The server generates appropriate feedback based on the analysis results and sends it to the terminal.

[0289] Step 4: Communicate feedback

[0290] The terminal receives the feedback sent from the server and transmits voice messages to the user through the headset, haptic feedback through the haptic device, and music through the terminal's speaker or headset. The input of this step is the feedback from the server, and the output is the provision of information in a form that the user can recognize. The terminal transmits the received feedback to the user through an appropriate output device.

[0291] Through these steps, visually impaired people can perceive their environment more safely and efficiently and receive customized support based on their emotional state.

[0292] (Application example 2)

[0293] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] Currently, when users, including visually impaired people, use autonomous vehicles, it is difficult for them to accurately recognize the surrounding environment and situation and travel safely and comfortably. In particular, it is difficult for them to understand the status of surrounding vehicles, obstacles, and traffic lights, and information is not provided or the environment is not adjusted according to the user's emotional state, which can increase anxiety and stress while traveling.

[0295] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0296] In this invention, the server includes a means for recognizing dynamic elements of the environment in real time and providing information, a means for accessing AI via an in-vehicle system or a headset, a means for analyzing input data from sensors and cameras to obtain environmental information around the vehicle, a means for recognizing surrounding vehicles, obstacles, and traffic light conditions, and a means for adjusting an emotional state based on voice, voice messages, haptic feedback, and biometric information. This allows the user to accurately understand the surrounding environment and situation and travel safely and comfortably. In addition, by providing information and adjusting the environment according to the user's emotional state, anxiety and stress during travel can be reduced, resulting in a more comfortable travel experience.

[0297] "Generative AI" is a type of artificial intelligence technology, a system that analyzes complex data and generates new information and inferences based on it.

[0298] "Means of recognizing dynamic elements of the environment in real time and providing information" refers to the function of analyzing input data from sensors, cameras, etc., instantly detecting changing conditions in the surroundings (vehicle movement, presence of obstacles, changes in traffic lights, etc.), and conveying that information to the user.

[0299] "Means of accessing AI via in-vehicle systems or headsets" refers to interfaces that communicate with the generative AI and exchange information through embedded systems in autonomous vehicles or headsets worn by the user.

[0300] "Means of analyzing input data from sensors and cameras" refers to the function of processing and analyzing in real time data collected by sensors and cameras installed to capture environmental data around the vehicle, and grasping environmental information.

[0301] "Means of recognizing surrounding vehicles, obstacles, and traffic signal conditions" refers to technologies and systems that enable autonomous vehicles to accurately understand the situation of surrounding objects, other vehicles, traffic signals, etc., and make safe driving decisions based on this information.

[0302] "Means for adjusting emotional state based on voice or voice messages, haptic feedback, and biometric information" refers to a function that provides information to the user through voice or haptic feedback, and further analyzes the user's biometric information to infer their emotional state, and provides information or adjusts the environment accordingly.

[0303] The embodiment of the present invention is a support system for users, including visually impaired people, to move safely and comfortably in an autonomous vehicle. This system is composed of a combination of on-board sensors and cameras, a generative AI model, a voice synthesis engine, a haptic feedback device, a biometric sensor, etc.

[0304] The server receives input data from the vehicle's sensors and cameras, analyzes it, and recognizes the vehicle's surroundings. This environmental information includes other vehicles, obstacles, traffic lights, etc. Using a generative AI model, the server generates voice messages and haptic feedback for the user based on this information. It also analyzes data collected from the user's biometric sensors (e.g., heart rate and skin electrical conductivity) to infer the user's emotional state. Depending on the user's emotional state, the method of providing information and the environment are adjusted to improve the user's psychological comfort.

[0305] For example, if the server detects another vehicle approaching from the right side of the vehicle, it will use the generative AI model to generate a voice message saying, "A vehicle is approaching from the right. Please be careful," and convey this to the user through the voice synthesis engine. At the same time, if the user feels tense, it will send a prompt to the generative AI model saying, "Please provide relaxing music," and play music to relax the user.

[0306] In this form, hardware and software such as cameras and sensors installed in the vehicle, haptic feedback devices, biometric sensors such as heart rate sensors and skin conductivity sensors, generative AI models, and voice synthesis engines play an important role. By combining these appropriately, it is possible to provide all users, including the visually impaired, with support to better understand the surrounding environment and move around safely and comfortably.

[0307] The flow of the identification process will be described with reference to FIG.

[0308] Step 1: Collect environmental data

[0309] The server collects surrounding environment data sent from the vehicle's sensors and cameras. This data includes the location of obstacles, the movement of nearby vehicles, the status of traffic lights, etc. Based on this input data, the server processes and calculates the data to understand the surrounding situation.

[0310] Step 2: Analyze environmental data

[0311] The collected environmental data is analyzed by a generative AI model, which recognizes the location of obstacles around the vehicle, the behavior of nearby vehicles, the status of traffic lights, etc. The results of the analysis are used as the basis for providing information to the user.

[0312] Step 3: Collect and analyze user biometric information

[0313] The server collects data from the user's biometric sensors, such as heart rate and skin conductivity, which are then analyzed and used to infer the user's emotional state, which in turn helps determine how to provide information and adjust the environment.

[0314] Step 4: Decide how to provide information

[0315] The server takes into account the analyzed environmental data and the user's emotional state to determine the optimal way to provide information. For example, if the user feels tense, it will decide to provide relaxing music. This decision-making process is driven by a generative AI model.

[0316] Step 5: Provide information

[0317] Based on the information provision method determined, the server will provide voice messages, haptic feedback, appropriate music, etc. These information provision will be customized to provide the best experience for the user. For example, a voice message may be conveyed to the user saying, "A vehicle is approaching from the right. Please be careful."

[0318] This program processing allows users of self-driving vehicles to better understand their surroundings, enabling them to travel safely and comfortably, and by providing information and adjusting the environment according to the user's emotional state, it can reduce anxiety and stress while traveling.

[0319] Example 3

[0320] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] The goal is to reduce the barriers to information access that visually impaired people face in their daily lives and social activities. Specifically, these challenges include understanding the reactions and atmosphere of people around them at meetings and events, accessing the contents of books and materials they want to read, and providing haptic feedback that adapts to their own emotional state. These challenges are the main factors that limit the improved social participation and independent living of visually impaired people.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0323] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the atmosphere of the environment and people's reactions from the text data using a generative AI model, a means for converting the text data into voice information and conveying it to the user, a means for monitoring the user's emotional state, a means for adjusting the intensity and pattern of haptic feedback based on the user's emotional state, and a means for collecting voices of meetings and events in real time. This allows visually impaired people to have a deeper understanding of their surroundings, actively participate in social activities, easily access the content they want to read as voice information, and receive appropriate haptic feedback according to their emotional state.

[0324] "Means for converting voice data into text data" refers to a process or device that converts collected voice information into text data. This conversion is accomplished using voice recognition technology.

[0325] "Means of analyzing environmental atmosphere and people's reactions from text data using generative AI models" refers to techniques and methods for analyzing text information and using that information to understand the surrounding situation, people's emotions, reactions, etc. Natural language processing technology is used for this analysis.

[0326] "Means for converting text data into voice information and conveying it to the user" refers to a process or device that reproduces the analyzed character information as voice and conveys it to the user. This conversion uses text-to-voice technology.

[0327] "Means for monitoring the user's emotional state" refers to technologies or methods for identifying and analyzing the user's emotional state from the user's tone of voice, facial expressions, body movements, etc. Voice analysis technology and sensor technology may be used for this monitoring.

[0328] "Means for adjusting the intensity or pattern of haptic feedback based on the user's emotional state" refers to techniques or methods for varying the type or degree of feedback provided through a haptic device depending on the user's current emotional state, using a feedback control algorithm based on the results of emotion analysis.

[0329] "Means of capturing audio in real time at meetings and events" refers to technologies and devices that directly capture live audio information in situ and record it as digital data, using microphones and audio recording systems.

[0330] In the embodiment of the present invention, a server, a terminal (smart device), and a user (visually impaired person) are at the center. The server is equipped with speech recognition technology, natural language processing technology, text-to-speech technology, emotion analysis technology, and haptic feedback control technology. The terminal is equipped with voice collection function, emotion state monitoring function, and haptic feedback providing function. Specific hardware and software include Google Cloud Speech-to-Text (speech recognition), OpenAI's GPT (text analysis and generation), Amazon Polly (text to speech conversion), and Tesseract OCR (character recognition).

[0331] The server converts the voice data received from the device into text data using Google Cloud Speech-to-Text. Next, this text data is analyzed using OpenAI's GPT to extract information such as the atmosphere of the meeting or event and people's reactions. The analysis results are converted into text data and into voice information using Amazon Polly. This voice information is transmitted to the user via the device.

[0332] The device uses a microphone to collect audio from meetings and events in real time and transmits it to the server. The device also monitors the user's tone of voice and body movements, and transmits data for analyzing the user's emotional state to the server. The server analyzes this data and transmits instructions to the device to adjust the intensity and pattern of haptic feedback according to the user's emotional state. Based on these instructions, the device provides appropriate feedback to the user through the haptic device.

[0333] A specific example is a scenario in which a server performs processing based on a prompt sentence such as "Analyze the atmosphere of the meeting and provide an overview of the atmosphere by voice," and a terminal conveys to a user voice information such as "The speaker just told a joke, and the participants are laughing." Another example includes a case in which a user scans a page of a book he or she wants to read with a camera on the terminal, and converts the text into voice information and provides it based on a prompt such as "Read the text on this page and read it aloud."

[0334] This format allows visually impaired people to more easily understand their surroundings and the content of meetings, access materials they want to read by voice, and receive richer information through haptic feedback that adapts to their emotional state.

[0335] The flow of the identification process will be described with reference to FIG.

[0336] Step 1:

[0337] The terminal uses a microphone to collect audio from a meeting or event. This audio data is sent to the server. The input is raw audio data, and the output is digitized audio data. This step involves the collection and digitization of audio data.

[0338] Step 2:

[0339] The server converts the received voice data into text data using Google Cloud Speech-to-Text. The input is digitized voice data, and the output is text data. In this process, data processing is performed to convert voice information into text information using voice recognition technology.

[0340] Step 3:

[0341] The server analyzes the converted text data using OpenAI's GPT to extract information such as the atmosphere of the meeting or event and people's reactions. The input is text data, and the output is the text data of the analysis results (a summary of the atmosphere and reactions). In this step, data calculations are performed using natural language processing technology to extract useful information from the text data.

[0342] Step 4:

[0343] The server converts the analysis results into voice information using Amazon Polly as text and sends this voice information to the terminal. The input is the text data of the analysis results, and the output is voice information. In this process, text-to-voice conversion is performed, and information is provided to the user in an easy-to-understand form.

[0344] Step 5:

[0345] The terminal transmits the voice information received from the server to the user. In this step, the voice information is played through the terminal's speaker. The input is the voice information, and the output is a voice message that the user can hear.

[0346] Step 6:

[0347] The terminal monitors the user's tone of voice and body movements and transmits the data to the server for analyzing the user's emotional state. The input is the user's voice and body movements data, and the output is the analysis result of the emotional state. In this step, the data is calculated using emotion analysis techniques to identify the user's emotional state.

[0348] Step 7:

[0349] The server sends instructions to the terminal to adjust the intensity and pattern of the haptic feedback according to the user's emotional state. The input is the analysis result of the emotional state, and the output is an instruction for haptic feedback. In this step, the haptic feedback is customized based on the user's emotional state.

[0350] Step 8:

[0351] The terminal provides appropriate feedback to the user through the haptic device based on the instruction from the server. In this step, the user receives a response according to his / her emotional state through haptic feedback. The input is an instruction for haptic feedback, and the output is the haptic feedback transmitted to the user's senses.

[0352] (Application example 3)

[0353] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0354] In modern society, visually impaired people and security workers face challenges such as understanding the environment, moving safely, and instantly recognizing abnormal behavior and dangerous situations. In particular, when visually impaired people participate in society and security workers need to respond quickly to emergency situations, there are limited means to effectively grasp the surrounding environment and people's emotional states. Another issue is the lack of support for leading a safer and more comfortable social life by providing appropriate haptic feedback according to the user's emotional state.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0356] In this invention, the server includes a means for providing information to substitute for vision using a generating AI, a means for analyzing the emotional state of people in the vicinity in real time and providing appropriate haptic feedback when abnormal behavior or situations are detected, and a means for adjusting the strength and pattern of the haptic feedback according to the emotional state of the user. This enables visually impaired people to engage in social activities more richly and independently, and enables security personnel to effectively grasp the surrounding environment and people's emotional states and respond quickly even in emergency situations.

[0357] "Generative AI" is a form of artificial intelligence that is a system that generates new information and responses independently from training data and performs specific tasks automatically.

[0358] A "means for providing information in lieu of vision" is a system or method that conveys necessary information to the user through audio or tactile feedback in place of visual information.

[0359] "Means of accessing AI via smart devices or headsets" refers to methods or systems for connecting to and using AI functions and services using devices such as smartphones, tablets, and head-mounted displays.

[0360] "Means of acquiring environmental information in real time" refers to systems and technologies that can instantly capture, analyze, and process the surrounding situation and changes.

[0361] "Means of recognizing surrounding objects and people" refers to systems or algorithms that identify objects and people from image and audio data obtained through cameras, sensors, etc., and analyze that information.

[0362] "Means of conveying information in the form of voice, voice message, haptic feedback, etc." refers to a method of conveying information to a user using voice synthesis technology or vibration patterns.

[0363] "Means for analyzing the emotional state of people in the surroundings in real time and providing appropriate haptic feedback when abnormal behavior or situations are detected" refers to a system or method that analyzes the facial expressions and tone of voice of people in the environment and instantly generates haptic feedback to warn of abnormalities based on the analysis.

[0364] "Means for adjusting the intensity and pattern of haptic feedback according to the emotional state of the user" refers to a system or technology that evaluates the user's current emotional or psychological state and adjusts the output of haptic feedback to match it.

[0365] An embodiment of the present invention includes a system that provides real-time alerts and haptic feedback to visually impaired people and security personnel based on environmental information and people's emotional states. The system is composed of a generative AI model, a smart device (smart glasses or a head-mounted display), emotion analysis software, and a haptic feedback device.

[0366] The server uses a generative AI model to analyze people's emotional states from the surrounding environmental data. Machine learning libraries such as OpenCV and TensorFlow are used for this analysis. The terminal (smart glasses or head-mounted display) collects environmental data via a camera and microphone and sends it to the server. The analysis results from the server are communicated to the user via the terminal, and a sensory warning is provided through a haptic feedback device if necessary.

[0367] The system allows users to gain a deeper understanding of their surroundings and people's emotional states, improving safety and comfort, enabling visually impaired people to be more independent in social activities and enabling security personnel to respond quickly and effectively in emergency situations.

[0368] A concrete example would be the following scenario: "A security guard wears smart glasses while on patrol. The application installed on his glasses scans the faces of people around him in real time and analyzes their emotional state from their facial expressions. If anger or aggressive emotions are detected, the guard's glasses will instantly emit a strong vibration to alert him." This process is intended to help security guards and visually impaired people better understand the situation around them and take appropriate action.

[0369] The flow of the identification process will be described with reference to FIG.

[0370] Step 1: The device collects environmental data using the camera and microphone. This collected data includes facial images and audio of people around it. It receives the surrounding images and audio as input and generates a data file as output.

[0371] Step 2: The terminal transmits the collected environmental data to the server. In this step, the collected video and audio data is transferred to the server. It receives data files from the terminal as input and transmits data to the server as output.

[0372] Step 3: The server uses the received environmental data to analyze the emotional state of the surrounding people through a generative AI model. This process uses image recognition and voice recognition technologies to generate emotion analysis results. It receives video and audio data as input and generates emotion analysis results as output.

[0373] Step 4: The server sends appropriate haptic feedback instructions to the terminal based on the emotion analysis result. The instructions include feedback intensity and pattern information. It receives the emotion analysis result as input and generates haptic feedback instructions as output.

[0374] Step 5: The terminal provides haptic feedback to the user based on the haptic feedback instruction received from the server. This feedback is provided through a haptic feedback device worn by the terminal or the user. The terminal receives the haptic feedback instruction as an input and performs specific haptic feedback to the user as an output.

[0375] Through these steps, the system can effectively grasp the surrounding environment and people's emotional states, and provide users with appropriate warnings and haptic feedback in real time, helping visually impaired people to move around safely and participate in society, and enabling security personnel to quickly grasp the situation and take action.

[0376] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0377] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0378] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0379] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0380] [Second embodiment]

[0381] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0382] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0383] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0384] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0385] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0386] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0387] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0388] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0389] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0390] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0391] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0392] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0393] "Example 1"

[0394] In this embodiment of the present invention, a visually impaired person wears a smart device or headset and accesses the generative AI. The generative AI analyzes information obtained from cameras and sensors and provides the visually impaired person with information about the surrounding environment. Specifically, the AI ​​recognizes the position, shape, and movement of objects and people, and communicates this to the visually impaired person through voice messages and haptic feedback.

[0395] "Example 2"

[0396] As a concrete example, when a visually impaired person walks through the city, the generative AI recognizes obstacles, intersections, traffic light status, etc. ahead and conveys them to the visually impaired person through voice messages. It also infers people's presence and movements from their movements and voices, and provides haptic feedback to the visually impaired person. This enables the visually impaired person to understand the situation around them and move around safely.

[0397] "Example 3"

[0398] Furthermore, generative AI can help visually impaired people participate in society. For example, it can analyze people's reactions and the atmosphere at meetings and events, and convey that to visually impaired people. It can also read the text information in books and documents that visually impaired people want to read, and read it aloud. This allows visually impaired people to participate in social activities more richly and independently.

[0399] The process flow of each embodiment will be described below.

[0400] "Example 1"

[0401] Step 1: A visually impaired person wears a smart device or headset and accesses the generative AI.

[0402] Step 2: The generative AI analyzes the information obtained from the camera and sensors.

[0403] Step 3: The AI ​​recognizes the position, shape, and movement of objects and people, and communicates this to the visually impaired through voice messages and haptic feedback.

[0404] "Example 2"

[0405] Step 1: When a visually impaired person walks around town, the generative AI recognizes obstacles, intersections, traffic light status, etc. ahead.

[0406] Step 2: The AI ​​infers people's presence and movements from their movements and voices.

[0407] Step 3: The AI ​​will communicate information to the visually impaired through voice messages and provide haptic feedback, allowing them to understand their surroundings and navigate safely.

[0408] "Example 3"

[0409] Step 1: Generative AI helps visually impaired people participate in society, for example by analyzing people's reactions and the atmosphere at meetings and events.

[0410] Step 2: The AI ​​reads the text information in the books and materials that the visually impaired person wants to read.

[0411] Step 3: The AI ​​will read out the information aloud and convey it to the visually impaired, enabling them to participate in social activities more richly and independently.

[0412] Example 1

[0413] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0414] With current technology, visually impaired people face many challenges when moving independently, and there are limited means to safely and effectively obtain information about their surroundings. In particular, the lack of a system that provides detailed information on the position, shape, and movement of people and objects in real time is a major factor that limits the freedom of movement of visually impaired people and reduces their opportunities for social participation. For this reason, new technological solutions are needed to help visually impaired people better understand their surrounding environment and move safely.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0416] In this invention, the server includes a means for analyzing environmental data using a generative AI model, a means for converting the analysis results into voice messages and haptic feedback, and a means for processing the collected environmental data in real time. This allows visually impaired people to receive detailed information about the positions, shapes, and movements of people and objects around them in real time through voice and haptic feedback. As a result, it is possible to support safe mobility for visually impaired people and expand opportunities for social participation.

[0417] "Generative AI" is a type of artificial intelligence that learns from data and generates models and algorithms to perform specific tasks.

[0418] A "smart device" is a portable electronic device that can connect to the Internet and provide multiple functions by running applications.

[0419] A "headset" is a device that generally has audio output (speaker) and audio input (microphone) functions and is worn on the head.

[0420] "Means of acquiring environmental information in real time" is a process or technology that instantly captures the current situation and changes and provides information without delay.

[0421] "Means for recognizing surrounding objects and people" refers to technology that analyzes visual, auditory, or other sensor information to determine the presence and location of objects or people within a particular environment.

[0422] "Means of conveying information in the form of voice, audio messages, tactile feedback, etc." are methods of providing information to users using sensory means such as sound or vibration.

[0423] The present invention relates to a system for assisting visually impaired people in understanding their surroundings and navigating safely, the system being implemented using specific hardware and software.

[0424] The servers host generative AI models that are designed to analyze data collected from the environment and identify information such as the location, shape, and movement of objects and people. The software used includes, for example, customized versions of OpenAI's GPT and Google's TensorFlow, which provide complex data analysis and learning capabilities that increase the accuracy and efficiency of the system.

[0425] The terminals are devices worn by the visually impaired, such as smart devices or headsets. These devices are equipped with cameras and sensors to collect environmental data. The collected data is sent to a server and analyzed by a generative AI model.

[0426] The analysis results are sent back to the device and provided to the visually impaired as audio messages and / or haptic feedback, helping users better understand their surroundings and providing them with information to navigate safely.

[0427] A specific example is the prompt, "Please identify the location and movement of people and objects around you and provide that information in a voice message," which dictates the type of data that will be analyzed by the generative AI model.

[0428] This embodiment of the invention focuses on leveraging advanced technology to solve everyday challenges faced by the visually impaired, with a combination of hardware and software to provide real-time environmental information to help them become more independent.

[0429] The flow of the identification process will be described with reference to FIG.

[0430] Step 1: The device collects surrounding environment data from the cameras and sensors installed in the smart device or headset worn by the visually impaired person. The input of this step is the video, audio, temperature, distance, and other data of the surrounding environment acquired by the device's camera and sensors. The output is the collected raw environmental data.

[0431] Step 2: The terminal transmits the collected environmental data to the server. In this step, data transmission is performed using wireless communication technology. The input is the environmental data collected in step 1, and the output is the data transmitted to the server.

[0432] Step 3: The server analyzes the received environmental data using the generative AI model. This analysis includes processes such as object recognition, person recognition, and motion detection. The input is the environmental data received in step 2, and the output is the analysis result. This analysis result includes information such as the position, shape, and movement of surrounding objects and people.

[0433] Step 4: The server generates information useful to visually impaired people based on the analysis results. This information is provided in the form of voice messages and haptic feedback. The input is the analysis results from step 3, and the output is information to convey to visually impaired people.

[0434] Step 5: The terminal conveys the information received from the server to the visually impaired person using the terminal's built-in speaker and haptic feedback device. The input is the information generated in step 4, and the output is a voice message or haptic feedback that can be recognized by the visually impaired person.

[0435] An example of a specific operation is processing based on the prompt, "Please identify the positions and movements of people and objects around you and convey that information in a voice message." In this case, the device uses a camera to capture images of the surroundings and sends the data to the server. The server analyzes the data, identifies the positions and movements of people and objects, and sends this information back to the device as a voice message for the visually impaired person. The device then conveys this message to the visually impaired person through a speaker.

[0436] (Application example 1)

[0437] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0438] Currently, the main challenge that visually impaired people face when shopping in brick-and-mortar stores is the difficulty of understanding the layout of the store and the location of products. Another major challenge is moving safely while avoiding obstacles. These challenges prevent visually impaired people from enjoying shopping independently and limit their opportunities for social participation.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0440] In this invention, the server includes a means for accessing the generating AI via a smart device or a headset, a means for acquiring and analyzing information on the in-store environment in real time, a means for recognizing surrounding objects and people and identifying their positions and shapes, and a means for conveying information to visually impaired people through voice messages and haptic feedback, thereby enabling visually impaired people to grasp the positions of products and the presence of obstacles in a physical store and to shop efficiently while moving around safely.

[0441] "Generative AI" is a type of artificial intelligence that analyzes environmental data and generates and provides useful information to users based on the results of that analysis.

[0442] A "smart device" is a portable electronic device that can connect to the Internet and perform multiple functions.

[0443] A "headset" is a device that can be worn on the ears to receive voice communication and voice guidance.

[0444] "Real-time environmental information acquisition" is the process of collecting and updating data about the surrounding environment in close sync with the current time.

[0445] "Means of recognizing surrounding objects and people" is a technology that identifies and identifies the presence, position, and shape of objects and people from input from cameras, sensors, etc.

[0446] "Voice or audio messaging" is a method of transmitting information in the form of sound waves that can be heard by the human ear.

[0447] "Haptic feedback" is a technology that provides users with information through their sense of touch, conveying information through vibrations and changes in pressure.

[0448] An embodiment of the invention includes the use of generative AI via a smart device or headset. The server uses a generative AI model to analyze environmental data sent from the smart device or headset. This analysis identifies the position, shape, and movement of surrounding objects and people, and conveys information to the visually impaired person through audio messages and haptic feedback.

[0449] The hardware used includes smart devices and headsets. These devices acquire real-time environmental data through built-in cameras and sensors and transmit it to a server. The server uses software libraries such as OpenCV and TensorFlow to perform object detection and image analysis, and conveys the obtained information to the user through a voice synthesis system.

[0450] For example, when a user puts on the smart glasses and enters a store, voice guidance is provided such as, "There is a product shelf ahead. The safe route is to go straight." This information is generated by analyzing video data captured by the smart glasses' camera. Also, if a user is looking for a specific product, the generating AI provides specific instructions such as, "Turn right and you will find the product." This allows visually impaired people to efficiently find the products they need while moving around safely.

[0451] The flow of the identification process will be described with reference to FIG.

[0452] Step 1: The device's camera and sensors acquire surrounding environmental data. In this step, the cameras and sensors in the smart device or headset worn by the user collect information on surrounding objects, people, obstacles, etc. in real time. The input is data from optical images and distance sensors, and the output is sent to the server as digitized environmental data.

[0453] Step 2: The server analyzes the environmental data. The received environmental data is analyzed by the generative AI model to identify the position, shape, and movement of surrounding objects and people. The input of this step is the digitized environmental data from step 1, and the output is the object identification information and movement patterns as the analysis results.

[0454] Step 3: Based on the analysis results, the server generates voice messages and haptic feedback in a format that is easy for the user to understand based on the analysis results. Examples include product location guidance and obstacle warnings. The input is the object identification information and movement patterns from step 2, and the output is the voice messages and haptic feedback provided through the user's smart device or headset.

[0455] Step 4: The user adjusts their behavior based on the provided information. Based on the voice messages and haptic feedback they receive, the user may adjust their route through the store or search for a specific product. The input to this step is the information from step 3, and the output is a change in the user's behavior, such as moving in a particular direction or picking up a product.

[0456] This process allows visually impaired people to shop safely and efficiently in physical stores.

[0457] Example 2

[0458] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0459] Currently, the biggest challenge that visually impaired people face when walking around town is that they cannot directly obtain the information they need to move safely, such as the presence of obstacles, traffic conditions, and the movements of people around them. This exposes them to great anxiety and danger when they move around. Another major problem is that it is difficult for visually impaired people to communicate and synchronize their activities with those around them when participating in society.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0461] In this invention, the server includes a means for collecting data from external sensors in real time, a means for analyzing the surrounding environment and the situation of people from the collected data using a generative AI model, and a means for providing appropriate instructions and information to visually impaired people in the form of voice messages and haptic feedback based on the analysis results, which enables visually impaired people to more accurately understand the surrounding environment, move safely, and easily communicate and synchronize their activities when participating in society.

[0462] "Means of collecting environmental data from external sensors in real time" refers to the process of instantly acquiring data about the current surrounding environment through sensors such as cameras, microphones, and GPS mounted on smart devices or dedicated devices, and sending it to a server for analysis.

[0463] "Means of analyzing the positions and movements of surrounding objects and people from collected environmental data using a generative AI model" refers to a method in which an artificial intelligence model trained using machine learning technology extracts and interprets information such as the positions of objects and the movements of people based on collected data.

[0464] "Means of providing instructions and information to visually impaired persons in the form of voice messages and haptic feedback based on analysis results" refers to a method of directly transmitting necessary information and appropriate instructions to users based on the results of analysis by AI, using formats such as voice and vibration that are easy for visually impaired persons to understand.

[0465] "Means of accessing AI via smart devices or dedicated devices and transmitting information in real time" refers to the process of communicating with AI on a server through internet-connected devices such as smartphones, tablets, and dedicated wearable devices, sending collected data, and receiving analysis results.

[0466] In the embodiment of the present invention, a system is provided to enable visually impaired people to understand their surrounding environment and navigate safely. The system mainly consists of three main components: data collection from external sensors, data analysis by generative AI models, and information provision through voice messages and haptic feedback.

[0467] The server uses cloud-based computing resources to host generative AI models that run on machine learning frameworks such as TensorFlow and PyTorch and are used to analyze data collected from external sensors, such as image data from cameras and audio data from microphones built into smart devices and dedicated devices.

[0468] The terminal is a smart device or dedicated device carried by the visually impaired person, which collects environmental data through external sensors such as a camera, microphone, and GPS, and transmits it to a server. In addition, it provides voice messages and haptic feedback to the visually impaired person based on instructions from the server. For example, if the server sends an instruction to the terminal saying, "There is an obstacle 3 meters ahead. Move left," the terminal outputs this message in voice and provides instructions to the visually impaired person to avoid the obstacle.

[0469] Based on the voice messages and haptic feedback provided, the user can understand the surrounding situation and move safely by avoiding obstacles. In addition, the haptic feedback indicating the movement and presence of people can help the user avoid collisions with other pedestrians. In this way, the present invention provides a concrete method for the visually impaired to move more independently and safely.

[0470] The flow of the identification process will be described with reference to FIG.

[0471] Step 1: Collect data from external sensors

[0472] The terminal collects environmental data from the visually impaired person's smart device or dedicated device through external sensors such as cameras, microphones, and GPS. The input of this step is the surrounding images, audio, and location information, and the output is a digitized form of these data. The terminal temporarily stores the data and converts it into a format that can be transmitted over the network in order to send it to the server.

[0473] Step 2: Analyze data and generate instructions

[0474] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding environment, the location of obstacles, people's movements, etc. from this data. The input for this step is image data, audio data, and location information, and the output is the analysis results such as the location of obstacles and people's movements. Based on the analysis results, the generative AI model generates specific instructions and information to be provided to the visually impaired person.

[0475] Step 3: Provide information

[0476] The terminal provides the instructions and information received from the server to the visually impaired person as voice messages and haptic feedback. The input of this step is the instructions and information sent from the server, and the output is the voice message and haptic feedback received directly by the visually impaired person. The terminal outputs the instructions as voice messages using a voice synthesis technology that converts text to speech, and provides the necessary information as a tactile sensation through a haptic feedback device.

[0477] Through these steps, visually impaired people can better understand their surroundings, receive specific instructions to avoid obstacles, and navigate safely with information about the movements and positions of people around them. This process happens in real time, significantly improving support for visually impaired people in participating in society.

[0478] (Application example 2)

[0479] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0480] In modern society, personal safety is a major concern. Accurately understanding the perception of the surrounding environment and people and their impact on safety is essential to ensure personal safety, especially when going out alone or at night. However, current technology has limited effective means to support personal safety in a given environment in real time and immediately warn of the approach of suspicious people or the occurrence of an emergency. Against this background, there is a need for the development of supportive technologies that allow individuals to actively manage their own safety and protect themselves from danger.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0482] In this invention, the server includes a means for acquiring environmental information in real time, a means for recognizing surrounding objects and people, and a means for transmitting information to the user via voice, voice message, and haptic feedback. This allows the user to grasp the surrounding situation in real time and increase safety awareness. In addition, by using this system, a warning can be received immediately when a suspicious person approaches or an emergency occurs, and appropriate measures can be taken. In this way, this invention provides important support for ensuring personal safety.

[0483] "Generative AI" is a type of artificial intelligence based on machine learning technology, a system that recognizes environmental information, objects, and the state of people based on input data, and generates and provides useful information to users.

[0484] "Means of acquiring environmental information in real time" refers to technologies and methods that use devices such as sensors, cameras, and microphones to instantly capture and analyze the surrounding situation and sounds at that moment.

[0485] "Means for recognizing surrounding objects and people" refers to techniques or methods that use image recognition technology or voice recognition technology to identify the characteristics of surrounding objects and people and understand that information.

[0486] "Voice or audio message" is a form of audible linguistic information used to convey system-generated information to the user.

[0487] "Haptic feedback" is a means of conveying information from a device to the user through physical sensations such as vibration or pressure.

[0488] In the embodiment of the present invention, a smartphone, smart glasses, or a head-mounted display is used as a terminal, and surrounding environmental information is acquired in real time through a camera or microphone built into these terminals. The acquired information is transmitted to a server equipped with a generative AI model. The server uses a deep learning framework to recognize surrounding objects and people through image recognition technology and voice recognition technology, and generates an appropriate response according to the situation. The generated information is communicated to the user in the form of voice, voice message, or haptic feedback.

[0489] The hardware used includes smartphones with Qualcomm Snapdragon processors and wearable devices such as Google Glass, while the software uses machine learning frameworks such as TensorFlow and PyTorch.

[0490] As a specific example, if a user is walking at night and the camera captures a suspicious person approaching, the server generates a voice message saying, "Caution: Someone is approaching from behind. Please move to a safe place immediately," and transmits this to the user through the terminal. In addition, in the event of an emergency, it is also possible to provide haptic feedback saying, "An emergency has occurred. Please evacuate to a safe place immediately." In this way, the present invention provides a concrete means for the user to grasp the situation around them in real time and ensure their safety.

[0491] The flow of the identification process will be described with reference to FIG.

[0492] Step 1: The device collects surrounding environment information. In this step, the device collects surrounding video and audio in real time using the camera and microphone built into the smartphone, smart glasses, or head-mounted display. At this time, the device takes video and audio data as input and prepares to send it to the server.

[0493] Step 2: Send the collected environmental information to the server. The terminal sends the video and audio data collected in step 1 to the server. In this process, the data is transferred to the server via the Internet.

[0494] Step 3: Server analyzes environmental information. The video and audio data sent to the server is analyzed by a generative AI model. This AI model uses a deep learning framework to recognize surrounding objects and people from the video, and analyze environmental sounds and conversations from the audio data. The output of this step is the analysis results, which include specific situational awareness and identification of potential hazards.

[0495] Step 4: Generate an appropriate response based on the analysis results. Based on the analysis results obtained in step 3, the server generates information to be communicated to the user. This information is about the appropriate action the user should take at that moment, such as safety warnings and instructions. The generated information can take the form of a voice message or haptic feedback.

[0496] Step 5: Communicate the generated information to the user. The server sends the information generated in step 4 to the terminal, which then communicates the information to the user. The information is communicated to the user through the speaker in the case of a voice message, or by using the device's vibration function in the case of haptic feedback. In this step, the user receives specific information to ensure their own safety.

[0497] Example 3

[0498] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0499] When visually impaired people participate in social activities such as meetings and events, they cannot visually grasp the reactions and atmosphere of those around them. In addition, it is difficult for them to independently read the text information in books and materials they want to read. These challenges are factors that limit the independent participation of visually impaired people in social activities.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0501] In this invention, the server includes a means for analyzing reactions and atmosphere at a meeting or event, a means for reading out text information of documents or materials, and a means for interpreting prompts input by voice and generating appropriate responses, thereby enabling visually impaired people to understand the surrounding situation in social activities and obtain necessary information independently.

[0502] "Generative AI" is a type of artificial intelligence that can generate new information based on learned data.

[0503] "A means of analyzing reactions and atmosphere at meetings and events" refers to the ability to understand the emotions and mood of participants at a particular gathering or event and express them in verbal or other forms.

[0504] "A means for converting the textual information of a document or material into audible speech" refers to a process or device that converts printed or electronically provided textual information into audio output.

[0505] "Means of accessing AI via smart devices and headsets" refers to methods of using intelligent electronic devices and audio output devices to communicate with and interact with artificial intelligence systems.

[0506] "Means of acquiring environmental information in real time" refers to the function of instantly collecting and analyzing the surrounding situation and data at the current time.

[0507] "Means of recognizing surrounding objects and people" refers to technologies and methods that use cameras, sensors, etc. to identify and identify nearby objects and people.

[0508] "Means of conveying information in the form of voice, audio messages, haptic feedback, etc." refers to methods of directly transmitting information to the user using sound waveforms, vibrations, etc.

[0509] "Ability to interpret voice prompts and generate appropriate responses" refers to the artificial intelligence's ability to understand spoken instructions or questions from a user and provide information or a response to them.

[0510] The embodiment of the present invention is mainly composed of three parties: a server, a terminal, and a user. The server is equipped with a generative AI model, which is designed to solve the problems that visually impaired people face when participating in social activities. The terminal mainly takes the form of a smart device or a headset, and plays a role in transmitting input from the user to the server. The user is a visually impaired person, and can acquire and deepen his / her understanding of information in social activities through this system.

[0511] The generative AI model that runs on the server is implemented using advanced technologies such as OpenAI's GPT and Google's BERT. This model analyzes video and audio data from meetings and events, and extracts the atmosphere and reactions of the event as text information. It also has the ability to read text information from images of documents and materials and convert it into audio data.

[0512] The device receives the user's voice prompts through a microphone, converts them into text data, and sends them to the server. It also uses the device's camera to take images of documents and materials and send them to the server. The server processes the received data with a generative AI model to generate an appropriate response.

[0513] As a specific example of use, if a user voice-inputs a prompt such as "Tell me about the atmosphere of the current meeting," the device converts this input into text and sends it to the server. The generative AI model on the server analyzes and extracts information about the atmosphere of the meeting and sends the results as text to the device. The device converts this text into speech and reads it aloud to the user. In addition, if a user requests "Read this page," the device takes an image of the document and sends it to the server. The server extracts text from the image, converts it into speech, and sends it to the device. The device reads this speech aloud to the user.

[0514] This format allows visually impaired people to receive specific support to help them acquire and understand information in social activities.

[0515] The flow of the identification process will be described with reference to FIG.

[0516] Step 1: The user speaks to the device, including a prompt about the mood of the meeting or event and a request for a document to be read aloud. The device receives the speech and converts it into text data.

[0517] Step 2: The converted text data is sent over the internet to a server, which receives the data and prepares it for processing by the generative AI model.

[0518] Step 3: For requests related to the atmosphere of a meeting or event, the server uses the generative AI model to analyze the atmosphere and reactions from the provided audio and video data. For requests to read a document, the server receives the image data of the document sent from the device and performs text extraction.

[0519] Step 4: The generative AI model generates an appropriate text response based on the analyzed or extracted information. For a request about the atmosphere of the meeting, it generates a response summarizing the emotions and key points of the meeting, and for a request to read a document, it generates the extracted text information.

[0520] Step 5: The generated text response is sent from the server to the device, which receives the text and converts it into audio data using its built-in text-to-speech conversion function.

[0521] Step 6: The converted voice data is output to the user, allowing the user to understand the atmosphere of the meeting or event and hear the content of the document.

[0522] Through this processing flow, visually impaired people can acquire information and deepen their understanding in social activities, enabling them to participate more independently.

[0523] (Application example 3)

[0524] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0525] When visually impaired people shop at a store, they face the challenge of being unable to grasp the location and information of products, the atmosphere of the store, and the reactions of people. For this reason, visually impaired people may feel restricted in their ability to shop independently, limiting their opportunities to participate in society. Another problem is the lack of systems that efficiently provide the information necessary for safe movement and product selection at a store.

[0526] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0527] In this invention, the server includes a means for providing audio guidance to visually impaired people based on analysis of product information and people's reactions in a sales area using a generation AI, a means for acquiring environmental information in real time, analyzing it, and generating audio guidance information, and a means for utilizing environmental data including product positions and sale information in the sales area. This enables visually impaired people to safely and efficiently select products and move around in sales areas, expanding opportunities for social participation.

[0528] "Generative AI" is an algorithm based on machine learning and deep learning techniques, and is a system that generates output from specific input in a format that is easy for humans to understand.

[0529] A "visually impaired person" refers to a person who has difficulty recognizing general visual information due to a visual impairment.

[0530] "Product information in the sales area" includes all information necessary for purchasing a product, such as the type of product, price, location, and sale information.

[0531] "Voice guidance based on analysis of people's reactions" refers to analyzing the facial expressions, words, actions, and other reactions of people around you and providing voice instructions and information based on that information.

[0532] "Means of acquiring real-time environmental information" refers to methods or technologies that continuously collect surrounding environmental data in accordance with the current time.

[0533] "Means for generating voice guidance information" refers to the process of analyzing collected information and data and using that information to provide useful instructions and information in voice format to visually impaired people.

[0534] "Environmental data including product location and sales information within the point of sale" is general data about the state and condition of the point of sale, including information indicating where products are located and which products are on sale.

[0535] A "server" is a computer system or software that collects, processes, stores data, and provides information to clients.

[0536] The embodiment of the present invention relates to a system that supports visually impaired people in shopping at a sales point. The system utilizes a generative AI model to provide voice guidance based on an analysis of product information and people's reactions in the sales point. The server has a function of acquiring environmental information in real time, analyzing it, and generating voice guidance information. The server also has a function of utilizing environmental data including product positions and sale information in the sales point. Terminals such as smart devices and headsets are used to implement the system, and visually impaired people access the generative AI through these terminals. The terminal has a function of conveying information to the user in the form of voice, audio messages, haptic feedback, etc.

[0537] The hardware used includes smart devices (e.g. smartphones), in-store cameras and sensors, and the software uses voice recognition and synthesis systems, location services, and generative AI models (e.g. OpenAI's GPT-4). This system allows visually impaired people to safely and efficiently select and navigate products at sales points, expanding their opportunities for social participation.

[0538] For example, when a user uses a smart device to ask, "What sweets are on sale nearby?", the server analyzes the in-store environmental data and uses a generative AI model to generate guidance information such as, "There is chocolate on sale near you. It's on the shelf about 3 meters to your right," and conveys this information to the user via a voice message on the device. This process combines the user's location information with in-store product information to provide optimal guidance.

[0539] The flow of the identification process will be described with reference to FIG.

[0540] Step 1: The user inputs a question into the terminal.

[0541] Users use their smart devices to voice-input questions about specific product information or in-store directions, which are then converted into text data by the device's on-board voice recognition system.

[0542] Step 2: The device sends the text data of the question to the server.

[0543] The converted text data is sent over the Internet to a server, which receives the text data and uses it as input for processing.

[0544] Step 3: The server parses the text data and sends prompts to the generative AI model to generate appropriate responses.

[0545] The server analyzes the received text data and forms prompts to generate specific responses based on the user's question, which are then sent to a generative AI model for use in generating responses.

[0546] Step 4: The generative AI model generates a response based on the prompt.

[0547] The generative AI model processes the prompts received from the server and generates appropriate responses to the user's questions, which may include in-store product locations, sales information, and descriptions of the in-store environment.

[0548] Step 5: Convert the server-generated response into audio data.

[0549] The server converts the response text received from the generative AI model into voice data using a voice synthesis system, which is then used to convey information to the user.

[0550] Step 6: The server sends the voice data to the terminal.

[0551] The voice data is again transmitted over the Internet to the user's terminal, which receives the voice data and outputs it to the user.

[0552] Step 7: The user listens to the audio output to obtain the information.

[0553] The user hears product information and in-store guidance information in response to questions through voice output from the terminal, allowing the user to obtain the necessary information and shop efficiently at the sales point.

[0554] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0555] "Example 1"

[0556] In one embodiment of the present invention, the generation AI has an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotions from the user's tone of voice, facial expressions, behavior, etc., and provides the information to the generation AI. The generation AI adjusts the method of providing information to the user based on the information provided by the emotion engine. For example, if the user feels angry, the generation AI will calm the way in which information is communicated, or will communicate only the necessary information. This makes it possible to respond appropriately according to the user's emotional state.

[0557] "Example 2"

[0558] In another embodiment of the present invention, the emotion engine also analyzes the user's biometric information (e.g., heart rate and skin electrical conductivity). This allows the user's emotional state to be grasped more accurately. The generation AI infers the user's emotional state based on this biometric information and adjusts the method of providing information. For example, if the user feels tense, the generation AI takes measures to improve the user's emotional state, such as providing relaxing music.

[0559] "Example 3"

[0560] Furthermore, in another embodiment of the present invention, the generation AI adjusts the intensity and pattern of the haptic feedback according to the user's emotional state. For example, if the user feels surprised, the generation AI provides strong haptic feedback to attract the user's attention. On the other hand, if the user feels relaxed, the generation AI provides gentle haptic feedback. This makes it possible to provide haptic feedback appropriate to the user's emotional state.

[0561] The process flow of each embodiment will be described below.

[0562] "Example 1"

[0563] Step 1: The emotion engine starts working, inferring emotions from the user's tone of voice, facial expressions, behavior, etc.

[0564] Step 2: The emotion engine provides the inferred emotion information to the generative AI.

[0565] Step 3: The generative AI adjusts how it presents information to the user based on the information provided by the emotion engine.

[0566] "Example 2"

[0567] Step 1: The emotion engine analyzes the user's biometric information (e.g., heart rate and skin conductivity).

[0568] Step 2: The generative AI uses this biometric information to infer the user's emotional state and adjusts the way it provides information.

[0569] Step 3: The generative AI responds to improve the user's emotional state. For example, if the user feels tense, the generative AI provides relaxing music.

[0570] "Example 3"

[0571] Step 1: The generative AI obtains the user's emotional state from the emotion engine.

[0572] Step 2: The generative AI adjusts the intensity and pattern of haptic feedback depending on the user's emotional state.

[0573] Step 3: The generative AI provides calibrated haptic feedback to the user.

[0574] Example 1

[0575] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0576] Visually impaired people face the challenge of having difficulty accurately understanding the surrounding environment when engaging in daily life and social activities. Conventional assistive technologies only provide environmental information one-way, and there is also a problem in that they do not provide information that takes into account the user's emotional state. This can make it difficult for users to accept information or to respond appropriately in emergency situations.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0578] In this invention, the server includes a means for analyzing environmental data transmitted from a mobile terminal or a headset, a means for estimating the user's emotional state, and a means for adjusting the information provision method based on the analyzed environmental information and emotional information. This allows visually impaired people to understand the surrounding environment more accurately and easily, and provides appropriate information according to their emotional state. This adaptive information provision also enables safe mobility support and improved social participation for visually impaired people.

[0579] "Generative AI" is an algorithm trained using machine learning and deep learning techniques, and is a system capable of generating or analyzing new information from input data.

[0580] "Environmental data" refers to information about the physical situation and conditions surrounding the user, such as images, audio, and location information, collected through sensors and cameras.

[0581] A "mobile terminal" is an electronic device, such as a smartphone or tablet, that is portable and has wireless communication capabilities.

[0582] A "headset" is a device worn on the ears that enables voice communication and listening to music. In support systems for the visually impaired, a headset is used to transmit voice messages and accept voice input.

[0583] "User's emotional state" refers to the user's current psychological and emotional state, and is inferred from tone of voice, facial expressions, behavior, etc.

[0584] "Adjusting the way information is presented" refers to appropriately changing the way information is conveyed (e.g., the tone of a voice message or the intensity of haptic feedback) based on the received environmental data and the user's emotional state.

[0585] The embodiment of the present invention is centered around a mobile device or headset worn by a visually impaired person and a generative AI model that runs on a server that controls them. The mobile device or headset includes a camera and sensors for recognizing the environment, and a microphone for inferring the user's emotional state. Data collected through these hardware devices is transmitted to the server using wireless communication technology.

[0586] On the server, a generative AI model using machine learning frameworks such as TensorFlow and PyTorch analyzes the transmitted data. This analysis includes identifying the location, shape, and movement of objects and people using image recognition technology, and inferring the user's emotional state using voice analysis technology. Based on the analysis results, the generative AI model determines the optimal way to provide information to the user and conveys that information to the user via a mobile device or headset in the form of a voice message or haptic feedback.

[0587] For example, while a user is walking in a park, the camera on the mobile device captures a bench and people walking in front of them, and this data is sent to the server. The generative AI model on the server analyzes the position of the bench and the movement of the people walking from this image, and provides the user with a voice message saying, "There is a bench 3 meters to the right. There is a person walking ahead." At the same time, it infers the emotion from the user's tone of voice and adjusts the tone of the message as necessary.

[0588] In this way, this invention provides an information provision system that integrates both environmental recognition and emotion estimation. This allows visually impaired people to understand their surroundings in more detail and more appropriately, and to receive information customized according to their emotional state. This system is expected to contribute to improving the quality of daily life of visually impaired people and promoting their participation in society.

[0589] The flow of the identification process will be described with reference to FIG.

[0590] Step 1: Collect environmental data

[0591] The device collects data about the surrounding environment using cameras and sensors. This data includes images, audio, and location information. The input is raw data from the environment, and the output is a digitalized version of the raw data. Through this conversion process, the device transmits visible light images, audio data, and location information from GPS to a server.

[0592] Step 2: Data analysis and sentiment estimation

[0593] The server analyzes the received environmental data using a generative AI model and an emotion engine. The generative AI model performs image recognition to identify the location, shape, and movement of people and objects. The emotion engine infers the user's emotional state from voice data. The input is the environmental data sent from the device, and the output is object identification information and the user's emotional state. Through this process, the server generates detailed information about both the surrounding environment and the user's emotions.

[0594] Step 3: Decide how to provide information

[0595] The generative AI model on the server combines the analyzed environmental information and emotional information to determine how to provide information to the user. In this process, it is possible to adapt the information delivery method to the user's emotional state. The inputs are environmental information and the user's emotional state, and the output is an instruction for the adapted information delivery method. The server sends the information delivery method to the terminal based on this instruction.

[0596] Step 4: Communicate

[0597] The device provides information to the user through voice messages and haptic feedback according to the instructions received from the server. In this step, the input is an instruction from the server, and the output is a transmission of information in a form that the user can understand. Through this process, the device provides the user with a voice message such as "There is a pedestrian crossing ahead" or haptic feedback indicating that someone is nearby.

[0598] (Application example 1)

[0599] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0600] In modern society, visually impaired people face many difficulties in moving around and in their daily lives. In particular, when they go out, it is difficult for them to accurately grasp their surroundings, which is an obstacle to safe movement and participation in society. In addition, because the methods of transmitting information are limited, there is a lack of means to reduce the anxiety and stress felt by visually impaired people. Under these circumstances, there is a demand for the development of support methods that enable visually impaired people to participate in society more safely and with confidence.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0602] In this invention, the server includes a means for analyzing environmental information using a generating AI and providing the user with a voice message or haptic feedback in place of visual information based on the analysis result, a means for accessing the AI ​​via a smart device or a headset and acquiring environmental data in real time, a means for recognizing surrounding objects and people, their movements and shapes using a camera or a sensor, and a means having an emotion engine for predicting the user's emotional state and adjusting the method of providing information based on that information. This enables visually impaired people to understand the surrounding environment more accurately and realize safe movement and social participation. Furthermore, by adopting an information transmission method according to the user's emotional state, it is possible to reduce the anxiety and stress felt by the user and support more comfortable social participation.

[0603] "Generative AI" is a system that uses artificial intelligence technology to analyze environmental data and provide information to users based on the results of that analysis.

[0604] A "voice message" is a means of verbal communication, a message that conveys information to the user by voice.

[0605] "Haptic feedback" is a means of conveying information through the sense of touch, a method of providing information from a device to the user through physical stimulation.

[0606] A "smart device" is a portable electronic device that has the ability to connect to the Internet and integrates multiple smart functions.

[0607] A "headset" is a device worn on the head that allows for voice communication and audio output, and in this context is used to receive audio messages and haptic feedback.

[0608] "Cameras and sensors" are devices that capture the surrounding environment and objects as images and data, and this information is used by the generative AI to analyze it.

[0609] An "emotion engine" is a technology or system that infers a user's emotional state from their tone of voice, facial expressions, behavior, etc., and adjusts the way information is provided based on that information.

[0610] The embodiment of this invention is a system for visually impaired people to understand the surrounding environment more safely and effectively when going out or in daily life. The system includes generative AI, smart devices and headsets, cameras and sensors, and an emotion engine.

[0611] The server is equipped with a generative AI that analyzes the environmental data sent from the cameras and sensors. The analyzed data includes information such as the position, shape, and movement of objects, and this information is conveyed to the user through voice messages and haptic feedback. The smart device or headset acts as an interface for the user to receive this information and transmits the environmental data to the generative AI in real time.

[0612] The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and behavior, and provides that information to the generation AI. The generation AI then adjusts the way it provides information based on this emotional information. For example, if the user is feeling anxious or scared, it will provide the necessary information in a calmer tone of voice.

[0613] To realize this system, cameras, sensors, and an emotion engine must be built into the smart device or headset, and algorithms must be developed on the server side for the generative AI to process this data and generate appropriate feedback.

[0614] For example, when a user is approaching an intersection, the camera and sensors capture the user's position and the movement of surrounding vehicles, and the generation AI uses this information to generate a voice message to the user through the headset saying, "A car is approaching from the right. Please cross when it is safe." Also, if the emotion engine detects that the user is nervous, the generation AI adjusts the way the information is conveyed to be more gentle.

[0615] This enables visually impaired people to understand their surroundings more safely and effectively, achieving greater independence in their daily lives and social participation.

[0616] The flow of the identification process will be described with reference to FIG.

[0617] Step 1: Collect environmental data

[0618] The camera and sensors installed on the device collect data about the user's surroundings, including the position, shape, and movement of objects, and the collected data is sent to a server in real time.

[0619] Step 2: Inferring emotional state

[0620] The device's on-board emotion engine infers the user's emotional state from their tone of voice, facial expressions, and behavior, and this inferred emotional state is sent to the server to tailor the way information is presented.

[0621] Step 3: Analyze data and generate information

[0622] The generative AI installed on the server analyzes the environmental data and emotional state data collected and sent in steps 1 and 2. Based on this analysis, useful information for the user is generated, such as how to cross an intersection safely or information about surrounding objects.

[0623] Step 4: Communicate

[0624] The generated information is sent from the server to the device, which then conveys it to the user in the form of voice messages and haptic feedback. The method of conveying the information is adjusted based on the user's emotional state. For example, if the user is feeling anxious, the information is conveyed in a calmer tone of voice.

[0625] Step 5: Receiving and responding to feedback

[0626] The user takes action based on the information provided, while the user's reactions and additional emotional state are fed back to the server via the device, and this feedback is used to improve the accuracy of the system.

[0627] Through this process, users will gain a deeper understanding of their surroundings, allowing them to safely and effectively participate in their daily lives and society. The system will also be more responsive to the user's emotional state, providing more personalized information.

[0628] Example 2

[0629] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0630] One of the main challenges faced by visually impaired people is the difficulty of safely and effectively perceiving the surrounding environment when going out. This is due to various factors, such as the presence of obstacles, people's movements, and changing traffic conditions. Furthermore, if the way information is presented is not adapted to the user's emotional state and individual needs, it can cause anxiety and stress. Conventional assistive technologies cannot provide sufficient solutions to these challenges.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0632] In this invention, the server includes a means for providing information to substitute for the field of view using a generating AI, a means for acquiring environmental information in real time and recognizing surrounding objects and people, and a means for analyzing the user's biometric information and inferring the user's emotional state. This allows visually impaired people to safely recognize their surroundings and receive voice messages and haptic feedback to support their movement. In addition, the method of providing information is adjusted according to the user's emotional state, reducing the user's stress and allowing them to go out more comfortably.

[0633] "Generative AI" is a type of artificial intelligence, a technology that generates new information and data from input data.

[0634] "Alternatives to vision" are information based on senses other than vision (hearing, touch, etc.) that are used by people who cannot rely on vision to understand their surroundings and situations.

[0635] A "smart device" is an electronic device that can connect to the Internet and other devices and has multiple functions.

[0636] A "headset" is a device, typically worn on the head, that provides both audio input (microphone) and output (headphones).

[0637] "Means for acquiring environmental information in real time" refers to techniques and processes for instantly collecting and analyzing information from the user's surrounding environment.

[0638] "Means of recognizing surrounding objects and people" refers to technologies that use cameras, sensors, voice recognition technology, etc. to identify and understand the presence, location, and actions of objects and people.

[0639] "Voice, audio messages, and haptic feedback" are means of communicating information to the user through non-visual means such as sound or vibration.

[0640] "User's biological information" is data that indicates the user's physical condition, such as heart rate, electrical conductivity of the skin, etc.

[0641] "Means for inferring emotional state" is a technology that analyzes and understands a user's emotional and psychological state at any given time from their biometric information and behavioral patterns.

[0642] "Means for adjusting the method of information provision" are techniques or processes that optimize the method of information delivery (e.g., message content and presentation) according to the user's condition and needs.

[0643] The embodiment of the present invention is a system for supporting visually impaired people to understand the surrounding environment and move safely when they go out. This system is configured by utilizing a generative AI model, a smart device, a sensor, a headset, and a biometric information measuring device.

[0644] The generative AI model deployed on the server analyzes the location and movement of surrounding objects and people, as well as the user's biometric information, based on data sent from smart devices and sensors. This analysis involves advanced data processing and data calculation. The hardware used includes high-resolution cameras, microphones, heart rate sensors, and skin conductivity sensors, while the software includes image recognition algorithms, voice recognition algorithms, and emotional state analysis algorithms.

[0645] The terminal refers to a smart device (smartphone or tablet) that collects information about the surroundings using a built-in camera and microphone and transmits it to the server. The terminal also receives responses from the server and conveys them to the user as voice messages or haptic feedback. The headset is used to convey voice messages directly to the user.

[0646] As a concrete example, consider a situation where a user is walking down a street. The device's camera captures obstacles ahead, and the generative AI model generates a voice message based on the prompt sentence "There is a chair 3 meters ahead," which is conveyed to the user through the headset. At the same time, the device uses a microphone to capture surrounding sounds, analyzes people's movements, and provides haptic feedback through a haptic device worn on the wrist. Based on the emotional state indicated by the user's biometric information, if it determines that "the user's heart rate is high and therefore tense," the generative AI model processes the prompt sentence to provide relaxing music.

[0647] This format allows visually impaired people to obtain a wealth of information about their surroundings, and the provision of information that takes into account their emotional state increases their sense of security when out and about.

[0648] The flow of the identification process will be described with reference to FIG.

[0649] Step 1: Collect environmental data

[0650] The device uses its built-in camera, microphone, and biometric sensors to collect environmental data and biometric information about the user. The inputs for this step are the sights and sounds of the surroundings, the user's heart rate, and the electrical conductivity of the skin, and the output is a digital representation of the collected environmental data and biometric information. The device converts this data into digital information for processing.

[0651] Step 2: Send and analyze data

[0652] The collected data is transmitted to a server via wireless communication. The generative AI model on the server analyzes the transmitted environmental data and biometric information to identify the positions, movements, and emotional state of surrounding objects and people. The inputs of this step are environmental data and biometric information, and the outputs are the identification information of objects and people and the emotional state of the user. The server analyzes the data and infers the identification information and the emotional state.

[0653] Step 3: Coordinate input and generate feedback

[0654] The generative AI model adjusts the content and format of the information provided to the user based on the analysis results. If the user is tense, it selects music to relax them, provides the location of obstacles with voice messages, and conveys the movements of people around them with haptic feedback. The input for this step is the identification information of objects and people and the user's emotional state, and the output is the information provided, such as voice messages, haptic feedback, and music. The server generates appropriate feedback based on the analysis results and sends it to the terminal.

[0655] Step 4: Communicate feedback

[0656] The terminal receives the feedback sent from the server and transmits voice messages to the user through the headset, haptic feedback through the haptic device, and music through the terminal's speaker or headset. The input of this step is the feedback from the server, and the output is the provision of information in a form that the user can recognize. The terminal transmits the received feedback to the user through an appropriate output device.

[0657] Through these steps, visually impaired people can perceive their surroundings more safely and efficiently and receive customized support based on their emotional state.

[0658] (Application example 2)

[0659] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0660] Currently, when users, including visually impaired people, use autonomous vehicles, it is difficult for them to accurately recognize the surrounding environment and situation and travel safely and comfortably. In particular, it is difficult for them to understand the status of surrounding vehicles, obstacles, and traffic lights, and information is not provided or the environment is not adjusted according to the user's emotional state, which can increase anxiety and stress while traveling.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0662] In this invention, the server includes a means for recognizing dynamic elements of the environment in real time and providing information, a means for accessing AI via an in-vehicle system or a headset, a means for analyzing input data from sensors and cameras to obtain environmental information around the vehicle, a means for recognizing surrounding vehicles, obstacles, and traffic light conditions, and a means for adjusting an emotional state based on voice, voice messages, haptic feedback, and biometric information. This allows the user to accurately understand the surrounding environment and situation and travel safely and comfortably. In addition, by providing information and adjusting the environment according to the user's emotional state, anxiety and stress during travel can be reduced, resulting in a more comfortable travel experience.

[0663] "Generative AI" is a type of artificial intelligence technology, a system that analyzes complex data and generates new information and inferences based on it.

[0664] "Means of recognizing dynamic elements of the environment in real time and providing information" refers to the function of analyzing input data from sensors, cameras, etc., instantly detecting changing conditions in the surroundings (vehicle movement, presence of obstacles, changes in traffic lights, etc.), and conveying that information to the user.

[0665] "Means of accessing AI via in-vehicle systems or headsets" refers to interfaces that communicate with the generative AI and exchange information through embedded systems in autonomous vehicles or headsets worn by the user.

[0666] "Means of analyzing input data from sensors and cameras" refers to the function of processing and analyzing in real time data collected by sensors and cameras installed to capture environmental data around the vehicle, and grasping environmental information.

[0667] "Means of recognizing surrounding vehicles, obstacles, and traffic signal conditions" refers to technologies and systems that enable autonomous vehicles to accurately understand the situation of surrounding objects, other vehicles, traffic signals, etc., and make safe driving decisions based on this information.

[0668] "Means for adjusting emotional state based on voice or voice messages, haptic feedback, and biometric information" refers to a function that provides information to the user through voice or haptic feedback, and further analyzes the user's biometric information to infer their emotional state, and provides information or adjusts the environment accordingly.

[0669] The embodiment of the present invention is a support system for users, including visually impaired people, to move safely and comfortably in an autonomous vehicle. This system is composed of a combination of on-board sensors and cameras, a generative AI model, a voice synthesis engine, a haptic feedback device, a biometric sensor, etc.

[0670] The server receives input data from the vehicle's sensors and cameras, analyzes it, and recognizes the vehicle's surroundings. This environmental information includes other vehicles, obstacles, traffic lights, etc. Using a generative AI model, the server generates voice messages and haptic feedback for the user based on this information. It also analyzes data collected from the user's biometric sensors (e.g., heart rate and skin electrical conductivity) to infer the user's emotional state. Depending on the user's emotional state, the method of providing information and the environment are adjusted to improve the user's psychological comfort.

[0671] For example, if the server detects another vehicle approaching from the right side of the vehicle, it will use the generative AI model to generate a voice message saying, "A vehicle is approaching from the right. Please be careful," and convey this to the user through the voice synthesis engine. At the same time, if the user feels tense, it will send a prompt to the generative AI model saying, "Please provide relaxing music," and play music to relax the user.

[0672] In this form, hardware and software such as cameras and sensors installed in the vehicle, haptic feedback devices, biometric sensors such as heart rate sensors and skin conductivity sensors, generative AI models, and voice synthesis engines play an important role. By combining these appropriately, it is possible to provide all users, including the visually impaired, with support to better understand the surrounding environment and move around safely and comfortably.

[0673] The flow of the identification process will be described with reference to FIG.

[0674] Step 1: Collect environmental data

[0675] The server collects surrounding environment data sent from the vehicle's sensors and cameras. This data includes the location of obstacles, the movement of nearby vehicles, the status of traffic lights, etc. Based on this input data, the server processes and calculates the data to understand the surrounding situation.

[0676] Step 2: Analyze environmental data

[0677] The collected environmental data is analyzed by a generative AI model, which recognizes the location of obstacles around the vehicle, the behavior of nearby vehicles, the status of traffic lights, etc. The results of the analysis are used as the basis for providing information to the user.

[0678] Step 3: Collect and analyze user biometric information

[0679] The server collects data from the user's biometric sensors, such as heart rate and skin conductivity, which are then analyzed and used to infer the user's emotional state, which in turn helps determine how to provide information and adjust the environment.

[0680] Step 4: Decide how to provide information

[0681] The server takes into account the analyzed environmental data and the user's emotional state to determine the optimal way to provide information. For example, if the user feels tense, it will decide to provide relaxing music. A generative AI model is used in this decision-making process.

[0682] Step 5: Provide information

[0683] Based on the information provision method determined, the server will provide voice messages, haptic feedback, appropriate music, etc. These information provision will be customized to provide the best experience for the user. For example, a voice message may be conveyed to the user saying, "A vehicle is approaching from the right. Please be careful."

[0684] This program processing allows users of self-driving vehicles to better understand their surroundings, enabling them to travel safely and comfortably, and by providing information and adjusting the environment according to the user's emotional state, it can reduce anxiety and stress while traveling.

[0685] Example 3

[0686] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0687] The goal is to reduce the barriers to information access that visually impaired people face in their daily lives and social activities. Specifically, these challenges include understanding the reactions and atmosphere of people around them at meetings and events, accessing the contents of books and materials they want to read, and providing haptic feedback that adapts to their own emotional state. These challenges are the main factors that limit the improved social participation and independent living of visually impaired people.

[0688] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0689] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the atmosphere of the environment and people's reactions from the text data using a generative AI model, a means for converting the text data into voice information and conveying it to the user, a means for monitoring the user's emotional state, a means for adjusting the intensity and pattern of haptic feedback based on the user's emotional state, and a means for collecting voices of meetings and events in real time. This allows visually impaired people to have a deeper understanding of their surroundings, actively participate in social activities, easily access the content they want to read as voice information, and receive appropriate haptic feedback according to their emotional state.

[0690] "Means for converting voice data into text data" refers to a process or device that converts collected voice information into text data. This conversion is accomplished using voice recognition technology.

[0691] "Means of analyzing environmental atmosphere and people's reactions from text data using generative AI models" refers to techniques and methods for analyzing text information and using that information to understand the surrounding situation, people's emotions, reactions, etc. Natural language processing technology is used for this analysis.

[0692] "Means for converting text data into voice information and conveying it to the user" refers to a process or device that reproduces the analyzed character information as voice and conveys it to the user. This conversion uses text-to-voice technology.

[0693] "Means for monitoring the user's emotional state" refers to technologies or methods for identifying and analyzing the user's emotional state from the user's tone of voice, facial expressions, body movements, etc. Voice analysis technology and sensor technology may be used for this monitoring.

[0694] "Means for adjusting the intensity or pattern of haptic feedback based on the user's emotional state" refers to techniques or methods for varying the type or degree of feedback provided through a haptic device depending on the user's current emotional state, using a feedback control algorithm based on the results of emotion analysis.

[0695] "Means of capturing audio in real time at meetings and events" refers to technologies and devices that directly capture live audio information in situ and record it as digital data, using microphones and audio recording systems.

[0696] In the embodiment of the present invention, a server, a terminal (smart device), and a user (visually impaired person) are at the center. The server is equipped with speech recognition technology, natural language processing technology, text-to-speech technology, emotion analysis technology, and haptic feedback control technology. The terminal is equipped with voice collection function, emotion state monitoring function, and haptic feedback providing function. Specific hardware and software include Google Cloud Speech-to-Text (speech recognition), OpenAI's GPT (text analysis and generation), Amazon Polly (text to speech conversion), and Tesseract OCR (character recognition).

[0697] The server converts the voice data received from the device into text data using Google Cloud Speech-to-Text. Next, this text data is analyzed using OpenAI's GPT to extract information such as the atmosphere of the meeting or event and people's reactions. The analysis results are converted into text data and into voice information using Amazon Polly. This voice information is transmitted to the user via the device.

[0698] The device uses a microphone to collect audio from meetings and events in real time and transmits it to the server. The device also monitors the user's tone of voice and body movements, and transmits data for analyzing the user's emotional state to the server. The server analyzes this data and transmits instructions to the device to adjust the intensity and pattern of haptic feedback according to the user's emotional state. Based on these instructions, the device provides appropriate feedback to the user through the haptic device.

[0699] A specific example is a scenario in which a server performs processing based on a prompt sentence such as "Analyze the atmosphere of the meeting and provide an overview of the atmosphere by voice," and a terminal conveys to a user voice information such as "The speaker just told a joke, and the participants are laughing." Another example includes a case in which a user scans a page of a book he or she wants to read with a camera on the terminal, and converts the text into voice information and provides it based on a prompt such as "Read the text on this page and read it aloud."

[0700] This format allows visually impaired people to more easily understand their surroundings and the content of meetings, access materials they want to read by voice, and receive richer information through haptic feedback that adapts to their emotional state.

[0701] The flow of the identification process will be described with reference to FIG.

[0702] Step 1:

[0703] The terminal uses a microphone to collect audio from a meeting or event. This audio data is sent to the server. The input is raw audio data, and the output is digitized audio data. This step involves the collection and digitization of audio data.

[0704] Step 2:

[0705] The server converts the received voice data into text data using Google Cloud Speech-to-Text. The input is digitized voice data, and the output is text data. In this process, data processing is performed to convert voice information into text information using voice recognition technology.

[0706] Step 3:

[0707] The server analyzes the converted text data using OpenAI's GPT to extract information such as the atmosphere of the meeting or event and people's reactions. The input is text data, and the output is the text data of the analysis results (a summary of the atmosphere and reactions). In this step, data calculations are performed using natural language processing technology to extract useful information from the text data.

[0708] Step 4:

[0709] The server converts the analysis results into voice information using Amazon Polly as text and sends this voice information to the terminal. The input is the text data of the analysis results, and the output is voice information. In this process, text-to-voice conversion is performed, and information is provided to the user in an easy-to-understand form.

[0710] Step 5:

[0711] The terminal transmits the voice information received from the server to the user. In this step, the voice information is played through the terminal's speaker. The input is the voice information, and the output is a voice message that the user can hear.

[0712] Step 6:

[0713] The terminal monitors the user's tone of voice and body movements and transmits the data to the server for analyzing the user's emotional state. The input is the user's voice and body movements data, and the output is the analysis result of the emotional state. In this step, the data is calculated using emotion analysis techniques to identify the user's emotional state.

[0714] Step 7:

[0715] The server sends instructions to the terminal to adjust the intensity and pattern of the haptic feedback according to the user's emotional state. The input is the analysis result of the emotional state, and the output is an instruction for haptic feedback. In this step, the haptic feedback is customized based on the user's emotional state.

[0716] Step 8:

[0717] The terminal provides appropriate feedback to the user through the haptic device based on the instruction from the server. In this step, the user receives a response according to his / her emotional state through haptic feedback. The input is an instruction for haptic feedback, and the output is the haptic feedback transmitted to the user's senses.

[0718] (Application example 3)

[0719] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0720] In modern society, visually impaired people and security workers face challenges such as understanding the environment, moving safely, and instantly recognizing abnormal behavior and dangerous situations. In particular, when visually impaired people participate in society and security workers need to respond quickly to emergency situations, there are limited means to effectively grasp the surrounding environment and people's emotional states. Another issue is the lack of support for leading a safer and more comfortable social life by providing appropriate haptic feedback according to the user's emotional state.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0722] In this invention, the server includes a means for providing information to substitute for vision using a generating AI, a means for analyzing the emotional state of people in the vicinity in real time and providing appropriate haptic feedback when abnormal behavior or situations are detected, and a means for adjusting the strength and pattern of the haptic feedback according to the emotional state of the user. This enables visually impaired people to engage in social activities more richly and independently, and enables security personnel to effectively grasp the surrounding environment and people's emotional states and respond quickly even in emergency situations.

[0723] "Generative AI" is a form of artificial intelligence that is a system that generates new information and responses independently from training data and performs specific tasks automatically.

[0724] A "means for providing information in place of vision" is a system or method that conveys necessary information to the user through audio or tactile feedback in place of visual information.

[0725] "Means of accessing AI via smart devices or headsets" refers to methods or systems for connecting to and using AI functions and services using devices such as smartphones, tablets, and head-mounted displays.

[0726] "Means of acquiring environmental information in real time" refers to systems and technologies that can instantly capture, analyze, and process the surrounding situation and changes.

[0727] "Means of recognizing surrounding objects and people" refers to systems or algorithms that identify objects and people from image and audio data obtained through cameras, sensors, etc., and analyze that information.

[0728] "Means of conveying information in the form of voice, voice message, haptic feedback, etc." refers to a method of conveying information to a user using voice synthesis technology or vibration patterns.

[0729] "Means for analyzing the emotional state of people in the surroundings in real time and providing appropriate haptic feedback when abnormal behavior or situations are detected" refers to a system or method that analyzes the facial expressions and tone of voice of people in the environment and instantly generates haptic feedback to warn of abnormalities based on the analysis.

[0730] "Means for adjusting the intensity and pattern of haptic feedback according to the emotional state of the user" refers to a system or technology that evaluates the user's current emotional or psychological state and adjusts the output of haptic feedback to match it.

[0731] An embodiment of the present invention includes a system that provides real-time alerts and haptic feedback to visually impaired people and security personnel based on environmental information and people's emotional states. The system is composed of a generative AI model, a smart device (smart glasses or a head-mounted display), emotion analysis software, and a haptic feedback device.

[0732] The server uses a generative AI model to analyze people's emotional states from the surrounding environmental data. Machine learning libraries such as OpenCV and TensorFlow are used for this analysis. The terminal (smart glasses or head-mounted display) collects environmental data via a camera and microphone and sends it to the server. The analysis results from the server are communicated to the user via the terminal, and a sensory warning is provided through a haptic feedback device if necessary.

[0733] The system allows users to gain a deeper understanding of their surroundings and people's emotional states, improving safety and comfort, enabling visually impaired people to be more independent in social activities and enabling security personnel to respond quickly and effectively in emergency situations.

[0734] A concrete example would be the following scenario: "A security guard wears smart glasses while on patrol. The application installed on his glasses scans the faces of people around him in real time and analyzes their emotional state from their facial expressions. If anger or aggressive emotions are detected, the guard's glasses will instantly emit a strong vibration to alert him." This process is intended to help security guards and visually impaired people better understand the situation around them and take appropriate action.

[0735] The flow of the identification process will be described with reference to FIG.

[0736] Step 1: The device collects environmental data using the camera and microphone. This collected data includes facial images and audio of people around it. It receives the surrounding images and audio as input and generates a data file as output.

[0737] Step 2: The terminal transmits the collected environmental data to the server. In this step, the collected video and audio data is transferred to the server. It receives a data file from the terminal as input and transmits data to the server as output.

[0738] Step 3: The server uses the received environmental data to analyze the emotional state of the surrounding people through a generative AI model. This process uses image recognition and voice recognition technologies to generate emotion analysis results. It receives video and audio data as input and generates emotion analysis results as output.

[0739] Step 4: The server sends appropriate haptic feedback instructions to the terminal based on the emotion analysis result. The instructions include feedback intensity and pattern information. It receives the emotion analysis result as input and generates haptic feedback instructions as output.

[0740] Step 5: The terminal provides haptic feedback to the user based on the haptic feedback instruction received from the server. This feedback is provided through a haptic feedback device worn by the terminal or the user. The terminal receives the haptic feedback instruction as an input and performs specific haptic feedback to the user as an output.

[0741] Through these steps, the system can effectively grasp the surrounding environment and people's emotional states, and provide users with appropriate warnings and haptic feedback in real time, helping visually impaired people to move around safely and participate in society, and enabling security personnel to quickly grasp the situation and take action.

[0742] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0743] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0744] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0745] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0746] [Third embodiment]

[0747] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0748] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0749] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0750] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0751] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0752] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0753] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0754] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0755] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0756] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0757] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0758] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0759] "Example 1"

[0760] In this embodiment of the present invention, a visually impaired person wears a smart device or headset and accesses the generative AI. The generative AI analyzes information obtained from cameras and sensors and provides the visually impaired person with information about the surrounding environment. Specifically, the AI ​​recognizes the position, shape, and movement of objects and people, and communicates this to the visually impaired person through voice messages and haptic feedback.

[0761] "Example 2"

[0762] As a concrete example, when a visually impaired person walks through the city, the generative AI recognizes obstacles, intersections, traffic light status, etc. ahead and conveys them to the visually impaired person through voice messages. It also infers people's presence and movements from their movements and voices, and provides haptic feedback to the visually impaired person. This enables the visually impaired person to understand the situation around them and move around safely.

[0763] "Example 3"

[0764] Furthermore, generative AI can help visually impaired people participate in society. For example, it can analyze people's reactions and the atmosphere at meetings and events, and convey that to visually impaired people. It can also read the text information in books and documents that visually impaired people want to read, and read it aloud. This allows visually impaired people to participate in social activities more richly and independently.

[0765] The process flow of each embodiment will be described below.

[0766] "Example 1"

[0767] Step 1: A visually impaired person wears a smart device or headset and accesses the generative AI.

[0768] Step 2: The generative AI analyzes the information obtained from the camera and sensors.

[0769] Step 3: The AI ​​recognizes the position, shape, and movement of objects and people, and sends voice messages and tactile

[0770] Communicating with visually impaired people through feedback.

[0771] "Example 2"

[0772] Step 1: When a visually impaired person walks around town, the generative AI recognizes obstacles, intersections, traffic light status, etc. ahead.

[0773] Step 2: The AI ​​infers people's presence and movements from their movements and voices.

[0774] Step 3: The AI ​​will communicate information to the visually impaired through voice messages and provide haptic feedback, allowing them to understand their surroundings and navigate safely.

[0775] "Example 3"

[0776] Step 1: Generative AI helps visually impaired people participate in society, for example by analyzing people's reactions and the atmosphere at meetings and events.

[0777] Step 2: The AI ​​reads the text information in the books and materials that the visually impaired person wants to read.

[0778] Step 3: The AI ​​will read out the information aloud and convey it to the visually impaired, enabling them to participate in social activities more richly and independently.

[0779] Example 1

[0780] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0781] With current technology, visually impaired people face many challenges when moving independently, and there are limited means to safely and effectively obtain information about their surroundings. In particular, the lack of a system that provides detailed information on the position, shape, and movement of people and objects in real time is a major factor that limits the freedom of movement of visually impaired people and reduces their opportunities for social participation. For this reason, new technological solutions are needed to help visually impaired people better understand their surrounding environment and move safely.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0783] In this invention, the server includes a means for analyzing environmental data using a generative AI model, a means for converting the analysis results into voice messages and haptic feedback, and a means for processing the collected environmental data in real time. This allows visually impaired people to receive detailed information about the positions, shapes, and movements of people and objects around them in real time through voice and haptic feedback. As a result, it is possible to support safe mobility for visually impaired people and expand opportunities for social participation.

[0784] "Generative AI" is a type of artificial intelligence that learns from data and generates models and algorithms to perform specific tasks.

[0785] A "smart device" is a portable electronic device that can connect to the Internet and provide multiple functions by running applications.

[0786] A "headset" is a device that generally has audio output (speaker) and audio input (microphone) functions and is worn on the head.

[0787] "Means of acquiring environmental information in real time" is a process or technology that instantly captures the current situation and changes and provides information without delay.

[0788] "Means for recognizing surrounding objects and people" refers to technology that analyzes visual, auditory, or other sensor information to determine the presence and location of objects or people within a particular environment.

[0789] "Means of conveying information in the form of voice, audio messages, tactile feedback, etc." are methods of providing information to users using sensory means such as sound or vibration.

[0790] The present invention relates to a system for assisting visually impaired people in understanding their surroundings and navigating safely, the system being implemented using specific hardware and software.

[0791] The servers host generative AI models that are designed to analyze data collected from the environment and identify information such as the location, shape, and movement of objects and people. The software used includes, for example, customized versions of OpenAI's GPT and Google's TensorFlow, which provide complex data analysis and learning capabilities that increase the accuracy and efficiency of the system.

[0792] The terminals are devices worn by the visually impaired, such as smart devices or headsets. These devices are equipped with cameras and sensors to collect environmental data. The collected data is sent to a server and analyzed by a generative AI model.

[0793] The analysis results are sent back to the device and provided to the visually impaired as audio messages and / or haptic feedback, helping users better understand their surroundings and providing them with information to navigate safely.

[0794] A specific example is the prompt, "Please identify the location and movement of people and objects around you and provide that information in a voice message," which dictates the type of data that will be analyzed by the generative AI model.

[0795] This embodiment of the invention focuses on leveraging advanced technology to solve everyday challenges faced by the visually impaired, with a combination of hardware and software to provide real-time environmental information to help them become more independent.

[0796] The flow of the identification process will be described with reference to FIG.

[0797] Step 1: The device collects surrounding environment data from the cameras and sensors installed in the smart device or headset worn by the visually impaired person. The input of this step is the video, audio, temperature, distance, and other data of the surrounding environment acquired by the device's camera and sensors. The output is the collected raw environmental data.

[0798] Step 2: The terminal transmits the collected environmental data to the server. In this step, data transmission is performed using wireless communication technology. The input is the environmental data collected in step 1, and the output is the data transmitted to the server.

[0799] Step 3: The server analyzes the received environmental data using the generative AI model. This analysis includes processes such as object recognition, person recognition, and motion detection. The input is the environmental data received in step 2, and the output is the analysis result. This analysis result includes information such as the position, shape, and movement of surrounding objects and people.

[0800] Step 4: The server generates information useful to visually impaired people based on the analysis results. This information is provided in the form of voice messages and haptic feedback. The input is the analysis results from step 3, and the output is information to convey to visually impaired people.

[0801] Step 5: The terminal conveys the information received from the server to the visually impaired person using the terminal's built-in speaker and haptic feedback device. The input is the information generated in step 4, and the output is a voice message or haptic feedback that can be recognized by the visually impaired person.

[0802] An example of a specific operation is processing based on the prompt, "Please identify the positions and movements of people and objects around you and convey that information in a voice message." In this case, the device uses a camera to capture images of the surroundings and sends the data to the server. The server analyzes the data, identifies the positions and movements of people and objects, and sends this information back to the device as a voice message for the visually impaired person. The device then conveys this message to the visually impaired person through a speaker.

[0803] (Application example 1)

[0804] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0805] Currently, the main challenge that visually impaired people face when shopping in brick-and-mortar stores is the difficulty of understanding the layout of the store and the location of products. Another major challenge is moving safely while avoiding obstacles. These challenges prevent visually impaired people from enjoying shopping independently and limit their opportunities for social participation.

[0806] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0807] In this invention, the server includes a means for accessing the generating AI via a smart device or a headset, a means for acquiring and analyzing information on the in-store environment in real time, a means for recognizing surrounding objects and people and identifying their positions and shapes, and a means for conveying information to visually impaired people through voice messages and haptic feedback, thereby enabling visually impaired people to grasp the positions of products and the presence of obstacles in a physical store and to shop efficiently while moving around safely.

[0808] "Generative AI" is a type of artificial intelligence that analyzes environmental data and generates and provides useful information to users based on the results of that analysis.

[0809] A "smart device" is a portable electronic device that can connect to the Internet and perform multiple functions.

[0810] A "headset" is a device that can be worn on the ears to receive voice communication and voice guidance.

[0811] "Real-time environmental information acquisition" is the process of collecting and updating data about the surrounding environment in close sync with the current time.

[0812] "Means of recognizing surrounding objects and people" is a technology that identifies and identifies the presence, position, and shape of objects and people from input from cameras, sensors, etc.

[0813] A "voice or audio message" is a method of transmitting information in the form of sound waves that can be heard by the human ear.

[0814] "Haptic feedback" is a technology that provides users with information through their sense of touch, conveying information through vibrations and changes in pressure.

[0815] An embodiment of the invention includes the use of generative AI via a smart device or headset. The server uses a generative AI model to analyze environmental data sent from the smart device or headset. This analysis identifies the position, shape, and movement of surrounding objects and people, and conveys information to the visually impaired person through audio messages and haptic feedback.

[0816] The hardware used includes smart devices and headsets. These devices acquire real-time environmental data through built-in cameras and sensors and transmit it to a server. The server uses software libraries such as OpenCV and TensorFlow to perform object detection and image analysis, and conveys the obtained information to the user through a voice synthesis system.

[0817] For example, when a user puts on the smart glasses and enters a store, voice guidance is provided such as, "There is a product shelf ahead. The safe route is to go straight." This information is generated by analyzing video data captured by the smart glasses' camera. Also, if a user is looking for a specific product, the generating AI provides specific instructions such as, "Turn right and you will find the product." This allows visually impaired people to efficiently find the products they need while moving around safely.

[0818] The flow of the identification process will be described with reference to FIG.

[0819] Step 1: The device's camera and sensors acquire surrounding environmental data. In this step, the cameras and sensors in the smart device or headset worn by the user collect information on surrounding objects, people, obstacles, etc. in real time. The input is data from optical images and distance sensors, and the output is sent to the server as digitized environmental data.

[0820] Step 2: The server analyzes the environmental data. The received environmental data is analyzed by the generative AI model to identify the position, shape, and movement of surrounding objects and people. The input of this step is the digitized environmental data from step 1, and the output is the object identification information and movement patterns as the analysis results.

[0821] Step 3: Based on the analysis results, the server generates voice messages and haptic feedback in a format that is easy for the user to understand based on the analysis results. Examples include product location guidance and obstacle warnings. The input is the object identification information and movement patterns from step 2, and the output is the voice messages and haptic feedback provided through the user's smart device or headset.

[0822] Step 4: The user adjusts their behavior based on the provided information. Based on the voice messages and haptic feedback they receive, the user may adjust their route through the store or search for a specific product. The input to this step is the information from step 3, and the output is a change in the user's behavior, such as moving in a particular direction or picking up a product.

[0823] This process allows visually impaired people to shop safely and efficiently in physical stores.

[0824] Example 2

[0825] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0826] Currently, the biggest challenge that visually impaired people face when walking around town is that they cannot directly obtain the information they need to move safely, such as the presence of obstacles, traffic conditions, and the movements of people around them. This exposes them to great anxiety and danger when they move around. Another major problem is that it is difficult for visually impaired people to communicate and synchronize their activities with those around them when participating in society.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0828] In this invention, the server includes a means for collecting data from external sensors in real time, a means for analyzing the surrounding environment and the situation of people from the collected data using a generative AI model, and a means for providing appropriate instructions and information to visually impaired people in the form of voice messages and haptic feedback based on the analysis results, which enables visually impaired people to more accurately understand the surrounding environment, move safely, and easily communicate and synchronize their activities when participating in society.

[0829] "Means of collecting environmental data from external sensors in real time" refers to the process of instantly acquiring data about the current surrounding environment through sensors such as cameras, microphones, and GPS mounted on smart devices or dedicated devices, and sending it to a server for analysis.

[0830] "Means of analyzing the positions and movements of surrounding objects and people from collected environmental data using a generative AI model" refers to a method in which an artificial intelligence model trained using machine learning technology extracts and interprets information such as the positions of objects and the movements of people based on collected data.

[0831] "Means of providing instructions and information to visually impaired persons in the form of voice messages and haptic feedback based on analysis results" refers to a method of directly transmitting necessary information and appropriate instructions to users based on the results of analysis by AI, using formats such as voice and vibration that are easy for visually impaired persons to understand.

[0832] "Means of accessing AI via smart devices or dedicated devices and transmitting information in real time" refers to the process of communicating with AI on a server through internet-connected devices such as smartphones, tablets, and dedicated wearable devices, sending collected data, and receiving analysis results.

[0833] In the embodiment of the present invention, a system is provided to enable visually impaired people to understand their surrounding environment and navigate safely. The system mainly consists of three main components: data collection from external sensors, data analysis by generative AI models, and information provision through voice messages and haptic feedback.

[0834] The server uses cloud-based computing resources to host generative AI models that run on machine learning frameworks such as TensorFlow and PyTorch and are used to analyze data collected from external sensors, such as image data from cameras and audio data from microphones built into smart devices and dedicated devices.

[0835] The terminal is a smart device or dedicated device carried by the visually impaired person, which collects environmental data through external sensors such as a camera, microphone, and GPS, and transmits it to a server. In addition, it provides voice messages and haptic feedback to the visually impaired person based on instructions from the server. For example, if the server sends an instruction to the terminal saying, "There is an obstacle 3 meters ahead. Move left," the terminal outputs this message in voice and provides instructions to the visually impaired person to avoid the obstacle.

[0836] Based on the voice messages and haptic feedback provided, the user can understand the surrounding situation and move safely by avoiding obstacles. In addition, the haptic feedback indicating the movement and presence of people can help the user avoid collisions with other pedestrians. In this way, the present invention provides a concrete method for the visually impaired to move more independently and safely.

[0837] The flow of the identification process will be described with reference to FIG.

[0838] Step 1: Collect data from external sensors

[0839] The terminal collects environmental data from the visually impaired person's smart device or dedicated device through external sensors such as cameras, microphones, and GPS. The input of this step is the surrounding images, audio, and location information, and the output is a digitized form of these data. The terminal temporarily stores the data and converts it into a format that can be transmitted over the network in order to send it to the server.

[0840] Step 2: Analyze data and generate instructions

[0841] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding environment, the location of obstacles, people's movements, etc. from this data. The input for this step is image data, audio data, and location information, and the output is the analysis results such as the location of obstacles and people's movements. Based on the analysis results, the generative AI model generates specific instructions and information to be provided to the visually impaired person.

[0842] Step 3: Provide information

[0843] The terminal provides the instructions and information received from the server to the visually impaired person as voice messages and haptic feedback. The input of this step is the instructions and information sent from the server, and the output is the voice message and haptic feedback received directly by the visually impaired person. The terminal outputs the instructions as voice messages using a voice synthesis technology that converts text to speech, and provides the necessary information as a tactile sensation through a haptic feedback device.

[0844] Through these steps, visually impaired people can better understand their surroundings, receive specific instructions to avoid obstacles, and navigate safely with information about the movements and positions of people around them. This process happens in real time, significantly improving support for visually impaired people in participating in society.

[0845] (Application example 2)

[0846] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0847] In modern society, personal safety is a major concern. Accurately understanding the perception of the surrounding environment and people and their impact on safety is essential to ensure personal safety, especially when going out alone or at night. However, current technology has limited effective means to support personal safety in a given environment in real time and immediately warn of the approach of suspicious people or the occurrence of an emergency. Against this background, there is a need for the development of supportive technologies that allow individuals to actively manage their own safety and protect themselves from danger.

[0848] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0849] In this invention, the server includes a means for acquiring environmental information in real time, a means for recognizing surrounding objects and people, and a means for transmitting information to the user via voice, voice message, and haptic feedback. This allows the user to grasp the surrounding situation in real time and increase safety awareness. In addition, by using this system, a warning can be received immediately when a suspicious person approaches or an emergency occurs, and appropriate measures can be taken. In this way, this invention provides important support for ensuring personal safety.

[0850] "Generative AI" is a type of artificial intelligence based on machine learning technology, a system that recognizes environmental information, objects, and the state of people based on input data, and generates and provides useful information to users.

[0851] "Means of acquiring environmental information in real time" refers to technologies and methods that use devices such as sensors, cameras, and microphones to instantly capture and analyze the surrounding situation and sounds at that moment.

[0852] "Means for recognizing surrounding objects and people" refers to techniques or methods that use image recognition technology or voice recognition technology to identify the characteristics of surrounding objects and people and understand that information.

[0853] "Voice or audio message" is a form of audible linguistic information used to convey system-generated information to the user.

[0854] "Haptic feedback" is a means of conveying information from a device to the user through physical sensations such as vibration or pressure.

[0855] In the embodiment of the present invention, a smartphone, smart glasses, or a head-mounted display is used as a terminal, and surrounding environmental information is acquired in real time through a camera or microphone built into these terminals. The acquired information is transmitted to a server equipped with a generative AI model. The server uses a deep learning framework to recognize surrounding objects and people through image recognition technology and voice recognition technology, and generates an appropriate response according to the situation. The generated information is communicated to the user in the form of voice, voice message, or haptic feedback.

[0856] The hardware used includes smartphones with Qualcomm Snapdragon processors and wearable devices such as Google Glass, while the software uses machine learning frameworks such as TensorFlow and PyTorch.

[0857] As a specific example, if a user is walking at night and the camera captures a suspicious person approaching, the server generates a voice message saying, "Caution: Someone is approaching from behind. Please move to a safe place immediately," and transmits this to the user through the terminal. In addition, in the event of an emergency, it is also possible to provide haptic feedback saying, "An emergency has occurred. Please evacuate to a safe place immediately." In this way, the present invention provides a concrete means for the user to grasp the situation around them in real time and ensure their safety.

[0858] The flow of the identification process will be described with reference to FIG.

[0859] Step 1: The device collects surrounding environment information. In this step, the device collects surrounding video and audio in real time using the camera and microphone built into the smartphone, smart glasses, or head-mounted display. At this time, the device takes video and audio data as input and prepares to send it to the server.

[0860] Step 2: Send the collected environmental information to the server. The terminal sends the video and audio data collected in step 1 to the server. In this process, the data is transferred to the server via the Internet.

[0861] Step 3: Server analyzes environmental information. The video and audio data sent to the server is analyzed by a generative AI model. This AI model uses a deep learning framework to recognize surrounding objects and people from the video, and analyze environmental sounds and conversations from the audio data. The output of this step is the analysis results, which include specific situational awareness and identification of potential hazards.

[0862] Step 4: Generate an appropriate response based on the analysis results. Based on the analysis results obtained in step 3, the server generates information to be communicated to the user. This information is about the appropriate action the user should take at that moment, such as safety warnings and instructions. The generated information can take the form of a voice message or haptic feedback.

[0863] Step 5: Communicate the generated information to the user. The server sends the information generated in step 4 to the terminal, which then communicates the information to the user. The information is communicated to the user through the speaker in the case of a voice message, or by using the device's vibration function in the case of haptic feedback. In this step, the user receives specific information to ensure their own safety.

[0864] Example 3

[0865] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0866] When visually impaired people participate in social activities such as meetings and events, they cannot visually grasp the reactions and atmosphere of those around them. In addition, it is difficult for them to independently read the text information in books and materials they want to read. These challenges are factors that limit the independent participation of visually impaired people in social activities.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0868] In this invention, the server includes a means for analyzing reactions and atmosphere at a meeting or event, a means for reading out text information of documents or materials, and a means for interpreting prompts input by voice and generating appropriate responses, thereby enabling visually impaired people to understand the surrounding situation in social activities and obtain necessary information independently.

[0869] "Generative AI" is a type of artificial intelligence that can generate new information based on learned data.

[0870] "A means of analyzing reactions and atmosphere at meetings and events" refers to the ability to understand the emotions and mood of participants at a particular gathering or event and express them in verbal or other forms.

[0871] "A means for converting the textual information of a document or material into audible speech" refers to a process or device that converts printed or electronically provided textual information into audio output.

[0872] "Means of accessing AI via smart devices and headsets" refers to methods of using intelligent electronic devices and audio output devices to communicate with and interact with artificial intelligence systems.

[0873] "Means of acquiring environmental information in real time" refers to the function of instantly collecting and analyzing the surrounding situation and data at the current time.

[0874] "Means of recognizing surrounding objects and people" refers to technologies and methods that use cameras, sensors, etc. to identify and identify nearby objects and people.

[0875] "Means of conveying information in the form of voice, audio messages, haptic feedback, etc." refers to methods of directly transmitting information to the user using sound waveforms, vibrations, etc.

[0876] "Ability to interpret voice prompts and generate appropriate responses" refers to the artificial intelligence's ability to understand spoken instructions or questions from a user and provide information or a response to them.

[0877] The embodiment of the present invention is mainly composed of three parties: a server, a terminal, and a user. The server is equipped with a generative AI model, which is designed to solve the problems that visually impaired people face when participating in social activities. The terminal mainly takes the form of a smart device or a headset, and plays a role in transmitting input from the user to the server. The user is a visually impaired person, and can acquire and deepen his / her understanding of information in social activities through this system.

[0878] The generative AI model that runs on the server is implemented using advanced technologies such as OpenAI's GPT and Google's BERT. This model analyzes video and audio data from meetings and events, and extracts the atmosphere and reactions of the event as text information. It also has the ability to read text information from images of documents and materials and convert it into audio data.

[0879] The device receives the user's voice prompts through a microphone, converts them into text data, and sends them to the server. It also uses the device's camera to take images of documents and materials and send them to the server. The server processes the received data with a generative AI model to generate an appropriate response.

[0880] As a specific example of use, if a user voice-inputs a prompt such as "Tell me about the atmosphere of the current meeting," the device converts this input into text and sends it to the server. The generative AI model on the server analyzes and extracts information about the atmosphere of the meeting and sends the results as text to the device. The device converts this text into speech and reads it aloud to the user. In addition, if a user requests "Read this page," the device takes an image of the document and sends it to the server. The server extracts text from the image, converts it into speech, and sends it to the device. The device reads this speech aloud to the user.

[0881] This format allows visually impaired people to receive specific support to help them acquire and understand information in social activities.

[0882] The flow of the identification process will be described with reference to FIG.

[0883] Step 1: The user speaks to the device, including a prompt about the mood of the meeting or event and a request for a document to be read aloud. The device receives the speech and converts it into text data.

[0884] Step 2: The converted text data is sent over the internet to a server, which receives the data and prepares it for processing by the generative AI model.

[0885] Step 3: For requests related to the atmosphere of a meeting or event, the server uses the generative AI model to analyze the atmosphere and reactions from the provided audio and video data. For requests to read a document aloud, the server receives the image data of the document sent from the device and performs text extraction.

[0886] Step 4: The generative AI model generates an appropriate text response based on the analyzed or extracted information. For a request about the atmosphere of the meeting, it generates a response summarizing the emotions and key points of the meeting, and for a request to read a document, it generates the extracted text information.

[0887] Step 5: The generated text response is sent from the server to the device, which receives the text and converts it into audio data using its built-in text-to-speech conversion function.

[0888] Step 6: The converted voice data is output to the user, allowing the user to understand the atmosphere of the meeting or event and hear the content of the document.

[0889] Through this processing flow, visually impaired people can acquire information and deepen their understanding in social activities, enabling them to participate more independently.

[0890] (Application example 3)

[0891] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0892] When visually impaired people shop at a store, they face the challenge of being unable to grasp the location and information of products, the atmosphere of the store, and the reactions of people. For this reason, visually impaired people may feel restricted in their ability to shop independently, limiting their opportunities to participate in society. Another problem is the lack of systems that efficiently provide the information necessary for safe movement and product selection at a store.

[0893] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0894] In this invention, the server includes a means for providing audio guidance to visually impaired people based on analysis of product information and people's reactions in a sales area using a generation AI, a means for acquiring environmental information in real time, analyzing it, and generating audio guidance information, and a means for utilizing environmental data including product positions and sale information in the sales area. This enables visually impaired people to safely and efficiently select products and move around in sales areas, expanding opportunities for social participation.

[0895] "Generative AI" is an algorithm based on machine learning and deep learning techniques, and is a system that generates output from specific input in a format that is easy for humans to understand.

[0896] A "visually impaired person" refers to a person who has difficulty recognizing general visual information due to a visual impairment.

[0897] "Product information in the sales area" includes all information necessary for purchasing a product, such as the type of product, price, location, and sale information.

[0898] "Voice guidance based on analysis of people's reactions" refers to analyzing the facial expressions, words, actions, and other reactions of people around you and providing voice instructions and information based on that information.

[0899] "Means of acquiring real-time environmental information" refers to methods or technologies that continuously collect surrounding environmental data in accordance with the current time.

[0900] "Means for generating voice guidance information" refers to the process of analyzing collected information and data and using that information to provide useful instructions and information in voice format to visually impaired people.

[0901] "Environmental data including product location and sales information within the point of sale" is general data about the state and condition of the point of sale, including information indicating where products are located and which products are on sale.

[0902] A "server" is a computer system or software that collects, processes, stores data, and provides information to clients.

[0903] The embodiment of the present invention relates to a system that supports visually impaired people in shopping at a sales point. The system utilizes a generative AI model to provide voice guidance based on an analysis of product information and people's reactions in the sales point. The server has a function of acquiring environmental information in real time, analyzing it, and generating voice guidance information. The server also has a function of utilizing environmental data including product positions and sale information in the sales point. Terminals such as smart devices and headsets are used to implement the system, and visually impaired people access the generative AI through these terminals. The terminal has a function of conveying information to the user in the form of voice, audio messages, haptic feedback, etc.

[0904] The hardware used includes smart devices (e.g. smartphones), in-store cameras and sensors, and the software uses voice recognition and synthesis systems, location services, and generative AI models (e.g. OpenAI's GPT-4). This system allows visually impaired people to safely and efficiently select and navigate products at sales points, expanding their opportunities for social participation.

[0905] For example, when a user uses a smart device to ask, "What sweets are on sale nearby?", the server analyzes the in-store environmental data and uses a generative AI model to generate guidance information such as, "There is chocolate on sale near you. It's on the shelf about 3 meters to your right," and conveys this information to the user via a voice message on the device. This process combines the user's location information with in-store product information to provide optimal guidance.

[0906] The flow of the identification process will be described with reference to FIG.

[0907] Step 1: The user inputs a question into the terminal.

[0908] Users use their smart devices to voice-input questions about specific product information or in-store directions, which are then converted into text data by the device's on-board voice recognition system.

[0909] Step 2: The device sends the text data of the question to the server.

[0910] The converted text data is sent over the Internet to a server, which receives the text data and uses it as input for processing.

[0911] Step 3: The server parses the text data and sends prompts to the generative AI model to generate appropriate responses.

[0912] The server analyzes the received text data and forms prompts to generate specific responses based on the user's question, which are then sent to a generative AI model for use in generating responses.

[0913] Step 4: The generative AI model generates a response based on the prompt.

[0914] The generative AI model processes the prompts received from the server and generates appropriate responses to the user's questions, which may include in-store product locations, sales information, and descriptions of the in-store environment.

[0915] Step 5: Convert the server-generated response into audio data.

[0916] The server converts the response text received from the generative AI model into voice data using a voice synthesis system, which is then used to convey information to the user.

[0917] Step 6: The server sends the voice data to the terminal.

[0918] The voice data is again transmitted over the Internet to the user's terminal, which receives the voice data and outputs it to the user.

[0919] Step 7: The user listens to the audio output to obtain the information.

[0920] The user hears product information and in-store guidance information in response to questions through voice output from the terminal, allowing the user to obtain the necessary information and shop efficiently at the sales point.

[0921] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0922] "Example 1"

[0923] In one embodiment of the present invention, the generation AI has an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotions from the user's tone of voice, facial expressions, behavior, etc., and provides the information to the generation AI. The generation AI adjusts the method of providing information to the user based on the information provided by the emotion engine. For example, if the user feels angry, the generation AI will calm the way in which information is communicated, or will communicate only the necessary information. This makes it possible to respond appropriately according to the user's emotional state.

[0924] "Example 2"

[0925] In another embodiment of the present invention, the emotion engine also analyzes the user's biometric information (e.g., heart rate and skin electrical conductivity). This allows the user's emotional state to be grasped more accurately. The generation AI infers the user's emotional state based on this biometric information and adjusts the method of providing information. For example, if the user feels tense, the generation AI takes measures to improve the user's emotional state, such as providing relaxing music.

[0926] "Example 3"

[0927] Furthermore, in another embodiment of the present invention, the generation AI adjusts the intensity and pattern of the haptic feedback according to the user's emotional state. For example, if the user feels surprised, the generation AI provides strong haptic feedback to attract the user's attention. On the other hand, if the user feels relaxed, the generation AI provides gentle haptic feedback. This makes it possible to provide haptic feedback appropriate to the user's emotional state.

[0928] The process flow of each embodiment will be described below.

[0929] "Example 1"

[0930] Step 1: The emotion engine starts working, inferring emotions from the user's tone of voice, facial expressions, behavior, etc.

[0931] Step 2: The emotion engine provides the inferred emotion information to the generative AI.

[0932] Step 3: The generative AI adjusts how it presents information to the user based on the information provided by the emotion engine.

[0933] "Example 2"

[0934] Step 1: The emotion engine analyzes the user's biometric information (e.g., heart rate and skin conductivity).

[0935] Step 2: The generative AI uses this biometric information to infer the user's emotional state and adjusts the way it provides information.

[0936] Step 3: The generative AI responds to improve the user's emotional state. For example, if the user feels tense, the generative AI provides relaxing music.

[0937] "Example 3"

[0938] Step 1: The generative AI obtains the user's emotional state from the emotion engine.

[0939] Step 2: The generative AI adjusts the intensity and pattern of haptic feedback depending on the user's emotional state.

[0940] Step 3: The generative AI provides calibrated haptic feedback to the user.

[0941] Example 1

[0942] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0943] Visually impaired people face the challenge of having difficulty accurately understanding the surrounding environment when engaging in daily life and social activities. Conventional assistive technologies only provide environmental information one-way, and there is also a problem in that they do not provide information that takes into account the user's emotional state. This can make it difficult for users to accept information or to respond appropriately in emergency situations.

[0944] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0945] In this invention, the server includes a means for analyzing environmental data transmitted from a mobile terminal or a headset, a means for estimating the user's emotional state, and a means for adjusting the information provision method based on the analyzed environmental information and emotional information. This allows visually impaired people to understand the surrounding environment more accurately and easily, and provides appropriate information according to their emotional state. This adaptive information provision also enables safe mobility support and improved social participation for visually impaired people.

[0946] "Generative AI" is an algorithm trained using machine learning and deep learning techniques, and is a system capable of generating or analyzing new information from input data.

[0947] "Environmental data" refers to information about the physical situation and conditions surrounding the user, such as images, audio, and location information, collected through sensors and cameras.

[0948] A "mobile terminal" is an electronic device, such as a smartphone or tablet, that is portable and has wireless communication capabilities.

[0949] A "headset" is a device worn on the ears that enables voice communication and listening to music. In support systems for the visually impaired, a headset is used to transmit voice messages and accept voice input.

[0950] "User's emotional state" refers to the user's current psychological and emotional state, and is inferred from tone of voice, facial expressions, behavior, etc.

[0951] "Adjusting the way information is presented" refers to appropriately changing the way information is conveyed (e.g., the tone of a voice message or the intensity of haptic feedback) based on the received environmental data and the user's emotional state.

[0952] The embodiment of the present invention is centered around a mobile device or headset worn by a visually impaired person and a generative AI model that runs on a server that controls them. The mobile device or headset includes a camera and sensors for recognizing the environment, and a microphone for inferring the user's emotional state. Data collected through these hardware devices is transmitted to the server using wireless communication technology.

[0953] On the server, a generative AI model using machine learning frameworks such as TensorFlow and PyTorch analyzes the transmitted data. This analysis includes identifying the location, shape, and movement of objects and people using image recognition technology, and inferring the user's emotional state using voice analysis technology. Based on the analysis results, the generative AI model determines the optimal way to provide information to the user and conveys that information to the user via a mobile device or headset in the form of a voice message or haptic feedback.

[0954] For example, while a user is walking in a park, the camera on the mobile device captures a bench and people walking in front of them, and this data is sent to the server. The generative AI model on the server analyzes the position of the bench and the movement of the people walking from this image, and provides the user with a voice message saying, "There is a bench 3 meters to the right. There is a person walking ahead." At the same time, it infers the emotion from the user's tone of voice and adjusts the tone of the message as necessary.

[0955] In this way, this invention provides an information provision system that integrates both environmental recognition and emotion estimation. This allows visually impaired people to understand their surroundings in more detail and more appropriately, and to receive information customized according to their emotional state. This system is expected to contribute to improving the quality of daily life of visually impaired people and promoting their participation in society.

[0956] The flow of the identification process will be described with reference to FIG.

[0957] Step 1: Collect environmental data

[0958] The device collects data about the surrounding environment using cameras and sensors. This data includes images, audio, and location information. The input is raw data from the environment, and the output is a digitalized version of the raw data. Through this conversion process, the device transmits visible light images, audio data, and location information from GPS to a server.

[0959] Step 2: Data analysis and sentiment estimation

[0960] The server analyzes the received environmental data using a generative AI model and an emotion engine. The generative AI model performs image recognition to identify the location, shape, and movement of people and objects. The emotion engine infers the user's emotional state from voice data. The input is the environmental data sent from the device, and the output is object identification information and the user's emotional state. Through this process, the server generates detailed information about both the surrounding environment and the user's emotions.

[0961] Step 3: Decide how to provide information

[0962] The generative AI model on the server combines the analyzed environmental information and emotional information to determine how to provide information to the user. In this process, it is possible to adapt the information delivery method to the user's emotional state. The inputs are environmental information and the user's emotional state, and the output is an instruction for the adapted information delivery method. The server sends the information delivery method to the terminal based on this instruction.

[0963] Step 4: Communicate information

[0964] The device provides information to the user through voice messages and haptic feedback according to the instructions received from the server. In this step, the input is an instruction from the server, and the output is a transmission of information in a form that the user can understand. Through this process, the device provides the user with a voice message such as "There is a pedestrian crossing ahead" or haptic feedback indicating that someone is nearby.

[0965] (Application example 1)

[0966] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0967] In modern society, visually impaired people face many difficulties in moving around and in their daily lives. In particular, when they go out, it is difficult for them to accurately grasp their surroundings, which is an obstacle to safe movement and participation in society. In addition, because the methods of transmitting information are limited, there is a lack of means to reduce the anxiety and stress felt by visually impaired people. Under these circumstances, there is a demand for the development of support methods that enable visually impaired people to participate in society more safely and with confidence.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0969] In this invention, the server includes a means for analyzing environmental information using a generating AI and providing the user with a voice message or haptic feedback in place of visual information based on the analysis result, a means for accessing the AI ​​via a smart device or a headset and acquiring environmental data in real time, a means for recognizing surrounding objects and people, their movements and shapes using a camera or a sensor, and a means having an emotion engine for predicting the user's emotional state and adjusting the method of providing information based on that information. This enables visually impaired people to understand the surrounding environment more accurately and realize safe movement and social participation. Furthermore, by adopting an information transmission method according to the user's emotional state, it is possible to reduce the anxiety and stress felt by the user and support more comfortable social participation.

[0970] "Generative AI" is a system that uses artificial intelligence technology to analyze environmental data and provide information to users based on the results of that analysis.

[0971] A "voice message" is a means of verbal communication, a message that conveys information to the user by voice.

[0972] "Haptic feedback" is a means of conveying information through the sense of touch, a method of providing information from a device to the user through physical stimulation.

[0973] A "smart device" is a portable electronic device that has the ability to connect to the Internet and integrates multiple smart functions.

[0974] A "headset" is a device worn on the head that allows for voice communication and audio output, and in this context is used to receive audio messages and haptic feedback.

[0975] "Cameras and sensors" are devices that capture the surrounding environment and objects as images and data, and this information is used by the generative AI to analyze it.

[0976] An "emotion engine" is a technology or system that infers a user's emotional state from their tone of voice, facial expressions, behavior, etc., and adjusts the way information is provided based on that information.

[0977] The embodiment of this invention is a system for visually impaired people to understand the surrounding environment more safely and effectively when going out or in daily life. The system includes generative AI, smart devices and headsets, cameras and sensors, and an emotion engine.

[0978] The server is equipped with a generative AI that analyzes the environmental data sent from the cameras and sensors. The analyzed data includes information such as the position, shape, and movement of objects, and this information is conveyed to the user through voice messages and haptic feedback. The smart device or headset acts as an interface for the user to receive this information and transmits the environmental data to the generative AI in real time.

[0979] The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and behavior, and provides that information to the generation AI. The generation AI then adjusts the way it provides information based on this emotional information. For example, if the user is feeling anxious or scared, it will provide the necessary information in a calmer tone of voice.

[0980] To realize this system, cameras, sensors, and an emotion engine must be built into the smart device or headset, and algorithms must be developed on the server side for the generative AI to process this data and generate appropriate feedback.

[0981] For example, when a user is approaching an intersection, the camera and sensors capture the user's position and the movement of surrounding vehicles, and the generation AI uses this information to generate a voice message to the user through the headset saying, "A car is approaching from the right. Please cross when it is safe." Also, if the emotion engine detects that the user is nervous, the generation AI adjusts the way the information is conveyed to be more gentle.

[0982] This enables visually impaired people to understand their surroundings more safely and effectively, achieving greater independence in their daily lives and social participation.

[0983] The flow of the identification process will be described with reference to FIG.

[0984] Step 1: Collect environmental data

[0985] The camera and sensors installed on the device collect data about the user's surroundings, including the position, shape, and movement of objects, and the collected data is sent to a server in real time.

[0986] Step 2: Inferring emotional state

[0987] The device's on-board emotion engine infers the user's emotional state from their tone of voice, facial expressions, and behavior, and this inferred emotional state is sent to the server to tailor the way information is presented.

[0988] Step 3: Analyze data and generate information

[0989] The generative AI installed on the server analyzes the environmental data and emotional state data collected and sent in steps 1 and 2. Based on this analysis, useful information for the user is generated, such as how to cross an intersection safely or information about surrounding objects.

[0990] Step 4: Communicate information

[0991] The generated information is sent from the server to the device, which then conveys it to the user in the form of voice messages and haptic feedback. The method of conveying the information is adjusted based on the user's emotional state. For example, if the user is feeling anxious, the information is conveyed in a calmer tone of voice.

[0992] Step 5: Receiving and responding to feedback

[0993] The user takes action based on the information provided, while the user's reactions and additional emotional state are fed back to the server via the device, and this feedback is used to improve the accuracy of the system.

[0994] Through this process, users will gain a deeper understanding of their surroundings, allowing them to safely and effectively participate in their daily lives and society. The system will also be more responsive to the user's emotional state, providing more personalized information.

[0995] Example 2

[0996] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0997] One of the main challenges faced by visually impaired people is the difficulty of safely and effectively perceiving the surrounding environment when going out. This is due to various factors, such as the presence of obstacles, people's movements, and changing traffic conditions. Furthermore, if the way information is presented is not adapted to the user's emotional state and individual needs, it can cause anxiety and stress. Conventional assistive technologies cannot provide sufficient solutions to these challenges.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[0999] In this invention, the server includes a means for providing information to substitute for the field of view using a generating AI, a means for acquiring environmental information in real time and recognizing surrounding objects and people, and a means for analyzing the user's biometric information and inferring the user's emotional state. This allows visually impaired people to safely recognize their surroundings and receive voice messages and haptic feedback to support their movement. In addition, the method of providing information is adjusted according to the user's emotional state, reducing the user's stress and allowing them to go out more comfortably.

[1000] "Generative AI" is a type of artificial intelligence, a technology that generates new information and data from input data.

[1001] "Alternatives to vision" are information based on senses other than vision (hearing, touch, etc.) that are used by people who cannot rely on vision to understand their surroundings and situations.

[1002] A "smart device" is an electronic device that can connect to the Internet and other devices and has multiple functions.

[1003] A "headset" is a device, typically worn on the head, that provides both audio input (microphone) and output (headphones).

[1004] "Means for acquiring environmental information in real time" refers to techniques and processes for instantly collecting and analyzing information from the user's surrounding environment.

[1005] "Means of recognizing surrounding objects and people" refers to technologies that use cameras, sensors, voice recognition technology, etc. to identify and understand the presence, location, and actions of objects and people.

[1006] "Voice, audio messages, and haptic feedback" are means of communicating information to the user through non-visual means such as sound or vibration.

[1007] "User's biological information" is data that indicates the user's physical condition, such as heart rate, electrical conductivity of the skin, etc.

[1008] "Means for inferring emotional state" is a technology that analyzes and understands a user's emotional and psychological state at any given time from their biometric information and behavioral patterns.

[1009] "Means for adjusting the method of information provision" are techniques or processes that optimize the method of information delivery (e.g., message content and presentation) according to the user's condition and needs.

[1010] The embodiment of the present invention is a system for supporting visually impaired people to understand the surrounding environment and move safely when they go out. This system is configured by utilizing a generative AI model, a smart device, a sensor, a headset, and a biometric information measuring device.

[1011] The generative AI model deployed on the server analyzes the location and movement of surrounding objects and people, as well as the user's biometric information, based on data sent from smart devices and sensors. This analysis involves advanced data processing and data calculation. The hardware used includes high-resolution cameras, microphones, heart rate sensors, and skin conductivity sensors, while the software includes image recognition algorithms, voice recognition algorithms, and emotional state analysis algorithms.

[1012] The terminal refers to a smart device (smartphone or tablet) that collects information about the surroundings using a built-in camera and microphone and transmits it to the server. The terminal also receives responses from the server and conveys them to the user as voice messages or haptic feedback. The headset is used to convey voice messages directly to the user.

[1013] As a concrete example, consider a situation where a user is walking down a street. The device's camera captures obstacles ahead, and the generative AI model generates a voice message based on the prompt sentence "There is a chair 3 meters ahead," which is conveyed to the user through the headset. At the same time, the device uses a microphone to capture surrounding sounds, analyzes people's movements, and provides haptic feedback through a haptic device worn on the wrist. Based on the emotional state indicated by the user's biometric information, if it determines that "the user's heart rate is high and therefore tense," the generative AI model processes the prompt sentence to provide relaxing music.

[1014] This format allows visually impaired people to obtain a wealth of information about their surroundings, and the provision of information that takes into account their emotional state increases their sense of security when out and about.

[1015] The flow of the identification process will be described with reference to FIG.

[1016] Step 1: Collect environmental data

[1017] The device uses its built-in camera, microphone, and biometric sensors to collect environmental data and biometric information about the user. The inputs for this step are the sights and sounds of the surroundings, the user's heart rate, and the electrical conductivity of the skin, and the output is a digital representation of the collected environmental data and biometric information. The device converts this data into digital information for processing.

[1018] Step 2: Send and analyze data

[1019] The collected data is transmitted to a server via wireless communication. The generative AI model on the server analyzes the transmitted environmental data and biometric information to identify the positions, movements, and emotional state of surrounding objects and people. The inputs of this step are environmental data and biometric information, and the outputs are the identification information of objects and people and the emotional state of the user. The server analyzes the data and infers the identification information and the emotional state.

[1020] Step 3: Coordinate input and generate feedback

[1021] The generative AI model adjusts the content and format of the information provided to the user based on the analysis results. If the user is tense, it selects music to relax them, provides the location of obstacles with voice messages, and conveys the movements of people around them with haptic feedback. The input for this step is the identification information of objects and people and the user's emotional state, and the output is the information provided, such as voice messages, haptic feedback, and music. The server generates appropriate feedback based on the analysis results and sends it to the terminal.

[1022] Step 4: Communicate feedback

[1023] The terminal receives the feedback sent from the server and transmits voice messages to the user through the headset, haptic feedback through the haptic device, and music through the terminal's speaker or headset. The input of this step is the feedback from the server, and the output is the provision of information in a form that the user can recognize. The terminal transmits the received feedback to the user through an appropriate output device.

[1024] Through these steps, visually impaired people can perceive their surroundings more safely and efficiently and receive customized support based on their emotional state.

[1025] (Application example 2)

[1026] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[1027] Currently, when users, including visually impaired people, use autonomous vehicles, it is difficult for them to accurately recognize the surrounding environment and situation and travel safely and comfortably. In particular, it is difficult for them to understand the status of surrounding vehicles, obstacles, and traffic lights, and information is not provided or the environment is not adjusted according to the user's emotional state, which can increase anxiety and stress while traveling.

[1028] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[1029] In this invention, the server includes a means for recognizing dynamic elements of the environment in real time and providing information, a means for accessing AI via an in-vehicle system or a headset, a means for analyzing input data from sensors and cameras to obtain environmental information around the vehicle, a means for recognizing surrounding vehicles, obstacles, and traffic light conditions, and a means for adjusting an emotional state based on voice, voice messages, haptic feedback, and biometric information. This allows the user to accurately understand the surrounding environment and situation and travel safely and comfortably. In addition, by providing information and adjusting the environment according to the user's emotional state, anxiety and stress during travel can be reduced, resulting in a more comfortable travel experience.

[1030] "Generative AI" is a type of artificial intelligence technology, a system that analyzes complex data and generates new information and inferences based on it.

[1031] "Means of recognizing dynamic elements of the environment in real time and providing information" refers to the function of analyzing input data from sensors, cameras, etc., instantly detecting changing conditions in the surroundings (vehicle movement, presence of obstacles, changes in traffic lights, etc.), and conveying that information to the user.

[1032] "Means of accessing AI via in-vehicle systems or headsets" refers to interfaces that communicate with the generative AI and exchange information through embedded systems in autonomous vehicles or headsets worn by the user.

[1033] "Means of analyzing input data from sensors and cameras" refers to the function of processing and analyzing in real time data collected by sensors and cameras installed to capture environmental data around the vehicle, and grasping environmental information.

[1034] "Means of recognizing surrounding vehicles, obstacles, and traffic signal conditions" refers to technologies and systems that enable autonomous vehicles to accurately understand the situation of surrounding objects, other vehicles, traffic signals, etc., and make safe driving decisions based on this information.

[1035] "Means for adjusting emotional state based on voice or voice messages, haptic feedback, and biometric information" refers to a function that provides information to the user through voice or haptic feedback, and further analyzes the user's biometric information to infer their emotional state, and provides information or adjusts the environment accordingly.

[1036] The embodiment of the present invention is a support system for users, including visually impaired people, to move safely and comfortably in an autonomous vehicle. This system is composed of a combination of on-board sensors and cameras, a generative AI model, a voice synthesis engine, a haptic feedback device, a biometric sensor, etc.

[1037] The server receives input data from the vehicle's sensors and cameras, analyzes it, and recognizes the vehicle's surroundings. This environmental information includes other vehicles, obstacles, traffic lights, etc. Using a generative AI model, the server generates voice messages and haptic feedback for the user based on this information. It also analyzes data collected from the user's biometric sensors (e.g., heart rate and skin electrical conductivity) to infer the user's emotional state. Depending on the user's emotional state, the method of providing information and the environment are adjusted to improve the user's psychological comfort.

[1038] For example, if the server detects another vehicle approaching from the right side of the vehicle, it will use the generative AI model to generate a voice message saying, "A vehicle is approaching from the right. Please be careful," and convey this to the user through the voice synthesis engine. At the same time, if the user feels tense, it will send a prompt to the generative AI model saying, "Please provide relaxing music," and play music to relax the user.

[1039] In this form, hardware and software such as cameras and sensors installed in the vehicle, haptic feedback devices, biometric sensors such as heart rate sensors and skin conductivity sensors, generative AI models, and voice synthesis engines play an important role. By combining these appropriately, it is possible to provide all users, including the visually impaired, with support to better understand the surrounding environment and move around safely and comfortably.

[1040] The flow of the identification process will be described with reference to FIG.

[1041] Step 1: Collect environmental data

[1042] The server collects surrounding environment data sent from the vehicle's sensors and cameras. This data includes the location of obstacles, the movement of nearby vehicles, the status of traffic lights, etc. Based on this input data, the server processes and calculates the data to understand the surrounding situation.

[1043] Step 2: Analyze environmental data

[1044] The collected environmental data is analyzed by a generative AI model, which recognizes the location of obstacles around the vehicle, the behavior of nearby vehicles, the status of traffic lights, etc. The results of the analysis are used as the basis for providing information to the user.

[1045] Step 3: Collect and analyze user biometric information

[1046] The server collects data from the user's biometric sensors, such as heart rate and skin conductivity, which are then analyzed and used to infer the user's emotional state, which in turn helps determine how to provide information and adjust the environment.

[1047] Step 4: Decide how to provide information

[1048] The server takes into account the analyzed environmental data and the user's emotional state to determine the optimal way to provide information. For example, if the user feels tense, it will decide to provide relaxing music. A generative AI model is used in this decision-making process.

[1049] Step 5: Provide information

[1050] Based on the information provision method determined, the server will provide voice messages, haptic feedback, appropriate music, etc. These information provision will be customized to provide the best experience for the user. For example, a voice message may be conveyed to the user saying, "A vehicle is approaching from the right. Please be careful."

[1051] This program processing allows users of self-driving vehicles to better understand their surroundings, enabling them to travel safely and comfortably, and by providing information and adjusting the environment according to the user's emotional state, it can reduce anxiety and stress while traveling.

[1052] Example 3

[1053] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1054] The goal is to reduce the barriers to information access that visually impaired people face in their daily lives and social activities. Specifically, these challenges include understanding the reactions and atmosphere of people around them at meetings and events, accessing the contents of books and materials they want to read, and providing haptic feedback that adapts to their own emotional state. These challenges are the main factors that limit the improved social participation and independent living of visually impaired people.

[1055] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[1056] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the atmosphere of the environment and people's reactions from the text data using a generative AI model, a means for converting the text data into voice information and conveying it to the user, a means for monitoring the user's emotional state, a means for adjusting the intensity and pattern of haptic feedback based on the user's emotional state, and a means for collecting voices of meetings and events in real time. This allows visually impaired people to have a deeper understanding of their surroundings, actively participate in social activities, easily access the content they want to read as voice information, and receive appropriate haptic feedback according to their emotional state.

[1057] "Means for converting voice data into text data" refers to a process or device that converts collected voice information into text data. This conversion is accomplished using voice recognition technology.

[1058] "Means of analyzing environmental atmosphere and people's reactions from text data using generative AI models" refers to techniques and methods for analyzing text information and using that information to understand the surrounding situation, people's emotions, reactions, etc. Natural language processing technology is used for this analysis.

[1059] "Means of converting text data into voice information and conveying it to the user" refers to a process or device that reproduces the analyzed character information as voice and conveys it to the user. This conversion uses text-to-voice technology.

[1060] "Means for monitoring the user's emotional state" refers to technologies or methods for identifying and analyzing the user's emotional state from the user's tone of voice, facial expressions, body movements, etc. Voice analysis technology and sensor technology may be used for this monitoring.

[1061] "Means for adjusting the intensity or pattern of haptic feedback based on the user's emotional state" refers to techniques or methods for varying the type or degree of feedback provided through a haptic device depending on the user's current emotional state, using a feedback control algorithm based on the results of emotion analysis.

[1062] "Means of capturing audio in real time at meetings and events" refers to technologies and devices that directly capture live audio information in situ and record it as digital data, using microphones and audio recording systems.

[1063] In the embodiment of the present invention, a server, a terminal (smart device), and a user (visually impaired person) are at the center. The server is equipped with speech recognition technology, natural language processing technology, text-to-speech technology, emotion analysis technology, and haptic feedback control technology. The terminal is equipped with voice collection function, emotion state monitoring function, and haptic feedback providing function. Specific hardware and software include Google Cloud Speech-to-Text (speech recognition), OpenAI's GPT (text analysis and generation), Amazon Polly (text to speech conversion), and Tesseract OCR (character recognition).

[1064] The server converts the voice data received from the device into text data using Google Cloud Speech-to-Text. Next, this text data is analyzed using OpenAI's GPT to extract information such as the atmosphere of the meeting or event and people's reactions. The analysis results are converted into text data and into voice information using Amazon Polly. This voice information is transmitted to the user via the device.

[1065] The device uses a microphone to collect audio from meetings and events in real time and transmits it to the server. The device also monitors the user's tone of voice and body movements, and transmits data for analyzing the user's emotional state to the server. The server analyzes this data and transmits instructions to the device to adjust the intensity and pattern of haptic feedback according to the user's emotional state. Based on these instructions, the device provides appropriate feedback to the user through the haptic device.

[1066] A specific example is a scenario in which a server performs processing based on a prompt sentence such as "Analyze the atmosphere of the meeting and provide an overview of the atmosphere by voice," and a terminal conveys to a user voice information such as "The speaker just told a joke, and the participants are laughing." Another example includes a case in which a user scans a page of a book he or she wants to read with a camera on the terminal, and converts the text into voice information and provides it based on a prompt such as "Read the text on this page and read it aloud."

[1067] This format allows visually impaired people to more easily understand their surroundings and the content of meetings, access materials they want to read by voice, and receive richer information through haptic feedback that adapts to their emotional state.

[1068] The flow of the identification process will be described with reference to FIG.

[1069] Step 1:

[1070] The terminal uses a microphone to collect audio from a meeting or event. This audio data is sent to the server. The input is raw audio data, and the output is digitized audio data. This step involves the collection and digitization of audio data.

[1071] Step 2:

[1072] The server converts the received voice data into text data using Google Cloud Speech-to-Text. The input is digitized voice data, and the output is text data. In this process, data processing is performed to convert voice information into text information using voice recognition technology.

[1073] Step 3:

[1074] The server analyzes the converted text data using OpenAI's GPT to extract information such as the atmosphere of the meeting or event and people's reactions. The input is text data, and the output is the text data of the analysis results (a summary of the atmosphere and reactions). In this step, data calculations are performed using natural language processing technology to extract useful information from the text data.

[1075] Step 4:

[1076] The server converts the analysis results into voice information using Amazon Polly as text and sends this voice information to the terminal. The input is the text data of the analysis results, and the output is voice information. In this process, text-to-voice conversion is performed, and information is provided to the user in an easy-to-understand form.

[1077] Step 5:

[1078] The terminal transmits the voice information received from the server to the user. In this step, the voice information is played through the terminal's speaker. The input is the voice information, and the output is a voice message that the user can hear.

[1079] Step 6:

[1080] The terminal monitors the user's tone of voice and body movements and transmits the data to the server for analyzing the user's emotional state. The input is the user's voice and body movements data, and the output is the analysis result of the emotional state. In this step, the data is calculated using emotion analysis techniques to identify the user's emotional state.

[1081] Step 7:

[1082] The server sends instructions to the terminal to adjust the intensity and pattern of the haptic feedback according to the user's emotional state. The input is the analysis result of the emotional state, and the output is an instruction for haptic feedback. In this step, the haptic feedback is customized based on the user's emotional state.

[1083] Step 8:

[1084] The terminal provides appropriate feedback to the user through the haptic device based on the instruction from the server. In this step, the user receives a response according to his / her emotional state through haptic feedback. The input is an instruction for haptic feedback, and the output is the haptic feedback transmitted to the user's senses.

[1085] (Application example 3)

[1086] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1087] In modern society, visually impaired people and security workers face challenges such as understanding the environment, moving safely, and instantly recognizing abnormal behavior and dangerous situations. In particular, when visually impaired people participate in society and security workers need to respond quickly to emergency situations, there are limited means to effectively grasp the surrounding environment and people's emotional states. Another issue is the lack of support for leading a safer and more comfortable social life by providing appropriate haptic feedback according to the user's emotional state.

[1088] The specific processing by the specific processing unit 290 of the data processing device 12 is realized by the following means.

[1089] In this invention, the server includes a means for providing information to substitute for vision using a generating AI, a means for analyzing the emotional state of people in the vicinity in real time and providing appropriate haptic feedback when abnormal behavior or situations are detected, and a means for adjusting the strength and pattern of the haptic feedback according to the emotional state of the user. This enables visually impaired people to engage in social activities more richly and independently, and enables security personnel to effectively grasp the surrounding environment and people's emotional states and respond quickly even in emergency situations.

[1090] "Generative AI" is a form of artificial intelligence that is a system that generates new information and responses independently from training data and performs specific tasks automatically.

[1091] A "means for providing information in place of vision" is a system or method that conveys necessary information to the user through audio or tactile feedback in place of visual information.

[1092] "Means of accessing AI via smart devices or headsets" refers to methods or systems for connecting to and using AI functions and services using devices such as smartphones, tablets, and head-mounted displays.

[1093] "Means of acquiring environmental information in real time" refers to systems and technologies that can instantly capture, analyze, and process the surrounding situation and changes.

[1094] "Means of recognizing surrounding objects and people" refers to systems or algorithms that identify objects and people from image and audio data obtained through cameras, sensors, etc., and analyze that information.

[1095] "Means of conveying information in the form of voice, voice message, haptic feedback, etc." refers to a method of conveying information to a user using voice synthesis technology or vibration patterns.

[1096] "Means for analyzing the emotional state of ...

Claims

1. A means of analyzing surrounding environmental information using generative AI, A means of accessing the AI ​​via an information and communications technology device; and a means for acquiring environmental data in real time; A means for identifying features of objects and people in the environment; A system that includes means for transmitting information through voice and touch.

2. The acquiring means acquires surrounding environmental data using a camera or a sensor; The accessing means accesses the generating AI via an information and communication technology device; The analyzing means and the identifying means use the generating AI to analyze the acquired environmental data to identify the position, shape, or movement of a surrounding object or person; 2. The system according to claim 1, wherein the means for transmitting information provides the user with a result of identifying the position, shape, or movement of a surrounding object or person by a voice message or haptic feedback.

3. The acquiring means acquires environmental data of the surroundings of the conference or event in which the user is participating, and acquires documents or materials used in the conference or event, using a camera or a sensor; The accessing means accesses the generating AI via an information and communication technology device; The analyzing means and the identifying means use the generation AI to analyze the acquired environmental data to analyze the reactions or atmosphere at the meeting or event, and generate a response that reads out the text information of the document or material by voice using the generation AI; 2. The system according to claim 1, wherein the means for transmitting information provides the user with an analysis of the reaction or atmosphere at the meeting or event by a voice message or haptic feedback, and plays back the response.

4. The acquiring means acquires environmental data including product information indicating a position of a product in the store where the user is located, the analyzing means and the identifying means use the generating AI to analyze the acquired environmental data, thereby generating a response corresponding to a question about a product input by a user from a combination of the user's location information and the product information; 2. The system of claim 1, wherein said means for communicating provides said generated response to a user by a voice message.

5. means for inferring an emotional state of the user using an emotion engine; The system of claim 1 , wherein the means for transmitting information uses the generation AI to adjust the voice message to be provided to the user in accordance with the emotional state of the user, and provides the voice message to the user.

6. means for inferring an emotional state of the user using an emotion engine; The system according to claim 1 , wherein the means for transmitting information uses the generating AI to generate music corresponding to the emotional state of the user and provides the music to the user.

7. means for inferring an emotional state of the user using an emotion engine; The system according to claim 1 , wherein the means for transmitting information provides the haptic feedback to the user by adjusting an intensity or a pattern of the haptic feedback provided to the user depending on an emotional state of the user.

8. means for inferring an emotional state of people around the user using an emotion engine; The system according to claim 1 , wherein the means for transmitting information provides the haptic feedback to the user by adjusting an intensity or pattern of the haptic feedback provided to the user depending on the emotional state of the people around the user.

Images