System

Abstract

To provide a system for analyzing the line-of-sight information and environmental data of a user in real time, and for providing individually optimal information.SOLUTION: The system includes an eye tracker for tracking the user's eye in real time, an environmental information collector for collecting environmental information, a generative AI model for analyzing the user's eye information and environmental information and generating optimal information for the user, and an information displayer for displaying the generated information in the user's field of view.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the provision of information tailored to individual needs and situations is becoming increasingly important. However, conventional smart glasses and assistant services have limited ability to adapt to the different requirements of each user, and are unable to provide advanced support, such as providing real-time, appropriate guidance and warnings to help visually impaired people walk safely, or quickly providing important information to doctors during surgery. Therefore, there is a need for systems that can analyze users' gaze information and environmental data in real time and provide individually optimized information to support their daily lives and work efficiently and safely. [Means for solving the problem]

[0005] To solve this problem, the present invention provides a system including a gaze tracking means for tracking a user's gaze in real time, an environmental data collection means for acquiring surrounding environmental data, a generative AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user, and an information display means for displaying the generated information in the user's field of view.The system further includes a learning model update means for collecting user feedback and improving the performance of the generative AI model, a user recognition means for authenticating users and personalizing information based on their settings and profiles, a highlighting means for detecting important information and notifying the user by highlighting it, and a voice notification means for providing guidance and warnings to the user using voice in real time, thereby achieving highly flexible and accurate information provision and support in a variety of situations.

[0006] An "eye tracking means" is a device that detects a user's gaze in real time and identifies its position and direction.

[0007] "Environmental data collection means" refers to devices such as sensors and cameras that acquire information about the user's surroundings (such as location, illuminance, sound, and temperature).

[0008] "Generative AI model means" refers to an artificial intelligence algorithm or system that analyzes collected gaze information and environmental data to generate information that is optimal for the user.

[0009] The "information display means" refers to a display or projection device for displaying the generated information in the user's field of vision.

[0010] The "learning model update means" is a data processing system used to analyze user feedback and improve the performance of the generative AI model.

[0011] "User Recognition" means an identification system for authenticating users and personalizing information based on their individual profiles and preferences.

[0012] "Highlighting means" refers to a technique (such as a flash effect or color change) for detecting important information and displaying that information in an emphasized manner.

[0013] The "voice notification means" refers to a voice synthesizer or speaker that provides guidance or warnings to the user using voice in real time. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that analyzes a user's line of sight and environmental data in real time and provides individually optimized information. The program processing of this system will be explained below in natural language.

[0036] Program processing flow

[0037] Initialization and environment data collection

[0038] Device: When the smart glasses are turned on, the internal software starts up and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect environmental data (location, light intensity, sound, etc.) in real time.

[0039] User Awareness and Customization

[0040] User: Wears smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[0041] Terminal: Sends authentication data to the server.

[0042] Server: Retrieves user profile data from the database, analyzes past usage history and preferences, and predicts necessary information. This data is then sent back to the device.

[0043] Terminal: Receives and applies profile settings sent from the server.

[0044] Eye tracking and information provision

[0045] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[0046] Terminal: Sends gaze information and environmental data to the server.

[0047] Server: Analyzes the received gaze information and environmental data, and uses a generative AI to create information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object). This information is then sent to the device.

[0048] Terminal: Displays the received information in the field of view and provides it to the user.

[0049] Specific examples

[0050] Example 1: Shopping support

[0051] User: Looking for a product at a shopping mall in town.

[0052] Device: Detects when the user looks at a specific product.

[0053] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[0054] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[0055] Example 2: Supporting the visually impaired

[0056] User: A visually impaired user is walking around town.

[0057] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[0058] Server: Generates optimal travel routes and points of interest from environmental data.

[0059] Device: Provides directions and alerts to users through a voice assistant.

[0060] Real-time feedback and learning

[0061] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0062] Terminal: Sends feedback data to the server.

[0063] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[0064] Highlighting important information

[0065] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[0066] Device: Notify the user using sound, vibration, and visual enhancements.

[0067] These functions enable smart glasses to provide users with personalized information in a variety of situations, supporting their daily lives and work efficiently and safely. The system of the present invention serves as a flexible and advanced information providing tool that meets the diverse needs of modern society.

[0068] The processing flow will be explained below.

[0069] Specific processing flow of the program

[0070] Initialization and environment data collection

[0071] Step 1:

[0072] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[0073] Step 2:

[0074] Device: The camera tracks the user's gaze, and built-in sensors capture environmental data such as location, light, and sound in real time.

[0075] User Awareness and Customization

[0076] Step 3:

[0077] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[0078] Step 4:

[0079] Terminal: Sends the user's authentication data to the server.

[0080] Step 5:

[0081] Server: Retrieves user profile data from a database and generates configuration information based on past usage history and preferences.

[0082] Step 6:

[0083] Server: Returns the generated profile settings to the device.

[0084] Step 7:

[0085] Terminal: Receives and applies profile settings sent from the server.

[0086] Eye tracking and information provision

[0087] Step 8:

[0088] Device: The camera tracks the user's gaze in real time and identifies the object of their attention.

[0089] Step 9:

[0090] Terminal: Sends gaze information and environmental data to the server.

[0091] Step 10:

[0092] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[0093] Step 11:

[0094] Server: Sends the generated information to the terminal.

[0095] Step 12:

[0096] Terminal: Displays the received information in the field of view and provides it to the user.

[0097] Real-time feedback and learning

[0098] Step 13:

[0099] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0100] Step 14:

[0101] Terminal: Sends user feedback data to the server.

[0102] Step 15:

[0103] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[0104] Highlighting important information

[0105] Step 16:

[0106] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[0107] Step 17:

[0108] Device: Notify the user using sound, vibration, and visual enhancements.

[0109] Specific examples

[0110] Example 1: Shopping support

[0111] Step 1:

[0112] User: Looking for a product at a shopping mall in town.

[0113] Step 8:

[0114] Device: Detects when the user looks at a specific product.

[0115] Step 9:

[0116] Terminal: Sends gaze information and environmental data to the server.

[0117] Step 10:

[0118] Server: Generates detailed product information and reviews, and also generates recommended items based on the user's preferences.

[0119] Step 12:

[0120] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[0121] Example 2: Supporting the visually impaired

[0122] Step 1:

[0123] User: A visually impaired user is walking around town.

[0124] Step 2:

[0125] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[0126] Step 9:

[0127] Terminal: Sends environmental data to the server.

[0128] Step 10:

[0129] Server: Generates optimal travel routes and points of interest from environmental data.

[0130] Step 12:

[0131] Device: Provides directions and alerts to users through a voice assistant.

[0132] Example 1

[0133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Conventional eye-tracking and environmental data analysis technologies have faced challenges in providing information to users effectively and quickly. Furthermore, personalization based on user profiles was insufficient, making it difficult to provide information tailored to individual users' needs and preferences. Furthermore, there was a lack of a means to quickly incorporate user feedback and improve the accuracy of the system. The highlighting function for important information and obstacles was also incomplete, failing to adequately ensure user safety and convenience.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0136] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, and user recognition means for authenticating users and customizing information based on each user's settings and profile. This enables the server to quickly and accurately provide the information the user desires. The server also includes learning model update means for collecting user feedback and improving the performance of the information generation means, and highlighting means for detecting obstacles and important information and highlighting them. This improves the accuracy of the system and enhances user safety and convenience by highlighting important information.

[0137] An "eye tracking means" is a technology or device that detects a user's gaze in real time and tracks its position and direction.

[0138] "Environmental data collection means" refers to sensors or devices for acquiring surrounding environmental information (position information, illuminance, sound, etc.).

[0139] "Information generation means" refers to a system or software that analyzes the user's line of sight and environmental data, and generates information that is optimal for the user.

[0140] The "information display means" refers to a display or projection device for displaying the generated information in the user's field of vision in real time.

[0141] "User recognition" means technologies or systems that authenticate users and customize information based on their individual preferences and profiles.

[0142] A "learning model update means" is an algorithm or system that collects user feedback and uses it to improve the performance of the information generation means.

[0143] The "highlighting means" is a function for detecting obstacles and important information and highlighting them for the user.

[0144] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time and provides individually optimized information. The system includes gaze tracking means, environmental data collection means, information generation means, information display means, user recognition means, learning model update means, and highlighting means.

[0145] Initialization and environment data collection

[0146] Device: When the smart glasses are turned on, the internal software starts up. The camera and sensors are initialized and various data collection begins. Specifically, the eye-tracking camera tracks the user's gaze, location information is acquired by the GPS sensor, ambient light is measured by the light sensor, and audio data is collected by the built-in microphone.

[0147] User Awareness and Customization

[0148] User: Wears smart glasses and logs in to their profile using facial recognition, voice recognition, or password entry. For example, when performing facial recognition, the built-in camera recognizes the user's face, and dedicated software analyzes the facial data.

[0149] Terminal: The authentication data is sent to the server. The user recognition means then retrieves and analyzes profile data from the database on the server. Based on past usage history and preferences, the necessary information is predicted and sent back to the terminal.

[0150] Device: Receives and applies profile settings sent by the server, so that information is provided according to the user's preferences.

[0151] Eye tracking and information provision

[0152] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[0153] Terminal: Sends gaze information and environmental data to the server. For example, it detects when a user looks at a specific product and sends that information to the server.

[0154] Server: Analyzes the received gaze information and environmental data. Using a generative AI model, it generates information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object) and sends it to the device.

[0155] Device: Displays the received information in the user's field of view, for example, displaying detailed information and reviews about a specific product.

[0156] Specific examples

[0157] Shopping Support

[0158] User: Looking for a product at a shopping mall in town.

[0159] Device: Detects when a user looks at a specific product.

[0160] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[0161] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[0162] Support for the visually impaired

[0163] User: A visually impaired user is walking around town.

[0164] Device: The device uses a camera to recognize the user's surroundings and collect information about obstacles and directions.

[0165] Server: Generates optimal travel routes and points of interest from environmental data.

[0166] Device: Provides users with directions and alerts through a voice assistant.

[0167] Real-time feedback and learning

[0168] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0169] Terminal: Sends feedback data to the server.

[0170] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[0171] Highlighting important information

[0172] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[0173] Device: Notify the user using sound, vibration, and visual enhancements.

[0174] This allows smart glasses to provide users with personalized information tailored to their needs in a variety of situations, supporting their daily lives and work efficiently and safely. This system serves as a flexible and advanced means of providing information, meeting the diverse needs of modern society.

[0175] Example prompt: "Show me more information about the product the user is interested in."

[0176] "Please provide directions to the nearest hospital from my current location."

[0177] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0178] Step 1: Initialization and Environmental Data Collection

[0179] Terminal: When the smart glasses are powered on, the internal software starts up, performs initial configuration, and initializes the camera and sensors.

[0180] Specific operation: The firmware starts up, the camera starts tracking the user's gaze, the light sensor detects the ambient brightness, and the GPS module obtains location information.

[0181] Input: Power ON signal

[0182] Output: Initialization completion signal, gaze data, location information, illuminance data, surrounding audio data

[0183] Step 2: User awareness and customization

[0184] User: Puts on smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[0185] How it works: The built-in camera recognizes your face and the facial recognition software analyzes the data, or you speak a specific phrase for voice recognition and the voice recognition software analyzes it.

[0186] Input: Facial image data or audio data

[0187] Output: Authenticated data

[0188] Step 3: Send authentication data

[0189] Terminal: The user authentication means sends authentication data to the server.

[0190] Specific operation: The communication module in the smart glasses sends authentication data to the server endpoint.

[0191] Input: Authenticated Data

[0192] Output: Signal that authentication data has been sent to the server

[0193] Step 4: Analyze the profile data

[0194] Server: The server receives the authentication data and retrieves the user's profile data from the database. It analyzes the required information based on past usage history and preferences.

[0195] What it does: Executes database queries to retrieve the required profile data, analyzes the data using machine learning models, and generates predictions.

[0196] Input: Authentication data

[0197] Output: Profile data, analysis results

[0198] Step 5: Applying Profile Settings

[0199] Terminal: Receives and applies profile settings sent from the server.

[0200] Specific operation: The received profile settings are reflected in the display settings and interface settings of the smart glasses.

[0201] Input: Profile data, analysis results

[0202] Output: Apply completion signal, customized settings

[0203] Step 6: Eye tracking and data collection

[0204] Device: The camera tracks the user's gaze in real time and simultaneously collects environmental data.

[0205] How it works: Eye-tracking software monitors your eye movements and detects specific objects. Light sensors, GPS, and microphones collect data accordingly.

[0206] Input: gaze data, environmental data

[0207] Output: Real-time gaze information, environmental data

[0208] Step 7: Sending gaze and environment data

[0209] Terminal: Sends gaze information and environmental data to the server.

[0210] Specific operation: A communication module is used to integrate gaze data and environmental data and send them to the server.

[0211] Input: gaze information, environmental data

[0212] Output: Data transmission completion signal to the server

[0213] Step 8: Information Generation

[0214] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to generate information about the object the user is looking at.

[0215] Specific operation: Using gaze information and environmental data as input, the AI ​​model creates prompt sentences and generates related information.

[0216] Input: gaze information, environmental data

[0217] Output: Generated information, prompt

[0218] Step 9: Viewing Information

[0219] Terminal: Displays the received information in the field of view and provides it to the user.

[0220] Specific operation: The generated information is overlaid on the smart glasses display and presented to the user in real time.

[0221] Input: Generated information

[0222] Output: Information displayed in the field of view

[0223] (Application example 1)

[0224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] In traditional shopping experiences, users must use devices such as smartphones to check product details and reviews, which is time-consuming. Furthermore, simply viewing products visually often does not immediately provide the information users actually need. Furthermore, the information provided is not optimized for each user's individual preferences or history, which prevents sufficient user satisfaction. A system that can solve these problems and provide users with the most appropriate information in real time is needed.

[0226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0227] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating information optimal for the user, information display means for displaying the generated information in the user's field of view, and product information presentation means for providing detailed information about products the user is looking at in a physical store. This allows users to improve their shopping experience in physical stores, instantly obtain necessary information, and select products optimal for them.

[0228] An "eye-gaze tracker" is a device or software that detects and tracks a user's gaze in real time.

[0229] The "environmental data collection means" is a device or software for acquiring surrounding environmental data (position information, illuminance, sound, etc.).

[0230] The "generative AI model means" is an artificial intelligence-based data analysis system that analyzes the user's gaze information and environmental data, and generates information that is optimal for the user.

[0231] "Information display means" refers to a device or software that displays the generated information in the user's field of vision in real time.

[0232] A "product information presentation means" is a device or software that provides detailed information and reviews of products that a user looks at in a physical store.

[0233] The "learning model update means" is a data analysis and model update system that collects user feedback and improves the performance of the generative AI model.

[0234] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[0235] This invention relates to a system that provides optimal information in real time when a user is shopping for products in a physical store. This system is composed of an eye tracking means, an environmental data collection means, a generative AI model means, an information display means, and a product information presentation means.

[0236] System Configuration

[0237] 1. Eye-tracking methods:

[0238] Smart glasses are used as devices that detect and track a user's gaze in real time. They are equipped with a camera and internal software for tracking the user's gaze.

[0239] 2. Environmental data collection methods:

[0240] To obtain environmental data, the smart glasses are equipped with built-in sensors that collect location information, light intensity, audio data, and other information in real time.

[0241] 3. Generative AI model means:

[0242] An AI information analysis server located on the cloud is used to analyze the user's gaze information and environmental data and generate individually optimized information. This server analyzes and generates data using a generative AI model (e.g., OpenAI® GPT-4®, Google® BERT).

[0243] 4. Information display means:

[0244] The smart glasses display is used to display the generated information in real time in the user's field of view, including product details, reviews, and recommendations.

[0245] 5. Product information presentation means:

[0246] To provide detailed information about the product the user is looking at, the server refers to the database on the cloud to retrieve relevant information, which is then analyzed by a generative AI model, and the generated information is displayed on the smart glasses.

[0247] Specific operation example

[0248] 1. Eye tracking and information provision:

[0249] A user puts on smart glasses in a physical store and focuses their gaze on a specific product.

[0250] The smart glasses' eye-tracking camera detects the user's gaze and sends the data to a cloud server.

[0251] The server analyzes gaze information and environmental data to generate detailed product information and recommended items based on the user's preferences.

[0252] The generated information is displayed in real time on the smart glasses display.

[0253] 2. User Feedback and Learning:

[0254] Users provide feedback on the information provided, such as "helpful" or "not helpful."

[0255] Feedback data is sent to the server, and the generative AI model is trained and updated.

[0256] Hardware and software used

[0257] Cloud server: AWS (registered trademark), Google Cloud, Azure (registered trademark)

[0258] AI model generation: OpenAI GPT-4, Google BERT

[0259] Data analysis: TENSORFLOW(registered trademark), PyTorch

[0260] Eye tracking and sensors: Smart glasses (e.g., Google Glass, Vuzix Blade)

[0261] Prompt Sentence Examples

[0262] Write a Python program that uses user gaze data and environmental data to generate product information in real time and display it on the smart glasses screen. OpenAI GPT-4 is used for the generation AI, and data is analyzed and managed in the AWS cloud.

[0263] In this way, the system can improve the user's shopping experience and provide the necessary information in real time.

[0264] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0265] Step 1:

[0266] When a user puts on the smart glasses and turns them on, the smart glasses' internal software starts up and initializes the eye-tracking camera and environmental sensors. The input here is the smart glasses' startup signal, and the output is the initialization completion state. Specifically, the smart glasses' camera begins to track the user's gaze, and the sensors collect location information, illuminance, and audio data in real time.

[0267] Step 2:

[0268] Users log in to their profile using facial or voice recognition. The input is the user's authentication data, and the output is an authentication success or failure message. The device sends the authentication data to a cloud server, which retrieves the user's profile data from a database and analyzes their past usage history and preferences to predict what information they will need. This profile data is then sent back to the device.

[0269] Step 3:

[0270] When a user directs their gaze toward a product in a physical store, an eye-tracking camera detects the gaze information in real time. The input is gaze data, and the output is the identification information of the specific product. The device sends the gaze information and environmental data to a cloud server, which then analyzes the received data. Specifically, it determines that the gaze has been focused on a specific product.

[0271] Step 4:

[0272] The server analyzes gaze information and environmental data and uses a generative AI model to generate optimal information for the user (detailed product information, reviews, recommended items, etc.). The input is gaze information and environmental data, and the output is the generated product information. Data processing involves detecting which product the gaze is focused on, and the generative AI generates related information about that product.

[0273] Step 5:

[0274] The generated information is sent to the device and displayed in real time on the smart glasses display. The input is the generated information sent from the server, and the output is the information displayed in the field of view. Specifically, product prices, reviews, related products, etc. are displayed on the smart glasses screen.

[0275] Step 6:

[0276] The user provides feedback on the provided information, such as "it was helpful" or "it wasn't helpful." The input is the user's feedback, and the output is the feedback data. This feedback data is sent from the terminal to the server.

[0277] Step 7:

[0278] The server analyzes the received feedback data and updates and trains the generative AI model. The input is the feedback data, and the output is an improved generative AI model. As a data calculation, the model parameters are adjusted based on the feedback, improving the accuracy of information provided from the next time onwards.

[0279] In this way, the system improves the user's shopping experience and allows them to get the information they need instantly.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Below, the program processing of this system is explained in natural language.

[0282] Program processing flow

[0283] Initialization and environment data collection

[0284] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[0285] User Awareness and Customization

[0286] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[0287] Terminal: Sends the user's authentication data to the server.

[0288] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[0289] Terminal: Receives and applies profile settings sent from the server.

[0290] Eye tracking and information provision

[0291] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[0292] Terminal: Sends gaze information and environmental data to the server.

[0293] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[0294] Terminal: Displays the received information in the field of view and provides it to the user.

[0295] Emotion recognition and information regulation

[0296] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[0297] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[0298] Terminal: Displays tailored information in the user's field of view.

[0299] Specific examples

[0300] Example 1: Emotion recognition in shopping support

[0301] User: Looking for a product at a shopping mall in town.

[0302] Device: Detects when the user looks at a specific product. Furthermore, analyzes the user's facial expressions to recognize emotions such as interest or dissatisfaction.

[0303] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. If the user is interested, additional information such as special offers is also displayed.

[0304] Device: The generated information is displayed in the user's field of view. Users can check product prices, reviews, and related product information, and obtain information appropriate to their emotions.

[0305] Example 2: Emotion recognition in support of the visually impaired

[0306] User: A visually impaired user is walking around town.

[0307] Device: The device uses a camera to recognize the user's surroundings, collects information about obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[0308] Server: Generates optimal routes and points to be aware of based on environmental data and emotional information. If the user feels anxious, provides voice guidance to encourage and relax them.

[0309] Device: Provides users with directions, warnings, and emotional support through a voice assistant.

[0310] Real-time feedback and learning

[0311] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0312] Terminal: Sends feedback data to the server.

[0313] Server: Analyzes feedback and updates generative AI models and emotion engines to improve their accuracy.

[0314] Highlighting important information

[0315] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[0316] Device: Notify the user using sound, vibration, and visual enhancements.

[0317] In this way, the system of the present invention provides information that takes the user's emotions into consideration, thereby realizing a more personalized and accurate assistant function, enabling advanced assistance tailored to the individual needs of the user.

[0318] The processing flow will be explained below.

[0319] Specific processing flow of a program that combines emotion engines

[0320] Initialization and environment data collection

[0321] Step 1:

[0322] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[0323] Step 2:

[0324] Device: The camera tracks the user's gaze, and built-in sensors capture real-time environmental data such as location, light, and sound.

[0325] User Awareness and Customization

[0326] Step 3:

[0327] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[0328] Step 4:

[0329] Terminal: Sends the user's authentication data to the server.

[0330] Step 5:

[0331] Server: Retrieves user profile data from a database and generates settings based on past usage history and preferences.

[0332] Step 6:

[0333] Server: Returns the generated profile settings to the device.

[0334] Step 7:

[0335] Terminal: Receives and applies profile settings sent from the server.

[0336] Eye tracking and information provision

[0337] Step 8:

[0338] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[0339] Step 9:

[0340] Terminal: Sends gaze information and environmental data to the server.

[0341] Step 10:

[0342] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[0343] Step 11:

[0344] Server: Sends the generated information to the terminal.

[0345] Step 12:

[0346] Terminal: Displays the received information in the field of view and provides it to the user.

[0347] Emotion recognition and information regulation

[0348] Step 13:

[0349] Device: Uses a camera and microphone to monitor the user's facial expressions, voice tone, and body movements in real time.

[0350] Step 14:

[0351] Terminal: Analyzes the acquired data using an emotion engine means and estimates the user's emotions (joy, interest, stress, anxiety, etc.).

[0352] Step 15:

[0353] Terminal: Sends emotion information to the server.

[0354] Step 16:

[0355] Server: Uses emotional information to adjust the content and presentation of information provided to the user. For example, if the user is feeling stressed, the server displays information and suggestions to help them relax.

[0356] Step 17:

[0357] Terminal: Displays tailored information in the user's field of view.

[0358] Specific examples

[0359] Example 1: Emotion recognition in shopping support

[0360] Step 1:

[0361] User: Looking for a product at a shopping mall in town.

[0362] Step 8:

[0363] Device: Detects when the user looks at a specific product.

[0364] Step 9:

[0365] Terminal: Sends gaze information and environmental data to the server.

[0366] Step 10:

[0367] Server: Generates detailed product information, reviews, and recommended items. It also analyzes the user's facial expressions to recognize emotions such as interest and dissatisfaction.

[0368] Step 11:

[0369] Server: Adds and sends information that may interest the user, as well as special offers.

[0370] Step 12:

[0371] Terminal: The generated information is displayed in the user's field of view, allowing them to check product prices, reviews, and related product information.

[0372] Step 13:

[0373] Device: Recognizes the user's facial expressions and tone of voice and analyzes them with an emotion engine.

[0374] Step 14:

[0375] Terminal: Sends emotion information to the server.

[0376] Step 15:

[0377] Server: Uses emotional information to adapt the content of information provided to the user, for example, highlighting campaigns or discount information if the user shows interest.

[0378] Example 2: Emotion recognition in support of the visually impaired

[0379] Step 1:

[0380] User: A visually impaired user is walking around town.

[0381] Step 2:

[0382] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[0383] Step 9:

[0384] Terminal: Sends environmental data to the server.

[0385] Step 10:

[0386] Server: Generates optimal travel routes and points of interest from environmental data.

[0387] Step 11:

[0388] Server: Sends the generated information to the terminal.

[0389] Step 12:

[0390] Device: Provides directions to users through a voice assistant.

[0391] Step 13:

[0392] Device: Recognizes the user's vocal tone and physical reactions and analyzes them with an emotion engine.

[0393] Step 14:

[0394] Terminal: Sends emotion information to the server.

[0395] Step 15:

[0396] Server: Based on the emotional information, if the user feels anxious, it provides voice guidance to encourage and relax the user.

[0397] Step 17:

[0398] Terminal: Provides psychological support to the user using sound and vibration.

[0399] These steps allow the system of the present invention to provide information that takes into account not only the user's gaze information and environmental data, but also their emotional information, thereby providing more personalized and accurate assistance to the user.

[0400] Example 2

[0401] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0402] Current information delivery systems have difficulty tracking the user's gaze and environmental data in real time, and provide uniform information without considering the user's emotions, making it impossible to provide optimal information to the user. For this reason, there is a demand for more individually optimized, highly accurate information delivery methods that respond to the user's situation and emotions.

[0403] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0404] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user, information display means for displaying the generated information in the user's field of view, emotion recognition means for monitoring the user's voice, facial expressions, and physical reactions and estimating emotions, and information adjustment means for adjusting the information provided to the user based on the emotion recognition means. This makes it possible to provide optimally personalized information taking into account the user's gaze, environmental data, and emotional state.

[0405] An "eye tracking device" is a device or software that tracks a user's eye gaze in real time and analyzes that movement.

[0406] "Environmental data collection means" refers to devices or software such as sensors, cameras, microphones, etc. that are used to acquire surrounding environmental data.

[0407] The "generative AI model means" refers to an artificial intelligence model and its execution environment that analyzes the user's gaze information and environmental data and generates information that is optimal for the user.

[0408] "Information display means" refers to a device or software for displaying the generated information in the user's field of vision, such as a head-mounted display or smart glasses.

[0409] An "emotion recognition means" is a device or software that monitors the user's voice, facial expressions, and physical reactions and infers the user's emotions based on them.

[0410] The "information adjustment means" is a device or software that adjusts the information provided to the user based on the emotion information obtained by the emotion recognition means.

[0411] "Learning model update means" refers to the functions and processes used to collect user feedback and improve the performance of the generative AI model based on that feedback.

[0412] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[0413] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Specific embodiments of this system will be described below.

[0414] Initialization and environment data collection

[0415] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera begins tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[0416] User Awareness and Customization

[0417] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[0418] Terminal: Sends the user's authentication data to the server.

[0419] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[0420] Terminal: Receives and applies profile settings sent from the server.

[0421] Eye tracking and information provision

[0422] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[0423] Terminal: Sends gaze information and environmental data to the server.

[0424] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[0425] Terminal: Displays the received information in the field of view and provides it to the user.

[0426] Emotion recognition and information regulation

[0427] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[0428] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[0429] Terminal: Displays tailored information in the user's field of view.

[0430] Examples of specific examples and prompts

[0431] Examples:

[0432] Example 1: Emotion recognition in shopping support

[0433] User: Looking for a product at a shopping mall in town.

[0434] Device: Detects when the user looks at a specific product and analyzes facial expressions to recognize emotions.

[0435] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. It also displays information about additional benefits.

[0436] Terminal: The generated information is displayed in the user's field of view. The user can check the product information and obtain the appropriate information.

[0437] Example 2: Emotion recognition in support of the visually impaired

[0438] User: A visually impaired user is walking around town.

[0439] Device: Recognizes the surrounding environment using a camera, collects information on obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[0440] Server: Generates optimal travel routes and points to note based on environmental data and emotional information, and provides relaxing audio guidance.

[0441] Device: Provides directions, warnings, and emotional support through a voice assistant.

[0442] Prompt Sentence Examples

[0443] "Show me the most popular products at the shopping malls in town."

[0444] "I'm lost, can you tell me the shortest route from my current location?"

[0445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0446] Step 1:

[0447] Device: Power on your smart glasses.

[0448] Input: The smart glasses power switch is turned on.

[0449] Processing: The built-in software starts up and initializes the camera and sensors.

[0450] Output: The camera and sensors are ready to use.

[0451] Specific operation: The OS built into the smart glasses begins its boot sequence, loading each device driver, starting the camera, and environmental sensors (location sensor, illuminance sensor, sound sensor, etc.)

[0452] Step 2:

[0453] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[0454] Input: User's facial image data, voice data, and password.

[0455] Processing: Analyzes the input authentication data and executes authentication procedures.

[0456] Output: The result of authentication, success or failure.

[0457] How it works: The smart glasses use a camera to capture a facial image, which is then analyzed by a facial recognition algorithm. For voice recognition, a microphone records the user's voice and compares it with pre-registered voice patterns. For password entry, the user enters their password using the touchpad or voice input.

[0458] Step 3:

[0459] Terminal: Sends authentication data to the server.

[0460] Input: Authentication data.

[0461] Processing: The authentication data is sent using a communication protocol for sending the data to the server.

[0462] Output: Packet of authentication data to send to the server.

[0463] Specific operation: The smart glasses use wireless communication such as Wi-Fi or Bluetooth to send authentication data to the server.

[0464] Step 4:

[0465] Server: Receives authentication data and retrieves user profile information from a database.

[0466] Input: Authentication data sent from the terminal.

[0467] Processing: Query and retrieve the user's profile information from the database.

[0468] Output: User profile information.

[0469] Specific operation: The server receives the authentication data and sends a query to a database management system (DBMS) to obtain the corresponding user information.

[0470] Step 5:

[0471] Server: Generates configuration information based on the user's profile data and sends it to the device.

[0472] Input: User profile information.

[0473] Processing: Generates settings based on your past usage history and preferences.

[0474] Output: Configuration information to send to the device.

[0475] How it works: The server's algorithm analyzes the user's usage history and selects the settings that best suit the current usage situation. The selected information is then sent to the device using a communication protocol.

[0476] Step 6:

[0477] Terminal: Receives and applies profile settings sent from the server.

[0478] Input: Configuration information sent by the server.

[0479] Processing: Analyze the configuration information and apply it to the smart glasses' functions.

[0480] Output: The profile settings that were applied to the user.

[0481] Specific operation: Based on the received setting information, the display settings of the smart glasses and the initial settings of the application are performed.

[0482] Step 7:

[0483] Device: Tracks user gaze information in real time.

[0484] Input: Gaze data from the smartglasses camera.

[0485] Processing: Uses an eye-tracking algorithm to analyze the user's gaze direction.

[0486] Output: Real-time gaze data.

[0487] Specific operation: Analyzes the image data captured by the camera and calculates which direction the user is looking.

[0488] Step 8:

[0489] Terminal: Sends gaze information and environmental data to the server.

[0490] Input: Gaze data and environment data.

[0491] Processing: Gaze data and environmental data are packetized and sent to the server.

[0492] Output: Gaze data and environment data sent to the server.

[0493] Specific operation: Gaze data and environmental data are sent to the server via wireless communication.

[0494] Step 9:

[0495] Server: Analyzes gaze information and environmental data and generates related information.

[0496] Input: Gaze data and environment data.

[0497] Processing: Using a generative AI model, generate information related to the user's focus.

[0498] Output: Relevant information.

[0499] Specific operation: Using gaze data and environmental data as input, the generative AI model generates detailed information and recommendation information about the object.

[0500] Step 10:

[0501] Server: Sends the generated information to the terminal.

[0502] Input: Generated relevant information.

[0503] Processing: The generated information is packetized and sent to the terminal.

[0504] Output: Relevant information sent to the terminal.

[0505] Specific operation: The generated information is transmitted to the terminal via wireless communication.

[0506] Step 11:

[0507] Terminal: Displays the received information in the user's field of view.

[0508] Input: Relevant information sent by the server.

[0509] Processing: Display algorithms are used to properly display the information in the field of view.

[0510] Output: Information displayed in the field of view.

[0511] Specific operation: The generated information is overlaid on the smart glasses display.

[0512] Step 12:

[0513] Terminal: Monitors voice, facial expressions, and physical reactions to estimate emotions.

[0514] Input: User's voice data, facial expression data, and physical reaction data.

[0515] Processing: Emotion recognition algorithms are used to analyze this data and infer emotional states.

[0516] Output: User sentiment information.

[0517] Specific operation: Analyzes data collected by microphones, cameras, and sensors, and determines the user's emotions using an emotion estimation engine.

[0518] Step 13:

[0519] Server: Adjusts the information provided based on emotional information.

[0520] Input: Emotion information sent from the device.

[0521] Processing: Re-adjusting the information we generate based on emotional information.

[0522] Output: The adjusted information.

[0523] Specific operation: Analyzes emotional information and generates relevant information and encouraging messages to help users relax.

[0524] Step 14:

[0525] Terminal: Displays adjusted information in field of view.

[0526] Input: Adjusted information sent from the server.

[0527] Processing: Final processing is performed to display the adjusted information in the user's field of view.

[0528] Output: Calibration information displayed in the field of view.

[0529] Specific operation: Adjusted information is overlaid on the display to provide the user with the most appropriate information.

[0530] (Application example 2)

[0531] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0532] Conventional food delivery systems have difficulty considering the stress and anxiety that delivery personnel feel during deliveries, and have problems with providing optimal route guidance and important delivery-related information. This can lead to reduced efficiency and safety for delivery personnel, as well as reduced customer satisfaction. The present invention aims to solve these problems by providing optimal information to delivery personnel in real time, enabling them to perform delivery work efficiently and safely.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0534] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, emotion engine means for recognizing the user's emotions and adjusting information based on the emotions, generation AI model means for analyzing the user's gaze information, environmental data, and emotion information and generating optimal information for the user, and information display means for displaying the generated information in the user's field of view. This enables delivery personnel to obtain optimal route guidance and important information in real time, improving the efficiency and safety of delivery work.

[0535] An "eye tracking device" is a device or technology that tracks a user's gaze in real time and detects where the user is looking.

[0536] "Environmental data collection means" refers to devices or technologies that acquire data about the surrounding environment, including location information, illuminance, sound, etc.

[0537] "Emotion engine means" refers to a device or technology that analyzes the user's voice, facial expressions, and physical reactions and estimates the user's emotions.

[0538] The "generative AI model means" is an artificial intelligence model that analyzes gaze information, environmental data, and emotional information to generate information that is optimal for the user.

[0539] "Information display means" refers to a device or technology that displays the generated information in the user's field of vision.

[0540] "Learning model update means" is a machine learning technology that collects user feedback and improves the performance and accuracy of the generative AI model.

[0541] "User recognition means" refers to devices or technologies that authenticate users and personalize information based on their individual settings and profiles.

[0542] The system of the present invention uses a device such as smart glasses to collect a user's gaze and environmental data in real time, recognizes the user's emotions, and provides optimal information. A specific embodiment of this system will be described below.

[0543] Hardware and software used

[0544] Hardware:

[0545] Smart glasses: Equipped with a camera and various sensors (location information, light intensity, sound).

[0546] Server: Analyzes data and generates information.

[0547] software:

[0548] Eye-tracking software: Analyzes video data from the camera in real time to track gaze direction.

[0549] Environmental data collection software: Collects data from sensors and integrates it into environmental information.

[0550] Emotion engine: Analyzes the user's voice and facial expressions to estimate their emotional state.

[0551] Generative AI model: An artificial intelligence model that generates information based on gaze information, environmental data, and emotional information.

[0552] Information display software: Displays information optimally suited to the field of view of the smart glasses.

[0553] Data processing and calculation

[0554] The server receives gaze information, environmental data, and emotion information collected from the smart glasses, and processes these data in the following steps.

[0555] 1. Gaze analysis: Video data captured by the camera is analyzed using gaze tracking software to identify what the user is looking at.

[0556] 2. Environmental data analysis: Location information, illuminance, and audio data are collected and processed as environmental data.

[0557] 3. Emotion analysis: The emotion engine analyzes the user's voice and facial expression data to estimate their emotional state (e.g., stress or anxiety).

[0558] 4. Information generation: Based on gaze information, environmental data, and emotional state, the generative AI model generates optimal information. This information is created based on the prompt text.

[0559] 5. Information display: The generated information is displayed in the field of view of the smart glasses.

[0560] Specific examples

[0561] Example 1: Food delivery support

[0562] For delivery workers, the smart glasses collect real-time information about the surrounding environment during deliveries, providing optimal route guidance and detailed delivery destination information. Furthermore, an emotion engine detects stress and anxiety in the delivery worker and displays encouraging messages.

[0563] Example prompt sentence:

[0564] Collect visual and environmental data in front of your customer's home to provide optimal routes and important information to your delivery drivers during their delivery, as well as provide reassuring information if they are feeling anxious.

[0565] In this way, by using the system of the present invention, food delivery personnel can carry out their work efficiently and safely.

[0566] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0567] Step 1:

[0568] The device powers on the smart glasses and initializes the camera and sensors, allowing the camera to track the user's gaze and the sensors to capture real-time environmental data such as location, light, and audio.

[0569] Input: Power on the smart glasses

[0570] Output: Camera and sensor initialization complete

[0571] Step 2:

[0572] The user wears the smart glasses and logs in using facial or voice authentication. The results of the facial or voice authentication are sent from the device to the server.

[0573] Input: User's facial or voice authentication data

[0574] Output: User authentication result

[0575] Step 3:

[0576] The server receives the user's authentication data, retrieves the user's profile data from the database, and generates settings information based on the user's past usage history and preferences, which it then sends back to the device.

[0577] Input: Authentication data

[0578] Output: Profile data

[0579] Step 4:

[0580] The device receives profile setting information from the server and applies it to the smart glasses, completing customization for each user.

[0581] Input: Profile setting information

[0582] Output: Smart glasses customization completed

[0583] Step 5:

[0584] The device uses a camera to track the user's gaze in real time, identifying the direction of their gaze and the object they are looking at. The gaze information and environmental data are then sent to a server.

[0585] Input: Camera image

[0586] Output: Gaze information, environmental data

[0587] Step 6:

[0588] The server analyzes gaze information, environmental data, and user emotional information. The emotion engine estimates the user's emotions (e.g., stress, anxiety), and the generative AI model generates optimal information. The generated information is sent to the device.

[0589] Input: gaze information, environmental data, emotional information

[0590] Output: Best information

[0591] Step 7:

[0592] The device receives information from the server and displays it in the field of view of the smart glasses, such as delivery route and destination information.

[0593] Input: Best information

[0594] Output: Smart glasses visual field information

[0595] Step 8:

[0596] The user provides feedback on the provided information, such as "it was useful" or "it wasn't useful." The feedback data is sent from the terminal to the server.

[0597] Input: User feedback

[0598] Output: Feedback data

[0599] Step 9:

[0600] The server analyzes the feedback data and updates the learning model to improve the performance of the generative AI model, which is then reflected in the next information generation.

[0601] Input: Feedback data

[0602] Output: Updated generative AI model

[0603] In this way, through processing at each step, it is possible to provide users with the most appropriate information in real time, supporting efficient and safe food delivery.

[0604] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0605] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0606] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0607] [Second embodiment]

[0608] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0609] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0610] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0611] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0612] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0613] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0614] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0615] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0616] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0617] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0618] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0619] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0620] The present invention relates to a system that analyzes a user's line of sight and environmental data in real time and provides individually optimized information. The program processing of this system will be explained below in natural language.

[0621] Program processing flow

[0622] Initialization and environment data collection

[0623] Device: When the smart glasses are turned on, the internal software starts up and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect environmental data (location, light intensity, sound, etc.) in real time.

[0624] User Awareness and Customization

[0625] User: Wears smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[0626] Terminal: Sends authentication data to the server.

[0627] Server: Retrieves user profile data from the database, analyzes past usage history and preferences, and predicts necessary information. This data is then sent back to the device.

[0628] Terminal: Receives and applies profile settings sent from the server.

[0629] Eye tracking and information provision

[0630] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[0631] Terminal: Sends gaze information and environmental data to the server.

[0632] Server: Analyzes the received gaze information and environmental data, and uses a generative AI to create information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object). This information is then sent to the device.

[0633] Terminal: Displays the received information in the field of view and provides it to the user.

[0634] Specific examples

[0635] Example 1: Shopping support

[0636] User: Looking for a product at a shopping mall in town.

[0637] Device: Detects when the user looks at a specific product.

[0638] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[0639] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[0640] Example 2: Supporting the visually impaired

[0641] User: A visually impaired user is walking around town.

[0642] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[0643] Server: Generates optimal travel routes and points of interest from environmental data.

[0644] Device: Provides directions and alerts to users through a voice assistant.

[0645] Real-time feedback and learning

[0646] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0647] Terminal: Sends feedback data to the server.

[0648] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[0649] Highlighting important information

[0650] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[0651] Device: Notify the user using sound, vibration, and visual enhancements.

[0652] These functions enable smart glasses to provide users with personalized information in a variety of situations, supporting their daily lives and work efficiently and safely. The system of the present invention serves as a flexible and advanced information providing tool that meets the diverse needs of modern society.

[0653] The processing flow will be explained below.

[0654] Specific processing flow of the program

[0655] Initialization and environment data collection

[0656] Step 1:

[0657] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[0658] Step 2:

[0659] Device: The camera tracks the user's gaze, and built-in sensors capture environmental data such as location, light, and sound in real time.

[0660] User Awareness and Customization

[0661] Step 3:

[0662] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[0663] Step 4:

[0664] Terminal: Sends the user's authentication data to the server.

[0665] Step 5:

[0666] Server: Retrieves user profile data from a database and generates configuration information based on past usage history and preferences.

[0667] Step 6:

[0668] Server: Returns the generated profile settings to the device.

[0669] Step 7:

[0670] Terminal: Receives and applies profile settings sent from the server.

[0671] Eye tracking and information provision

[0672] Step 8:

[0673] Device: The camera tracks the user's gaze in real time and identifies the object of their attention.

[0674] Step 9:

[0675] Terminal: Sends gaze information and environmental data to the server.

[0676] Step 10:

[0677] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[0678] Step 11:

[0679] Server: Sends the generated information to the terminal.

[0680] Step 12:

[0681] Terminal: Displays the received information in the field of view and provides it to the user.

[0682] Real-time feedback and learning

[0683] Step 13:

[0684] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0685] Step 14:

[0686] Terminal: Sends user feedback data to the server.

[0687] Step 15:

[0688] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[0689] Highlighting important information

[0690] Step 16:

[0691] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[0692] Step 17:

[0693] Device: Notify the user using sound, vibration, and visual enhancements.

[0694] Specific examples

[0695] Example 1: Shopping support

[0696] Step 1:

[0697] User: Looking for a product at a shopping mall in town.

[0698] Step 8:

[0699] Device: Detects when the user looks at a specific product.

[0700] Step 9:

[0701] Terminal: Sends gaze information and environmental data to the server.

[0702] Step 10:

[0703] Server: Generates detailed product information and reviews, and also generates recommended items based on the user's preferences.

[0704] Step 12:

[0705] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[0706] Example 2: Supporting the visually impaired

[0707] Step 1:

[0708] User: A visually impaired user is walking around town.

[0709] Step 2:

[0710] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[0711] Step 9:

[0712] Terminal: Sends environmental data to the server.

[0713] Step 10:

[0714] Server: Generates optimal travel routes and points of interest from environmental data.

[0715] Step 12:

[0716] Device: Provides directions and alerts to users through a voice assistant.

[0717] Example 1

[0718] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0719] Conventional eye-tracking and environmental data analysis technologies have faced challenges in providing information to users effectively and quickly. Furthermore, personalization based on user profiles was insufficient, making it difficult to provide information tailored to individual users' needs and preferences. Furthermore, there was a lack of a means to quickly incorporate user feedback and improve the accuracy of the system. The highlighting function for important information and obstacles was also incomplete, failing to adequately ensure user safety and convenience.

[0720] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0721] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, and user recognition means for authenticating users and customizing information based on each user's settings and profile. This enables the server to quickly and accurately provide the information the user desires. The server also includes learning model update means for collecting user feedback and improving the performance of the information generation means, and highlighting means for detecting obstacles and important information and highlighting them. This improves the accuracy of the system and enhances user safety and convenience by highlighting important information.

[0722] An "eye tracking means" is a technology or device that detects a user's gaze in real time and tracks its position and direction.

[0723] "Environmental data collection means" refers to sensors or devices for acquiring surrounding environmental information (position information, illuminance, sound, etc.).

[0724] "Information generation means" refers to a system or software that analyzes the user's line of sight and environmental data, and generates information that is optimal for the user.

[0725] The "information display means" refers to a display or projection device for displaying the generated information in the user's field of vision in real time.

[0726] "User recognition" means technologies or systems that authenticate users and customize information based on their individual preferences and profiles.

[0727] A "learning model update means" is an algorithm or system that collects user feedback and uses it to improve the performance of the information generation means.

[0728] The "highlighting means" is a function for detecting obstacles and important information and highlighting them for the user.

[0729] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time and provides individually optimized information. The system includes gaze tracking means, environmental data collection means, information generation means, information display means, user recognition means, learning model update means, and highlighting means.

[0730] Initialization and environment data collection

[0731] Device: When the smart glasses are turned on, the internal software starts up. The camera and sensors are initialized and various data collection begins. Specifically, the eye-tracking camera tracks the user's gaze, location information is acquired by the GPS sensor, ambient light is measured by the light sensor, and audio data is collected by the built-in microphone.

[0732] User Awareness and Customization

[0733] User: Wears smart glasses and logs in to their profile using facial recognition, voice recognition, or password entry. For example, when performing facial recognition, the built-in camera recognizes the user's face, and dedicated software analyzes the facial data.

[0734] Terminal: The authentication data is sent to the server. The user recognition means then retrieves and analyzes profile data from the database on the server. Based on past usage history and preferences, the necessary information is predicted and sent back to the terminal.

[0735] Device: Receives and applies profile settings sent by the server, so that information is provided according to the user's preferences.

[0736] Eye tracking and information provision

[0737] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[0738] Terminal: Sends gaze information and environmental data to the server. For example, it detects when a user looks at a specific product and sends that information to the server.

[0739] Server: Analyzes the received gaze information and environmental data. Using a generative AI model, it generates information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object) and sends it to the device.

[0740] Device: Displays the received information in the user's field of view, for example, displaying detailed information and reviews about a specific product.

[0741] Specific examples

[0742] Shopping Support

[0743] User: Looking for a product at a shopping mall in town.

[0744] Device: Detects when a user looks at a specific product.

[0745] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[0746] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[0747] Support for the visually impaired

[0748] User: A visually impaired user is walking around town.

[0749] Device: The device uses a camera to recognize the user's surroundings and collect information about obstacles and directions.

[0750] Server: Generates optimal travel routes and points of interest from environmental data.

[0751] Device: Provides users with directions and alerts through a voice assistant.

[0752] Real-time feedback and learning

[0753] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0754] Terminal: Sends feedback data to the server.

[0755] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[0756] Highlighting important information

[0757] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[0758] Device: Notify the user using sound, vibration, and visual enhancements.

[0759] This allows smart glasses to provide users with personalized information tailored to their needs in a variety of situations, supporting their daily lives and work efficiently and safely. This system serves as a flexible and advanced means of providing information, meeting the diverse needs of modern society.

[0760] Example prompt: "Show me more information about the product the user is interested in."

[0761] "Please provide directions to the nearest hospital from my current location."

[0762] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0763] Step 1: Initialization and Environmental Data Collection

[0764] Terminal: When the smart glasses are powered on, the internal software starts up, performs initial configuration, and initializes the camera and sensors.

[0765] Specific operation: The firmware starts up, the camera starts tracking the user's gaze, the light sensor detects the ambient brightness, and the GPS module obtains location information.

[0766] Input: Power ON signal

[0767] Output: Initialization completion signal, gaze data, location information, illuminance data, surrounding audio data

[0768] Step 2: User awareness and customization

[0769] User: Puts on smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[0770] How it works: The built-in camera recognizes your face and the facial recognition software analyzes the data, or you speak a specific phrase for voice recognition and the voice recognition software analyzes it.

[0771] Input: Facial image data or audio data

[0772] Output: Authenticated data

[0773] Step 3: Send authentication data

[0774] Terminal: The user authentication means sends authentication data to the server.

[0775] Specific operation: The communication module in the smart glasses sends authentication data to the server endpoint.

[0776] Input: Authenticated Data

[0777] Output: Signal that authentication data has been sent to the server

[0778] Step 4: Analyze the profile data

[0779] Server: The server receives the authentication data and retrieves the user's profile data from the database. It analyzes the required information based on past usage history and preferences.

[0780] What it does: Executes database queries to retrieve the required profile data, analyzes the data using machine learning models, and generates predictions.

[0781] Input: Authentication data

[0782] Output: Profile data, analysis results

[0783] Step 5: Applying Profile Settings

[0784] Terminal: Receives and applies profile settings sent from the server.

[0785] Specific operation: The received profile settings are reflected in the display settings and interface settings of the smart glasses.

[0786] Input: Profile data, analysis results

[0787] Output: Apply completion signal, customized settings

[0788] Step 6: Eye tracking and data collection

[0789] Device: The camera tracks the user's gaze in real time and simultaneously collects environmental data.

[0790] How it works: Eye-tracking software monitors your eye movements and detects specific objects. Light sensors, GPS, and microphones collect data accordingly.

[0791] Input: gaze data, environmental data

[0792] Output: Real-time gaze information, environmental data

[0793] Step 7: Sending gaze and environment data

[0794] Terminal: Sends gaze information and environmental data to the server.

[0795] Specific operation: A communication module is used to integrate gaze data and environmental data and send them to the server.

[0796] Input: gaze information, environmental data

[0797] Output: Data transmission completion signal to the server

[0798] Step 8: Information Generation

[0799] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to generate information about the object the user is looking at.

[0800] Specific operation: Using gaze information and environmental data as input, the AI ​​model creates prompt sentences and generates related information.

[0801] Input: gaze information, environmental data

[0802] Output: Generated information, prompt

[0803] Step 9: Viewing Information

[0804] Terminal: Displays the received information in the field of view and provides it to the user.

[0805] Specific operation: The generated information is overlaid on the smart glasses display and presented to the user in real time.

[0806] Input: Generated information

[0807] Output: Information displayed in the field of view

[0808] (Application example 1)

[0809] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0810] In traditional shopping experiences, users must use devices such as smartphones to check product details and reviews, which is time-consuming. Furthermore, simply viewing products visually often does not immediately provide the information users actually need. Furthermore, the information provided is not optimized for each user's individual preferences or history, which prevents sufficient user satisfaction. A system that can solve these problems and provide users with the most appropriate information in real time is needed.

[0811] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0812] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating information optimal for the user, information display means for displaying the generated information in the user's field of view, and product information presentation means for providing detailed information about products the user is looking at in a physical store. This allows users to improve their shopping experience in physical stores, instantly obtain necessary information, and select products optimal for them.

[0813] An "eye-gaze tracker" is a device or software that detects and tracks a user's gaze in real time.

[0814] The "environmental data collection means" is a device or software for acquiring surrounding environmental data (position information, illuminance, sound, etc.).

[0815] The "generative AI model means" is an artificial intelligence-based data analysis system that analyzes the user's gaze information and environmental data, and generates information that is optimal for the user.

[0816] "Information display means" refers to a device or software that displays the generated information in the user's field of vision in real time.

[0817] A "product information presentation means" is a device or software that provides detailed information and reviews of products that a user looks at in a physical store.

[0818] The "learning model update means" is a data analysis and model update system that collects user feedback and improves the performance of the generative AI model.

[0819] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[0820] This invention relates to a system that provides optimal information in real time when a user is shopping for products in a physical store. This system is composed of an eye tracking means, an environmental data collection means, a generative AI model means, an information display means, and a product information presentation means.

[0821] System Configuration

[0822] 1. Eye-tracking methods:

[0823] Smart glasses are used as devices that detect and track a user's gaze in real time. They are equipped with a camera and internal software for tracking the user's gaze.

[0824] 2. Environmental data collection methods:

[0825] To obtain environmental data, the smart glasses are equipped with built-in sensors that collect location information, light intensity, audio data, and other information in real time.

[0826] 3. Generative AI model means:

[0827] An AI information analysis server located on the cloud is used to analyze the user's gaze information and environmental data and generate individually optimized information. This server analyzes and generates data using a generative AI model (e.g., OpenAI GPT-4, Google BERT).

[0828] 4. Information display means:

[0829] The smart glasses display is used to display the generated information in real time in the user's field of view, including product details, reviews, and recommendations.

[0830] 5. Product information presentation means:

[0831] To provide detailed information about the product the user is looking at, the server refers to the database on the cloud to retrieve relevant information, which is then analyzed by a generative AI model, and the generated information is displayed on the smart glasses.

[0832] Specific operation example

[0833] 1. Eye tracking and information provision:

[0834] A user puts on smart glasses in a physical store and focuses their gaze on a specific product.

[0835] The smart glasses' eye-tracking camera detects the user's gaze and sends the data to a cloud server.

[0836] The server analyzes gaze information and environmental data to generate detailed product information and recommended items based on the user's preferences.

[0837] The generated information is displayed in real time on the smart glasses display.

[0838] 2. User Feedback and Learning:

[0839] Users provide feedback on the information provided, such as "helpful" or "not helpful."

[0840] Feedback data is sent to the server, and the generative AI model is trained and updated.

[0841] Hardware and software used

[0842] Cloud servers: AWS, Google Cloud, Azure

[0843] AI model generation: OpenAI GPT-4, Google BERT

[0844] Data analysis: TensorFlow, PyTorch

[0845] Eye tracking and sensors: Smart glasses (e.g., Google Glass, Vuzix Blade)

[0846] Prompt Sentence Examples

[0847] Write a Python program that uses user gaze data and environmental data to generate product information in real time and display it on the smart glasses screen. OpenAI GPT-4 is used for the generation AI, and data is analyzed and managed in the AWS cloud.

[0848] In this way, the system can improve the user's shopping experience and provide the necessary information in real time.

[0849] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0850] Step 1:

[0851] When a user puts on the smart glasses and turns them on, the smart glasses' internal software starts up and initializes the eye-tracking camera and environmental sensors. The input here is the smart glasses' startup signal, and the output is the initialization completion state. Specifically, the smart glasses' camera begins to track the user's gaze, and the sensors collect location information, illuminance, and audio data in real time.

[0852] Step 2:

[0853] Users log in to their profile using facial or voice recognition. The input is the user's authentication data, and the output is an authentication success or failure message. The device sends the authentication data to a cloud server, which retrieves the user's profile data from a database and analyzes their past usage history and preferences to predict what information they will need. This profile data is then sent back to the device.

[0854] Step 3:

[0855] When a user directs their gaze toward a product in a physical store, an eye-tracking camera detects the gaze information in real time. The input is gaze data, and the output is the identification information of the specific product. The device sends the gaze information and environmental data to a cloud server, which then analyzes the received data. Specifically, it determines that the gaze has been focused on a specific product.

[0856] Step 4:

[0857] The server analyzes gaze information and environmental data and uses a generative AI model to generate optimal information for the user (detailed product information, reviews, recommended items, etc.). The input is gaze information and environmental data, and the output is the generated product information. Data processing involves detecting which product the gaze is focused on, and the generative AI generates related information about that product.

[0858] Step 5:

[0859] The generated information is sent to the device and displayed in real time on the smart glasses display. The input is the generated information sent from the server, and the output is the information displayed in the field of view. Specifically, product prices, reviews, related products, etc. are displayed on the smart glasses screen.

[0860] Step 6:

[0861] The user provides feedback on the provided information, such as "it was helpful" or "it wasn't helpful." The input is the user's feedback, and the output is the feedback data. This feedback data is sent from the terminal to the server.

[0862] Step 7:

[0863] The server analyzes the received feedback data and updates and trains the generative AI model. The input is the feedback data, and the output is an improved generative AI model. As a data calculation, the model parameters are adjusted based on the feedback, improving the accuracy of information provided from the next time onwards.

[0864] In this way, the system improves the user's shopping experience and allows them to get the information they need instantly.

[0865] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0866] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Below, the program processing of this system is explained in natural language.

[0867] Program processing flow

[0868] Initialization and environment data collection

[0869] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[0870] User Awareness and Customization

[0871] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[0872] Terminal: Sends the user's authentication data to the server.

[0873] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[0874] Terminal: Receives and applies profile settings sent from the server.

[0875] Eye tracking and information provision

[0876] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[0877] Terminal: Sends gaze information and environmental data to the server.

[0878] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[0879] Terminal: Displays the received information in the field of view and provides it to the user.

[0880] Emotion recognition and information regulation

[0881] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[0882] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[0883] Terminal: Displays tailored information in the user's field of view.

[0884] Specific examples

[0885] Example 1: Emotion recognition in shopping support

[0886] User: Looking for a product at a shopping mall in town.

[0887] Device: Detects when the user looks at a specific product. Furthermore, analyzes the user's facial expressions to recognize emotions such as interest or dissatisfaction.

[0888] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. If the user is interested, additional information such as special offers is also displayed.

[0889] Device: The generated information is displayed in the user's field of view. Users can check product prices, reviews, and related product information, and obtain information appropriate to their emotions.

[0890] Example 2: Emotion recognition in support of the visually impaired

[0891] User: A visually impaired user is walking around town.

[0892] Device: The device uses a camera to recognize the user's surroundings, collects information about obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[0893] Server: Generates optimal routes and points to be aware of based on environmental data and emotional information. If the user feels anxious, provides voice guidance to encourage and relax them.

[0894] Device: Provides users with directions, warnings, and emotional support through a voice assistant.

[0895] Real-time feedback and learning

[0896] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[0897] Terminal: Sends feedback data to the server.

[0898] Server: Analyzes feedback and updates generative AI models and emotion engines to improve their accuracy.

[0899] Highlighting important information

[0900] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[0901] Device: Notify the user using sound, vibration, and visual enhancements.

[0902] In this way, the system of the present invention provides information that takes the user's emotions into consideration, thereby realizing a more personalized and accurate assistant function, enabling advanced assistance tailored to the individual needs of the user.

[0903] The processing flow will be explained below.

[0904] Specific processing flow of a program that combines emotion engines

[0905] Initialization and environment data collection

[0906] Step 1:

[0907] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[0908] Step 2:

[0909] Device: The camera tracks the user's gaze, and built-in sensors capture real-time environmental data such as location, light, and sound.

[0910] User Awareness and Customization

[0911] Step 3:

[0912] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[0913] Step 4:

[0914] Terminal: Sends the user's authentication data to the server.

[0915] Step 5:

[0916] Server: Retrieves user profile data from a database and generates settings based on past usage history and preferences.

[0917] Step 6:

[0918] Server: Returns the generated profile settings to the device.

[0919] Step 7:

[0920] Terminal: Receives and applies profile settings sent from the server.

[0921] Eye tracking and information provision

[0922] Step 8:

[0923] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[0924] Step 9:

[0925] Terminal: Sends gaze information and environmental data to the server.

[0926] Step 10:

[0927] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[0928] Step 11:

[0929] Server: Sends the generated information to the terminal.

[0930] Step 12:

[0931] Terminal: Displays the received information in the field of view and provides it to the user.

[0932] Emotion recognition and information regulation

[0933] Step 13:

[0934] Device: Uses a camera and microphone to monitor the user's facial expressions, voice tone, and body movements in real time.

[0935] Step 14:

[0936] Terminal: Analyzes the acquired data using an emotion engine means and estimates the user's emotions (joy, interest, stress, anxiety, etc.).

[0937] Step 15:

[0938] Terminal: Sends emotion information to the server.

[0939] Step 16:

[0940] Server: Uses emotional information to adjust the content and presentation of information provided to the user. For example, if the user is feeling stressed, the server displays information and suggestions to help them relax.

[0941] Step 17:

[0942] Terminal: Displays tailored information in the user's field of view.

[0943] Specific examples

[0944] Example 1: Emotion recognition in shopping support

[0945] Step 1:

[0946] User: Looking for a product at a shopping mall in town.

[0947] Step 8:

[0948] Device: Detects when the user looks at a specific product.

[0949] Step 9:

[0950] Terminal: Sends gaze information and environmental data to the server.

[0951] Step 10:

[0952] Server: Generates detailed product information, reviews, and recommended items. It also analyzes the user's facial expressions to recognize emotions such as interest and dissatisfaction.

[0953] Step 11:

[0954] Server: Adds and sends information that may interest the user, as well as special offers.

[0955] Step 12:

[0956] Terminal: The generated information is displayed in the user's field of view, allowing them to check product prices, reviews, and related product information.

[0957] Step 13:

[0958] Device: Recognizes the user's facial expressions and tone of voice and analyzes them with an emotion engine.

[0959] Step 14:

[0960] Terminal: Sends emotion information to the server.

[0961] Step 15:

[0962] Server: Uses emotional information to adapt the content of information provided to the user, for example, highlighting campaigns or discount information if the user shows interest.

[0963] Example 2: Emotion recognition in support of the visually impaired

[0964] Step 1:

[0965] User: A visually impaired user is walking around town.

[0966] Step 2:

[0967] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[0968] Step 9:

[0969] Terminal: Sends environmental data to the server.

[0970] Step 10:

[0971] Server: Generates optimal travel routes and points of interest from environmental data.

[0972] Step 11:

[0973] Server: Sends the generated information to the terminal.

[0974] Step 12:

[0975] Device: Provides directions to users through a voice assistant.

[0976] Step 13:

[0977] Device: Recognizes the user's vocal tone and physical reactions and analyzes them with an emotion engine.

[0978] Step 14:

[0979] Terminal: Sends emotion information to the server.

[0980] Step 15:

[0981] Server: Based on the emotional information, if the user feels anxious, it provides voice guidance to encourage and relax the user.

[0982] Step 17:

[0983] Terminal: Provides psychological support to the user using sound and vibration.

[0984] These steps allow the system of the present invention to provide information that takes into account not only the user's gaze information and environmental data, but also their emotional information, thereby providing more personalized and accurate assistance to the user.

[0985] Example 2

[0986] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0987] Current information delivery systems have difficulty tracking the user's gaze and environmental data in real time, and provide uniform information without considering the user's emotions, making it impossible to provide optimal information to the user. For this reason, there is a demand for more individually optimized, highly accurate information delivery methods that respond to the user's situation and emotions.

[0988] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0989] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user, information display means for displaying the generated information in the user's field of view, emotion recognition means for monitoring the user's voice, facial expressions, and physical reactions and estimating emotions, and information adjustment means for adjusting the information provided to the user based on the emotion recognition means. This makes it possible to provide optimally personalized information taking into account the user's gaze, environmental data, and emotional state.

[0990] An "eye tracking device" is a device or software that tracks a user's eye gaze in real time and analyzes that movement.

[0991] "Environmental data collection means" refers to devices or software such as sensors, cameras, microphones, etc. that are used to acquire surrounding environmental data.

[0992] The "generative AI model means" refers to an artificial intelligence model and its execution environment that analyzes the user's gaze information and environmental data and generates information that is optimal for the user.

[0993] "Information display means" refers to a device or software for displaying the generated information in the user's field of vision, such as a head-mounted display or smart glasses.

[0994] An "emotion recognition means" is a device or software that monitors the user's voice, facial expressions, and physical reactions and infers the user's emotions based on them.

[0995] The "information adjustment means" is a device or software that adjusts the information provided to the user based on the emotion information obtained by the emotion recognition means.

[0996] "Learning model update means" refers to the functions and processes used to collect user feedback and improve the performance of the generative AI model based on that feedback.

[0997] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[0998] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Specific embodiments of this system will be described below.

[0999] Initialization and environment data collection

[1000] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera begins tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[1001] User Awareness and Customization

[1002] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[1003] Terminal: Sends the user's authentication data to the server.

[1004] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[1005] Terminal: Receives and applies profile settings sent from the server.

[1006] Eye tracking and information provision

[1007] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[1008] Terminal: Sends gaze information and environmental data to the server.

[1009] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[1010] Terminal: Displays the received information in the field of view and provides it to the user.

[1011] Emotion recognition and information regulation

[1012] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[1013] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[1014] Terminal: Displays tailored information in the user's field of view.

[1015] Examples of specific examples and prompts

[1016] Examples:

[1017] Example 1: Emotion recognition in shopping support

[1018] User: Looking for a product at a shopping mall in town.

[1019] Device: Detects when the user looks at a specific product and analyzes facial expressions to recognize emotions.

[1020] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. It also displays information about additional benefits.

[1021] Terminal: The generated information is displayed in the user's field of view. The user can check the product information and obtain the appropriate information.

[1022] Example 2: Emotion recognition in support of the visually impaired

[1023] User: A visually impaired user is walking around town.

[1024] Device: Recognizes the surrounding environment using a camera, collects information on obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[1025] Server: Generates optimal travel routes and points to note based on environmental data and emotional information, and provides relaxing audio guidance.

[1026] Device: Provides directions, warnings, and emotional support through a voice assistant.

[1027] Prompt Sentence Examples

[1028] "Show me the most popular products at the shopping malls in town."

[1029] "I'm lost, can you tell me the shortest route from my current location?"

[1030] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1031] Step 1:

[1032] Device: Power on your smart glasses.

[1033] Input: The smart glasses power switch is turned on.

[1034] Processing: The built-in software starts up and initializes the camera and sensors.

[1035] Output: The camera and sensors are ready to use.

[1036] Specific operation: The OS built into the smart glasses begins its boot sequence, loading each device driver, starting the camera, and environmental sensors (location sensor, illuminance sensor, sound sensor, etc.)

[1037] Step 2:

[1038] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[1039] Input: User's facial image data, voice data, and password.

[1040] Processing: Analyzes the input authentication data and executes authentication procedures.

[1041] Output: The result of authentication, success or failure.

[1042] How it works: The smart glasses use a camera to capture a facial image, which is then analyzed by a facial recognition algorithm. For voice recognition, a microphone records the user's voice and compares it with pre-registered voice patterns. For password entry, the user enters their password using the touchpad or voice input.

[1043] Step 3:

[1044] Terminal: Sends authentication data to the server.

[1045] Input: Authentication data.

[1046] Processing: The authentication data is sent using a communication protocol for sending the data to the server.

[1047] Output: Packet of authentication data to send to the server.

[1048] Specific operation: The smart glasses use wireless communication such as Wi-Fi or Bluetooth to send authentication data to the server.

[1049] Step 4:

[1050] Server: Receives authentication data and retrieves user profile information from a database.

[1051] Input: Authentication data sent from the terminal.

[1052] Processing: Query and retrieve the user's profile information from the database.

[1053] Output: User profile information.

[1054] Specific operation: The server receives the authentication data and sends a query to a database management system (DBMS) to obtain the corresponding user information.

[1055] Step 5:

[1056] Server: Generates configuration information based on the user's profile data and sends it to the device.

[1057] Input: User profile information.

[1058] Processing: Generates settings based on your past usage history and preferences.

[1059] Output: Configuration information to send to the device.

[1060] How it works: The server's algorithm analyzes the user's usage history and selects the settings that best suit the current usage situation. The selected information is then sent to the device using a communication protocol.

[1061] Step 6:

[1062] Terminal: Receives and applies profile settings sent from the server.

[1063] Input: Configuration information sent by the server.

[1064] Processing: Analyze the configuration information and apply it to the smart glasses' functions.

[1065] Output: The profile settings that were applied to the user.

[1066] Specific operation: Based on the received setting information, the display settings of the smart glasses and the initial settings of the application are performed.

[1067] Step 7:

[1068] Device: Tracks user gaze information in real time.

[1069] Input: Gaze data from the smartglasses camera.

[1070] Processing: Uses an eye-tracking algorithm to analyze the user's gaze direction.

[1071] Output: Real-time gaze data.

[1072] Specific operation: Analyzes the image data captured by the camera and calculates which direction the user is looking.

[1073] Step 8:

[1074] Terminal: Sends gaze information and environmental data to the server.

[1075] Input: Gaze data and environment data.

[1076] Processing: Gaze data and environmental data are packetized and sent to the server.

[1077] Output: Gaze data and environment data sent to the server.

[1078] Specific operation: Gaze data and environmental data are sent to the server via wireless communication.

[1079] Step 9:

[1080] Server: Analyzes gaze information and environmental data and generates related information.

[1081] Input: Gaze data and environment data.

[1082] Processing: Using a generative AI model, generate information related to the user's focus.

[1083] Output: Relevant information.

[1084] Specific operation: Using gaze data and environmental data as input, the generative AI model generates detailed information and recommendation information about the object.

[1085] Step 10:

[1086] Server: Sends the generated information to the terminal.

[1087] Input: Generated relevant information.

[1088] Processing: The generated information is packetized and sent to the terminal.

[1089] Output: Relevant information sent to the terminal.

[1090] Specific operation: The generated information is transmitted to the terminal via wireless communication.

[1091] Step 11:

[1092] Terminal: Displays the received information in the user's field of view.

[1093] Input: Relevant information sent by the server.

[1094] Processing: Display algorithms are used to properly display the information in the field of view.

[1095] Output: Information displayed in the field of view.

[1096] Specific operation: The generated information is overlaid on the smart glasses display.

[1097] Step 12:

[1098] Terminal: Monitors voice, facial expressions, and physical reactions to estimate emotions.

[1099] Input: User's voice data, facial expression data, and physical reaction data.

[1100] Processing: Emotion recognition algorithms are used to analyze this data and infer emotional states.

[1101] Output: User sentiment information.

[1102] Specific operation: Analyzes data collected by microphones, cameras, and sensors, and determines the user's emotions using an emotion estimation engine.

[1103] Step 13:

[1104] Server: Adjusts the information provided based on emotional information.

[1105] Input: Emotion information sent from the device.

[1106] Processing: Re-adjusting the information we generate based on emotional information.

[1107] Output: The adjusted information.

[1108] Specific operation: Analyzes emotional information and generates relevant information and encouraging messages to help users relax.

[1109] Step 14:

[1110] Terminal: Displays adjusted information in field of view.

[1111] Input: Adjusted information sent from the server.

[1112] Processing: Final processing is performed to display the adjusted information in the user's field of view.

[1113] Output: Calibration information displayed in the field of view.

[1114] Specific operation: Adjusted information is overlaid on the display to provide the user with the most appropriate information.

[1115] (Application example 2)

[1116] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1117] Conventional food delivery systems have difficulty considering the stress and anxiety that delivery personnel feel during deliveries, and have problems with providing optimal route guidance and important delivery-related information. This can lead to reduced efficiency and safety for delivery personnel, as well as reduced customer satisfaction. The present invention aims to solve these problems by providing optimal information to delivery personnel in real time, enabling them to perform delivery work efficiently and safely.

[1118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1119] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, emotion engine means for recognizing the user's emotions and adjusting information based on the emotions, generation AI model means for analyzing the user's gaze information, environmental data, and emotion information and generating optimal information for the user, and information display means for displaying the generated information in the user's field of view. This enables delivery personnel to obtain optimal route guidance and important information in real time, improving the efficiency and safety of delivery work.

[1120] An "eye tracking device" is a device or technology that tracks a user's gaze in real time and detects where the user is looking.

[1121] "Environmental data collection means" refers to devices or technologies that acquire data about the surrounding environment, including location information, illuminance, sound, etc.

[1122] "Emotion engine means" refers to a device or technology that analyzes the user's voice, facial expressions, and physical reactions and estimates the user's emotions.

[1123] The "generative AI model means" is an artificial intelligence model that analyzes gaze information, environmental data, and emotional information to generate information that is optimal for the user.

[1124] "Information display means" refers to a device or technology that displays the generated information in the user's field of vision.

[1125] "Learning model update means" is a machine learning technology that collects user feedback and improves the performance and accuracy of the generative AI model.

[1126] "User recognition means" refers to devices or technologies that authenticate users and personalize information based on their individual settings and profiles.

[1127] The system of the present invention uses a device such as smart glasses to collect a user's gaze and environmental data in real time, recognizes the user's emotions, and provides optimal information. A specific embodiment of this system will be described below.

[1128] Hardware and software used

[1129] Hardware:

[1130] Smart glasses: Equipped with a camera and various sensors (location information, light intensity, sound).

[1131] Server: Analyzes data and generates information.

[1132] software:

[1133] Eye-tracking software: Analyzes video data from the camera in real time to track gaze direction.

[1134] Environmental data collection software: Collects data from sensors and integrates it into environmental information.

[1135] Emotion engine: Analyzes the user's voice and facial expressions to estimate their emotional state.

[1136] Generative AI model: An artificial intelligence model that generates information based on gaze information, environmental data, and emotional information.

[1137] Information display software: Displays information optimally suited to the field of view of the smart glasses.

[1138] Data processing and calculation

[1139] The server receives gaze information, environmental data, and emotion information collected from the smart glasses, and processes these data in the following steps.

[1140] 1. Gaze analysis: Video data captured by the camera is analyzed using gaze tracking software to identify what the user is looking at.

[1141] 2. Environmental data analysis: Location information, illuminance, and audio data are collected and processed as environmental data.

[1142] 3. Emotion analysis: The emotion engine analyzes the user's voice and facial expression data to estimate their emotional state (e.g., stress or anxiety).

[1143] 4. Information generation: Based on gaze information, environmental data, and emotional state, the generative AI model generates optimal information. This information is created based on the prompt text.

[1144] 5. Information display: The generated information is displayed in the field of view of the smart glasses.

[1145] Specific examples

[1146] Example 1: Food delivery support

[1147] For delivery workers, the smart glasses collect real-time information about the surrounding environment during deliveries, providing optimal route guidance and detailed delivery destination information. Furthermore, an emotion engine detects stress and anxiety in the delivery worker and displays encouraging messages.

[1148] Example prompt sentence:

[1149] Collect visual and environmental data in front of your customer's home to provide optimal routes and important information to your delivery drivers during their delivery, as well as provide reassuring information if they are feeling anxious.

[1150] In this way, by using the system of the present invention, food delivery personnel can carry out their work efficiently and safely.

[1151] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1152] Step 1:

[1153] The device powers on the smart glasses and initializes the camera and sensors, allowing the camera to track the user's gaze and the sensors to capture real-time environmental data such as location, light, and audio.

[1154] Input: Power on the smart glasses

[1155] Output: Camera and sensor initialization complete

[1156] Step 2:

[1157] The user wears the smart glasses and logs in using facial or voice authentication. The results of the facial or voice authentication are sent from the device to the server.

[1158] Input: User's facial or voice authentication data

[1159] Output: User authentication result

[1160] Step 3:

[1161] The server receives the user's authentication data, retrieves the user's profile data from the database, and generates settings information based on the user's past usage history and preferences, which it then sends back to the device.

[1162] Input: Authentication data

[1163] Output: Profile data

[1164] Step 4:

[1165] The device receives profile setting information from the server and applies it to the smart glasses, completing customization for each user.

[1166] Input: Profile setting information

[1167] Output: Smart glasses customization completed

[1168] Step 5:

[1169] The device uses a camera to track the user's gaze in real time, identifying the direction of their gaze and the object they are looking at. The gaze information and environmental data are then sent to a server.

[1170] Input: Camera image

[1171] Output: Gaze information, environmental data

[1172] Step 6:

[1173] The server analyzes gaze information, environmental data, and user emotional information. The emotion engine estimates the user's emotions (e.g., stress, anxiety), and the generative AI model generates optimal information. The generated information is sent to the device.

[1174] Input: gaze information, environmental data, emotional information

[1175] Output: Best information

[1176] Step 7:

[1177] The device receives information from the server and displays it in the field of view of the smart glasses, such as delivery route and destination information.

[1178] Input: Best information

[1179] Output: Smart glasses visual field information

[1180] Step 8:

[1181] The user provides feedback on the provided information, such as "it was useful" or "it wasn't useful." The feedback data is sent from the terminal to the server.

[1182] Input: User feedback

[1183] Output: Feedback data

[1184] Step 9:

[1185] The server analyzes the feedback data and updates the learning model to improve the performance of the generative AI model, which is then reflected in the next information generation.

[1186] Input: Feedback data

[1187] Output: Updated generative AI model

[1188] In this way, through processing at each step, it is possible to provide users with the most appropriate information in real time, supporting efficient and safe food delivery.

[1189] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1190] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1191] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1192] [Third embodiment]

[1193] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1194] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1195] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1196] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1197] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1198] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1199] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1200] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1201] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1202] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1203] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1204] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1205] The present invention relates to a system that analyzes a user's line of sight and environmental data in real time and provides individually optimized information. The program processing of this system will be explained below in natural language.

[1206] Program processing flow

[1207] Initialization and environment data collection

[1208] Device: When the smart glasses are turned on, the internal software starts up and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect environmental data (location, light intensity, sound, etc.) in real time.

[1209] User Awareness and Customization

[1210] User: Wears smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[1211] Terminal: Sends authentication data to the server.

[1212] Server: Retrieves user profile data from the database, analyzes past usage history and preferences, and predicts necessary information. This data is then sent back to the device.

[1213] Terminal: Receives and applies profile settings sent from the server.

[1214] Eye tracking and information provision

[1215] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[1216] Terminal: Sends gaze information and environmental data to the server.

[1217] Server: Analyzes the received gaze information and environmental data, and uses a generative AI to create information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object). This information is then sent to the device.

[1218] Terminal: Displays the received information in the field of view and provides it to the user.

[1219] Specific examples

[1220] Example 1: Shopping support

[1221] User: Looking for a product at a shopping mall in town.

[1222] Device: Detects when the user looks at a specific product.

[1223] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[1224] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[1225] Example 2: Supporting the visually impaired

[1226] User: A visually impaired user is walking around town.

[1227] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[1228] Server: Generates optimal travel routes and points of interest from environmental data.

[1229] Device: Provides directions and alerts to users through a voice assistant.

[1230] Real-time feedback and learning

[1231] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1232] Terminal: Sends feedback data to the server.

[1233] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[1234] Highlighting important information

[1235] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[1236] Device: Notify the user using sound, vibration, and visual enhancements.

[1237] These functions enable smart glasses to provide users with personalized information in a variety of situations, supporting their daily lives and work efficiently and safely. The system of the present invention serves as a flexible and advanced information providing tool that meets the diverse needs of modern society.

[1238] The processing flow will be explained below.

[1239] Specific processing flow of the program

[1240] Initialization and environment data collection

[1241] Step 1:

[1242] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[1243] Step 2:

[1244] Device: The camera tracks the user's gaze, and built-in sensors capture environmental data such as location, light, and sound in real time.

[1245] User Awareness and Customization

[1246] Step 3:

[1247] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[1248] Step 4:

[1249] Terminal: Sends the user's authentication data to the server.

[1250] Step 5:

[1251] Server: Retrieves user profile data from a database and generates configuration information based on past usage history and preferences.

[1252] Step 6:

[1253] Server: Returns the generated profile settings to the device.

[1254] Step 7:

[1255] Terminal: Receives and applies profile settings sent from the server.

[1256] Eye tracking and information provision

[1257] Step 8:

[1258] Device: The camera tracks the user's gaze in real time and identifies the object of their attention.

[1259] Step 9:

[1260] Terminal: Sends gaze information and environmental data to the server.

[1261] Step 10:

[1262] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[1263] Step 11:

[1264] Server: Sends the generated information to the terminal.

[1265] Step 12:

[1266] Terminal: Displays the received information in the field of view and provides it to the user.

[1267] Real-time feedback and learning

[1268] Step 13:

[1269] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1270] Step 14:

[1271] Terminal: Sends user feedback data to the server.

[1272] Step 15:

[1273] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[1274] Highlighting important information

[1275] Step 16:

[1276] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[1277] Step 17:

[1278] Device: Notify the user using sound, vibration, and visual enhancements.

[1279] Specific examples

[1280] Example 1: Shopping support

[1281] Step 1:

[1282] User: Looking for a product at a shopping mall in town.

[1283] Step 8:

[1284] Device: Detects when the user looks at a specific product.

[1285] Step 9:

[1286] Terminal: Sends gaze information and environmental data to the server.

[1287] Step 10:

[1288] Server: Generates detailed product information and reviews, and also generates recommended items based on the user's preferences.

[1289] Step 12:

[1290] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[1291] Example 2: Supporting the visually impaired

[1292] Step 1:

[1293] User: A visually impaired user is walking around town.

[1294] Step 2:

[1295] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[1296] Step 9:

[1297] Terminal: Sends environmental data to the server.

[1298] Step 10:

[1299] Server: Generates optimal travel routes and points of interest from environmental data.

[1300] Step 12:

[1301] Device: Provides directions and alerts to users through a voice assistant.

[1302] Example 1

[1303] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1304] Conventional eye-tracking and environmental data analysis technologies have faced challenges in providing information to users effectively and quickly. Furthermore, personalization based on user profiles was insufficient, making it difficult to provide information tailored to individual users' needs and preferences. Furthermore, there was a lack of a means to quickly incorporate user feedback and improve the accuracy of the system. The highlighting function for important information and obstacles was also incomplete, failing to adequately ensure user safety and convenience.

[1305] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1306] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, and user recognition means for authenticating users and customizing information based on each user's settings and profile. This enables the server to quickly and accurately provide the information the user desires. The server also includes learning model update means for collecting user feedback and improving the performance of the information generation means, and highlighting means for detecting obstacles and important information and highlighting them. This improves the accuracy of the system and enhances user safety and convenience by highlighting important information.

[1307] An "eye tracking means" is a technology or device that detects a user's gaze in real time and tracks its position and direction.

[1308] "Environmental data collection means" refers to sensors or devices for acquiring surrounding environmental information (position information, illuminance, sound, etc.).

[1309] "Information generation means" refers to a system or software that analyzes the user's line of sight and environmental data, and generates information that is optimal for the user.

[1310] The "information display means" refers to a display or projection device for displaying the generated information in the user's field of vision in real time.

[1311] "User recognition" means technologies or systems that authenticate users and customize information based on their individual preferences and profiles.

[1312] A "learning model update means" is an algorithm or system that collects user feedback and uses it to improve the performance of the information generation means.

[1313] The "highlighting means" is a function for detecting obstacles and important information and highlighting them for the user.

[1314] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time and provides individually optimized information. The system includes gaze tracking means, environmental data collection means, information generation means, information display means, user recognition means, learning model update means, and highlighting means.

[1315] Initialization and environment data collection

[1316] Device: When the smart glasses are turned on, the internal software starts up. The camera and sensors are initialized and various data collection begins. Specifically, the eye-tracking camera tracks the user's gaze, location information is acquired by the GPS sensor, ambient light is measured by the light sensor, and audio data is collected by the built-in microphone.

[1317] User Awareness and Customization

[1318] User: Wears smart glasses and logs in to their profile using facial recognition, voice recognition, or password entry. For example, when performing facial recognition, the built-in camera recognizes the user's face, and dedicated software analyzes the facial data.

[1319] Terminal: The authentication data is sent to the server. The user recognition means then retrieves and analyzes profile data from the database on the server. Based on past usage history and preferences, the necessary information is predicted and sent back to the terminal.

[1320] Device: Receives and applies profile settings sent by the server, so that information is provided according to the user's preferences.

[1321] Eye tracking and information provision

[1322] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[1323] Terminal: Sends gaze information and environmental data to the server. For example, it detects when a user looks at a specific product and sends that information to the server.

[1324] Server: Analyzes the received gaze information and environmental data. Using a generative AI model, it generates information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object) and sends it to the device.

[1325] Device: Displays the received information in the user's field of view, for example, displaying detailed information and reviews about a specific product.

[1326] Specific examples

[1327] Shopping Support

[1328] User: Looking for a product at a shopping mall in town.

[1329] Device: Detects when a user looks at a specific product.

[1330] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[1331] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[1332] Support for the visually impaired

[1333] User: A visually impaired user is walking around town.

[1334] Device: The device uses a camera to recognize the user's surroundings and collect information about obstacles and directions.

[1335] Server: Generates optimal travel routes and points of interest from environmental data.

[1336] Device: Provides users with directions and alerts through a voice assistant.

[1337] Real-time feedback and learning

[1338] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1339] Terminal: Sends feedback data to the server.

[1340] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[1341] Highlighting important information

[1342] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[1343] Device: Notify the user using sound, vibration, and visual enhancements.

[1344] This allows smart glasses to provide users with personalized information tailored to their needs in a variety of situations, supporting their daily lives and work efficiently and safely. This system serves as a flexible and advanced means of providing information, meeting the diverse needs of modern society.

[1345] Example prompt: "Show me more information about the product the user is interested in."

[1346] "Please provide directions to the nearest hospital from my current location."

[1347] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1348] Step 1: Initialization and Environmental Data Collection

[1349] Terminal: When the smart glasses are powered on, the internal software starts up, performs initial configuration, and initializes the camera and sensors.

[1350] Specific operation: The firmware starts up, the camera starts tracking the user's gaze, the light sensor detects the ambient brightness, and the GPS module obtains location information.

[1351] Input: Power ON signal

[1352] Output: Initialization completion signal, gaze data, location information, illuminance data, surrounding audio data

[1353] Step 2: User awareness and customization

[1354] User: Puts on smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[1355] How it works: The built-in camera recognizes your face and the facial recognition software analyzes the data, or you speak a specific phrase for voice recognition and the voice recognition software analyzes it.

[1356] Input: Facial image data or audio data

[1357] Output: Authenticated data

[1358] Step 3: Send authentication data

[1359] Terminal: The user authentication means sends authentication data to the server.

[1360] Specific operation: The communication module in the smart glasses sends authentication data to the server endpoint.

[1361] Input: Authenticated Data

[1362] Output: Signal that authentication data has been sent to the server

[1363] Step 4: Analyze the profile data

[1364] Server: The server receives the authentication data and retrieves the user's profile data from the database. It analyzes the required information based on past usage history and preferences.

[1365] What it does: Executes database queries to retrieve the required profile data, analyzes the data using machine learning models, and generates predictions.

[1366] Input: Authentication data

[1367] Output: Profile data, analysis results

[1368] Step 5: Applying Profile Settings

[1369] Terminal: Receives and applies profile settings sent from the server.

[1370] Specific operation: The received profile settings are reflected in the display settings and interface settings of the smart glasses.

[1371] Input: Profile data, analysis results

[1372] Output: Apply completion signal, customized settings

[1373] Step 6: Eye tracking and data collection

[1374] Device: The camera tracks the user's gaze in real time and simultaneously collects environmental data.

[1375] How it works: Eye-tracking software monitors your eye movements and detects specific objects. Light sensors, GPS, and microphones collect data accordingly.

[1376] Input: gaze data, environmental data

[1377] Output: Real-time gaze information, environmental data

[1378] Step 7: Sending gaze and environment data

[1379] Terminal: Sends gaze information and environmental data to the server.

[1380] Specific operation: A communication module is used to integrate gaze data and environmental data and send them to the server.

[1381] Input: gaze information, environmental data

[1382] Output: Data transmission completion signal to the server

[1383] Step 8: Information Generation

[1384] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to generate information about the object the user is looking at.

[1385] Specific operation: Using gaze information and environmental data as input, the AI ​​model creates prompt sentences and generates related information.

[1386] Input: gaze information, environmental data

[1387] Output: Generated information, prompt

[1388] Step 9: Viewing Information

[1389] Terminal: Displays the received information in the field of view and provides it to the user.

[1390] Specific operation: The generated information is overlaid on the smart glasses display and presented to the user in real time.

[1391] Input: Generated information

[1392] Output: Information displayed in the field of view

[1393] (Application example 1)

[1394] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1395] In traditional shopping experiences, users must use devices such as smartphones to check product details and reviews, which is time-consuming. Furthermore, simply viewing products visually often does not immediately provide the information users actually need. Furthermore, the information provided is not optimized for each user's individual preferences or history, which prevents sufficient user satisfaction. A system that can solve these problems and provide users with the most appropriate information in real time is needed.

[1396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1397] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating information optimal for the user, information display means for displaying the generated information in the user's field of view, and product information presentation means for providing detailed information about products the user is looking at in a physical store. This allows users to improve their shopping experience in physical stores, instantly obtain necessary information, and select products optimal for them.

[1398] An "eye-gaze tracker" is a device or software that detects and tracks a user's gaze in real time.

[1399] The "environmental data collection means" is a device or software for acquiring surrounding environmental data (position information, illuminance, sound, etc.).

[1400] The "generative AI model means" is an artificial intelligence-based data analysis system that analyzes the user's gaze information and environmental data, and generates information that is optimal for the user.

[1401] "Information display means" refers to a device or software that displays the generated information in the user's field of vision in real time.

[1402] A "product information presentation means" is a device or software that provides detailed information and reviews of products that a user looks at in a physical store.

[1403] The "learning model update means" is a data analysis and model update system that collects user feedback and improves the performance of the generative AI model.

[1404] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[1405] This invention relates to a system that provides optimal information in real time when a user is shopping for products in a physical store. This system is composed of an eye tracking means, an environmental data collection means, a generative AI model means, an information display means, and a product information presentation means.

[1406] System Configuration

[1407] 1. Eye-tracking methods:

[1408] Smart glasses are used as devices that detect and track a user's gaze in real time. They are equipped with a camera and internal software for tracking the user's gaze.

[1409] 2. Environmental data collection methods:

[1410] To obtain environmental data, the smart glasses are equipped with built-in sensors that collect location information, light intensity, audio data, and other information in real time.

[1411] 3. Generative AI model means:

[1412] An AI information analysis server located on the cloud is used to analyze the user's gaze information and environmental data and generate individually optimized information. This server analyzes and generates data using a generative AI model (e.g., OpenAI GPT-4, Google BERT).

[1413] 4. Information display means:

[1414] The smart glasses display is used to display the generated information in real time in the user's field of view, including product details, reviews, and recommendations.

[1415] 5. Product information presentation means:

[1416] To provide detailed information about the product the user is looking at, the server refers to the database on the cloud to retrieve relevant information, which is then analyzed by a generative AI model, and the generated information is displayed on the smart glasses.

[1417] Specific operation example

[1418] 1. Eye tracking and information provision:

[1419] A user puts on smart glasses in a physical store and focuses their gaze on a specific product.

[1420] The smart glasses' eye-tracking camera detects the user's gaze and sends the data to a cloud server.

[1421] The server analyzes gaze information and environmental data to generate detailed product information and recommended items based on the user's preferences.

[1422] The generated information is displayed in real time on the smart glasses display.

[1423] 2. User Feedback and Learning:

[1424] Users provide feedback on the information provided, such as "helpful" or "not helpful."

[1425] Feedback data is sent to the server, and the generative AI model is trained and updated.

[1426] Hardware and software used

[1427] Cloud servers: AWS, Google Cloud, Azure

[1428] AI model generation: OpenAI GPT-4, Google BERT

[1429] Data analysis: TensorFlow, PyTorch

[1430] Eye tracking and sensors: Smart glasses (e.g., Google Glass, Vuzix Blade)

[1431] Prompt Sentence Examples

[1432] Write a Python program that uses user gaze data and environmental data to generate product information in real time and display it on the smart glasses screen. OpenAI GPT-4 is used for the generation AI, and data is analyzed and managed in the AWS cloud.

[1433] In this way, the system can improve the user's shopping experience and provide the necessary information in real time.

[1434] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1435] Step 1:

[1436] When a user puts on the smart glasses and turns them on, the smart glasses' internal software starts up and initializes the eye-tracking camera and environmental sensors. The input here is the smart glasses' startup signal, and the output is the initialization completion state. Specifically, the smart glasses' camera begins to track the user's gaze, and the sensors collect location information, illuminance, and audio data in real time.

[1437] Step 2:

[1438] Users log in to their profile using facial or voice recognition. The input is the user's authentication data, and the output is an authentication success or failure message. The device sends the authentication data to a cloud server, which retrieves the user's profile data from a database and analyzes their past usage history and preferences to predict what information they will need. This profile data is then sent back to the device.

[1439] Step 3:

[1440] When a user directs their gaze toward a product in a physical store, an eye-tracking camera detects the gaze information in real time. The input is gaze data, and the output is the identification information of the specific product. The device sends the gaze information and environmental data to a cloud server, which then analyzes the received data. Specifically, it determines that the gaze has been focused on a specific product.

[1441] Step 4:

[1442] The server analyzes gaze information and environmental data and uses a generative AI model to generate optimal information for the user (detailed product information, reviews, recommended items, etc.). The input is gaze information and environmental data, and the output is the generated product information. Data processing involves detecting which product the gaze is focused on, and the generative AI generates related information about that product.

[1443] Step 5:

[1444] The generated information is sent to the device and displayed in real time on the smart glasses display. The input is the generated information sent from the server, and the output is the information displayed in the field of view. Specifically, product prices, reviews, related products, etc. are displayed on the smart glasses screen.

[1445] Step 6:

[1446] The user provides feedback on the provided information, such as "it was helpful" or "it wasn't helpful." The input is the user's feedback, and the output is the feedback data. This feedback data is sent from the terminal to the server.

[1447] Step 7:

[1448] The server analyzes the received feedback data and updates and trains the generative AI model. The input is the feedback data, and the output is an improved generative AI model. As a data calculation, the model parameters are adjusted based on the feedback, improving the accuracy of information provided from the next time onwards.

[1449] In this way, the system improves the user's shopping experience and allows them to get the information they need instantly.

[1450] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1451] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Below, the program processing of this system is explained in natural language.

[1452] Program processing flow

[1453] Initialization and environment data collection

[1454] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[1455] User Awareness and Customization

[1456] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[1457] Terminal: Sends the user's authentication data to the server.

[1458] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[1459] Terminal: Receives and applies profile settings sent from the server.

[1460] Eye tracking and information provision

[1461] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[1462] Terminal: Sends gaze information and environmental data to the server.

[1463] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[1464] Terminal: Displays the received information in the field of view and provides it to the user.

[1465] Emotion recognition and information regulation

[1466] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[1467] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[1468] Terminal: Displays tailored information in the user's field of view.

[1469] Specific examples

[1470] Example 1: Emotion recognition in shopping support

[1471] User: Looking for a product at a shopping mall in town.

[1472] Device: Detects when the user looks at a specific product. Furthermore, analyzes the user's facial expressions to recognize emotions such as interest or dissatisfaction.

[1473] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. If the user is interested, additional information such as special offers is also displayed.

[1474] Device: The generated information is displayed in the user's field of view. Users can check product prices, reviews, and related product information, and obtain information appropriate to their emotions.

[1475] Example 2: Emotion recognition in support of the visually impaired

[1476] User: A visually impaired user is walking around town.

[1477] Device: The device uses a camera to recognize the user's surroundings, collects information about obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[1478] Server: Generates optimal routes and points to be aware of based on environmental data and emotional information. If the user feels anxious, provides voice guidance to encourage and relax them.

[1479] Device: Provides users with directions, warnings, and emotional support through a voice assistant.

[1480] Real-time feedback and learning

[1481] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1482] Terminal: Sends feedback data to the server.

[1483] Server: Analyzes feedback and updates generative AI models and emotion engines to improve their accuracy.

[1484] Highlighting important information

[1485] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[1486] Device: Notify the user using sound, vibration, and visual enhancements.

[1487] In this way, the system of the present invention provides information that takes the user's emotions into consideration, thereby realizing a more personalized and accurate assistant function, enabling advanced assistance tailored to the individual needs of the user.

[1488] The processing flow will be explained below.

[1489] Specific processing flow of a program that combines emotion engines

[1490] Initialization and environment data collection

[1491] Step 1:

[1492] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[1493] Step 2:

[1494] Device: The camera tracks the user's gaze, and built-in sensors capture real-time environmental data such as location, light, and sound.

[1495] User Awareness and Customization

[1496] Step 3:

[1497] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[1498] Step 4:

[1499] Terminal: Sends the user's authentication data to the server.

[1500] Step 5:

[1501] Server: Retrieves user profile data from a database and generates settings based on past usage history and preferences.

[1502] Step 6:

[1503] Server: Returns the generated profile settings to the device.

[1504] Step 7:

[1505] Terminal: Receives and applies profile settings sent from the server.

[1506] Eye tracking and information provision

[1507] Step 8:

[1508] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[1509] Step 9:

[1510] Terminal: Sends gaze information and environmental data to the server.

[1511] Step 10:

[1512] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[1513] Step 11:

[1514] Server: Sends the generated information to the terminal.

[1515] Step 12:

[1516] Terminal: Displays the received information in the field of view and provides it to the user.

[1517] Emotion recognition and information regulation

[1518] Step 13:

[1519] Device: Uses a camera and microphone to monitor the user's facial expressions, voice tone, and body movements in real time.

[1520] Step 14:

[1521] Terminal: Analyzes the acquired data using an emotion engine means and estimates the user's emotions (joy, interest, stress, anxiety, etc.).

[1522] Step 15:

[1523] Terminal: Sends emotion information to the server.

[1524] Step 16:

[1525] Server: Uses emotional information to adjust the content and presentation of information provided to the user. For example, if the user is feeling stressed, the server displays information and suggestions to help them relax.

[1526] Step 17:

[1527] Terminal: Displays tailored information in the user's field of view.

[1528] Specific examples

[1529] Example 1: Emotion recognition in shopping support

[1530] Step 1:

[1531] User: Looking for a product at a shopping mall in town.

[1532] Step 8:

[1533] Device: Detects when the user looks at a specific product.

[1534] Step 9:

[1535] Terminal: Sends gaze information and environmental data to the server.

[1536] Step 10:

[1537] Server: Generates detailed product information, reviews, and recommended items. It also analyzes the user's facial expressions to recognize emotions such as interest and dissatisfaction.

[1538] Step 11:

[1539] Server: Adds and sends information that may interest the user, as well as special offers.

[1540] Step 12:

[1541] Terminal: The generated information is displayed in the user's field of view, allowing them to check product prices, reviews, and related product information.

[1542] Step 13:

[1543] Device: Recognizes the user's facial expressions and tone of voice and analyzes them with an emotion engine.

[1544] Step 14:

[1545] Terminal: Sends emotion information to the server.

[1546] Step 15:

[1547] Server: Uses emotional information to adapt the content of information provided to the user, for example, highlighting campaigns or discount information if the user shows interest.

[1548] Example 2: Emotion recognition in support of the visually impaired

[1549] Step 1:

[1550] User: A visually impaired user is walking around town.

[1551] Step 2:

[1552] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[1553] Step 9:

[1554] Terminal: Sends environmental data to the server.

[1555] Step 10:

[1556] Server: Generates optimal travel routes and points of interest from environmental data.

[1557] Step 11:

[1558] Server: Sends the generated information to the terminal.

[1559] Step 12:

[1560] Device: Provides directions to users through a voice assistant.

[1561] Step 13:

[1562] Device: Recognizes the user's vocal tone and physical reactions and analyzes them with an emotion engine.

[1563] Step 14:

[1564] Terminal: Sends emotion information to the server.

[1565] Step 15:

[1566] Server: Based on the emotional information, if the user feels anxious, it provides voice guidance to encourage and relax the user.

[1567] Step 17:

[1568] Terminal: Provides psychological support to the user using sound and vibration.

[1569] These steps allow the system of the present invention to provide information that takes into account not only the user's gaze information and environmental data, but also their emotional information, thereby providing more personalized and accurate assistance to the user.

[1570] Example 2

[1571] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1572] Current information delivery systems have difficulty tracking the user's gaze and environmental data in real time, and provide uniform information without considering the user's emotions, making it impossible to provide optimal information to the user. For this reason, there is a demand for more individually optimized, highly accurate information delivery methods that respond to the user's situation and emotions.

[1573] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1574] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user, information display means for displaying the generated information in the user's field of view, emotion recognition means for monitoring the user's voice, facial expressions, and physical reactions and estimating emotions, and information adjustment means for adjusting the information provided to the user based on the emotion recognition means. This makes it possible to provide optimally personalized information taking into account the user's gaze, environmental data, and emotional state.

[1575] An "eye tracking device" is a device or software that tracks a user's eye gaze in real time and analyzes that movement.

[1576] "Environmental data collection means" refers to devices or software such as sensors, cameras, microphones, etc. that are used to acquire surrounding environmental data.

[1577] The "generative AI model means" refers to an artificial intelligence model and its execution environment that analyzes the user's gaze information and environmental data and generates information that is optimal for the user.

[1578] "Information display means" refers to a device or software for displaying the generated information in the user's field of vision, such as a head-mounted display or smart glasses.

[1579] An "emotion recognition means" is a device or software that monitors the user's voice, facial expressions, and physical reactions and infers the user's emotions based on them.

[1580] The "information adjustment means" is a device or software that adjusts the information provided to the user based on the emotion information obtained by the emotion recognition means.

[1581] "Learning model update means" refers to the functions and processes used to collect user feedback and improve the performance of the generative AI model based on that feedback.

[1582] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[1583] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Specific embodiments of this system will be described below.

[1584] Initialization and environment data collection

[1585] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera begins tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[1586] User Awareness and Customization

[1587] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[1588] Terminal: Sends the user's authentication data to the server.

[1589] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[1590] Terminal: Receives and applies profile settings sent from the server.

[1591] Eye tracking and information provision

[1592] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[1593] Terminal: Sends gaze information and environmental data to the server.

[1594] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[1595] Terminal: Displays the received information in the field of view and provides it to the user.

[1596] Emotion recognition and information regulation

[1597] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[1598] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[1599] Terminal: Displays tailored information in the user's field of view.

[1600] Examples of specific examples and prompts

[1601] Examples:

[1602] Example 1: Emotion recognition in shopping support

[1603] User: Looking for a product at a shopping mall in town.

[1604] Device: Detects when the user looks at a specific product and analyzes facial expressions to recognize emotions.

[1605] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. It also displays information about additional benefits.

[1606] Terminal: The generated information is displayed in the user's field of view. The user can check the product information and obtain the appropriate information.

[1607] Example 2: Emotion recognition in support of the visually impaired

[1608] User: A visually impaired user is walking around town.

[1609] Device: Recognizes the surrounding environment using a camera, collects information on obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[1610] Server: Generates optimal travel routes and points to note based on environmental data and emotional information, and provides relaxing audio guidance.

[1611] Device: Provides directions, warnings, and emotional support through a voice assistant.

[1612] Prompt Sentence Examples

[1613] "Show me the most popular products at the shopping malls in town."

[1614] "I'm lost, can you tell me the shortest route from my current location?"

[1615] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1616] Step 1:

[1617] Device: Power on your smart glasses.

[1618] Input: The smart glasses power switch is turned on.

[1619] Processing: The built-in software starts up and initializes the camera and sensors.

[1620] Output: The camera and sensors are ready to use.

[1621] Specific operation: The OS built into the smart glasses begins its boot sequence, loading each device driver, starting the camera, and environmental sensors (location sensor, illuminance sensor, sound sensor, etc.)

[1622] Step 2:

[1623] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[1624] Input: User's facial image data, voice data, and password.

[1625] Processing: Analyzes the input authentication data and executes authentication procedures.

[1626] Output: The result of authentication, success or failure.

[1627] How it works: The smart glasses use a camera to capture a facial image, which is then analyzed by a facial recognition algorithm. For voice recognition, a microphone records the user's voice and compares it with pre-registered voice patterns. For password entry, the user enters their password using the touchpad or voice input.

[1628] Step 3:

[1629] Terminal: Sends authentication data to the server.

[1630] Input: Authentication data.

[1631] Processing: The authentication data is sent using a communication protocol for sending the data to the server.

[1632] Output: Packet of authentication data to send to the server.

[1633] Specific operation: The smart glasses use wireless communication such as Wi-Fi or Bluetooth to send authentication data to the server.

[1634] Step 4:

[1635] Server: Receives authentication data and retrieves user profile information from a database.

[1636] Input: Authentication data sent from the terminal.

[1637] Processing: Query and retrieve the user's profile information from the database.

[1638] Output: User profile information.

[1639] Specific operation: The server receives the authentication data and sends a query to a database management system (DBMS) to obtain the corresponding user information.

[1640] Step 5:

[1641] Server: Generates configuration information based on the user's profile data and sends it to the device.

[1642] Input: User profile information.

[1643] Processing: Generates settings based on your past usage history and preferences.

[1644] Output: Configuration information to send to the device.

[1645] How it works: The server's algorithm analyzes the user's usage history and selects the settings that best suit the current usage situation. The selected information is then sent to the device using a communication protocol.

[1646] Step 6:

[1647] Terminal: Receives and applies profile settings sent from the server.

[1648] Input: Configuration information sent by the server.

[1649] Processing: Analyze the configuration information and apply it to the smart glasses' functions.

[1650] Output: The profile settings that were applied to the user.

[1651] Specific operation: Based on the received setting information, the display settings of the smart glasses and the initial settings of the application are performed.

[1652] Step 7:

[1653] Device: Tracks user gaze information in real time.

[1654] Input: Gaze data from the smartglasses camera.

[1655] Processing: Uses an eye-tracking algorithm to analyze the user's gaze direction.

[1656] Output: Real-time gaze data.

[1657] Specific operation: Analyzes the image data captured by the camera and calculates which direction the user is looking.

[1658] Step 8:

[1659] Terminal: Sends gaze information and environmental data to the server.

[1660] Input: Gaze data and environment data.

[1661] Processing: Gaze data and environmental data are packetized and sent to the server.

[1662] Output: Gaze data and environment data sent to the server.

[1663] Specific operation: Gaze data and environmental data are sent to the server via wireless communication.

[1664] Step 9:

[1665] Server: Analyzes gaze information and environmental data and generates related information.

[1666] Input: Gaze data and environment data.

[1667] Processing: Using a generative AI model, generate information related to the user's focus.

[1668] Output: Relevant information.

[1669] Specific operation: Using gaze data and environmental data as input, the generative AI model generates detailed information and recommendation information about the object.

[1670] Step 10:

[1671] Server: Sends the generated information to the terminal.

[1672] Input: Generated relevant information.

[1673] Processing: The generated information is packetized and sent to the terminal.

[1674] Output: Relevant information sent to the terminal.

[1675] Specific operation: The generated information is transmitted to the terminal via wireless communication.

[1676] Step 11:

[1677] Terminal: Displays the received information in the user's field of view.

[1678] Input: Relevant information sent by the server.

[1679] Processing: Display algorithms are used to properly display the information in the field of view.

[1680] Output: Information displayed in the field of view.

[1681] Specific operation: The generated information is overlaid on the smart glasses display.

[1682] Step 12:

[1683] Terminal: Monitors voice, facial expressions, and physical reactions to estimate emotions.

[1684] Input: User's voice data, facial expression data, and physical reaction data.

[1685] Processing: Emotion recognition algorithms are used to analyze this data and infer emotional states.

[1686] Output: User sentiment information.

[1687] Specific operation: Analyzes data collected by microphones, cameras, and sensors, and determines the user's emotions using an emotion estimation engine.

[1688] Step 13:

[1689] Server: Adjusts the information provided based on emotional information.

[1690] Input: Emotion information sent from the device.

[1691] Processing: Re-adjusting the information we generate based on emotional information.

[1692] Output: The adjusted information.

[1693] Specific operation: Analyzes emotional information and generates relevant information and encouraging messages to help users relax.

[1694] Step 14:

[1695] Terminal: Displays adjusted information in field of view.

[1696] Input: Adjusted information sent from the server.

[1697] Processing: Final processing is performed to display the adjusted information in the user's field of view.

[1698] Output: Calibration information displayed in the field of view.

[1699] Specific operation: Adjusted information is overlaid on the display to provide the user with the most appropriate information.

[1700] (Application example 2)

[1701] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1702] Conventional food delivery systems have difficulty considering the stress and anxiety that delivery personnel feel during deliveries, and have problems with providing optimal route guidance and important delivery-related information. This can lead to reduced efficiency and safety for delivery personnel, as well as reduced customer satisfaction. The present invention aims to solve these problems by providing optimal information to delivery personnel in real time, enabling them to perform delivery work efficiently and safely.

[1703] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1704] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, emotion engine means for recognizing the user's emotions and adjusting information based on the emotions, generation AI model means for analyzing the user's gaze information, environmental data, and emotion information and generating optimal information for the user, and information display means for displaying the generated information in the user's field of view. This enables delivery personnel to obtain optimal route guidance and important information in real time, improving the efficiency and safety of delivery work.

[1705] An "eye tracking device" is a device or technology that tracks a user's gaze in real time and detects where the user is looking.

[1706] "Environmental data collection means" refers to devices or technologies that acquire data about the surrounding environment, including location information, illuminance, sound, etc.

[1707] "Emotion engine means" refers to a device or technology that analyzes the user's voice, facial expressions, and physical reactions and estimates the user's emotions.

[1708] The "generative AI model means" is an artificial intelligence model that analyzes gaze information, environmental data, and emotional information to generate information that is optimal for the user.

[1709] "Information display means" refers to a device or technology that displays the generated information in the user's field of vision.

[1710] "Learning model update means" is a machine learning technology that collects user feedback and improves the performance and accuracy of the generative AI model.

[1711] "User recognition means" refers to devices or technologies that authenticate users and personalize information based on their individual settings and profiles.

[1712] The system of the present invention uses a device such as smart glasses to collect a user's gaze and environmental data in real time, recognizes the user's emotions, and provides optimal information. A specific embodiment of this system will be described below.

[1713] Hardware and software used

[1714] Hardware:

[1715] Smart glasses: Equipped with a camera and various sensors (location information, light intensity, sound).

[1716] Server: Analyzes data and generates information.

[1717] software:

[1718] Eye-tracking software: Analyzes video data from the camera in real time to track gaze direction.

[1719] Environmental data collection software: Collects data from sensors and integrates it into environmental information.

[1720] Emotion engine: Analyzes the user's voice and facial expressions to estimate their emotional state.

[1721] Generative AI model: An artificial intelligence model that generates information based on gaze information, environmental data, and emotional information.

[1722] Information display software: Displays information optimally suited to the field of view of the smart glasses.

[1723] Data processing and calculation

[1724] The server receives gaze information, environmental data, and emotion information collected from the smart glasses, and processes these data in the following steps.

[1725] 1. Gaze analysis: Video data captured by the camera is analyzed using gaze tracking software to identify what the user is looking at.

[1726] 2. Environmental data analysis: Location information, illuminance, and audio data are collected and processed as environmental data.

[1727] 3. Emotion analysis: The emotion engine analyzes the user's voice and facial expression data to estimate their emotional state (e.g., stress or anxiety).

[1728] 4. Information generation: Based on gaze information, environmental data, and emotional state, the generative AI model generates optimal information. This information is created based on the prompt text.

[1729] 5. Information display: The generated information is displayed in the field of view of the smart glasses.

[1730] Specific examples

[1731] Example 1: Food delivery support

[1732] For delivery workers, the smart glasses collect real-time information about the surrounding environment during deliveries, providing optimal route guidance and detailed delivery destination information. Furthermore, an emotion engine detects stress and anxiety in the delivery worker and displays encouraging messages.

[1733] Example prompt sentence:

[1734] Collect visual and environmental data in front of your customer's home to provide optimal routes and important information to your delivery drivers during their delivery, as well as provide reassuring information if they are feeling anxious.

[1735] In this way, by using the system of the present invention, food delivery personnel can carry out their work efficiently and safely.

[1736] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1737] Step 1:

[1738] The device powers on the smart glasses and initializes the camera and sensors, allowing the camera to track the user's gaze and the sensors to capture real-time environmental data such as location, light, and audio.

[1739] Input: Power on the smart glasses

[1740] Output: Camera and sensor initialization complete

[1741] Step 2:

[1742] The user wears the smart glasses and logs in using facial or voice authentication. The results of the facial or voice authentication are sent from the device to the server.

[1743] Input: User's facial or voice authentication data

[1744] Output: User authentication result

[1745] Step 3:

[1746] The server receives the user's authentication data, retrieves the user's profile data from the database, and generates settings information based on the user's past usage history and preferences, which it then sends back to the device.

[1747] Input: Authentication data

[1748] Output: Profile data

[1749] Step 4:

[1750] The device receives profile setting information from the server and applies it to the smart glasses, completing customization for each user.

[1751] Input: Profile setting information

[1752] Output: Smart glasses customization completed

[1753] Step 5:

[1754] The device uses a camera to track the user's gaze in real time, identifying the direction of their gaze and the object they are looking at. The gaze information and environmental data are then sent to a server.

[1755] Input: Camera image

[1756] Output: Gaze information, environmental data

[1757] Step 6:

[1758] The server analyzes gaze information, environmental data, and user emotional information. The emotion engine estimates the user's emotions (e.g., stress, anxiety), and the generative AI model generates optimal information. The generated information is sent to the device.

[1759] Input: gaze information, environmental data, emotional information

[1760] Output: Best information

[1761] Step 7:

[1762] The device receives information from the server and displays it in the field of view of the smart glasses, such as delivery route and destination information.

[1763] Input: Best information

[1764] Output: Smart glasses visual field information

[1765] Step 8:

[1766] The user provides feedback on the provided information, such as "it was useful" or "it wasn't useful." The feedback data is sent from the terminal to the server.

[1767] Input: User feedback

[1768] Output: Feedback data

[1769] Step 9:

[1770] The server analyzes the feedback data and updates the learning model to improve the performance of the generative AI model, which is then reflected in the next information generation.

[1771] Input: Feedback data

[1772] Output: Updated generative AI model

[1773] In this way, through processing at each step, it is possible to provide users with the most appropriate information in real time, supporting efficient and safe food delivery.

[1774] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1775] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1776] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1777] [Fourth embodiment]

[1778] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1779] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1780] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1781] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1782] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1783] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1784] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1785] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1786] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1787] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1788] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1789] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1790] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1791] The present invention relates to a system that analyzes a user's line of sight and environmental data in real time and provides individually optimized information. The program processing of this system will be explained below in natural language.

[1792] Program processing flow

[1793] Initialization and environment data collection

[1794] Device: When the smart glasses are turned on, the internal software starts up and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect environmental data (location, light intensity, sound, etc.) in real time.

[1795] User Awareness and Customization

[1796] User: Wears smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[1797] Terminal: Sends authentication data to the server.

[1798] Server: Retrieves user profile data from the database, analyzes past usage history and preferences, and predicts necessary information. This data is then sent back to the device.

[1799] Terminal: Receives and applies profile settings sent from the server.

[1800] Eye tracking and information provision

[1801] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[1802] Terminal: Sends gaze information and environmental data to the server.

[1803] Server: Analyzes the received gaze information and environmental data, and uses a generative AI to create information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object). This information is then sent to the device.

[1804] Terminal: Displays the received information in the field of view and provides it to the user.

[1805] Specific examples

[1806] Example 1: Shopping support

[1807] User: Looking for a product at a shopping mall in town.

[1808] Device: Detects when the user looks at a specific product.

[1809] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[1810] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[1811] Example 2: Supporting the visually impaired

[1812] User: A visually impaired user is walking around town.

[1813] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[1814] Server: Generates optimal travel routes and points of interest from environmental data.

[1815] Device: Provides directions and alerts to users through a voice assistant.

[1816] Real-time feedback and learning

[1817] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1818] Terminal: Sends feedback data to the server.

[1819] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[1820] Highlighting important information

[1821] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[1822] Device: Notify the user using sound, vibration, and visual enhancements.

[1823] These functions enable smart glasses to provide users with personalized information in a variety of situations, supporting their daily lives and work efficiently and safely. The system of the present invention serves as a flexible and advanced information providing tool that meets the diverse needs of modern society.

[1824] The processing flow will be explained below.

[1825] Specific processing flow of the program

[1826] Initialization and environment data collection

[1827] Step 1:

[1828] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[1829] Step 2:

[1830] Device: The camera tracks the user's gaze, and built-in sensors capture environmental data such as location, light, and sound in real time.

[1831] User Awareness and Customization

[1832] Step 3:

[1833] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[1834] Step 4:

[1835] Terminal: Sends the user's authentication data to the server.

[1836] Step 5:

[1837] Server: Retrieves user profile data from a database and generates configuration information based on past usage history and preferences.

[1838] Step 6:

[1839] Server: Returns the generated profile settings to the device.

[1840] Step 7:

[1841] Terminal: Receives and applies profile settings sent from the server.

[1842] Eye tracking and information provision

[1843] Step 8:

[1844] Device: The camera tracks the user's gaze in real time and identifies the object of their attention.

[1845] Step 9:

[1846] Terminal: Sends gaze information and environmental data to the server.

[1847] Step 10:

[1848] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[1849] Step 11:

[1850] Server: Sends the generated information to the terminal.

[1851] Step 12:

[1852] Terminal: Displays the received information in the field of view and provides it to the user.

[1853] Real-time feedback and learning

[1854] Step 13:

[1855] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1856] Step 14:

[1857] Terminal: Sends user feedback data to the server.

[1858] Step 15:

[1859] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[1860] Highlighting important information

[1861] Step 16:

[1862] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[1863] Step 17:

[1864] Device: Notify the user using sound, vibration, and visual enhancements.

[1865] Specific examples

[1866] Example 1: Shopping support

[1867] Step 1:

[1868] User: Looking for a product at a shopping mall in town.

[1869] Step 8:

[1870] Device: Detects when the user looks at a specific product.

[1871] Step 9:

[1872] Terminal: Sends gaze information and environmental data to the server.

[1873] Step 10:

[1874] Server: Generates detailed product information and reviews, and also generates recommended items based on the user's preferences.

[1875] Step 12:

[1876] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[1877] Example 2: Supporting the visually impaired

[1878] Step 1:

[1879] User: A visually impaired user is walking around town.

[1880] Step 2:

[1881] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[1882] Step 9:

[1883] Terminal: Sends environmental data to the server.

[1884] Step 10:

[1885] Server: Generates optimal travel routes and points of interest from environmental data.

[1886] Step 12:

[1887] Device: Provides directions and alerts to users through a voice assistant.

[1888] Example 1

[1889] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1890] Conventional eye-tracking and environmental data analysis technologies have faced challenges in providing information to users effectively and quickly. Furthermore, personalization based on user profiles was insufficient, making it difficult to provide information tailored to individual users' needs and preferences. Furthermore, there was a lack of a means to quickly incorporate user feedback and improve the accuracy of the system. The highlighting function for important information and obstacles was also incomplete, failing to adequately ensure user safety and convenience.

[1891] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1892] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, and user recognition means for authenticating users and customizing information based on each user's settings and profile. This enables the server to quickly and accurately provide the information the user desires. The server also includes learning model update means for collecting user feedback and improving the performance of the information generation means, and highlighting means for detecting obstacles and important information and highlighting them. This improves the accuracy of the system and enhances user safety and convenience by highlighting important information.

[1893] An "eye tracking means" is a technology or device that detects a user's gaze in real time and tracks its position and direction.

[1894] "Environmental data collection means" refers to sensors or devices for acquiring surrounding environmental information (position information, illuminance, sound, etc.).

[1895] "Information generation means" refers to a system or software that analyzes the user's line of sight and environmental data, and generates information that is optimal for the user.

[1896] The "information display means" refers to a display or projection device for displaying the generated information in the user's field of vision in real time.

[1897] "User recognition" means technologies or systems that authenticate users and customize information based on their individual preferences and profiles.

[1898] A "learning model update means" is an algorithm or system that collects user feedback and uses it to improve the performance of the information generation means.

[1899] The "highlighting means" is a function for detecting obstacles and important information and highlighting them for the user.

[1900] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time and provides individually optimized information. The system includes gaze tracking means, environmental data collection means, information generation means, information display means, user recognition means, learning model update means, and highlighting means.

[1901] Initialization and environment data collection

[1902] Device: When the smart glasses are turned on, the internal software starts up. The camera and sensors are initialized and various data collection begins. Specifically, the eye-tracking camera tracks the user's gaze, location information is acquired by the GPS sensor, ambient light is measured by the light sensor, and audio data is collected by the built-in microphone.

[1903] User Awareness and Customization

[1904] User: Wears smart glasses and logs in to their profile using facial recognition, voice recognition, or password entry. For example, when performing facial recognition, the built-in camera recognizes the user's face, and dedicated software analyzes the facial data.

[1905] Terminal: The authentication data is sent to the server. The user recognition means then retrieves and analyzes profile data from the database on the server. Based on past usage history and preferences, the necessary information is predicted and sent back to the terminal.

[1906] Device: Receives and applies profile settings sent by the server, so that information is provided according to the user's preferences.

[1907] Eye tracking and information provision

[1908] Device: The camera tracks the user's gaze in real time and detects what they are looking at.

[1909] Terminal: Sends gaze information and environmental data to the server. For example, it detects when a user looks at a specific product and sends that information to the server.

[1910] Server: Analyzes the received gaze information and environmental data. Using a generative AI model, it generates information about the object the user is looking at (such as product details, the history of a tourist spot, or a translation of the object) and sends it to the device.

[1911] Device: Displays the received information in the user's field of view, for example, displaying detailed information and reviews about a specific product.

[1912] Specific examples

[1913] Shopping Support

[1914] User: Looking for a product at a shopping mall in town.

[1915] Device: Detects when a user looks at a specific product.

[1916] Server: Generates detailed product information and reviews, and also provides recommendations based on the user's preferences.

[1917] Terminal: The generated information is displayed in the user's field of view, allowing the user to check product prices, reviews, related product information, etc.

[1918] Support for the visually impaired

[1919] User: A visually impaired user is walking around town.

[1920] Device: The device uses a camera to recognize the user's surroundings and collect information about obstacles and directions.

[1921] Server: Generates optimal travel routes and points of interest from environmental data.

[1922] Device: Provides users with directions and alerts through a voice assistant.

[1923] Real-time feedback and learning

[1924] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[1925] Terminal: Sends feedback data to the server.

[1926] Server: Analyzes feedback and updates the generative AI model to improve accuracy.

[1927] Highlighting important information

[1928] Server: If it detects a significant obstacle or important information in the medical field, it sends special highlighting instructions to the device.

[1929] Device: Notify the user using sound, vibration, and visual enhancements.

[1930] This allows smart glasses to provide users with personalized information tailored to their needs in a variety of situations, supporting their daily lives and work efficiently and safely. This system serves as a flexible and advanced means of providing information, meeting the diverse needs of modern society.

[1931] Example prompt: "Show me more information about the product the user is interested in."

[1932] "Please provide directions to the nearest hospital from my current location."

[1933] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1934] Step 1: Initialization and Environmental Data Collection

[1935] Terminal: When the smart glasses are powered on, the internal software starts up, performs initial configuration, and initializes the camera and sensors.

[1936] Specific operation: The firmware starts up, the camera starts tracking the user's gaze, the light sensor detects the ambient brightness, and the GPS module obtains location information.

[1937] Input: Power ON signal

[1938] Output: Initialization completion signal, gaze data, location information, illuminance data, surrounding audio data

[1939] Step 2: User awareness and customization

[1940] User: Puts on smart glasses and logs into their profile using facial recognition, voice recognition, or password entry.

[1941] How it works: The built-in camera recognizes your face and the facial recognition software analyzes the data, or you speak a specific phrase for voice recognition and the voice recognition software analyzes it.

[1942] Input: Facial image data or audio data

[1943] Output: Authenticated data

[1944] Step 3: Send authentication data

[1945] Terminal: The user authentication means sends authentication data to the server.

[1946] Specific operation: The communication module in the smart glasses sends authentication data to the server endpoint.

[1947] Input: Authenticated Data

[1948] Output: Signal that authentication data has been sent to the server

[1949] Step 4: Analyze the profile data

[1950] Server: The server receives the authentication data and retrieves the user's profile data from the database. It analyzes the required information based on past usage history and preferences.

[1951] What it does: Executes database queries to retrieve the required profile data, analyzes the data using machine learning models, and generates predictions.

[1952] Input: Authentication data

[1953] Output: Profile data, analysis results

[1954] Step 5: Applying Profile Settings

[1955] Terminal: Receives and applies profile settings sent from the server.

[1956] Specific operation: The received profile settings are reflected in the display settings and interface settings of the smart glasses.

[1957] Input: Profile data, analysis results

[1958] Output: Apply completion signal, customized settings

[1959] Step 6: Eye tracking and data collection

[1960] Device: The camera tracks the user's gaze in real time and simultaneously collects environmental data.

[1961] How it works: Eye-tracking software monitors your eye movements and detects specific objects. Light sensors, GPS, and microphones collect data accordingly.

[1962] Input: gaze data, environmental data

[1963] Output: Real-time gaze information, environmental data

[1964] Step 7: Sending gaze and environment data

[1965] Terminal: Sends gaze information and environmental data to the server.

[1966] Specific operation: A communication module is used to integrate gaze data and environmental data and send them to the server.

[1967] Input: gaze information, environmental data

[1968] Output: Data transmission completion signal to the server

[1969] Step 8: Information Generation

[1970] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to generate information about the object the user is looking at.

[1971] Specific operation: Using gaze information and environmental data as input, the AI ​​model creates prompt sentences and generates related information.

[1972] Input: gaze information, environmental data

[1973] Output: Generated information, prompt

[1974] Step 9: Viewing Information

[1975] Terminal: Displays the received information in the field of view and provides it to the user.

[1976] Specific operation: The generated information is overlaid on the smart glasses display and presented to the user in real time.

[1977] Input: Generated information

[1978] Output: Information displayed in the field of view

[1979] (Application example 1)

[1980] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1981] In traditional shopping experiences, users must use devices such as smartphones to check product details and reviews, which is time-consuming. Furthermore, simply viewing products visually often does not immediately provide the information users actually need. Furthermore, the information provided is not optimized for each user's individual preferences or history, which prevents sufficient user satisfaction. A system that can solve these problems and provide users with the most appropriate information in real time is needed.

[1982] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1983] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating information optimal for the user, information display means for displaying the generated information in the user's field of view, and product information presentation means for providing detailed information about products the user is looking at in a physical store. This allows users to improve their shopping experience in physical stores, instantly obtain necessary information, and select products optimal for them.

[1984] An "eye-gaze tracker" is a device or software that detects and tracks a user's gaze in real time.

[1985] The "environmental data collection means" is a device or software for acquiring surrounding environmental data (position information, illuminance, sound, etc.).

[1986] The "generative AI model means" is an artificial intelligence-based data analysis system that analyzes the user's gaze information and environmental data, and generates information that is optimal for the user.

[1987] "Information display means" refers to a device or software that displays the generated information in the user's field of vision in real time.

[1988] A "product information presentation means" is a device or software that provides detailed information and reviews of products that a user looks at in a physical store.

[1989] The "learning model update means" is a data analysis and model update system that collects user feedback and improves the performance of the generative AI model.

[1990] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[1991] This invention relates to a system that provides optimal information in real time when a user is shopping for products in a physical store. This system is composed of an eye tracking means, an environmental data collection means, a generative AI model means, an information display means, and a product information presentation means.

[1992] System Configuration

[1993] 1. Eye-tracking methods:

[1994] Smart glasses are used as devices that detect and track a user's gaze in real time. They are equipped with a camera and internal software for tracking the user's gaze.

[1995] 2. Environmental data collection methods:

[1996] To obtain environmental data, the smart glasses are equipped with built-in sensors that collect location information, light intensity, audio data, and other information in real time.

[1997] 3. Generative AI model means:

[1998] An AI information analysis server located on the cloud is used to analyze the user's gaze information and environmental data and generate individually optimized information. This server analyzes and generates data using a generative AI model (e.g., OpenAI GPT-4, Google BERT).

[1999] 4. Information display means:

[2000] The smart glasses display is used to display the generated information in real time in the user's field of view, including product details, reviews, and recommendations.

[2001] 5. Product information presentation means:

[2002] To provide detailed information about the product the user is looking at, the server refers to the database on the cloud to retrieve relevant information, which is then analyzed by a generative AI model, and the generated information is displayed on the smart glasses.

[2003] Specific operation example

[2004] 1. Eye tracking and information provision:

[2005] A user puts on smart glasses in a physical store and focuses their gaze on a specific product.

[2006] The smart glasses' eye-tracking camera detects the user's gaze and sends the data to a cloud server.

[2007] The server analyzes gaze information and environmental data to generate detailed product information and recommended items based on the user's preferences.

[2008] The generated information is displayed in real time on the smart glasses display.

[2009] 2. User Feedback and Learning:

[2010] Users provide feedback on the information provided, such as "helpful" or "not helpful."

[2011] Feedback data is sent to the server, and the generative AI model is trained and updated.

[2012] Hardware and software used

[2013] Cloud servers: AWS, Google Cloud, Azure

[2014] AI model generation: OpenAI GPT-4, Google BERT

[2015] Data analysis: TensorFlow, PyTorch

[2016] Eye tracking and sensors: Smart glasses (e.g., Google Glass, Vuzix Blade)

[2017] Prompt Sentence Examples

[2018] Write a Python program that uses user gaze data and environmental data to generate product information in real time and display it on the smart glasses screen. OpenAI GPT-4 is used for the generation AI, and data is analyzed and managed in the AWS cloud.

[2019] In this way, the system can improve the user's shopping experience and provide the necessary information in real time.

[2020] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2021] Step 1:

[2022] When a user puts on the smart glasses and turns them on, the smart glasses' internal software starts up and initializes the eye-tracking camera and environmental sensors. The input here is the smart glasses' startup signal, and the output is the initialization completion state. Specifically, the smart glasses' camera begins to track the user's gaze, and the sensors collect location information, illuminance, and audio data in real time.

[2023] Step 2:

[2024] Users log in to their profile using facial or voice recognition. The input is the user's authentication data, and the output is an authentication success or failure message. The device sends the authentication data to a cloud server, which retrieves the user's profile data from a database and analyzes their past usage history and preferences to predict what information they will need. This profile data is then sent back to the device.

[2025] Step 3:

[2026] When a user directs their gaze toward a product in a physical store, an eye-tracking camera detects the gaze information in real time. The input is gaze data, and the output is the identification information of the specific product. The device sends the gaze information and environmental data to a cloud server, which then analyzes the received data. Specifically, it determines that the gaze has been focused on a specific product.

[2027] Step 4:

[2028] The server analyzes gaze information and environmental data and uses a generative AI model to generate optimal information for the user (detailed product information, reviews, recommended items, etc.). The input is gaze information and environmental data, and the output is the generated product information. Data processing involves detecting which product the gaze is focused on, and the generative AI generates related information about that product.

[2029] Step 5:

[2030] The generated information is sent to the device and displayed in real time on the smart glasses display. The input is the generated information sent from the server, and the output is the information displayed in the field of view. Specifically, product prices, reviews, related products, etc. are displayed on the smart glasses screen.

[2031] Step 6:

[2032] The user provides feedback on the provided information, such as "it was helpful" or "it wasn't helpful." The input is the user's feedback, and the output is the feedback data. This feedback data is sent from the terminal to the server.

[2033] Step 7:

[2034] The server analyzes the received feedback data and updates and trains the generative AI model. The input is the feedback data, and the output is an improved generative AI model. As a data calculation, the model parameters are adjusted based on the feedback, improving the accuracy of information provided from the next time onwards.

[2035] In this way, the system improves the user's shopping experience and allows them to get the information they need instantly.

[2036] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2037] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Below, the program processing of this system is explained in natural language.

[2038] Program processing flow

[2039] Initialization and environment data collection

[2040] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera starts tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[2041] User Awareness and Customization

[2042] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[2043] Terminal: Sends the user's authentication data to the server.

[2044] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[2045] Terminal: Receives and applies profile settings sent from the server.

[2046] Eye tracking and information provision

[2047] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[2048] Terminal: Sends gaze information and environmental data to the server.

[2049] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[2050] Terminal: Displays the received information in the field of view and provides it to the user.

[2051] Emotion recognition and information regulation

[2052] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[2053] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[2054] Terminal: Displays tailored information in the user's field of view.

[2055] Specific examples

[2056] Example 1: Emotion recognition in shopping support

[2057] User: Looking for a product at a shopping mall in town.

[2058] Device: Detects when the user looks at a specific product. Furthermore, analyzes the user's facial expressions to recognize emotions such as interest or dissatisfaction.

[2059] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. If the user is interested, additional information such as special offers is also displayed.

[2060] Device: The generated information is displayed in the user's field of view. Users can check product prices, reviews, and related product information, and obtain information appropriate to their emotions.

[2061] Example 2: Emotion recognition in support of the visually impaired

[2062] User: A visually impaired user is walking around town.

[2063] Device: The device uses a camera to recognize the user's surroundings, collects information about obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[2064] Server: Generates optimal routes and points to be aware of based on environmental data and emotional information. If the user feels anxious, provides voice guidance to encourage and relax them.

[2065] Device: Provides users with directions, warnings, and emotional support through a voice assistant.

[2066] Real-time feedback and learning

[2067] User: Gives feedback on the information provided, such as "helpful" or "not helpful."

[2068] Terminal: Sends feedback data to the server.

[2069] Server: Analyzes feedback and updates generative AI models and emotion engines to improve their accuracy.

[2070] Highlighting important information

[2071] Server: When a serious obstacle or important information in the medical field is detected, it sends highlighting instructions to the terminal.

[2072] Device: Notify the user using sound, vibration, and visual enhancements.

[2073] In this way, the system of the present invention provides information that takes the user's emotions into consideration, thereby realizing a more personalized and accurate assistant function, enabling advanced assistance tailored to the individual needs of the user.

[2074] The processing flow will be explained below.

[2075] Specific processing flow of a program that combines emotion engines

[2076] Initialization and environment data collection

[2077] Step 1:

[2078] Device: When the smart glasses are turned on, the built-in software starts up and initializes the camera and sensors.

[2079] Step 2:

[2080] Device: The camera tracks the user's gaze, and built-in sensors capture real-time environmental data such as location, light, and sound.

[2081] User Awareness and Customization

[2082] Step 3:

[2083] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[2084] Step 4:

[2085] Terminal: Sends the user's authentication data to the server.

[2086] Step 5:

[2087] Server: Retrieves user profile data from a database and generates settings based on past usage history and preferences.

[2088] Step 6:

[2089] Server: Returns the generated profile settings to the device.

[2090] Step 7:

[2091] Terminal: Receives and applies profile settings sent from the server.

[2092] Eye tracking and information provision

[2093] Step 8:

[2094] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[2095] Step 9:

[2096] Terminal: Sends gaze information and environmental data to the server.

[2097] Step 10:

[2098] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object).

[2099] Step 11:

[2100] Server: Sends the generated information to the terminal.

[2101] Step 12:

[2102] Terminal: Displays the received information in the field of view and provides it to the user.

[2103] Emotion recognition and information regulation

[2104] Step 13:

[2105] Device: Uses a camera and microphone to monitor the user's facial expressions, voice tone, and body movements in real time.

[2106] Step 14:

[2107] Terminal: Analyzes the acquired data using an emotion engine means and estimates the user's emotions (joy, interest, stress, anxiety, etc.).

[2108] Step 15:

[2109] Terminal: Sends emotion information to the server.

[2110] Step 16:

[2111] Server: Uses emotional information to adjust the content and presentation of information provided to the user. For example, if the user is feeling stressed, the server displays information and suggestions to help them relax.

[2112] Step 17:

[2113] Terminal: Displays tailored information in the user's field of view.

[2114] Specific examples

[2115] Example 1: Emotion recognition in shopping support

[2116] Step 1:

[2117] User: Looking for a product at a shopping mall in town.

[2118] Step 8:

[2119] Device: Detects when the user looks at a specific product.

[2120] Step 9:

[2121] Terminal: Sends gaze information and environmental data to the server.

[2122] Step 10:

[2123] Server: Generates detailed product information, reviews, and recommended items. It also analyzes the user's facial expressions to recognize emotions such as interest and dissatisfaction.

[2124] Step 11:

[2125] Server: Adds and sends information that may interest the user, as well as special offers.

[2126] Step 12:

[2127] Terminal: The generated information is displayed in the user's field of view, allowing them to check product prices, reviews, and related product information.

[2128] Step 13:

[2129] Device: Recognizes the user's facial expressions and tone of voice and analyzes them with an emotion engine.

[2130] Step 14:

[2131] Terminal: Sends emotion information to the server.

[2132] Step 15:

[2133] Server: Uses emotional information to adapt the content of information provided to the user, for example, highlighting campaigns or discount information if the user shows interest.

[2134] Example 2: Emotion recognition in support of the visually impaired

[2135] Step 1:

[2136] User: A visually impaired user is walking around town.

[2137] Step 2:

[2138] Terminal: The device uses a camera to recognize the user's surroundings and collects information about obstacles and directions.

[2139] Step 9:

[2140] Terminal: Sends environmental data to the server.

[2141] Step 10:

[2142] Server: Generates optimal travel routes and points of interest from environmental data.

[2143] Step 11:

[2144] Server: Sends the generated information to the terminal.

[2145] Step 12:

[2146] Device: Provides directions to users through a voice assistant.

[2147] Step 13:

[2148] Device: Recognizes the user's vocal tone and physical reactions and analyzes them with an emotion engine.

[2149] Step 14:

[2150] Terminal: Sends emotion information to the server.

[2151] Step 15:

[2152] Server: Based on the emotional information, if the user feels anxious, it provides voice guidance to encourage and relax the user.

[2153] Step 17:

[2154] Terminal: Provides psychological support to the user using sound and vibration.

[2155] These steps allow the system of the present invention to provide information that takes into account not only the user's gaze information and environmental data, but also their emotional information, thereby providing more personalized and accurate assistance to the user.

[2156] Example 2

[2157] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2158] Current information delivery systems have difficulty tracking the user's gaze and environmental data in real time, and provide uniform information without considering the user's emotions, making it impossible to provide optimal information to the user. For this reason, there is a demand for more individually optimized, highly accurate information delivery methods that respond to the user's situation and emotions.

[2159] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2160] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, generation AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user, information display means for displaying the generated information in the user's field of view, emotion recognition means for monitoring the user's voice, facial expressions, and physical reactions and estimating emotions, and information adjustment means for adjusting the information provided to the user based on the emotion recognition means. This makes it possible to provide optimally personalized information taking into account the user's gaze, environmental data, and emotional state.

[2161] An "eye tracking device" is a device or software that tracks a user's eye gaze in real time and analyzes that movement.

[2162] "Environmental data collection means" refers to devices or software such as sensors, cameras, microphones, etc. that are used to acquire surrounding environmental data.

[2163] The "generative AI model means" refers to an artificial intelligence model and its execution environment that analyzes the user's gaze information and environmental data and generates information that is optimal for the user.

[2164] "Information display means" refers to a device or software for displaying the generated information in the user's field of vision, such as a head-mounted display or smart glasses.

[2165] An "emotion recognition means" is a device or software that monitors the user's voice, facial expressions, and physical reactions and infers the user's emotions based on them.

[2166] The "information adjustment means" is a device or software that adjusts the information provided to the user based on the emotion information obtained by the emotion recognition means.

[2167] "Learning model update means" refers to the functions and processes used to collect user feedback and improve the performance of the generative AI model based on that feedback.

[2168] A "user recognition means" is a device or software that authenticates a user and personalizes information based on the user's preferences or profile.

[2169] The present invention relates to a system that analyzes a user's gaze information and environmental data in real time, and further recognizes the user's emotions to provide individually optimized information. Specific embodiments of this system will be described below.

[2170] Initialization and environment data collection

[2171] Device: When the smart glasses are turned on, the built-in software launches and initializes the camera and sensors. The camera begins tracking the user's gaze, and the sensors collect real-time environmental data such as location, light level, and sound.

[2172] User Awareness and Customization

[2173] User: Wears smart glasses and logs in using facial recognition, voice recognition, or password entry.

[2174] Terminal: Sends the user's authentication data to the server.

[2175] Server: Retrieves user profile data from the database, generates configuration information based on past usage history and preferences, and sends it back to the device.

[2176] Terminal: Receives and applies profile settings sent from the server.

[2177] Eye tracking and information provision

[2178] Device: The camera tracks the user's gaze in real time and identifies what they are looking at.

[2179] Terminal: Sends gaze information and environmental data to the server.

[2180] Server: Analyzes the received gaze information and environmental data, and uses a generative AI model to create information related to the subject the user is looking at (such as product details, the history of a tourist spot, or a translation of an object). This information is then sent to the device.

[2181] Terminal: Displays the received information in the field of view and provides it to the user.

[2182] Emotion recognition and information regulation

[2183] Terminal: Monitors the user's voice, facial expressions, physical reactions, etc., and estimates the user's emotions using an emotion engine.

[2184] Server: Uses the emotion information from the emotion engine to adjust the generated information. For example, if the user is feeling stressed, it displays information and suggestions to help them relax.

[2185] Terminal: Displays tailored information in the user's field of view.

[2186] Examples of specific examples and prompts

[2187] Examples:

[2188] Example 1: Emotion recognition in shopping support

[2189] User: Looking for a product at a shopping mall in town.

[2190] Device: Detects when the user looks at a specific product and analyzes facial expressions to recognize emotions.

[2191] Server: Generates detailed product information, reviews, and recommended items, and displays information that matches the user's preferences. It also displays information about additional benefits.

[2192] Terminal: The generated information is displayed in the user's field of view. The user can check the product information and obtain the appropriate information.

[2193] Example 2: Emotion recognition in support of the visually impaired

[2194] User: A visually impaired user is walking around town.

[2195] Device: Recognizes the surrounding environment using a camera, collects information on obstacles and directions, and uses an emotion engine to estimate the user's psychological stress and anxiety.

[2196] Server: Generates optimal travel routes and points to note based on environmental data and emotional information, and provides relaxing audio guidance.

[2197] Device: Provides directions, warnings, and emotional support through a voice assistant.

[2198] Prompt Sentence Examples

[2199] "Show me the most popular products at the shopping malls in town."

[2200] "I'm lost, can you tell me the shortest route from my current location?"

[2201] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2202] Step 1:

[2203] Device: Power on your smart glasses.

[2204] Input: The smart glasses power switch is turned on.

[2205] Processing: The built-in software starts up and initializes the camera and sensors.

[2206] Output: The camera and sensors are ready to use.

[2207] Specific operation: The OS built into the smart glasses begins its boot sequence, loading each device driver, starting the camera, and environmental sensors (location sensor, illuminance sensor, sound sensor, etc.)

[2208] Step 2:

[2209] User: Wears smart glasses and logs in using facial recognition, voice recognition, password entry, etc.

[2210] Input: User's facial image data, voice data, and password.

[2211] Processing: Analyzes the input authentication data and executes authentication procedures.

[2212] Output: The result of authentication, success or failure.

[2213] How it works: The smart glasses use a camera to capture a facial image, which is then analyzed by a facial recognition algorithm. For voice recognition, a microphone records the user's voice and compares it with pre-registered voice patterns. For password entry, the user enters their password using the touchpad or voice input.

[2214] Step 3:

[2215] Terminal: Sends authentication data to the server.

[2216] Input: Authentication data.

[2217] Processing: The authentication data is sent using a communication protocol for sending the data to the server.

[2218] Output: Packet of authentication data to send to the server.

[2219] Specific operation: The smart glasses use wireless communication such as Wi-Fi or Bluetooth to send authentication data to the server.

[2220] Step 4:

[2221] Server: Receives authentication data and retrieves user profile information from a database.

[2222] Input: Authentication data sent from the terminal.

[2223] Processing: Query and retrieve the user's profile information from the database.

[2224] Output: User profile information.

[2225] Specific operation: The server receives the authentication data and sends a query to a database management system (DBMS) to obtain the corresponding user information.

[2226] Step 5:

[2227] Server: Generates configuration information based on the user's profile data and sends it to the device.

[2228] Input: User profile information.

[2229] Processing: Generates settings based on your past usage history and preferences.

[2230] Output: Configuration information to send to the device.

[2231] How it works: The server's algorithm analyzes the user's usage history and selects the settings that best suit the current usage situation. The selected information is then sent to the device using a communication protocol.

[2232] Step 6:

[2233] Terminal: Receives and applies profile settings sent from the server.

[2234] Input: Configuration information sent by the server.

[2235] Processing: Analyze the configuration information and apply it to the smart glasses' functions.

[2236] Output: The profile settings that were applied to the user.

[2237] Specific operation: Based on the received setting information, the display settings of the smart glasses and the initial settings of the application are performed.

[2238] Step 7:

[2239] Device: Tracks user gaze information in real time.

[2240] Input: Gaze data from the smartglasses camera.

[2241] Processing: Uses an eye-tracking algorithm to analyze the user's gaze direction.

[2242] Output: Real-time gaze data.

[2243] Specific operation: Analyzes the image data captured by the camera and calculates which direction the user is looking.

[2244] Step 8:

[2245] Terminal: Sends gaze information and environmental data to the server.

[2246] Input: Gaze data and environment data.

[2247] Processing: Gaze data and environmental data are packetized and sent to the server.

[2248] Output: Gaze data and environment data sent to the server.

[2249] Specific operation: Gaze data and environmental data are sent to the server via wireless communication.

[2250] Step 9:

[2251] Server: Analyzes gaze information and environmental data and generates related information.

[2252] Input: Gaze data and environment data.

[2253] Processing: Using a generative AI model, generate information related to the user's focus.

[2254] Output: Relevant information.

[2255] Specific operation: Using gaze data and environmental data as input, the generative AI model generates detailed information and recommendation information about the object.

[2256] Step 10:

[2257] Server: Sends the generated information to the terminal.

[2258] Input: Generated relevant information.

[2259] Processing: The generated information is packetized and sent to the terminal.

[2260] Output: Relevant information sent to the terminal.

[2261] Specific operation: The generated information is transmitted to the terminal via wireless communication.

[2262] Step 11:

[2263] Terminal: Displays the received information in the user's field of view.

[2264] Input: Relevant information sent by the server.

[2265] Processing: Display algorithms are used to properly display the information in the field of view.

[2266] Output: Information displayed in the field of view.

[2267] Specific operation: The generated information is overlaid on the smart glasses display.

[2268] Step 12:

[2269] Terminal: Monitors voice, facial expressions, and physical reactions to estimate emotions.

[2270] Input: User's voice data, facial expression data, and physical reaction data.

[2271] Processing: Emotion recognition algorithms are used to analyze this data and infer emotional states.

[2272] Output: User sentiment information.

[2273] Specific operation: Analyzes data collected by microphones, cameras, and sensors, and determines the user's emotions using an emotion estimation engine.

[2274] Step 13:

[2275] Server: Adjusts the information provided based on emotional information.

[2276] Input: Emotion information sent from the device.

[2277] Processing: Re-adjusting the information we generate based on emotional information.

[2278] Output: The adjusted information.

[2279] Specific operation: Analyzes emotional information and generates relevant information and encouraging messages to help users relax.

[2280] Step 14:

[2281] Terminal: Displays adjusted information in field of view.

[2282] Input: Adjusted information sent from the server.

[2283] Processing: Final processing is performed to display the adjusted information in the user's field of view.

[2284] Output: Calibration information displayed in the field of view.

[2285] Specific operation: Adjusted information is overlaid on the display to provide the user with the most appropriate information.

[2286] (Application example 2)

[2287] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2288] Conventional food delivery systems have difficulty considering the stress and anxiety that delivery personnel feel during deliveries, and have problems with providing optimal route guidance and important delivery-related information. This can lead to reduced efficiency and safety for delivery personnel, as well as reduced customer satisfaction. The present invention aims to solve these problems by providing optimal information to delivery personnel in real time, enabling them to perform delivery work efficiently and safely.

[2289] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2290] In this invention, the server includes gaze tracking means for tracking the user's gaze in real time, environmental data collection means for acquiring surrounding environmental data, emotion engine means for recognizing the user's emotions and adjusting information based on the emotions, generation AI model means for analyzing the user's gaze information, environmental data, and emotion information and generating optimal information for the user, and information display means for displaying the generated information in the user's field of view. This enables delivery personnel to obtain optimal route guidance and important information in real time, improving the efficiency and safety of delivery work.

[2291] An "eye tracking device" is a device or technology that tracks a user's gaze in real time and detects where the user is looking.

[2292] "Environmental data collection means" refers to devices or technologies that acquire data about the surrounding environment, including location information, illuminance, sound, etc.

[2293] "Emotion engine means" refers to a device or technology that analyzes the user's voice, facial expressions, and physical reactions and estimates the user's emotions.

[2294] The "generative AI model means" is an artificial intelligence model that analyzes gaze information, environmental data, and emotional information to generate information that is optimal for the user.

[2295] "Information display means" refers to a device or technology that displays the generated information in the user's field of vision.

[2296] "Learning model update means" is a machine learning technology that collects user feedback and improves the performance and accuracy of the generative AI model.

[2297] "User recognition means" refers to devices or technologies that authenticate users and personalize information based on their individual settings and profiles.

[2298] The system of the present invention uses a device such as smart glasses to collect a user's gaze and environmental data in real time, recognizes the user's emotions, and provides optimal information. A specific embodiment of this system will be described below.

[2299] Hardware and software used

[2300] Hardware:

[2301] Smart glasses: Equipped with a camera and various sensors (location information, light intensity, sound).

[2302] Server: Analyzes data and generates information.

[2303] software:

[2304] Eye-tracking software: Analyzes video data from the camera in real time to track gaze direction.

[2305] Environmental data collection software: Collects data from sensors and integrates it into environmental information.

[2306] Emotion engine: Analyzes the user's voice and facial expressions to estimate their emotional state.

[2307] Generative AI model: An artificial intelligence model that generates information based on gaze information, environmental data, and emotional information.

[2308] Information display software: Displays information optimally suited to the field of view of the smart glasses.

[2309] Data processing and calculation

[2310] The server receives gaze information, environmental data, and emotion information collected from the smart glasses, and processes these data in the following steps.

[2311] 1. Gaze analysis: Video data captured by the camera is analyzed using gaze tracking software to identify what the user is looking at.

[2312] 2. Environmental data analysis: Location information, illuminance, and audio data are collected and processed as environmental data.

[2313] 3. Emotion analysis: The emotion engine analyzes the user's voice and facial expression data to estimate their emotional state (e.g., stress or anxiety).

[2314] 4. Information generation: Based on gaze information, environmental data, and emotional state, the generative AI model generates optimal information. This information is created based on the prompt text.

[2315] 5. Information display: The generated information is displayed in the field of view of the smart glasses.

[2316] Specific examples

[2317] Example 1: Food delivery support

[2318] For delivery workers, the smart glasses collect real-time information about the surrounding environment during deliveries, providing optimal route guidance and detailed delivery destination information. Furthermore, an emotion engine detects stress and anxiety in the delivery worker and displays encouraging messages.

[2319] Example prompt sentence:

[2320] Collect visual and environmental data in front of your customer's home to provide optimal routes and important information to your delivery drivers during their delivery, as well as provide reassuring information if they are feeling anxious.

[2321] In this way, by using the system of the present invention, food delivery personnel can carry out their work efficiently and safely.

[2322] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2323] Step 1:

[2324] The device powers on the smart glasses and initializes the camera and sensors, allowing the camera to track the user's gaze and the sensors to capture real-time environmental data such as location, light, and audio.

[2325] Input: Power on the smart glasses

[2326] Output: Camera and sensor initialization complete

[2327] Step 2:

[2328] The user wears the smart glasses and logs in using facial or voice authentication. The results of the facial or voice authentication are sent from the device to the server.

[2329] Input: User's facial or voice authentication data

[2330] Output: User authentication result

[2331] Step 3:

[2332] The server receives the user's authentication data, retrieves the user's profile data from the database, and generates settings information based on the user's past usage history and preferences, which it then sends back to the device.

[2333] Input: Authentication data

[2334] Output: Profile data

[2335] Step 4:

[2336] The device receives profile setting information from the server and applies it to the smart glasses, completing customization for each user.

[2337] Input: Profile setting information

[2338] Output: Smart glasses customization completed

[2339] Step 5:

[2340] The device uses a camera to track the user's gaze in real time, identifying the direction of their gaze and the object they are looking at. The gaze information and environmental data are then sent to a server.

[2341] Input: Camera image

[2342] Output: Gaze information, environmental data

[2343] Step 6:

[2344] The server analyzes gaze information, environmental data, and user emotional information. The emotion engine estimates the user's emotions (e.g., stress, anxiety), and the generative AI model generates optimal information. The generated information is sent to the device.

[2345] Input: gaze information, environmental data, emotional information

[2346] Output: Best information

[2347] Step 7:

[2348] The device receives information from the server and displays it in the field of view of the smart glasses, such as delivery route and destination information.

[2349] Input: Best information

[2350] Output: Smart glasses visual field information

[2351] Step 8:

[2352] The user provides feedback on the provided information, such as "it was useful" or "it wasn't useful." The feedback data is sent from the terminal to the server.

[2353] Input: User feedback

[2354] Output: Feedback data

[2355] Step 9:

[2356] The server analyzes the feedback data and updates the learning model to improve the performance of the generative AI model, which is then reflected in the next information generation.

[2357] Input: Feedback data

[2358] Output: Updated generative AI model

[2359] In this way, through processing at each step, it is possible to provide users with the most appropriate information in real time, supporting efficient and safe food delivery.

[2360] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2361] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2362] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2363] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2364] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2365] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2366] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2367] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2368] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2369] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2370] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2371] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2372] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2373] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2374] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2375] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2376] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2377] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2378] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2379] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2380] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2381] The following is further disclosed regarding the above embodiment.

[2382] (Claim 1)

[2383] an eye tracking means for tracking the user's gaze in real time;

[2384] environmental data collection means for acquiring surrounding environmental data;

[2385] A generation AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user;

[2386] an information display means for displaying the generated information in the user's field of view;

[2387] A system including:

[2388] (Claim 2)

[2389] 10. The system of claim 1, further comprising a learning model update means for collecting user feedback and improving the performance of the generative AI model.

[2390] (Claim 3)

[2391] 10. The system of claim 1, further comprising a user recognition means for authenticating users and personalizing information based on user preferences and profiles.

[2392] (Claim 4)

[2393] 10. The system of claim 1, further comprising highlighting means for detecting important information and notifying the user by highlighting the information.

[2394] (Claim 5)

[2395] 2. The system according to claim 1, further comprising a voice notification means for providing guidance or warnings to the user using voice in real time.

[2396] "Example 1"

[2397] (Claim 1)

[2398] an eye tracking means for tracking the user's gaze in real time;

[2399] environmental data collection means for acquiring surrounding environmental data;

[2400] an information generating means for analyzing the user's line of sight information and environmental data and generating optimal information for the user;

[2401] an information display means for displaying the generated information in the user's field of view;

[2402] A user recognition method to authenticate u...

Claims

1. an eye tracking means for tracking the user's gaze in real time; environmental data collection means for acquiring surrounding environmental data; A generation AI model means for analyzing the user's gaze information and environmental data and generating optimal information for the user; an information display means for displaying the generated information in the user's field of view; A system including:

2. The system of claim 1 , further comprising a learning model update means for collecting user feedback and improving the performance of the generative AI model.

3. 10. The system of claim 1, further comprising a user recognition means for authenticating users and personalizing information based on user preferences and profiles.

4. 2. The system according to claim 1, further comprising highlighting means for detecting important information and notifying the user by highlighting the information.

5. 2. The system according to claim 1, further comprising a voice notification means for providing guidance or warnings to the user using voice in real time.

Images