System

Abstract

A system is provided.SOLUTION: A system, comprising: a wearable device including a camera and a sensor for tracking a gaze of a user; a gaze tracking unit configured to identify a position of the gaze of the user based on data acquired by the camera and the sensor; a unit configured to capture an image based on the position of the gaze identified by the gaze tracking unit; a unit configured to transmit the captured image to an external server; and a unit configured to analyze, by the external server, information based on the captured image, and personalize and provide the analyzed information to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's information society, users are required to quickly and efficiently obtain information tailored to their individual needs. However, conventional systems do not adequately provide personalized information for each user, causing significant inconvenience, particularly for the visually impaired and tourists. Furthermore, appropriate information cannot be obtained immediately in medical settings, work sites, and other settings, affecting the efficiency and accuracy of work. Given this current situation, there is an urgent need to provide a system that can provide appropriate information in real time based on the user's unique situation and line of sight. [Means for solving the problem]

[0005] The present invention provides a system including a wearable device equipped with a camera and a sensor for tracking a user's gaze, gaze tracking means for identifying the user's gaze position based on data acquired by the camera and the sensor, means for capturing images based on the gaze position identified by the gaze tracking means, means for transmitting the captured images to an external server, means for the external server to analyze information based on the captured images, and means for personalizing the analyzed information and providing it to the user. This allows users to accurately and quickly obtain information tailored to their individual needs even when in the same location or gaze position, and is adaptable to specific situations and for users with disabilities.

[0006] A "camera" is a photographic device that captures an image of an object in the user's line of sight.

[0007] A "sensor" is a device that detects a user's gaze and movements and collects data based on them.

[0008] "Eye tracking means" refers to a means for identifying the position of a user's gaze in real time based on data acquired by a camera and a sensor.

[0009] The "means for capturing an image" is a means for taking an image with a camera based on the gaze position identified by the gaze tracking means.

[0010] An "external server" is an external computer system that receives and analyzes the captured image data and generates personalized information.

[0011] The "analyzing means" is a means by which the server analyzes the content of the captured image and extracts necessary information.

[0012] "Personalization measures" are measures for customizing analyzed information to suit the individual needs and preferences of users.

[0013] A "wearable device" is a device that can be worn by a user and has a built-in camera and sensor. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Specifically, a wearable device equipped with a camera and sensors is used to track the user's gaze in real time, capture images based on the gaze position, and transmit them to an external server. The external server analyzes the images, generates and personalizes the necessary information, and provides it to the user in an appropriate format.

[0036] User Recognition and Eye Tracking

[0037] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[0038] Gaze data collection and image capture

[0039] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0040] Image analysis and information generation

[0041] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[0042] Personalized information provision

[0043] Based on the information analyzed by the server, personalized information is generated according to the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[0044] Example: Tourism support

[0045] Consider a case where a user visits a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[0046] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The user puts on the wearable device (smart glasses). The smart glasses start up and the camera and sensors begin operating.

[0050] Step 2:

[0051] The device identifies the user ID. Based on the user ID, it loads individual settings and history data. This data is necessary to provide information tailored to the user's past usage history and preferences.

[0052] Step 3:

[0053] The device uses a camera and sensors to track the user's gaze position in real time. The gaze tracking means sequentially acquires gaze position data and captures the position information.

[0054] Step 4:

[0055] The device captures an image of the object using the camera based on the gaze position, and the captured image data is temporarily stored in the device.

[0056] Step 5:

[0057] The device sends the captured image data to an external server, including gaze position information.

[0058] Step 6:

[0059] The server analyzes the received image data and uses advanced image recognition algorithms to identify objects and extract relevant information, such as the history and characteristics of statues at tourist attractions.

[0060] Step 7:

[0061] The server uses the analysis results to generate personalized information based on the user's history and settings, making it possible to provide information tailored to the individual needs and preferences of each user.

[0062] Step 8:

[0063] The server sends the generated information to the device, which then displays the received personalized information on the smart glasses display.

[0064] Step 9:

[0065] If the device is for the visually impaired, it will not only provide visual feedback but also audio feedback, conveying the necessary information to the user through audio guides.

[0066] Step 10:

[0067] Users review information and act on it, for example, learning about the history of a statue at a tourist spot and choosing a sightseeing route around the area.

[0068] This process allows users to obtain personalized information based on their gaze in real time, and audio information can be provided to visually impaired people, making it suitable for a wide range of users.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] Conventional eye-tracking systems have difficulty providing information about the object a user is looking at instantly and in line with individual needs. They also have difficulty supporting visually impaired users and lack real-time information provision. There is a need for a system that can resolve these issues and provide more convenient and useful information to users.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes means for analyzing information based on the captured image using an advanced image recognition algorithm, means for providing the analyzed information in a personalized manner according to the user's history and settings, and means for providing the analyzed information as audio feedback to the user, thereby enabling real-time personalized information provision based on the user's gaze.

[0074] - "Camera" means a device that converts light into electrical signals and is used to capture an image of an object in the user's line of sight.

[0075] A "sensor" is a device that detects physical phenomena and converts the signals into electrical signals, and is used to detect things like the user's line of sight.

[0076] "Wearable device" refers to any digital device that is worn on the body, and in this invention it primarily refers to devices with built-in cameras and sensors.

[0077] "Eye tracking means" refers to technology or devices that detect the movement of a user's eyes and identify the position of their gaze, and in this invention includes sensors and dedicated software algorithms.

[0078] "Image capturing means" means a function or device that takes an image of an object that a user is looking at based on an eye-tracking means.

[0079] "External Server" means a computer system that receives data sent from a wearable device and performs advanced analysis.

[0080] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying objects and textual information present within it.

[0081] "Personalization" means customizing information based on each user's history and preferences, and presenting it in a way that is most appropriate for the person providing it.

[0082] "Voice feedback" refers to a means of transmitting information that provides analyzed information to the user as audio, and is particularly suitable for visually impaired people.

[0083] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. This system uses a wearable device equipped with a camera and sensors to track the user's gaze in real time. As a result, an image is captured based on the gaze position and sent to an external server. The external server analyzes the image, generates the necessary information, personalizes it, and provides it to the user in an appropriate format.

[0084] User Recognition and Eye Tracking

[0085] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[0086] Gaze data collection and image capture

[0087] The device uses eye tracking to determine the user's gaze position. Specifically, sensors in the smart glasses detect the user's eye movements in real time and calculate the gaze position. Based on the determined gaze position, the device's camera automatically captures an image of the object at that position. This image is sent to an external server in real time.

[0088] Image analysis and information generation

[0089] The transmitted image data is received by an external server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the gaze. For example, if the image is a statue at a tourist spot, its historical background and related interesting facts can be extracted. Specific technologies used include Google Cloud Vision API and Microsoft Azure Computer Vision.

[0090] Personalized information provision

[0091] Based on the information analyzed by the server, personalized information is generated based on the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[0092] Example: Tourism support

[0093] Consider the example of a user visiting a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[0094] Example prompts for generative AI models

[0095] Example of input prompt:

[0096] "Describe a system that provides customized information about a specific object when the user is looking at that object. What hardware and software does this system use and how does it generate the information?"

[0097] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[0098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0099] Step 1:

[0100] User recognition and device initialization

[0101] When a user wears the smart glasses, the device uses the camera to recognize the user's face and uses a facial recognition algorithm (e.g., OpenCV's facial recognition function) to identify the user's ID.

[0102] Input: User's face image

[0103] Data processing: Facial feature points are extracted using a facial recognition algorithm, and the corresponding user ID is searched for in the database.

[0104] Output: User ID

[0105] What it does: The smart glasses' camera captures a picture of the user's face, applies a facial recognition algorithm, and, based on the identified user ID, loads the user's settings and history data from cloud storage or the glasses' internal memory.

[0106] Step 2:

[0107] Eye tracking and data collection

[0108] The device determines the user's gaze position using the eye tracking sensor in the smart glasses, which detects the user's eye movements in real time.

[0109] Input: User's eye movement data

[0110] Data processing: Eye tracking algorithm calculates eye gaze direction

[0111] Output: Gaze direction and position information

[0112] How it works: The eye tracking sensor detects the movement of the user's eyes, and the eye tracking algorithm processes that data to determine the position of the gaze.

[0113] Step 3:

[0114] Image capture and transmission

[0115] Based on the results of gaze tracking, the device's camera captures an image of the object in front of the user's eyes, which is then compressed, encrypted, and sent to an external server.

[0116] Input: gaze position information, image of object

[0117] Data processing: image capture, compression (e.g., JPEG format), encryption (e.g., SSL)

[0118] Output: Compressed and encrypted image data

[0119] Specific operation: When a fixed gaze is detected, the camera automatically releases the shutter and captures an image. The image is then compressed into JPEG format, encrypted using SSL, and sent to the server.

[0120] Step 4:

[0121] Image analysis and information generation

[0122] The server receives the transmitted image data and analyzes the image using advanced image recognition algorithms (e.g., Google Cloud Vision API or Microsoft Azure Computer Vision).

[0123] Input: Encrypted image data

[0124] Data processing: Decrypting data, applying image recognition algorithms, and identifying objects

[0125] Output: Information about the object (e.g., history, characteristics)

[0126] Specific operation: The received image data is decoded, and an image recognition API is called to identify the object and extract its information.

[0127] Step 5:

[0128] Personalized information provision

[0129] The server analyzes the information and generates personalized information based on the user's history and settings, which is then sent to the device and displayed visually on the smart glasses display or provided as audio feedback.

[0130] Input: Information about the object, user history and settings information

[0131] Data processing: Extraction of relevant information, personalized processing

[0132] Output: Personalized information (visual / audio)

[0133] What it does: It matches the analyzed information with the user's past history and preferences to select the most relevant information, and then provides that information as an AR display or audio guide.

[0134] Through the above process, this system provides real-time, personalized information based on the user's line of sight.

[0135] (Application example 1)

[0136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] Conventional wearable devices and eye-tracking technologies focused on tracking the user's gaze, but lacked the ability to instantly provide detailed information about the object in front of the user's eyes. Furthermore, there were insufficient means to provide real-time information about products that users were interested in in physical stores. Furthermore, voice feedback functions for the visually impaired were not fully developed, which led to issues with the efficiency of information acquisition.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0139] In this invention, the server includes a wearable device equipped with a camera and a sensor for tracking the user's gaze, gaze tracking means for identifying the user's gaze position based on data acquired by the camera and the sensor, means for capturing an image based on the gaze position identified by the gaze tracking means, means for transmitting the captured image to an external server, means for the external server to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, and means for providing information on products the user is looking at in a physical store in real time through a display device of smart glasses. This allows users to obtain information on products they are interested in in a physical store in real time, and also makes it possible to provide information to visually impaired people via voice feedback.

[0140] "Eye tracking" is a technology that tracks and identifies the position of a user's gaze.

[0141] A "wearable device" is an electronic device that is designed to be worn by the user.

[0142] "Cameras and Sensors" means the imaging devices and detectors used to capture user gaze data.

[0143] "Capturing an image" means taking a picture of an object at a particular line of sight.

[0144] An "external server" is a remote computer system that can be communicated with via the Internet.

[0145] "Analyzing information" means identifying details and features of an object based on the acquired data and extracting relevant information.

[0146] "Personalization" means adapting acquired information to the preferences and history of individual users.

[0147] "Blind people" means people who are visually impaired.

[0148] "Smart glasses" are a type of wearable eyeglass device that displays information based on the wearer's line of sight.

[0149] "Voice feedback" refers to a method of providing acquired information to the user in the form of voice.

[0150] The system for realizing this invention uses smart glasses as a wearable device worn by the user. The smart glasses have a built-in camera and sensor, and are equipped with an eye-tracking function that tracks the user's gaze. Below, we will explain the specific hardware and software used in each step, as well as the data processing and data calculation methods using them.

[0151] Hardware and software used

[0152] Hardware:

[0153] Smart glasses (with built-in camera and sensors)

[0154] Cloud server (general remote computer)

[0155] software:

[0156] Image recognition algorithms (TensorFlow, OpenCV)

[0157] User history management system (database)

[0158] Smart Glasses Display Control Software

[0159] Data Processing and Data Calculation

[0160] 1. Eye Tracking:

[0161] When a user wears the smart glasses, the sensors and cameras in the smart glasses track the user's gaze and identify the gaze position in real time. The gaze data is processed by a processor in the smart glasses and recorded as gaze position information.

[0162] 2. Image capture and transmission:

[0163] Based on the identified gaze position, the camera automatically captures an image of the object in front of the gaze, and the captured image is transmitted to an external cloud server via wireless communication.

[0164] 3. Image analysis and information extraction:

[0165] The server analyzes the received image data using advanced image recognition algorithms (TensorFlow, OpenCV) to extract information about the object in front of the gaze. For example, in the case of a product in a physical store, information such as its price, ingredients, and user reviews can be extracted.

[0166] 4. Personalization and Information:

[0167] The server then personalizes the extracted information based on the user's past purchase history and preferences (managed in a database). The personalized information is displayed on the smart glasses' display or provided as audio feedback. It is also possible to provide the user with appropriate suggestions (recommended products or promotional information) based on specific prompts.

[0168] Specific examples

[0169] Imagine a user browsing a store shelf. When the user directs their gaze at a particular product, the smartglasses' camera captures an image of that product and sends it to a cloud server. The server analyzes the received image and extracts the product's price, ingredients, and user reviews. Based on the user's past purchase history and preferences, the smartglasses display then displays personalized information, such as "This product is recommended based on your past purchase history."

[0170] Prompt Sentence Examples

[0171] When a user wants to know more about this product, a prompt like this is used:

[0172] "Track the user's gaze and capture the product they are looking at. Send the captured image to a server to retrieve product information. Customize and display the information based on the user's preferences."

[0173] Based on this prompt, the system will identify product information that interests the user and provide personalized information.

[0174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0175] Step 1:

[0176] The user puts on the smart glasses and activates the sensors and camera. The eye tracking system collects the user's gaze data in real time. Specifically, the camera captures the user's eye movements, and the sensor identifies the gaze position. Input: User's eye movements. Output: User's gaze data (gaze position information).

[0177] Step 2:

[0178] The device identifies the position of the object in front of the user's gaze based on the collected gaze data. Then, it uses the camera to capture an image of the object at that identified position. Input: User's gaze data. Output: Captured image.

[0179] Step 3:

[0180] The captured image is wirelessly transmitted from the device to an external server. Specifically, the image data is uploaded to a cloud server via a wireless communication module. Input: Captured image. Output: Image data stored on the server.

[0181] Step 4:

[0182] Based on the received image data, the server uses an image recognition algorithm (e.g., TensorFlow, OpenCV) to recognize the object in front of the user's line of sight and analyzes the information. Specifically, it extracts information such as the product name, price, ingredients, and user reviews. Input: Image data stored on the server. Output: Detailed information about the object.

[0183] Step 5:

[0184] The server uses the analyzed information to query a database of the user's history and preferences to personalize the information. For example, it generates related information and product recommendations based on the user's preferences and the product history of previous purchases. Input: Detailed information about the object, user history data. Output: Personalized information.

[0185] Step 6:

[0186] The server sends the personalized information to the device. Specifically, the information is sent to the smart glasses via wireless communication, where it is displayed visually on the smart glasses' display. In some cases, the information can also be provided as an audio guide. Input: Personalized information. Output: Smart glasses' display or audio guide.

[0187] Step 7:

[0188] Users can check the information displayed on the smart glasses display and obtain information about products and promotions that interest them. For visually impaired people, information is obtained through audio guidance. Input: Smart glasses display, audio guidance. Output: Information obtained by the user.

[0189] This series of processes allows users to obtain detailed product information and personalized recommendations in real time simply by looking at the product in a physical store.

[0190] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0191] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can further personalize the information provided and provide support tailored to the user's psychological state.

[0192] User Recognition and Eye Tracking

[0193] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[0194] Gaze data collection and image capture

[0195] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0196] Image analysis and information generation

[0197] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[0198] Emotion recognition by emotion engine

[0199] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[0200] Personalized information provision

[0201] Based on the information analyzed by the server, the system generates personalized information based on the user's history, settings, and emotional state using an emotion engine. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smartglasses' display or provide it as audio feedback.

[0202] Examples: Tourist assistance and emotional support

[0203] Consider a case where a user visits a famous tourist spot. As the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and artist. The analyzed information is then customized based on the user's past visits and interests and displayed visually on the smart glasses.

[0204] Furthermore, the emotion engine analyzes the user's emotional state and, for example, if they feel tired, provides the location of a nearby bench. For visually impaired users, this information is provided via audio guidance, and feedback is given according to their emotions.

[0205] In this way, users can receive personalized information based on the object they are looking at, while simultaneously receiving psychological support from the emotion engine. This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used in a wide range of applications, including medical and workplace settings, in addition to the visually impaired and tourists. This is expected to significantly improve the efficiency of users' daily lives and work.

[0206] The processing flow will be explained below.

[0207] Step 1:

[0208] The user puts on the wearable device (smart glasses). The device starts up and the camera and sensors begin operating.

[0209] Step 2:

[0210] Identify user ID: The device recognizes the user ID and loads individual settings and history data based on that ID.

[0211] Step 3:

[0212] Using eye tracking, the device tracks the user's eye position in real time, sequentially acquiring eye position data and capturing that position information.

[0213] Step 4:

[0214] Based on the gaze position, the device's camera automatically captures an image of the object at that position, and the captured image data is temporarily stored on the device.

[0215] Step 5:

[0216] The device sends the captured image data to an external server, including gaze position information.

[0217] Step 6:

[0218] The server analyzes the received image data. Using advanced image recognition algorithms, it identifies the objects and extracts relevant information. For example, the server analyzes the history and characteristics of statues at tourist attractions.

[0219] Step 7:

[0220] Using an emotion engine, the device recognizes the user's emotional state by analyzing the user's heart rate and facial expressions in real time based on gaze data and data from biometric sensors.

[0221] Step 8:

[0222] Based on the information analyzed by the server, personalized information is generated based on the user's history, settings, and emotional state using an emotion engine, including providing information according to the user's psychological state.

[0223] Step 9:

[0224] The server sends the generated personalized information to the device, which then visually displays the information on the smartglasses display.

[0225] Step 10:

[0226] For the visually impaired, the device provides audio feedback, conveying necessary information and emotional feedback to the user through audio guidance.

[0227] Step 11:

[0228] Users review information through displays or audio guides and use that information to make decisions about what to do at a tourist attraction, such as learning about the history of a statue or finding the location of a nearby bench.

[0229] This process allows users to obtain personalized information based on their gaze in real time, and with the support of an emotion engine, appropriate feedback tailored to their psychological state is provided, improving user satisfaction and enabling a wide range of applications.

[0230] Example 2

[0231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional wearable devices are limited to providing information by tracking the user's gaze, making it difficult to provide personalized information tailored to the user's emotional state. Furthermore, their support for visually impaired people is insufficient, so an improvement in the user experience is needed.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0234] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[0235] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[0236] means for capturing an image based on the gaze position identified by the gaze tracking means;

[0237] means for transmitting the captured image to an external server;

[0238] means for the external server to analyze information based on the captured image;

[0239] A means for personalizing the analyzed information and providing it to the user;

[0240] an emotion recognition means for identifying an emotional state of a user based on biometric data;

[0241] a means for personalizing information provided based on the user's emotional state;

[0242] a means for conveying the provided information to the user as a display or audio feedback;

[0243] This allows for personalized information to be provided based on the user's emotional state and the object in their line of sight.

[0244] "Eye tracking" is a feature that uses cameras and sensors to determine the position of a user's gaze in real time.

[0245] "Wearable devices" generally refer to electronic devices that can be worn by the user on the body, including smart glasses.

[0246] "Cameras and Sensors" means the imaging and detection devices used to track the user's gaze.

[0247] "Image capture" refers to the act of using a camera to obtain a still image of an object at a particular gaze position.

[0248] A "server" is a computer system that stores, analyzes, and provides data over a network.

[0249] "Information analysis" refers to the process of processing received data to extract useful information.

[0250] "Personalization" means providing information that is optimized based on the user's individual settings, history, and emotional state.

[0251] "Emotion recognition" is a technology that analyzes a user's biometric data to identify their current emotional state.

[0252] "Biometric data" means physiological data such as a user's heart rate, facial expression, and skin temperature.

[0253] A "display" is a screen device for presenting visual information to a user.

[0254] "Audio feedback" is a means of conveying information to the user through sound.

[0255] "Visually impaired people" refers to people who have problems with their eyesight and have difficulty obtaining normal visual information.

[0256] MODE FOR CARRYING OUT THE INVENTION

[0257] The present invention relates to a system that tracks a user's gaze and provides customized information based on the object in their line of sight. Furthermore, by combining it with an emotion engine, the system provides personalized information tailored to the user's emotional state.

[0258] User Recognition and Eye Tracking

[0259] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[0260] Gaze data collection and image capture

[0261] The device uses eye tracking to identify the user's gaze position. When the gaze is fixed at a specific position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0262] Image analysis and information generation

[0263] The transmitted image data is received by a server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the user's gaze. For example, if the user is looking at a statue at a tourist attraction, the server can extract the statue's historical background and related interesting facts. This analysis is performed using a generative AI model.

[0264] Emotion recognition by emotion engine

[0265] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[0266] Personalized information provision

[0267] The server then uses the analyzed information to generate personalized information based on the user's history, settings, and emotional state as determined by the emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide information about the statue's historical background and recommended sightseeing routes. The device then displays this information on the smart glasses' display or provides it as audio feedback.

[0268] Examples: Tourist assistance and emotional support

[0269] Specific examples

[0270] When a user visits a famous tourist spot and looks at a statue, the device tracks the user's gaze, capturing an image of the statue and sending it to a server. The server then analyzes the statue's detailed information, extracting data about its history and artist, for example. This information is customized based on the user's past visits and interests and displayed visually on the smart glasses. Furthermore, an emotion engine analyzes the user's emotional state, providing the location of a nearby bench if, for example, they are feeling tired. Visually impaired users can also receive this information through audio guidance and emotional feedback.

[0271] Prompt Sentence Examples

[0272] If the user wants to get information, they can enter a prompt like this:

[0273] "Please tell me more about the statue I'm looking at. I'd like to know its historical background and who made it."

[0274] This system allows users to receive personalized information based on the object in front of their eyes, while also receiving assistance from an emotion engine. This technology is expected to contribute not only to tourism support, but also to support for the visually impaired, medical care, and improving efficiency in various workplaces.

[0275] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0276] Step 1: User Awareness

[0277] When a user puts on a wearable device (smart glasses), the device uses its built-in camera and sensors to identify the user's ID. The device then uses facial recognition technology to identify the user's ID and reads the user's unique settings and history data from its built-in memory. This makes it possible to provide information that takes into account the user's preferences and past usage history.

[0278] input:

[0279] User's facial image, biometric data

[0280] Data processing / calculation:

[0281] It uses a facial recognition algorithm to identify the user ID and loads corresponding settings and history data.

[0282] output:

[0283] User ID, user settings and history data

[0284] Specific behavior:

[0285] The device uses a facial recognition sensor to capture the user's facial image and verify their identity by comparing it with an internal database.

[0286] Step 2: Eye tracking

[0287] The sensor tracks the user's eye movements in real time and identifies the position of the gaze. The device uses an infrared sensor to detect the user's gaze position and updates the data in real time, thereby identifying the position where the gaze is fixed.

[0288] input:

[0289] User eye position data

[0290] Data processing / calculation:

[0291] Uses an eye-tracking algorithm to determine the user's gaze position.

[0292] output:

[0293] Gaze position data

[0294] Specific behavior:

[0295] The device's infrared sensor detects the user's eye movements, analyzes the data, and displays the gaze position in real time.

[0296] Step 3: Collecting gaze data and capturing images

[0297] When a user's gaze is fixed at a specific location, the device automatically captures an image of the object at that location using its camera, and this image data is collected in real time and sent to an external server.

[0298] input:

[0299] Gaze position data

[0300] Data processing / calculation:

[0301] Controls the camera based on gaze position to capture images of the target object

[0302] output:

[0303] Captured image data

[0304] Specific behavior:

[0305] The camera automatically focuses on the position where the gaze is fixed and captures an image of the object.

[0306] Step 4: Sending image data

[0307] The captured image data is sent to an external server using a secure communication protocol. The device sends the image data to the server using HTTPS.

[0308] input:

[0309] Captured image data

[0310] Data processing / calculation:

[0311] Data encryption and transmission

[0312] output:

[0313] Image data sent to the server

[0314] Specific behavior:

[0315] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[0316] Step 5: Image analysis

[0317] The server analyzes the received image data and extracts information about the object in the line of sight. Using a generative AI model, it analyzes detailed information about the object in the line of sight.

[0318] input:

[0319] Received image data

[0320] Data processing / calculation:

[0321] Extract information using image recognition algorithms

[0322] output:

[0323] Analyzed object information

[0324] Specific behavior:

[0325] The server runs a generative AI model to extract detailed information and related data about the object.

[0326] Step 6: Emotion Recognition

[0327] The device analyzes the user's biometric data (heart rate, facial expressions, etc.) collected by the device, and the emotion engine identifies the user's emotional state. The emotion engine analyzes the user's biometric data in real time to determine the user's emotional state.

[0328] input:

[0329] User biometric data

[0330] Data processing / calculation:

[0331] Identify emotional states using emotion recognition algorithms

[0332] output:

[0333] The user's emotional state

[0334] Specific behavior:

[0335] The device analyzes data from a heart rate monitor and facial recognition sensors to determine whether the user is relaxed or tired.

[0336] Step 7: Personalized information generation

[0337] The server then uses the analyzed information to generate personalized information based on the user's history, preferences, and emotional state, as determined by an emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide historical background about the statue and recommend sightseeing routes.

[0338] input:

[0339] Analyzed object information, user emotional state, historical data and settings

[0340] Data processing / calculation:

[0341] Data Integration and Information Generation

[0342] output:

[0343] Personalized Information

[0344] Specific behavior:

[0345] The server integrates this data to generate customized information. The generative AI model is given a prompt: "Please tell me more about the statue I'm looking at. I'd also like to know its historical background and its creator.", and the required data is generated.

[0346] Step 8: Provide information

[0347] The device provides the generated information to the user, either visually displayed on the smart glasses display or as audio feedback.

[0348] input:

[0349] Personalized Information

[0350] Data processing / calculation:

[0351] Display format conversion

[0352] output:

[0353] Visual or audio information provided to the user

[0354] Specific behavior:

[0355] The device displays text about the statue's historical background and simultaneously provides audio feedback explaining the statue's creator.

[0356] (Application example 2)

[0357] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0358] In current factory work, workers must manually search or refer to manuals to obtain information about individual parts and tools. This procedure is inefficient and increases the burden on workers. In addition, there is a lack of a system to detect workers' stress and fatigue in real time and provide appropriate breaks and advice. As a result, work efficiency decreases and there is a risk of work errors and safety issues.

[0359] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a wearable device equipped with an optical device and a detection device for tracking the user's gaze, an gaze tracking device for identifying the user's gaze position based on data acquired by the optical device and the detection device, means for capturing an image based on the gaze position identified by the gaze tracking device, means for transmitting the captured image to an external installation device, means for the external installation device to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, an emotion recognition device for identifying the user's emotional state, and means for providing advice and work improvement suggestions to the user based on the emotional state identified by the emotion recognition device. This enables the worker to obtain information on the parts and tools in front of the worker's gaze in real time and to receive appropriate breaks and advice according to the worker's emotional state, thereby improving work efficiency and ensuring safety.

[0360] An "optical device" is a device that handles light, such as a camera or lens, to detect the user's line of sight.

[0361] A "detection device" is a sensor or device that acquires gaze data or biometric data.

[0362] A "wearable device" is a digital device that can be worn by the user, examples of which include smart glasses and head-mounted displays.

[0363] An "eye tracking device" is a system that tracks the position of a user's gaze in real time.

[0364] "Means for capturing" refers to a function for acquiring an image at a specified gaze position.

[0365] An "external device" is a server or cloud service that receives and analyzes data sent from a wearable device.

[0366] "Means for analyzing information" refers to the algorithms or software that analyze the captured image data and extract information about the object in the user's line of sight.

[0367] "Means for providing personalized information" refers to a function that customizes analyzed information based on the user's history and settings, and provides it in the most optimal form for the user.

[0368] An "emotion recognition device" refers to a sensor or analysis device used to detect a user's emotional state.

[0369] The "means for providing advice and work improvement suggestions" is a function that presents optimal breaks and suggestions for improving work procedures based on the user's emotional state detected by the emotion recognition device.

[0370] An embodiment of the present invention will be described. This system uses a head-mounted display (HMD) as a wearable device worn by a factory worker. The main elements and their operations are shown below.

[0371] Hardware and software used

[0372] 1. Wearable device (HMD): A device equipped with optical and detection devices to track the user's gaze.

[0373] 2. Eye tracking device: A system that uses the dlib library and other tools to track the user's gaze position in real time.

[0374] 3. External device (server): This device receives the captured image data and analyzes the information. TensorFlow is used for this analysis.

[0375] 4. Emotion Recognition Device: Used to analyze camera footage and identify the user's emotional state. Here, we use an emotion recognition model built with TensorFlow.

[0376] Data handling and calculations

[0377] 1. Gaze Tracking: The HMD camera captures the user's face and uses the dlib library to detect facial landmarks. The gaze direction is calculated based on these landmarks.

[0378] 2. Image capture and transmission: Images of the object are automatically captured based on the direction of gaze and transmitted to an external server, where they are analyzed in real time.

[0379] 3. Analysis and provision of information: The server analyzes the received image data using TensorFlow and extracts information about the parts and tools in front of the user's gaze. This information is then customized based on the user's history and settings and displayed on the HMD.

[0380] 4. Identifying and responding to emotional states: The HMD is equipped with an emotion recognition device that analyzes the user's facial expressions, heart rate, etc. to identify their emotional state. For example, if the user is feeling stressed, the system will suggest taking a break.

[0381] Specific examples

[0382] When a factory worker looks at a part of the robot arm they are operating, the HMD visually displays the manual and work procedures related to that part. Also, if the HMD detects that the worker is under stress, it will display instructions to take a break.

[0383] Example prompts for generative AI models

[0384] "Factory workers wear head-mounted displays and work while tracking their gaze. Generate an application program that displays information about parts and tools in front of their eyes in real time, analyzes the worker's emotional state, and suggests taking a break if the worker feels fatigued or stressed."

[0385] In this way, the system provides real-time information on the line of sight of factory workers and gives appropriate advice according to their emotional state, thereby improving work efficiency and ensuring safety.

[0386] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0387] Step 1:

[0388] The user wears a wearable device (HMD). The HMD has built-in optical and detection devices, which enable gaze tracking and emotion recognition. The input is the user's facial expression and gaze data, and the output is the capture of this data.

[0389] Step 2:

[0390] The device uses an eye-tracking device to track the user's gaze in real time. The dlib library is used to calculate the user's gaze direction from the captured facial landmark information. The input is the captured facial image data, and the output is the calculated gaze direction.

[0391] Step 3:

[0392] The gaze tracking device captures an image of the object in front of the gaze. The captured image is sent to an external device (server). The input is the calculated gaze direction and image data of the gaze destination, and the output is data sent to the server.

[0393] Step 4:

[0394] The server receives images sent from an external source and uses TensorFlow to analyze information about parts and tools in the line of sight. The analyzed information is personalized based on the user's history and settings. The input is the captured image data, and the output is the analyzed information about the line of sight.

[0395] Step 5:

[0396] The server sends the analyzed information to the HMD and displays it visually. The user can check detailed information about parts and tools in real time through the HMD. The input is the analyzed information, and the output is the information displayed on the HMD.

[0397] Step 6:

[0398] The device uses an emotion recognition device to analyze biometric data such as the user's facial expressions and heart rate to identify their emotional state. An emotion recognition model built with TensorFlow is used. The input is the user's biometric data, and the output is the identified emotional state.

[0399] Step 7:

[0400] The server sends appropriate advice and work improvement suggestions to the HMD based on the emotional state identified by the emotion recognition device. For example, if the user is recognized as being in a stressful state, a message suggesting a break will be displayed. The input is the identified emotional state, and the output is the advice displayed on the HMD.

[0401] Step 8:

[0402] The user performs the work according to the information and advice displayed through the HMD, which improves work efficiency and ensures safety. The input is the information displayed by the HMD, and the output is the user's work behavior.

[0403] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0404] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0405] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0406] [Second embodiment]

[0407] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0408] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0409] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0410] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0411] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0413] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0414] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0415] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0416] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0417] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0418] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0419] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Specifically, a wearable device equipped with a camera and sensors is used to track the user's gaze in real time, capture images based on the gaze position, and transmit them to an external server. The external server analyzes the images, generates and personalizes the necessary information, and provides it to the user in an appropriate format.

[0420] User Recognition and Eye Tracking

[0421] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[0422] Gaze data collection and image capture

[0423] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0424] Image analysis and information generation

[0425] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[0426] Personalized information provision

[0427] Based on the information analyzed by the server, personalized information is generated according to the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[0428] Example: Tourism support

[0429] Consider a case where a user visits a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[0430] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[0431] The processing flow will be explained below.

[0432] Step 1:

[0433] The user puts on the wearable device (smart glasses). The smart glasses start up and the camera and sensors begin operating.

[0434] Step 2:

[0435] The device identifies the user ID. Based on the user ID, it loads individual settings and history data. This data is necessary to provide information tailored to the user's past usage history and preferences.

[0436] Step 3:

[0437] The device uses a camera and sensors to track the user's gaze position in real time. The gaze tracking means sequentially acquires gaze position data and captures the position information.

[0438] Step 4:

[0439] The device captures an image of the object using the camera based on the gaze position, and the captured image data is temporarily stored in the device.

[0440] Step 5:

[0441] The device sends the captured image data to an external server, including gaze position information.

[0442] Step 6:

[0443] The server analyzes the received image data and uses advanced image recognition algorithms to identify objects and extract relevant information, such as the history and characteristics of statues at tourist attractions.

[0444] Step 7:

[0445] The server uses the analysis results to generate personalized information based on the user's history and settings, making it possible to provide information tailored to the individual needs and preferences of each user.

[0446] Step 8:

[0447] The server sends the generated information to the device, which then displays the received personalized information on the smart glasses display.

[0448] Step 9:

[0449] If the device is for the visually impaired, it will not only provide visual feedback but also audio feedback, conveying the necessary information to the user through audio guides.

[0450] Step 10:

[0451] Users review information and act on it, for example, learning about the history of a statue at a tourist spot and choosing a sightseeing route around the area.

[0452] This process allows users to obtain personalized information based on their gaze in real time, and audio information can be provided to visually impaired people, making it suitable for a wide range of users.

[0453] Example 1

[0454] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] Conventional eye-tracking systems have difficulty providing information about the object a user is looking at instantly and in line with individual needs. They also have difficulty supporting visually impaired users and lack real-time information provision. There is a need for a system that can resolve these issues and provide more convenient and useful information to users.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0457] In this invention, the server includes means for analyzing information based on the captured image using an advanced image recognition algorithm, means for providing the analyzed information in a personalized manner according to the user's history and settings, and means for providing the analyzed information as audio feedback to the user, thereby enabling real-time personalized information provision based on the user's gaze.

[0458] - "Camera" means a device that converts light into electrical signals and is used to capture an image of an object in the user's line of sight.

[0459] A "sensor" is a device that detects physical phenomena and converts the signals into electrical signals, and is used to detect things like the user's line of sight.

[0460] "Wearable device" refers to any digital device that is worn on the body, and in this invention it primarily refers to devices with built-in cameras and sensors.

[0461] "Eye tracking means" refers to technology or devices that detect the movement of a user's eyes and identify the position of their gaze, and in this invention includes sensors and dedicated software algorithms.

[0462] "Image capturing means" means a function or device that takes an image of an object that a user is looking at based on an eye-tracking means.

[0463] "External Server" means a computer system that receives data sent from a wearable device and performs advanced analysis.

[0464] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying objects and textual information present within it.

[0465] "Personalization" means customizing information based on each user's history and preferences, and presenting it in a way that is most appropriate for the person providing it.

[0466] "Voice feedback" refers to a means of transmitting information that provides analyzed information to the user as audio, and is particularly suitable for visually impaired people.

[0467] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. This system uses a wearable device equipped with a camera and sensors to track the user's gaze in real time. As a result, an image is captured based on the gaze position and sent to an external server. The external server analyzes the image, generates the necessary information, personalizes it, and provides it to the user in an appropriate format.

[0468] User Recognition and Eye Tracking

[0469] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[0470] Gaze data collection and image capture

[0471] The device uses eye tracking to determine the user's gaze position. Specifically, sensors in the smart glasses detect the user's eye movements in real time and calculate the gaze position. Based on the determined gaze position, the device's camera automatically captures an image of the object at that position. This image is sent to an external server in real time.

[0472] Image analysis and information generation

[0473] The transmitted image data is received by an external server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the gaze. For example, if the image is a statue at a tourist spot, its historical background and related interesting facts can be extracted. Specific technologies used include Google Cloud Vision API and Microsoft Azure Computer Vision.

[0474] Personalized information provision

[0475] Based on the information analyzed by the server, personalized information is generated based on the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[0476] Example: Tourism support

[0477] Consider the example of a user visiting a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[0478] Example prompts for generative AI models

[0479] Example of input prompt:

[0480] "Describe a system that provides customized information about a specific object when the user is looking at that object. What hardware and software does this system use and how does it generate the information?"

[0481] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[0482] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0483] Step 1:

[0484] User recognition and device initialization

[0485] When a user wears the smart glasses, the device uses the camera to recognize the user's face and uses a facial recognition algorithm (e.g., OpenCV's facial recognition function) to identify the user's ID.

[0486] Input: User's face image

[0487] Data processing: Facial feature points are extracted using a facial recognition algorithm, and the corresponding user ID is searched for in the database.

[0488] Output: User ID

[0489] What it does: The smart glasses' camera captures a picture of the user's face, applies a facial recognition algorithm, and, based on the identified user ID, loads the user's settings and history data from cloud storage or the glasses' internal memory.

[0490] Step 2:

[0491] Eye tracking and data collection

[0492] The device determines the user's gaze position using the eye tracking sensor in the smart glasses, which detects the user's eye movements in real time.

[0493] Input: User's eye movement data

[0494] Data processing: Eye tracking algorithm calculates eye gaze direction

[0495] Output: Gaze direction and position information

[0496] How it works: The eye tracking sensor detects the movement of the user's eyes, and the eye tracking algorithm processes that data to determine the position of the gaze.

[0497] Step 3:

[0498] Image capture and transmission

[0499] Based on the results of gaze tracking, the device's camera captures an image of the object in front of the user's eyes, which is then compressed, encrypted, and sent to an external server.

[0500] Input: gaze position information, image of object

[0501] Data processing: image capture, compression (e.g., JPEG format), encryption (e.g., SSL)

[0502] Output: Compressed and encrypted image data

[0503] Specific operation: When a fixed gaze is detected, the camera automatically releases the shutter and captures an image. The image is then compressed into JPEG format, encrypted using SSL, and sent to the server.

[0504] Step 4:

[0505] Image analysis and information generation

[0506] The server receives the transmitted image data and analyzes the image using advanced image recognition algorithms (e.g., Google Cloud Vision API or Microsoft Azure Computer Vision).

[0507] Input: Encrypted image data

[0508] Data processing: Decrypting data, applying image recognition algorithms, and identifying objects

[0509] Output: Information about the object (e.g., history, characteristics)

[0510] Specific operation: The received image data is decoded, and an image recognition API is called to identify the object and extract its information.

[0511] Step 5:

[0512] Personalized information provision

[0513] The server analyzes the information and generates personalized information based on the user's history and settings, which is then sent to the device and displayed visually on the smart glasses display or provided as audio feedback.

[0514] Input: Information about the object, user history and settings information

[0515] Data processing: Extraction of relevant information, personalized processing

[0516] Output: Personalized information (visual / audio)

[0517] What it does: It matches the analyzed information with the user's past history and preferences to select the most relevant information, and then provides that information as an AR display or audio guide.

[0518] Through the above process, this system provides real-time, personalized information based on the user's line of sight.

[0519] (Application example 1)

[0520] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0521] Conventional wearable devices and eye-tracking technologies focused on tracking the user's gaze, but lacked the ability to instantly provide detailed information about the object in front of the user's eyes. Furthermore, there were insufficient means to provide real-time information about products that users were interested in in physical stores. Furthermore, voice feedback functions for the visually impaired were not fully developed, which led to issues with the efficiency of information acquisition.

[0522] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0523] In this invention, the server includes a wearable device equipped with a camera and a sensor for tracking the user's gaze, gaze tracking means for identifying the user's gaze position based on data acquired by the camera and the sensor, means for capturing an image based on the gaze position identified by the gaze tracking means, means for transmitting the captured image to an external server, means for the external server to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, and means for providing information on products the user is looking at in a physical store in real time through a display device of smart glasses. This allows users to obtain information on products they are interested in in a physical store in real time, and also makes it possible to provide information to visually impaired people via voice feedback.

[0524] "Eye tracking" is a technology that tracks and identifies the position of a user's gaze.

[0525] A "wearable device" is an electronic device that is designed to be worn by the user.

[0526] "Cameras and Sensors" means the imaging devices and detectors used to capture user gaze data.

[0527] "Capturing an image" means taking a picture of an object at a particular line of sight.

[0528] An "external server" is a remote computer system that can be communicated with via the Internet.

[0529] "Analyzing information" means identifying details and features of an object based on the acquired data and extracting relevant information.

[0530] "Personalization" means adapting acquired information to the preferences and history of individual users.

[0531] "Blind people" means people who are visually impaired.

[0532] "Smart glasses" are a type of wearable eyeglass device that displays information based on the wearer's line of sight.

[0533] "Voice feedback" refers to a method of providing acquired information to the user in the form of voice.

[0534] The system for realizing this invention uses smart glasses as a wearable device worn by the user. The smart glasses have a built-in camera and sensor, and are equipped with an eye-tracking function that tracks the user's gaze. Below, we will explain the specific hardware and software used in each step, as well as the data processing and data calculation methods using them.

[0535] Hardware and software used

[0536] Hardware:

[0537] Smart glasses (with built-in camera and sensors)

[0538] Cloud server (general remote computer)

[0539] software:

[0540] Image recognition algorithms (TensorFlow, OpenCV)

[0541] User history management system (database)

[0542] Smart Glasses Display Control Software

[0543] Data Processing and Data Calculation

[0544] 1. Eye Tracking:

[0545] When a user wears the smart glasses, the sensors and cameras in the smart glasses track the user's gaze and identify the gaze position in real time. The gaze data is processed by a processor in the smart glasses and recorded as gaze position information.

[0546] 2. Image capture and transmission:

[0547] Based on the identified gaze position, the camera automatically captures an image of the object in front of the gaze, and the captured image is transmitted to an external cloud server via wireless communication.

[0548] 3. Image analysis and information extraction:

[0549] The server analyzes the received image data using advanced image recognition algorithms (TensorFlow, OpenCV) to extract information about the object in front of the gaze. For example, in the case of a product in a physical store, information such as its price, ingredients, and user reviews can be extracted.

[0550] 4. Personalization and Information:

[0551] The server then personalizes the extracted information based on the user's past purchase history and preferences (managed in a database). The personalized information is displayed on the smart glasses' display or provided as audio feedback. It is also possible to provide the user with appropriate suggestions (recommended products or promotional information) based on specific prompts.

[0552] Specific examples

[0553] Imagine a user browsing a store shelf. When the user directs their gaze at a particular product, the smartglasses' camera captures an image of that product and sends it to a cloud server. The server analyzes the received image and extracts the product's price, ingredients, and user reviews. Based on the user's past purchase history and preferences, the smartglasses display then displays personalized information, such as "This product is recommended based on your past purchase history."

[0554] Prompt Sentence Examples

[0555] When a user wants to know more about this product, a prompt like this is used:

[0556] "Track the user's gaze and capture the product they are looking at. Send the captured image to a server to retrieve product information. Customize and display the information based on the user's preferences."

[0557] Based on this prompt, the system will identify product information that interests the user and provide personalized information.

[0558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0559] Step 1:

[0560] The user puts on the smart glasses and activates the sensors and camera. The eye tracking system collects the user's gaze data in real time. Specifically, the camera captures the user's eye movements, and the sensor identifies the gaze position. Input: User's eye movements. Output: User's gaze data (gaze position information).

[0561] Step 2:

[0562] The device identifies the position of the object in front of the user's gaze based on the collected gaze data. Then, it uses the camera to capture an image of the object at that identified position. Input: User's gaze data. Output: Captured image.

[0563] Step 3:

[0564] The captured image is wirelessly transmitted from the device to an external server. Specifically, the image data is uploaded to a cloud server via a wireless communication module. Input: Captured image. Output: Image data stored on the server.

[0565] Step 4:

[0566] Based on the received image data, the server uses an image recognition algorithm (e.g., TensorFlow, OpenCV) to recognize the object in front of the user's line of sight and analyzes the information. Specifically, it extracts information such as the product name, price, ingredients, and user reviews. Input: Image data stored on the server. Output: Detailed information about the object.

[0567] Step 5:

[0568] The server uses the analyzed information to query a database of the user's history and preferences to personalize the information. For example, it generates related information and product recommendations based on the user's preferences and the product history of previous purchases. Input: Detailed information about the object, user history data. Output: Personalized information.

[0569] Step 6:

[0570] The server sends the personalized information to the device. Specifically, the information is sent to the smart glasses via wireless communication, where it is displayed visually on the smart glasses' display. In some cases, the information can also be provided as an audio guide. Input: Personalized information. Output: Smart glasses' display or audio guide.

[0571] Step 7:

[0572] Users can check the information displayed on the smart glasses display and obtain information about products and promotions that interest them. For visually impaired people, information is obtained through audio guidance. Input: Smart glasses display, audio guidance. Output: Information obtained by the user.

[0573] This series of processes allows users to obtain detailed product information and personalized recommendations in real time simply by looking at the product in a physical store.

[0574] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0575] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can further personalize the information provided and provide support tailored to the user's psychological state.

[0576] User Recognition and Eye Tracking

[0577] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[0578] Gaze data collection and image capture

[0579] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0580] Image analysis and information generation

[0581] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[0582] Emotion recognition by emotion engine

[0583] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[0584] Personalized information provision

[0585] Based on the information analyzed by the server, the system generates personalized information based on the user's history, settings, and emotional state using an emotion engine. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smartglasses' display or provide it as audio feedback.

[0586] Examples: Tourist assistance and emotional support

[0587] Consider a case where a user visits a famous tourist spot. As the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and artist. The analyzed information is then customized based on the user's past visits and interests and displayed visually on the smart glasses.

[0588] Furthermore, the emotion engine analyzes the user's emotional state and, for example, if they feel tired, provides the location of a nearby bench. For visually impaired users, this information is provided via audio guidance, and feedback is given according to their emotions.

[0589] In this way, users can receive personalized information based on the object they are looking at, while simultaneously receiving psychological support from the emotion engine. This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used in a wide range of applications, including medical and workplace settings, in addition to the visually impaired and tourists. This is expected to significantly improve the efficiency of users' daily lives and work.

[0590] The processing flow will be explained below.

[0591] Step 1:

[0592] The user puts on the wearable device (smart glasses). The device starts up and the camera and sensors begin operating.

[0593] Step 2:

[0594] Identify user ID: The device recognizes the user ID and loads individual settings and history data based on that ID.

[0595] Step 3:

[0596] Using eye tracking, the device tracks the user's eye position in real time, sequentially acquiring eye position data and capturing that position information.

[0597] Step 4:

[0598] Based on the gaze position, the device's camera automatically captures an image of the object at that position, and the captured image data is temporarily stored on the device.

[0599] Step 5:

[0600] The device sends the captured image data to an external server, including gaze position information.

[0601] Step 6:

[0602] The server analyzes the received image data. Using advanced image recognition algorithms, it identifies the objects and extracts relevant information. For example, the server analyzes the history and characteristics of statues at tourist attractions.

[0603] Step 7:

[0604] Using an emotion engine, the device recognizes the user's emotional state by analyzing the user's heart rate and facial expressions in real time based on gaze data and data from biometric sensors.

[0605] Step 8:

[0606] Based on the information analyzed by the server, personalized information is generated based on the user's history, settings, and emotional state using an emotion engine, including providing information according to the user's psychological state.

[0607] Step 9:

[0608] The server sends the generated personalized information to the device, which then visually displays the information on the smartglasses display.

[0609] Step 10:

[0610] For the visually impaired, the device provides audio feedback, conveying necessary information and emotional feedback to the user through audio guidance.

[0611] Step 11:

[0612] Users review information through displays or audio guides and use that information to make decisions about what to do at a tourist attraction, such as learning about the history of a statue or finding the location of a nearby bench.

[0613] This process allows users to obtain personalized information based on their gaze in real time, and with the support of an emotion engine, appropriate feedback tailored to their psychological state is provided, improving user satisfaction and enabling a wide range of applications.

[0614] Example 2

[0615] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0616] Conventional wearable devices are limited to providing information by tracking the user's gaze, making it difficult to provide personalized information tailored to the user's emotional state. Furthermore, their support for visually impaired people is insufficient, so an improvement in the user experience is needed.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0618] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[0619] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[0620] means for capturing an image based on the gaze position identified by the gaze tracking means;

[0621] means for transmitting the captured image to an external server;

[0622] means for the external server to analyze information based on the captured image;

[0623] A means for personalizing the analyzed information and providing it to the user;

[0624] an emotion recognition means for identifying an emotional state of a user based on biometric data;

[0625] a means for personalizing information provided based on the user's emotional state;

[0626] a means for conveying the provided information to the user as a display or audio feedback;

[0627] This allows for personalized information to be provided based on the user's emotional state and the object in their line of sight.

[0628] "Eye tracking" is a feature that uses cameras and sensors to determine the position of a user's gaze in real time.

[0629] "Wearable devices" generally refer to electronic devices that can be worn by the user on the body, including smart glasses.

[0630] "Cameras and Sensors" means the imaging and detection devices used to track the user's gaze.

[0631] "Image capture" refers to the act of using a camera to obtain a still image of an object at a particular gaze position.

[0632] A "server" is a computer system that stores, analyzes, and provides data over a network.

[0633] "Information analysis" refers to the process of processing received data to extract useful information.

[0634] "Personalization" means providing information that is optimized based on the user's individual settings, history, and emotional state.

[0635] "Emotion recognition" is a technology that analyzes a user's biometric data to identify their current emotional state.

[0636] "Biometric data" means physiological data such as a user's heart rate, facial expression, and skin temperature.

[0637] A "display" is a screen device for presenting visual information to a user.

[0638] "Audio feedback" is a means of conveying information to the user through sound.

[0639] "Visually impaired people" refers to people who have problems with their eyesight and have difficulty obtaining normal visual information.

[0640] MODE FOR CARRYING OUT THE INVENTION

[0641] The present invention relates to a system that tracks a user's gaze and provides customized information based on the object in their line of sight. Furthermore, by combining it with an emotion engine, the system provides personalized information tailored to the user's emotional state.

[0642] User Recognition and Eye Tracking

[0643] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[0644] Gaze data collection and image capture

[0645] The device uses eye tracking to identify the user's gaze position. When the gaze is fixed at a specific position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0646] Image analysis and information generation

[0647] The transmitted image data is received by a server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the user's gaze. For example, if the user is looking at a statue at a tourist attraction, the server can extract the statue's historical background and related interesting facts. This analysis is performed using a generative AI model.

[0648] Emotion recognition by emotion engine

[0649] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[0650] Personalized information provision

[0651] The server then uses the analyzed information to generate personalized information based on the user's history, settings, and emotional state as determined by the emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide information about the statue's historical background and recommended sightseeing routes. The device then displays this information on the smart glasses' display or provides it as audio feedback.

[0652] Examples: Tourist assistance and emotional support

[0653] Specific examples

[0654] When a user visits a famous tourist spot and looks at a statue, the device tracks the user's gaze, capturing an image of the statue and sending it to a server. The server then analyzes the statue's detailed information, extracting data about its history and artist, for example. This information is customized based on the user's past visits and interests and displayed visually on the smart glasses. Furthermore, an emotion engine analyzes the user's emotional state, providing the location of a nearby bench if, for example, they are feeling tired. Visually impaired users can also receive this information through audio guidance and emotional feedback.

[0655] Prompt Sentence Examples

[0656] If the user wants to get information, they can enter a prompt like this:

[0657] "Please tell me more about the statue I'm looking at. I'd like to know its historical background and who made it."

[0658] This system allows users to receive personalized information based on the object in front of their eyes, while also receiving assistance from an emotion engine. This technology is expected to contribute not only to tourism support, but also to support for the visually impaired, medical care, and improving efficiency in various workplaces.

[0659] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0660] Step 1: User Awareness

[0661] When a user puts on a wearable device (smart glasses), the device uses its built-in camera and sensors to identify the user's ID. The device then uses facial recognition technology to identify the user's ID and reads the user's unique settings and history data from its built-in memory. This makes it possible to provide information that takes into account the user's preferences and past usage history.

[0662] input:

[0663] User's facial image, biometric data

[0664] Data processing / calculation:

[0665] It uses a facial recognition algorithm to identify the user ID and loads corresponding settings and history data.

[0666] output:

[0667] User ID, user settings and history data

[0668] Specific behavior:

[0669] The device uses a facial recognition sensor to capture the user's facial image and verify their identity by comparing it with an internal database.

[0670] Step 2: Eye tracking

[0671] The sensor tracks the user's eye movements in real time and identifies the position of the gaze. The device uses an infrared sensor to detect the user's gaze position and updates the data in real time, thereby identifying the position where the gaze is fixed.

[0672] input:

[0673] User eye position data

[0674] Data processing / calculation:

[0675] Uses an eye-tracking algorithm to determine the user's gaze position.

[0676] output:

[0677] Gaze position data

[0678] Specific behavior:

[0679] The device's infrared sensor detects the user's eye movements, analyzes the data, and displays the gaze position in real time.

[0680] Step 3: Collecting gaze data and capturing images

[0681] When a user's gaze is fixed at a specific location, the device automatically captures an image of the object at that location using its camera, and this image data is collected in real time and sent to an external server.

[0682] input:

[0683] Gaze position data

[0684] Data processing / calculation:

[0685] Controls the camera based on gaze position to capture images of the target object

[0686] output:

[0687] Captured image data

[0688] Specific behavior:

[0689] The camera automatically focuses on the position where the gaze is fixed and captures an image of the object.

[0690] Step 4: Sending image data

[0691] The captured image data is sent to an external server using a secure communication protocol. The device sends the image data to the server using HTTPS.

[0692] input:

[0693] Captured image data

[0694] Data processing / calculation:

[0695] Data encryption and transmission

[0696] output:

[0697] Image data sent to the server

[0698] Specific behavior:

[0699] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[0700] Step 5: Image analysis

[0701] The server analyzes the received image data and extracts information about the object in the line of sight. Using a generative AI model, it analyzes detailed information about the object in the line of sight.

[0702] input:

[0703] Received image data

[0704] Data processing / calculation:

[0705] Extract information using image recognition algorithms

[0706] output:

[0707] Analyzed object information

[0708] Specific behavior:

[0709] The server runs a generative AI model to extract detailed information and related data about the object.

[0710] Step 6: Emotion Recognition

[0711] The device analyzes the user's biometric data (heart rate, facial expressions, etc.) collected by the device, and the emotion engine identifies the user's emotional state. The emotion engine analyzes the user's biometric data in real time to determine the user's emotional state.

[0712] input:

[0713] User biometric data

[0714] Data processing / calculation:

[0715] Identify emotional states using emotion recognition algorithms

[0716] output:

[0717] The user's emotional state

[0718] Specific behavior:

[0719] The device analyzes data from a heart rate monitor and facial recognition sensors to determine whether the user is relaxed or tired.

[0720] Step 7: Personalized information generation

[0721] The server then uses the analyzed information to generate personalized information based on the user's history, preferences, and emotional state, as determined by an emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide historical background about the statue and recommend sightseeing routes.

[0722] input:

[0723] Analyzed object information, user emotional state, historical data and settings

[0724] Data processing / calculation:

[0725] Data Integration and Information Generation

[0726] output:

[0727] Personalized Information

[0728] Specific behavior:

[0729] The server integrates this data to generate customized information. The generative AI model is given a prompt: "Please tell me more about the statue I'm looking at. I'd also like to know its historical background and its creator.", and the required data is generated.

[0730] Step 8: Provide information

[0731] The device provides the generated information to the user, either visually displayed on the smart glasses display or as audio feedback.

[0732] input:

[0733] Personalized Information

[0734] Data processing / calculation:

[0735] Display format conversion

[0736] output:

[0737] Visual or audio information provided to the user

[0738] Specific behavior:

[0739] The device displays text about the statue's historical background and simultaneously provides audio feedback explaining the statue's creator.

[0740] (Application example 2)

[0741] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] In current factory work, workers must manually search or refer to manuals to obtain information about individual parts and tools. This procedure is inefficient and increases the burden on workers. In addition, there is a lack of a system to detect workers' stress and fatigue in real time and provide appropriate breaks and advice. As a result, work efficiency decreases and there is a risk of work errors and safety issues.

[0743] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a wearable device equipped with an optical device and a detection device for tracking the user's gaze, an gaze tracking device for identifying the user's gaze position based on data acquired by the optical device and the detection device, means for capturing an image based on the gaze position identified by the gaze tracking device, means for transmitting the captured image to an external installation device, means for the external installation device to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, an emotion recognition device for identifying the user's emotional state, and means for providing advice and work improvement suggestions to the user based on the emotional state identified by the emotion recognition device. This enables the worker to obtain information on the parts and tools in front of the worker's gaze in real time and to receive appropriate breaks and advice according to the worker's emotional state, thereby improving work efficiency and ensuring safety.

[0744] An "optical device" is a device that handles light, such as a camera or lens, to detect the user's line of sight.

[0745] A "detection device" is a sensor or device that acquires gaze data or biometric data.

[0746] A "wearable device" is a digital device that can be worn by the user, examples of which include smart glasses and head-mounted displays.

[0747] An "eye tracking device" is a system that tracks the position of a user's gaze in real time.

[0748] "Means for capturing" refers to a function for acquiring an image at a specified gaze position.

[0749] An "external device" is a server or cloud service that receives and analyzes data sent from a wearable device.

[0750] "Means for analyzing information" refers to the algorithms or software that analyze the captured image data and extract information about the object in the user's line of sight.

[0751] "Means for providing personalized information" refers to a function that customizes analyzed information based on the user's history and settings, and provides it in the most optimal form for the user.

[0752] An "emotion recognition device" refers to a sensor or analysis device used to detect a user's emotional state.

[0753] The "means for providing advice and work improvement suggestions" is a function that presents optimal breaks and suggestions for improving work procedures based on the user's emotional state detected by the emotion recognition device.

[0754] An embodiment of the present invention will be described. This system uses a head-mounted display (HMD) as a wearable device worn by a factory worker. The main elements and their operations are shown below.

[0755] Hardware and software used

[0756] 1. Wearable device (HMD): A device equipped with optical and detection devices to track the user's gaze.

[0757] 2. Eye tracking device: A system that uses the dlib library and other tools to track the user's gaze position in real time.

[0758] 3. External device (server): This device receives the captured image data and analyzes the information. TensorFlow is used for this analysis.

[0759] 4. Emotion Recognition Device: Used to analyze camera footage and identify the user's emotional state. Here, we use an emotion recognition model built with TensorFlow.

[0760] Data handling and calculations

[0761] 1. Gaze Tracking: The HMD camera captures the user's face and uses the dlib library to detect facial landmarks. The gaze direction is calculated based on these landmarks.

[0762] 2. Image capture and transmission: Images of the object are automatically captured based on the direction of gaze and transmitted to an external server, where they are analyzed in real time.

[0763] 3. Analysis and provision of information: The server analyzes the received image data using TensorFlow and extracts information about the parts and tools in front of the user's gaze. This information is then customized based on the user's history and settings and displayed on the HMD.

[0764] 4. Identifying and responding to emotional states: The HMD is equipped with an emotion recognition device that analyzes the user's facial expressions, heart rate, etc. to identify their emotional state. For example, if the user is feeling stressed, the system will suggest taking a break.

[0765] Specific examples

[0766] When a factory worker looks at a part of the robot arm they are operating, the HMD visually displays the manual and work procedures related to that part. Also, if the HMD detects that the worker is under stress, it will display instructions to take a break.

[0767] Example prompts for generative AI models

[0768] "Factory workers wear head-mounted displays and work while tracking their gaze. Generate an application program that displays information about parts and tools in front of their eyes in real time, analyzes the worker's emotional state, and suggests taking a break if the worker feels fatigued or stressed."

[0769] In this way, the system provides real-time information on the line of sight of factory workers and gives appropriate advice according to their emotional state, thereby improving work efficiency and ensuring safety.

[0770] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0771] Step 1:

[0772] The user wears a wearable device (HMD). The HMD has built-in optical and detection devices, which enable gaze tracking and emotion recognition. The input is the user's facial expression and gaze data, and the output is the capture of this data.

[0773] Step 2:

[0774] The device uses an eye-tracking device to track the user's gaze in real time. The dlib library is used to calculate the user's gaze direction from the captured facial landmark information. The input is the captured facial image data, and the output is the calculated gaze direction.

[0775] Step 3:

[0776] The gaze tracking device captures an image of the object in front of the gaze. The captured image is sent to an external device (server). The input is the calculated gaze direction and image data of the gaze destination, and the output is data sent to the server.

[0777] Step 4:

[0778] The server receives images sent from an external source and uses TensorFlow to analyze information about parts and tools in the line of sight. The analyzed information is personalized based on the user's history and settings. The input is the captured image data, and the output is the analyzed information about the line of sight.

[0779] Step 5:

[0780] The server sends the analyzed information to the HMD and displays it visually. The user can check detailed information about parts and tools in real time through the HMD. The input is the analyzed information, and the output is the information displayed on the HMD.

[0781] Step 6:

[0782] The device uses an emotion recognition device to analyze biometric data such as the user's facial expressions and heart rate to identify their emotional state. An emotion recognition model built with TensorFlow is used. The input is the user's biometric data, and the output is the identified emotional state.

[0783] Step 7:

[0784] The server sends appropriate advice and work improvement suggestions to the HMD based on the emotional state identified by the emotion recognition device. For example, if the user is recognized as being in a stressful state, a message suggesting a break will be displayed. The input is the identified emotional state, and the output is the advice displayed on the HMD.

[0785] Step 8:

[0786] The user performs the work according to the information and advice displayed through the HMD, which improves work efficiency and ensures safety. The input is the information displayed by the HMD, and the output is the user's work behavior.

[0787] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0788] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0789] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0790] [Third embodiment]

[0791] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0792] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0793] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0794] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0795] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0796] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0797] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0798] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0799] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0800] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0801] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0802] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0803] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Specifically, a wearable device equipped with a camera and sensors is used to track the user's gaze in real time, capture images based on the gaze position, and transmit them to an external server. The external server analyzes the images, generates and personalizes the necessary information, and provides it to the user in an appropriate format.

[0804] User Recognition and Eye Tracking

[0805] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[0806] Gaze data collection and image capture

[0807] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0808] Image analysis and information generation

[0809] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[0810] Personalized information provision

[0811] Based on the information analyzed by the server, personalized information is generated according to the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[0812] Example: Tourism support

[0813] Consider a case where a user visits a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[0814] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[0815] The processing flow will be explained below.

[0816] Step 1:

[0817] The user puts on the wearable device (smart glasses). The smart glasses start up and the camera and sensors begin operating.

[0818] Step 2:

[0819] The device identifies the user ID. Based on the user ID, it loads individual settings and history data. This data is necessary to provide information tailored to the user's past usage history and preferences.

[0820] Step 3:

[0821] The device uses a camera and sensors to track the user's gaze position in real time. The gaze tracking means sequentially acquires gaze position data and captures the position information.

[0822] Step 4:

[0823] The device captures an image of the object using the camera based on the gaze position, and the captured image data is temporarily stored in the device.

[0824] Step 5:

[0825] The device sends the captured image data to an external server, including gaze position information.

[0826] Step 6:

[0827] The server analyzes the received image data and uses advanced image recognition algorithms to identify objects and extract relevant information, such as the history and characteristics of statues at tourist attractions.

[0828] Step 7:

[0829] The server uses the analysis results to generate personalized information based on the user's history and settings, making it possible to provide information tailored to the individual needs and preferences of each user.

[0830] Step 8:

[0831] The server sends the generated information to the device, which then displays the received personalized information on the smart glasses display.

[0832] Step 9:

[0833] If the device is for the visually impaired, it will not only provide visual feedback but also audio feedback, conveying the necessary information to the user through audio guides.

[0834] Step 10:

[0835] Users review information and act on it, for example, learning about the history of a statue at a tourist spot and choosing a sightseeing route around the area.

[0836] This process allows users to obtain personalized information based on their gaze in real time, and audio information can be provided to visually impaired people, making it suitable for a wide range of users.

[0837] Example 1

[0838] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] Conventional eye-tracking systems have difficulty providing information about the object a user is looking at instantly and in line with individual needs. They also have difficulty supporting visually impaired users and lack real-time information provision. There is a need for a system that can resolve these issues and provide more convenient and useful information to users.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0841] In this invention, the server includes means for analyzing information based on the captured image using an advanced image recognition algorithm, means for providing the analyzed information in a personalized manner according to the user's history and settings, and means for providing the analyzed information as audio feedback to the user, thereby enabling real-time personalized information provision based on the user's gaze.

[0842] - "Camera" means a device that converts light into electrical signals and is used to capture an image of an object in the user's line of sight.

[0843] A "sensor" is a device that detects physical phenomena and converts the signals into electrical signals, and is used to detect things like the user's line of sight.

[0844] "Wearable device" refers to any digital device that is worn on the body, and in this invention it primarily refers to devices with built-in cameras and sensors.

[0845] "Eye tracking means" refers to technology or devices that detect the movement of a user's eyes and identify the position of their gaze, and in this invention includes sensors and dedicated software algorithms.

[0846] "Image capturing means" means a function or device that takes an image of an object that a user is looking at based on an eye-tracking means.

[0847] "External Server" means a computer system that receives data sent from a wearable device and performs advanced analysis.

[0848] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying objects and textual information present within it.

[0849] "Personalization" means customizing information based on each user's history and preferences, and presenting it in a way that is most appropriate for the person providing it.

[0850] "Voice feedback" refers to a means of transmitting information that provides analyzed information to the user as audio, and is particularly suitable for visually impaired people.

[0851] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. This system uses a wearable device equipped with a camera and sensors to track the user's gaze in real time. As a result, an image is captured based on the gaze position and sent to an external server. The external server analyzes the image, generates the necessary information, personalizes it, and provides it to the user in an appropriate format.

[0852] User Recognition and Eye Tracking

[0853] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[0854] Gaze data collection and image capture

[0855] The device uses eye tracking to determine the user's gaze position. Specifically, sensors in the smart glasses detect the user's eye movements in real time and calculate the gaze position. Based on the determined gaze position, the device's camera automatically captures an image of the object at that position. This image is sent to an external server in real time.

[0856] Image analysis and information generation

[0857] The transmitted image data is received by an external server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the gaze. For example, if the image is a statue at a tourist spot, its historical background and related interesting facts can be extracted. Specific technologies used include Google Cloud Vision API and Microsoft Azure Computer Vision.

[0858] Personalized information provision

[0859] Based on the information analyzed by the server, personalized information is generated based on the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[0860] Example: Tourism support

[0861] Consider the example of a user visiting a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[0862] Example prompts for generative AI models

[0863] Example of input prompt:

[0864] "Describe a system that provides customized information about a specific object when the user is looking at that object. What hardware and software does this system use and how does it generate the information?"

[0865] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[0866] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0867] Step 1:

[0868] User recognition and device initialization

[0869] When a user wears the smart glasses, the device uses the camera to recognize the user's face and uses a facial recognition algorithm (e.g., OpenCV's facial recognition function) to identify the user's ID.

[0870] Input: User's face image

[0871] Data processing: Facial feature points are extracted using a facial recognition algorithm, and the corresponding user ID is searched for in the database.

[0872] Output: User ID

[0873] What it does: The smart glasses' camera captures a picture of the user's face, applies a facial recognition algorithm, and, based on the identified user ID, loads the user's settings and history data from cloud storage or the glasses' internal memory.

[0874] Step 2:

[0875] Eye tracking and data collection

[0876] The device determines the user's gaze position using the eye tracking sensor in the smart glasses, which detects the user's eye movements in real time.

[0877] Input: User's eye movement data

[0878] Data processing: Eye tracking algorithm calculates eye gaze direction

[0879] Output: Gaze direction and position information

[0880] How it works: The eye tracking sensor detects the movement of the user's eyes, and the eye tracking algorithm processes that data to determine the position of the gaze.

[0881] Step 3:

[0882] Image capture and transmission

[0883] Based on the results of gaze tracking, the device's camera captures an image of the object in front of the user's eyes, which is then compressed, encrypted, and sent to an external server.

[0884] Input: gaze position information, image of object

[0885] Data processing: image capture, compression (e.g., JPEG format), encryption (e.g., SSL)

[0886] Output: Compressed and encrypted image data

[0887] Specific operation: When a fixed gaze is detected, the camera automatically releases the shutter and captures an image. The image is then compressed into JPEG format, encrypted using SSL, and sent to the server.

[0888] Step 4:

[0889] Image analysis and information generation

[0890] The server receives the transmitted image data and analyzes the image using advanced image recognition algorithms (e.g., Google Cloud Vision API or Microsoft Azure Computer Vision).

[0891] Input: Encrypted image data

[0892] Data processing: Decrypting data, applying image recognition algorithms, and identifying objects

[0893] Output: Information about the object (e.g., history, characteristics)

[0894] Specific operation: The received image data is decoded, and an image recognition API is called to identify the object and extract its information.

[0895] Step 5:

[0896] Personalized information provision

[0897] The server analyzes the information and generates personalized information based on the user's history and settings, which is then sent to the device and displayed visually on the smart glasses display or provided as audio feedback.

[0898] Input: Information about the object, user history and settings information

[0899] Data processing: Extraction of relevant information, personalized processing

[0900] Output: Personalized information (visual / audio)

[0901] What it does: It matches the analyzed information with the user's past history and preferences to select the most relevant information, and then provides that information as an AR display or audio guide.

[0902] Through the above process, this system provides real-time, personalized information based on the user's line of sight.

[0903] (Application example 1)

[0904] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0905] Conventional wearable devices and eye-tracking technologies focused on tracking the user's gaze, but lacked the ability to instantly provide detailed information about the object in front of the user's eyes. Furthermore, there were insufficient means to provide real-time information about products that users were interested in in physical stores. Furthermore, voice feedback functions for the visually impaired were not fully developed, which led to issues with the efficiency of information acquisition.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0907] In this invention, the server includes a wearable device equipped with a camera and a sensor for tracking the user's gaze, gaze tracking means for identifying the user's gaze position based on data acquired by the camera and the sensor, means for capturing an image based on the gaze position identified by the gaze tracking means, means for transmitting the captured image to an external server, means for the external server to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, and means for providing information on products the user is looking at in a physical store in real time through a display device of smart glasses. This allows users to obtain information on products they are interested in in a physical store in real time, and also makes it possible to provide information to visually impaired people via voice feedback.

[0908] "Eye tracking" is a technology that tracks and identifies the position of a user's gaze.

[0909] A "wearable device" is an electronic device that is designed to be worn by the user.

[0910] "Cameras and Sensors" means the imaging devices and detectors used to capture user gaze data.

[0911] "Capturing an image" means taking a picture of an object at a particular line of sight.

[0912] An "external server" is a remote computer system that can be communicated with via the Internet.

[0913] "Analyzing information" means identifying details and features of an object based on the acquired data and extracting relevant information.

[0914] "Personalization" means adapting acquired information to the preferences and history of individual users.

[0915] "Blind people" means people who are visually impaired.

[0916] "Smart glasses" are a type of wearable eyeglass device that displays information based on the wearer's line of sight.

[0917] "Voice feedback" refers to a method of providing acquired information to the user in the form of voice.

[0918] The system for realizing this invention uses smart glasses as a wearable device worn by the user. The smart glasses have a built-in camera and sensor, and are equipped with an eye-tracking function that tracks the user's gaze. Below, we will explain the specific hardware and software used in each step, as well as the data processing and data calculation methods using them.

[0919] Hardware and software used

[0920] Hardware:

[0921] Smart glasses (with built-in camera and sensors)

[0922] Cloud server (general remote computer)

[0923] software:

[0924] Image recognition algorithms (TensorFlow, OpenCV)

[0925] User history management system (database)

[0926] Smart Glasses Display Control Software

[0927] Data Processing and Data Calculation

[0928] 1. Eye Tracking:

[0929] When a user wears the smart glasses, the sensors and cameras in the smart glasses track the user's gaze and identify the gaze position in real time. The gaze data is processed by a processor in the smart glasses and recorded as gaze position information.

[0930] 2. Image capture and transmission:

[0931] Based on the identified gaze position, the camera automatically captures an image of the object in front of the gaze, and the captured image is transmitted to an external cloud server via wireless communication.

[0932] 3. Image analysis and information extraction:

[0933] The server analyzes the received image data using advanced image recognition algorithms (TensorFlow, OpenCV) to extract information about the object in front of the gaze. For example, in the case of a product in a physical store, information such as its price, ingredients, and user reviews can be extracted.

[0934] 4. Personalization and Information:

[0935] The server then personalizes the extracted information based on the user's past purchase history and preferences (managed in a database). The personalized information is displayed on the smart glasses' display or provided as audio feedback. It is also possible to provide the user with appropriate suggestions (recommended products or promotional information) based on specific prompts.

[0936] Specific examples

[0937] Imagine a user browsing a store shelf. When the user directs their gaze at a particular product, the smartglasses' camera captures an image of that product and sends it to a cloud server. The server analyzes the received image and extracts the product's price, ingredients, and user reviews. Based on the user's past purchase history and preferences, the smartglasses display then displays personalized information, such as "This product is recommended based on your past purchase history."

[0938] Prompt Sentence Examples

[0939] When a user wants to know more about this product, a prompt like this is used:

[0940] "Track the user's gaze and capture the product they are looking at. Send the captured image to a server to retrieve product information. Customize and display the information based on the user's preferences."

[0941] Based on this prompt, the system will identify product information that interests the user and provide personalized information.

[0942] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0943] Step 1:

[0944] The user puts on the smart glasses and activates the sensors and camera. The eye tracking system collects the user's gaze data in real time. Specifically, the camera captures the user's eye movements, and the sensor identifies the gaze position. Input: User's eye movements. Output: User's gaze data (gaze position information).

[0945] Step 2:

[0946] The device identifies the position of the object in front of the user's gaze based on the collected gaze data. Then, it uses the camera to capture an image of the object at that identified position. Input: User's gaze data. Output: Captured image.

[0947] Step 3:

[0948] The captured image is wirelessly transmitted from the device to an external server. Specifically, the image data is uploaded to a cloud server via a wireless communication module. Input: Captured image. Output: Image data stored on the server.

[0949] Step 4:

[0950] Based on the received image data, the server uses an image recognition algorithm (e.g., TensorFlow, OpenCV) to recognize the object in front of the user's line of sight and analyzes the information. Specifically, it extracts information such as the product name, price, ingredients, and user reviews. Input: Image data stored on the server. Output: Detailed information about the object.

[0951] Step 5:

[0952] The server uses the analyzed information to query a database of the user's history and preferences to personalize the information. For example, it generates related information and product recommendations based on the user's preferences and the product history of previous purchases. Input: Detailed information about the object, user history data. Output: Personalized information.

[0953] Step 6:

[0954] The server sends the personalized information to the device. Specifically, the information is sent to the smart glasses via wireless communication, where it is displayed visually on the smart glasses' display. In some cases, the information can also be provided as an audio guide. Input: Personalized information. Output: Smart glasses' display or audio guide.

[0955] Step 7:

[0956] Users can check the information displayed on the smart glasses display and obtain information about products and promotions that interest them. For visually impaired people, information is obtained through audio guidance. Input: Smart glasses display, audio guidance. Output: Information obtained by the user.

[0957] This series of processes allows users to obtain detailed product information and personalized recommendations in real time simply by looking at the product in a physical store.

[0958] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0959] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can further personalize the information provided and provide support tailored to the user's psychological state.

[0960] User Recognition and Eye Tracking

[0961] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[0962] Gaze data collection and image capture

[0963] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[0964] Image analysis and information generation

[0965] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[0966] Emotion recognition by emotion engine

[0967] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[0968] Personalized information provision

[0969] Based on the information analyzed by the server, the system generates personalized information based on the user's history, settings, and emotional state using an emotion engine. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smartglasses' display or provide it as audio feedback.

[0970] Examples: Tourist assistance and emotional support

[0971] Consider a case where a user visits a famous tourist spot. As the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and artist. The analyzed information is then customized based on the user's past visits and interests and displayed visually on the smart glasses.

[0972] Furthermore, the emotion engine analyzes the user's emotional state and, for example, if they feel tired, provides the location of a nearby bench. For visually impaired users, this information is provided via audio guidance, and feedback is given according to their emotions.

[0973] In this way, users can receive personalized information based on the object they are looking at, while simultaneously receiving psychological support from the emotion engine. This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used in a wide range of applications, including medical and workplace settings, in addition to the visually impaired and tourists. This is expected to significantly improve the efficiency of users' daily lives and work.

[0974] The processing flow will be explained below.

[0975] Step 1:

[0976] The user puts on the wearable device (smart glasses). The device starts up and the camera and sensors begin operating.

[0977] Step 2:

[0978] Identify user ID: The device recognizes the user ID and loads individual settings and history data based on that ID.

[0979] Step 3:

[0980] Using eye tracking, the device tracks the user's eye position in real time, sequentially acquiring eye position data and capturing that position information.

[0981] Step 4:

[0982] Based on the gaze position, the device's camera automatically captures an image of the object at that position, and the captured image data is temporarily stored on the device.

[0983] Step 5:

[0984] The device sends the captured image data to an external server, including gaze position information.

[0985] Step 6:

[0986] The server analyzes the received image data. Using advanced image recognition algorithms, it identifies the objects and extracts relevant information. For example, the server analyzes the history and characteristics of statues at tourist attractions.

[0987] Step 7:

[0988] Using an emotion engine, the device recognizes the user's emotional state by analyzing the user's heart rate and facial expressions in real time based on gaze data and data from biometric sensors.

[0989] Step 8:

[0990] Based on the information analyzed by the server, personalized information is generated based on the user's history, settings, and emotional state using an emotion engine, including providing information according to the user's psychological state.

[0991] Step 9:

[0992] The server sends the generated personalized information to the device, which then visually displays the information on the smartglasses display.

[0993] Step 10:

[0994] For the visually impaired, the device provides audio feedback, conveying necessary information and emotional feedback to the user through audio guidance.

[0995] Step 11:

[0996] Users review information through displays or audio guides and use that information to make decisions about what to do at a tourist attraction, such as learning about the history of a statue or finding the location of a nearby bench.

[0997] This process allows users to obtain personalized information based on their gaze in real time, and with the support of an emotion engine, appropriate feedback tailored to their psychological state is provided, improving user satisfaction and enabling a wide range of applications.

[0998] Example 2

[0999] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1000] Conventional wearable devices are limited to providing information by tracking the user's gaze, making it difficult to provide personalized information tailored to the user's emotional state. Furthermore, their support for visually impaired people is insufficient, so an improvement in the user experience is needed.

[1001] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1002] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[1003] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[1004] means for capturing an image based on the gaze position identified by the gaze tracking means;

[1005] means for transmitting the captured image to an external server;

[1006] means for the external server to analyze information based on the captured image;

[1007] A means for personalizing the analyzed information and providing it to the user;

[1008] an emotion recognition means for identifying an emotional state of a user based on biometric data;

[1009] a means for personalizing information provided based on the user's emotional state;

[1010] a means for conveying the provided information to the user as a display or audio feedback;

[1011] This allows for personalized information to be provided based on the user's emotional state and the object in their line of sight.

[1012] "Eye tracking" is a feature that uses cameras and sensors to determine the position of a user's gaze in real time.

[1013] "Wearable devices" generally refer to electronic devices that can be worn by the user on the body, including smart glasses.

[1014] "Cameras and Sensors" means the imaging and detection devices used to track the user's gaze.

[1015] "Image capture" refers to the act of using a camera to obtain a still image of an object at a particular gaze position.

[1016] A "server" is a computer system that stores, analyzes, and provides data over a network.

[1017] "Information analysis" refers to the process of processing received data to extract useful information.

[1018] "Personalization" means providing information that is optimized based on the user's individual settings, history, and emotional state.

[1019] "Emotion recognition" is a technology that analyzes a user's biometric data to identify their current emotional state.

[1020] "Biometric data" means physiological data such as a user's heart rate, facial expression, and skin temperature.

[1021] A "display" is a screen device for presenting visual information to a user.

[1022] "Audio feedback" is a means of conveying information to the user through sound.

[1023] "Visually impaired people" refers to people who have problems with their eyesight and have difficulty obtaining normal visual information.

[1024] MODE FOR CARRYING OUT THE INVENTION

[1025] The present invention relates to a system that tracks a user's gaze and provides customized information based on the object in their line of sight. Furthermore, by combining it with an emotion engine, the system provides personalized information tailored to the user's emotional state.

[1026] User Recognition and Eye Tracking

[1027] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[1028] Gaze data collection and image capture

[1029] The device uses eye tracking to identify the user's gaze position. When the gaze is fixed at a specific position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[1030] Image analysis and information generation

[1031] The transmitted image data is received by a server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the user's gaze. For example, if the user is looking at a statue at a tourist attraction, the server can extract the statue's historical background and related interesting facts. This analysis is performed using a generative AI model.

[1032] Emotion recognition by emotion engine

[1033] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[1034] Personalized information provision

[1035] The server then uses the analyzed information to generate personalized information based on the user's history, settings, and emotional state as determined by the emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide information about the statue's historical background and recommended sightseeing routes. The device then displays this information on the smart glasses' display or provides it as audio feedback.

[1036] Examples: Tourist assistance and emotional support

[1037] Specific examples

[1038] When a user visits a famous tourist spot and looks at a statue, the device tracks the user's gaze, capturing an image of the statue and sending it to a server. The server then analyzes the statue's detailed information, extracting data about its history and artist, for example. This information is customized based on the user's past visits and interests and displayed visually on the smart glasses. Furthermore, an emotion engine analyzes the user's emotional state, providing the location of a nearby bench if, for example, they are feeling tired. Visually impaired users can also receive this information through audio guidance and emotional feedback.

[1039] Prompt Sentence Examples

[1040] If the user wants to get information, they can enter a prompt like this:

[1041] "Please tell me more about the statue I'm looking at. I'd like to know its historical background and who made it."

[1042] This system allows users to receive personalized information based on the object in front of their eyes, while also receiving assistance from an emotion engine. This technology is expected to contribute not only to tourism support, but also to support for the visually impaired, medical care, and improving efficiency in various workplaces.

[1043] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1044] Step 1: User Awareness

[1045] When a user puts on a wearable device (smart glasses), the device uses its built-in camera and sensors to identify the user's ID. The device then uses facial recognition technology to identify the user's ID and reads the user's unique settings and history data from its built-in memory. This makes it possible to provide information that takes into account the user's preferences and past usage history.

[1046] input:

[1047] User's facial image, biometric data

[1048] Data processing / calculation:

[1049] It uses a facial recognition algorithm to identify the user ID and loads corresponding settings and history data.

[1050] output:

[1051] User ID, user settings and history data

[1052] Specific behavior:

[1053] The device uses a facial recognition sensor to capture the user's facial image and verify their identity by comparing it with an internal database.

[1054] Step 2: Eye tracking

[1055] The sensor tracks the user's eye movements in real time and identifies the position of the gaze. The device uses an infrared sensor to detect the user's gaze position and updates the data in real time, thereby identifying the position where the gaze is fixed.

[1056] input:

[1057] User eye position data

[1058] Data processing / calculation:

[1059] Uses an eye-tracking algorithm to determine the user's gaze position.

[1060] output:

[1061] Gaze position data

[1062] Specific behavior:

[1063] The device's infrared sensor detects the user's eye movements, analyzes the data, and displays the gaze position in real time.

[1064] Step 3: Collecting gaze data and capturing images

[1065] When a user's gaze is fixed at a specific location, the device automatically captures an image of the object at that location using its camera, and this image data is collected in real time and sent to an external server.

[1066] input:

[1067] Gaze position data

[1068] Data processing / calculation:

[1069] Controls the camera based on gaze position to capture images of the target object

[1070] output:

[1071] Captured image data

[1072] Specific behavior:

[1073] The camera automatically focuses on the position where the gaze is fixed and captures an image of the object.

[1074] Step 4: Sending image data

[1075] The captured image data is sent to an external server using a secure communication protocol. The device sends the image data to the server using HTTPS.

[1076] input:

[1077] Captured image data

[1078] Data processing / calculation:

[1079] Data encryption and transmission

[1080] output:

[1081] Image data sent to the server

[1082] Specific behavior:

[1083] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[1084] Step 5: Image analysis

[1085] The server analyzes the received image data and extracts information about the object in the line of sight. Using a generative AI model, it analyzes detailed information about the object in the line of sight.

[1086] input:

[1087] Received image data

[1088] Data processing / calculation:

[1089] Extract information using image recognition algorithms

[1090] output:

[1091] Analyzed object information

[1092] Specific behavior:

[1093] The server runs a generative AI model to extract detailed information and related data about the object.

[1094] Step 6: Emotion Recognition

[1095] The device analyzes the user's biometric data (heart rate, facial expressions, etc.) collected by the device, and the emotion engine identifies the user's emotional state. The emotion engine analyzes the user's biometric data in real time to determine the user's emotional state.

[1096] input:

[1097] User biometric data

[1098] Data processing / calculation:

[1099] Identify emotional states using emotion recognition algorithms

[1100] output:

[1101] The user's emotional state

[1102] Specific behavior:

[1103] The device analyzes data from a heart rate monitor and facial recognition sensors to determine whether the user is relaxed or tired.

[1104] Step 7: Personalized information generation

[1105] The server then uses the analyzed information to generate personalized information based on the user's history, preferences, and emotional state, as determined by an emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide historical background about the statue and recommend sightseeing routes.

[1106] input:

[1107] Analyzed object information, user emotional state, historical data and settings

[1108] Data processing / calculation:

[1109] Data Integration and Information Generation

[1110] output:

[1111] Personalized Information

[1112] Specific behavior:

[1113] The server integrates this data to generate customized information. The generative AI model is given a prompt: "Please tell me more about the statue I'm looking at. I'd also like to know its historical background and its creator.", and the required data is generated.

[1114] Step 8: Provide information

[1115] The device provides the generated information to the user, either visually displayed on the smart glasses display or as audio feedback.

[1116] input:

[1117] Personalized Information

[1118] Data processing / calculation:

[1119] Display format conversion

[1120] output:

[1121] Visual or audio information provided to the user

[1122] Specific behavior:

[1123] The device displays text about the statue's historical background and simultaneously provides audio feedback explaining the statue's creator.

[1124] (Application example 2)

[1125] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1126] In current factory work, workers must manually search or refer to manuals to obtain information about individual parts and tools. This procedure is inefficient and increases the burden on workers. In addition, there is a lack of a system to detect workers' stress and fatigue in real time and provide appropriate breaks and advice. As a result, work efficiency decreases and there is a risk of work errors and safety issues.

[1127] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a wearable device equipped with an optical device and a detection device for tracking the user's gaze, an gaze tracking device for identifying the user's gaze position based on data acquired by the optical device and the detection device, means for capturing an image based on the gaze position identified by the gaze tracking device, means for transmitting the captured image to an external installation device, means for the external installation device to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, an emotion recognition device for identifying the user's emotional state, and means for providing advice and work improvement suggestions to the user based on the emotional state identified by the emotion recognition device. This enables the worker to obtain information on the parts and tools in front of the worker's gaze in real time and to receive appropriate breaks and advice according to the worker's emotional state, thereby improving work efficiency and ensuring safety.

[1128] An "optical device" is a device that handles light, such as a camera or lens, to detect the user's line of sight.

[1129] A "detection device" is a sensor or device that acquires gaze data or biometric data.

[1130] A "wearable device" is a digital device that can be worn by the user, examples of which include smart glasses and head-mounted displays.

[1131] An "eye tracking device" is a system that tracks the position of a user's gaze in real time.

[1132] "Means for capturing" refers to a function for acquiring an image at a specified gaze position.

[1133] An "external device" is a server or cloud service that receives and analyzes data sent from a wearable device.

[1134] "Means for analyzing information" refers to the algorithms or software that analyze the captured image data and extract information about the object in the user's line of sight.

[1135] "Means for providing personalized information" refers to a function that customizes analyzed information based on the user's history and settings, and provides it in the most optimal form for the user.

[1136] An "emotion recognition device" refers to a sensor or analysis device used to detect a user's emotional state.

[1137] The "means for providing advice and work improvement suggestions" is a function that presents optimal breaks and suggestions for improving work procedures based on the user's emotional state detected by the emotion recognition device.

[1138] An embodiment of the present invention will be described. This system uses a head-mounted display (HMD) as a wearable device worn by a factory worker. The main elements and their operations are shown below.

[1139] Hardware and software used

[1140] 1. Wearable device (HMD): A device equipped with optical and detection devices to track the user's gaze.

[1141] 2. Eye tracking device: A system that uses the dlib library and other tools to track the user's gaze position in real time.

[1142] 3. External device (server): This device receives the captured image data and analyzes the information. TensorFlow is used for this analysis.

[1143] 4. Emotion Recognition Device: Used to analyze camera footage and identify the user's emotional state. Here, we use an emotion recognition model built with TensorFlow.

[1144] Data handling and calculations

[1145] 1. Gaze Tracking: The HMD camera captures the user's face and uses the dlib library to detect facial landmarks. The gaze direction is calculated based on these landmarks.

[1146] 2. Image capture and transmission: Images of the object are automatically captured based on the direction of gaze and transmitted to an external server, where they are analyzed in real time.

[1147] 3. Analysis and provision of information: The server analyzes the received image data using TensorFlow and extracts information about the parts and tools in front of the user's gaze. This information is then customized based on the user's history and settings and displayed on the HMD.

[1148] 4. Identifying and responding to emotional states: The HMD is equipped with an emotion recognition device that analyzes the user's facial expressions, heart rate, etc. to identify their emotional state. For example, if the user is feeling stressed, the system will suggest taking a break.

[1149] Specific examples

[1150] When a factory worker looks at a part of the robot arm they are operating, the HMD visually displays the manual and work procedures related to that part. Also, if the HMD detects that the worker is under stress, it will display instructions to take a break.

[1151] Example prompts for generative AI models

[1152] "Factory workers wear head-mounted displays and work while tracking their gaze. Generate an application program that displays information about parts and tools in front of their eyes in real time, analyzes the worker's emotional state, and suggests taking a break if the worker feels fatigued or stressed."

[1153] In this way, the system provides real-time information on the line of sight of factory workers and gives appropriate advice according to their emotional state, thereby improving work efficiency and ensuring safety.

[1154] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1155] Step 1:

[1156] The user wears a wearable device (HMD). The HMD has built-in optical and detection devices, which enable gaze tracking and emotion recognition. The input is the user's facial expression and gaze data, and the output is the capture of this data.

[1157] Step 2:

[1158] The device uses an eye-tracking device to track the user's gaze in real time. The dlib library is used to calculate the user's gaze direction from the captured facial landmark information. The input is the captured facial image data, and the output is the calculated gaze direction.

[1159] Step 3:

[1160] The gaze tracking device captures an image of the object in front of the gaze. The captured image is sent to an external device (server). The input is the calculated gaze direction and image data of the gaze destination, and the output is data sent to the server.

[1161] Step 4:

[1162] The server receives images sent from an external source and uses TensorFlow to analyze information about parts and tools in the line of sight. The analyzed information is personalized based on the user's history and settings. The input is the captured image data, and the output is the analyzed information about the line of sight.

[1163] Step 5:

[1164] The server sends the analyzed information to the HMD and displays it visually. The user can check detailed information about parts and tools in real time through the HMD. The input is the analyzed information, and the output is the information displayed on the HMD.

[1165] Step 6:

[1166] The device uses an emotion recognition device to analyze biometric data such as the user's facial expressions and heart rate to identify their emotional state. An emotion recognition model built with TensorFlow is used. The input is the user's biometric data, and the output is the identified emotional state.

[1167] Step 7:

[1168] The server sends appropriate advice and work improvement suggestions to the HMD based on the emotional state identified by the emotion recognition device. For example, if the user is recognized as being in a stressful state, a message suggesting a break will be displayed. The input is the identified emotional state, and the output is the advice displayed on the HMD.

[1169] Step 8:

[1170] The user performs the work according to the information and advice displayed through the HMD, which improves work efficiency and ensures safety. The input is the information displayed by the HMD, and the output is the user's work behavior.

[1171] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1172] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1173] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1174] [Fourth embodiment]

[1175] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1176] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1177] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1178] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1179] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1180] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1181] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1182] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1183] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1184] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1185] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1186] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1187] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1188] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Specifically, a wearable device equipped with a camera and sensors is used to track the user's gaze in real time, capture images based on the gaze position, and transmit them to an external server. The external server analyzes the images, generates and personalizes the necessary information, and provides it to the user in an appropriate format.

[1189] User Recognition and Eye Tracking

[1190] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[1191] Gaze data collection and image capture

[1192] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[1193] Image analysis and information generation

[1194] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[1195] Personalized information provision

[1196] Based on the information analyzed by the server, personalized information is generated according to the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[1197] Example: Tourism support

[1198] Consider a case where a user visits a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[1199] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[1200] The processing flow will be explained below.

[1201] Step 1:

[1202] The user puts on the wearable device (smart glasses). The smart glasses start up and the camera and sensors begin operating.

[1203] Step 2:

[1204] The device identifies the user ID. Based on the user ID, it loads individual settings and history data. This data is necessary to provide information tailored to the user's past usage history and preferences.

[1205] Step 3:

[1206] The device uses a camera and sensors to track the user's gaze position in real time. The gaze tracking means sequentially acquires gaze position data and captures the position information.

[1207] Step 4:

[1208] The device captures an image of the object using the camera based on the gaze position, and the captured image data is temporarily stored in the device.

[1209] Step 5:

[1210] The device sends the captured image data to an external server, including gaze position information.

[1211] Step 6:

[1212] The server analyzes the received image data and uses advanced image recognition algorithms to identify objects and extract relevant information, such as the history and characteristics of statues at tourist attractions.

[1213] Step 7:

[1214] The server uses the analysis results to generate personalized information based on the user's history and settings, making it possible to provide information tailored to the individual needs and preferences of each user.

[1215] Step 8:

[1216] The server sends the generated information to the device, which then displays the received personalized information on the smart glasses display.

[1217] Step 9:

[1218] If the device is for the visually impaired, it will not only provide visual feedback but also audio feedback, conveying the necessary information to the user through audio guides.

[1219] Step 10:

[1220] Users review information and act on it, for example, learning about the history of a statue at a tourist spot and choosing a sightseeing route around the area.

[1221] This process allows users to obtain personalized information based on their gaze in real time, and audio information can be provided to visually impaired people, making it suitable for a wide range of users.

[1222] Example 1

[1223] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1224] Conventional eye-tracking systems have difficulty providing information about the object a user is looking at instantly and in line with individual needs. They also have difficulty supporting visually impaired users and lack real-time information provision. There is a need for a system that can resolve these issues and provide more convenient and useful information to users.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1226] In this invention, the server includes means for analyzing information based on the captured image using an advanced image recognition algorithm, means for providing the analyzed information in a personalized manner according to the user's history and settings, and means for providing the analyzed information as audio feedback to the user, thereby enabling real-time personalized information provision based on the user's gaze.

[1227] - "Camera" means a device that converts light into electrical signals and is used to capture an image of an object in the user's line of sight.

[1228] A "sensor" is a device that detects physical phenomena and converts the signals into electrical signals, and is used to detect things like the user's line of sight.

[1229] "Wearable device" refers to any digital device that is worn on the body, and in this invention it primarily refers to devices with built-in cameras and sensors.

[1230] "Eye tracking means" refers to technology or devices that detect the movement of a user's eyes and identify the position of their gaze, and in this invention includes sensors and dedicated software algorithms.

[1231] "Image capturing means" means a function or device that takes an image of an object that a user is looking at based on an eye-tracking means.

[1232] "External Server" means a computer system that receives data sent from a wearable device and performs advanced analysis.

[1233] "Image recognition algorithm" refers to a computational method for analyzing captured image data and identifying objects and textual information present within it.

[1234] "Personalization" means customizing information based on each user's history and preferences, and presenting it in a way that is most appropriate for the person providing it.

[1235] "Voice feedback" refers to a means of transmitting information that provides analyzed information to the user as audio, and is particularly suitable for visually impaired people.

[1236] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. This system uses a wearable device equipped with a camera and sensors to track the user's gaze in real time. As a result, an image is captured based on the gaze position and sent to an external server. The external server analyzes the image, generates the necessary information, personalizes it, and provides it to the user in an appropriate format.

[1237] User Recognition and Eye Tracking

[1238] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is used to provide personalized information based on the user's preferences and past usage history.

[1239] Gaze data collection and image capture

[1240] The device uses eye tracking to determine the user's gaze position. Specifically, sensors in the smart glasses detect the user's eye movements in real time and calculate the gaze position. Based on the determined gaze position, the device's camera automatically captures an image of the object at that position. This image is sent to an external server in real time.

[1241] Image analysis and information generation

[1242] The transmitted image data is received by an external server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the gaze. For example, if the image is a statue at a tourist spot, its historical background and related interesting facts can be extracted. Specific technologies used include Google Cloud Vision API and Microsoft Azure Computer Vision.

[1243] Personalized information provision

[1244] Based on the information analyzed by the server, personalized information is generated based on the user's history and settings. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smart glasses display or provide audio feedback.

[1245] Example: Tourism support

[1246] Consider the example of a user visiting a famous tourist spot. When the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and creator. The analyzed information is customized based on the user's past visit history and interests and displayed visually on the smart glasses. In addition, if the user is visually impaired, the device can also provide audio guidance.

[1247] Example prompts for generative AI models

[1248] Example of input prompt:

[1249] "Describe a system that provides customized information about a specific object when the user is looking at that object. What hardware and software does this system use and how does it generate the information?"

[1250] This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used for a wide range of purposes, including medical care and workplaces, in addition to visually impaired people and tourists. It is expected that this will significantly improve the efficiency of users' daily lives and work.

[1251] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1252] Step 1:

[1253] User recognition and device initialization

[1254] When a user wears the smart glasses, the device uses the camera to recognize the user's face and uses a facial recognition algorithm (e.g., OpenCV's facial recognition function) to identify the user's ID.

[1255] Input: User's face image

[1256] Data processing: Facial feature points are extracted using a facial recognition algorithm, and the corresponding user ID is searched for in the database.

[1257] Output: User ID

[1258] What it does: The smart glasses' camera captures a picture of the user's face, applies a facial recognition algorithm, and, based on the identified user ID, loads the user's settings and history data from cloud storage or the glasses' internal memory.

[1259] Step 2:

[1260] Eye tracking and data collection

[1261] The device determines the user's gaze position using the eye tracking sensor in the smart glasses, which detects the user's eye movements in real time.

[1262] Input: User's eye movement data

[1263] Data processing: Eye tracking algorithm calculates eye gaze direction

[1264] Output: Gaze direction and position information

[1265] How it works: The eye tracking sensor detects the movement of the user's eyes, and the eye tracking algorithm processes that data to determine the position of the gaze.

[1266] Step 3:

[1267] Image capture and transmission

[1268] Based on the results of gaze tracking, the device's camera captures an image of the object in front of the user's eyes, which is then compressed, encrypted, and sent to an external server.

[1269] Input: gaze position information, image of object

[1270] Data processing: image capture, compression (e.g., JPEG format), encryption (e.g., SSL)

[1271] Output: Compressed and encrypted image data

[1272] Specific operation: When a fixed gaze is detected, the camera automatically releases the shutter and captures an image. The image is then compressed into JPEG format, encrypted using SSL, and sent to the server.

[1273] Step 4:

[1274] Image analysis and information generation

[1275] The server receives the transmitted image data and analyzes the image using advanced image recognition algorithms (e.g., Google Cloud Vision API or Microsoft Azure Computer Vision).

[1276] Input: Encrypted image data

[1277] Data processing: Decrypting data, applying image recognition algorithms, and identifying objects

[1278] Output: Information about the object (e.g., history, characteristics)

[1279] Specific operation: The received image data is decoded, and an image recognition API is called to identify the object and extract its information.

[1280] Step 5:

[1281] Personalized information provision

[1282] The server analyzes the information and generates personalized information based on the user's history and settings, which is then sent to the device and displayed visually on the smart glasses display or provided as audio feedback.

[1283] Input: Information about the object, user history and settings information

[1284] Data processing: Extraction of relevant information, personalized processing

[1285] Output: Personalized information (visual / audio)

[1286] What it does: It matches the analyzed information with the user's past history and preferences to select the most relevant information, and then provides that information as an AR display or audio guide.

[1287] Through the above process, this system provides real-time, personalized information based on the user's line of sight.

[1288] (Application example 1)

[1289] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1290] Conventional wearable devices and eye-tracking technologies focused on tracking the user's gaze, but lacked the ability to instantly provide detailed information about the object in front of the user's eyes. Furthermore, there were insufficient means to provide real-time information about products that users were interested in in physical stores. Furthermore, voice feedback functions for the visually impaired were not fully developed, which led to issues with the efficiency of information acquisition.

[1291] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1292] In this invention, the server includes a wearable device equipped with a camera and a sensor for tracking the user's gaze, gaze tracking means for identifying the user's gaze position based on data acquired by the camera and the sensor, means for capturing an image based on the gaze position identified by the gaze tracking means, means for transmitting the captured image to an external server, means for the external server to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, and means for providing information on products the user is looking at in a physical store in real time through a display device of smart glasses. This allows users to obtain information on products they are interested in in a physical store in real time, and also makes it possible to provide information to visually impaired people via voice feedback.

[1293] "Eye tracking" is a technology that tracks and identifies the position of a user's gaze.

[1294] A "wearable device" is an electronic device that is designed to be worn by the user.

[1295] "Cameras and Sensors" means the imaging devices and detectors used to capture user gaze data.

[1296] "Capturing an image" means taking a picture of an object at a particular line of sight.

[1297] An "external server" is a remote computer system that can be communicated with via the Internet.

[1298] "Analyzing information" means identifying details and features of an object based on the acquired data and extracting relevant information.

[1299] "Personalization" means adapting acquired information to the preferences and history of individual users.

[1300] "Blind people" means people who are visually impaired.

[1301] "Smart glasses" are a type of wearable eyeglass device that displays information based on the wearer's line of sight.

[1302] "Voice feedback" refers to a method of providing acquired information to the user in the form of voice.

[1303] The system for realizing this invention uses smart glasses as a wearable device worn by the user. The smart glasses have a built-in camera and sensor, and are equipped with an eye-tracking function that tracks the user's gaze. Below, we will explain the specific hardware and software used in each step, as well as the data processing and data calculation methods using them.

[1304] Hardware and software used

[1305] Hardware:

[1306] Smart glasses (with built-in camera and sensors)

[1307] Cloud server (general remote computer)

[1308] software:

[1309] Image recognition algorithms (TensorFlow, OpenCV)

[1310] User history management system (database)

[1311] Smart Glasses Display Control Software

[1312] Data Processing and Data Calculation

[1313] 1. Eye Tracking:

[1314] When a user wears the smart glasses, the sensors and cameras in the smart glasses track the user's gaze and identify the gaze position in real time. The gaze data is processed by a processor in the smart glasses and recorded as gaze position information.

[1315] 2. Image capture and transmission:

[1316] Based on the identified gaze position, the camera automatically captures an image of the object in front of the gaze, and the captured image is transmitted to an external cloud server via wireless communication.

[1317] 3. Image analysis and information extraction:

[1318] The server analyzes the received image data using advanced image recognition algorithms (TensorFlow, OpenCV) to extract information about the object in front of the gaze. For example, in the case of a product in a physical store, information such as its price, ingredients, and user reviews can be extracted.

[1319] 4. Personalization and Information:

[1320] The server then personalizes the extracted information based on the user's past purchase history and preferences (managed in a database). The personalized information is displayed on the smart glasses' display or provided as audio feedback. It is also possible to provide the user with appropriate suggestions (recommended products or promotional information) based on specific prompts.

[1321] Specific examples

[1322] Imagine a user browsing a store shelf. When the user directs their gaze at a particular product, the smartglasses' camera captures an image of that product and sends it to a cloud server. The server analyzes the received image and extracts the product's price, ingredients, and user reviews. Based on the user's past purchase history and preferences, the smartglasses display then displays personalized information, such as "This product is recommended based on your past purchase history."

[1323] Prompt Sentence Examples

[1324] When a user wants to know more about this product, a prompt like this is used:

[1325] "Track the user's gaze and capture the product they are looking at. Send the captured image to a server to retrieve product information. Customize and display the information based on the user's preferences."

[1326] Based on this prompt, the system will identify product information that interests the user and provide personalized information.

[1327] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1328] Step 1:

[1329] The user puts on the smart glasses and activates the sensors and camera. The eye tracking system collects the user's gaze data in real time. Specifically, the camera captures the user's eye movements, and the sensor identifies the gaze position. Input: User's eye movements. Output: User's gaze data (gaze position information).

[1330] Step 2:

[1331] The device identifies the position of the object in front of the user's gaze based on the collected gaze data. Then, it uses the camera to capture an image of the object at that identified position. Input: User's gaze data. Output: Captured image.

[1332] Step 3:

[1333] The captured image is wirelessly transmitted from the device to an external server. Specifically, the image data is uploaded to a cloud server via a wireless communication module. Input: Captured image. Output: Image data stored on the server.

[1334] Step 4:

[1335] Based on the received image data, the server uses an image recognition algorithm (e.g., TensorFlow, OpenCV) to recognize the object in front of the user's line of sight and analyzes the information. Specifically, it extracts information such as the product name, price, ingredients, and user reviews. Input: Image data stored on the server. Output: Detailed information about the object.

[1336] Step 5:

[1337] The server uses the analyzed information to query a database of the user's history and preferences to personalize the information. For example, it generates related information and product recommendations based on the user's preferences and the product history of previous purchases. Input: Detailed information about the object, user history data. Output: Personalized information.

[1338] Step 6:

[1339] The server sends the personalized information to the device. Specifically, the information is sent to the smart glasses via wireless communication, where it is displayed visually on the smart glasses' display. In some cases, the information can also be provided as an audio guide. Input: Personalized information. Output: Smart glasses' display or audio guide.

[1340] Step 7:

[1341] Users can check the information displayed on the smart glasses display and obtain information about products and promotions that interest them. For visually impaired people, information is obtained through audio guidance. Input: Smart glasses display, audio guidance. Output: Information obtained by the user.

[1342] This series of processes allows users to obtain detailed product information and personalized recommendations in real time simply by looking at the product in a physical store.

[1343] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1344] The present invention relates to a system that tracks a user's gaze and provides customized information to the user based on the object they are looking at. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system can further personalize the information provided and provide support tailored to the user's psychological state.

[1345] User Recognition and Eye Tracking

[1346] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[1347] Gaze data collection and image capture

[1348] The device uses eye tracking to identify the user's gaze position. Based on the identified gaze position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[1349] Image analysis and information generation

[1350] The image data is then received by an external server, which uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the viewer's eyes. For example, if the image is a statue at a tourist spot, the server can extract historical background and other interesting facts about it.

[1351] Emotion recognition by emotion engine

[1352] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[1353] Personalized information provision

[1354] Based on the information analyzed by the server, the system generates personalized information based on the user's history, settings, and emotional state using an emotion engine. For example, if a user is looking at a statue at a tourist spot, the system will provide information about the statue's historical background and recommended sightseeing routes. The device will display this information visually on the smartglasses' display or provide it as audio feedback.

[1355] Examples: Tourist assistance and emotional support

[1356] Consider a case where a user visits a famous tourist spot. As the user looks at a statue at the tourist spot, the device tracks the user's gaze and captures an image of the statue. This image is sent to a server, which analyzes detailed information about the statue, such as its history and artist. The analyzed information is then customized based on the user's past visits and interests and displayed visually on the smart glasses.

[1357] Furthermore, the emotion engine analyzes the user's emotional state and, for example, if they feel tired, provides the location of a nearby bench. For visually impaired users, this information is provided via audio guidance, and feedback is given according to their emotions.

[1358] In this way, users can receive personalized information based on the object they are looking at, while simultaneously receiving psychological support from the emotion engine. This system allows users to effectively obtain information tailored to their individual needs in real time. It can also be used in a wide range of applications, including medical and workplace settings, in addition to the visually impaired and tourists. This is expected to significantly improve the efficiency of users' daily lives and work.

[1359] The processing flow will be explained below.

[1360] Step 1:

[1361] The user puts on the wearable device (smart glasses). The device starts up and the camera and sensors begin operating.

[1362] Step 2:

[1363] Identify user ID: The device recognizes the user ID and loads individual settings and history data based on that ID.

[1364] Step 3:

[1365] Using eye tracking, the device tracks the user's eye position in real time, sequentially acquiring eye position data and capturing that position information.

[1366] Step 4:

[1367] Based on the gaze position, the device's camera automatically captures an image of the object at that position, and the captured image data is temporarily stored on the device.

[1368] Step 5:

[1369] The device sends the captured image data to an external server, including gaze position information.

[1370] Step 6:

[1371] The server analyzes the received image data. Using advanced image recognition algorithms, it identifies the objects and extracts relevant information. For example, the server analyzes the history and characteristics of statues at tourist attractions.

[1372] Step 7:

[1373] Using an emotion engine, the device recognizes the user's emotional state by analyzing the user's heart rate and facial expressions in real time based on gaze data and data from biometric sensors.

[1374] Step 8:

[1375] Based on the information analyzed by the server, personalized information is generated based on the user's history, settings, and emotional state using an emotion engine, including providing information according to the user's psychological state.

[1376] Step 9:

[1377] The server sends the generated personalized information to the device, which then visually displays the information on the smartglasses display.

[1378] Step 10:

[1379] For the visually impaired, the device provides audio feedback, conveying necessary information and emotional feedback to the user through audio guidance.

[1380] Step 11:

[1381] Users review information through displays or audio guides and use that information to make decisions about what to do at a tourist attraction, such as learning about the history of a statue or finding the location of a nearby bench.

[1382] This process allows users to obtain personalized information based on their gaze in real time, and with the support of an emotion engine, appropriate feedback tailored to their psychological state is provided, improving user satisfaction and enabling a wide range of applications.

[1383] Example 2

[1384] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] Conventional wearable devices are limited to providing information by tracking the user's gaze, making it difficult to provide personalized information tailored to the user's emotional state. Furthermore, their support for visually impaired people is insufficient, so an improvement in the user experience is needed.

[1386] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1387] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[1388] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[1389] means for capturing an image based on the gaze position identified by the gaze tracking means;

[1390] means for transmitting the captured image to an external server;

[1391] means for the external server to analyze information based on the captured image;

[1392] A means for personalizing the analyzed information and providing it to the user;

[1393] an emotion recognition means for identifying an emotional state of a user based on biometric data;

[1394] a means for personalizing information provided based on the user's emotional state;

[1395] a means for conveying the provided information to the user as a display or audio feedback;

[1396] This allows for personalized information to be provided based on the user's emotional state and the object in their line of sight.

[1397] "Eye tracking" is a feature that uses cameras and sensors to determine the position of a user's gaze in real time.

[1398] "Wearable devices" generally refer to electronic devices that can be worn by the user on the body, including smart glasses.

[1399] "Cameras and Sensors" means the imaging and detection devices used to track the user's gaze.

[1400] "Image capture" refers to the act of using a camera to obtain a still image of an object at a particular gaze position.

[1401] A "server" is a computer system that stores, analyzes, and provides data over a network.

[1402] "Information analysis" refers to the process of processing received data to extract useful information.

[1403] "Personalization" means providing information that is optimized based on the user's individual settings, history, and emotional state.

[1404] "Emotion recognition" is a technology that analyzes a user's biometric data to identify their current emotional state.

[1405] "Biometric data" means physiological data such as a user's heart rate, facial expression, and skin temperature.

[1406] A "display" is a screen device for presenting visual information to a user.

[1407] "Audio feedback" is a means of conveying information to the user through sound.

[1408] "Visually impaired people" refers to people who have problems with their eyesight and have difficulty obtaining normal visual information.

[1409] MODE FOR CARRYING OUT THE INVENTION

[1410] The present invention relates to a system that tracks a user's gaze and provides customized information based on the object in their line of sight. Furthermore, by combining it with an emotion engine, the system provides personalized information tailored to the user's emotional state.

[1411] User Recognition and Eye Tracking

[1412] A user puts on a wearable device, such as smart glasses. The smart glasses are equipped with multiple cameras and sensors that track the user's gaze in real time. The device identifies the user's ID and loads the user's settings and history data based on that ID. This data is necessary to provide information based on the user's preferences and past usage history.

[1413] Gaze data collection and image capture

[1414] The device uses eye tracking to identify the user's gaze position. When the gaze is fixed at a specific position, the device's camera automatically captures an image of the object at that position. The image is collected in real time and sent to an external server.

[1415] Image analysis and information generation

[1416] The transmitted image data is received by a server. The server uses advanced image recognition algorithms to analyze the captured image and extract information about the object in front of the user's gaze. For example, if the user is looking at a statue at a tourist attraction, the server can extract the statue's historical background and related interesting facts. This analysis is performed using a generative AI model.

[1417] Emotion recognition by emotion engine

[1418] The device is equipped with an emotion engine that identifies the user's emotional state based on the user's gaze data and data from biometric sensors. The emotion engine analyzes the user's biometric data, such as heart rate and facial expressions, in real time to determine the user's emotional state.

[1419] Personalized information provision

[1420] The server then uses the analyzed information to generate personalized information based on the user's history, settings, and emotional state as determined by the emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide information about the statue's historical background and recommended sightseeing routes. The device then displays this information on the smart glasses' display or provides it as audio feedback.

[1421] Examples: Tourist assistance and emotional support

[1422] Specific examples

[1423] When a user visits a famous tourist spot and looks at a statue, the device tracks the user's gaze, capturing an image of the statue and sending it to a server. The server then analyzes the statue's detailed information, extracting data about its history and artist, for example. This information is customized based on the user's past visits and interests and displayed visually on the smart glasses. Furthermore, an emotion engine analyzes the user's emotional state, providing the location of a nearby bench if, for example, they are feeling tired. Visually impaired users can also receive this information through audio guidance and emotional feedback.

[1424] Prompt Sentence Examples

[1425] If the user wants to get information, they can enter a prompt like this:

[1426] "Please tell me more about the statue I'm looking at. I'd like to know its historical background and who made it."

[1427] This system allows users to receive personalized information based on the object in front of their eyes, while also receiving assistance from an emotion engine. This technology is expected to contribute not only to tourism support, but also to support for the visually impaired, medical care, and improving efficiency in various workplaces.

[1428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1429] Step 1: User Awareness

[1430] When a user puts on a wearable device (smart glasses), the device uses its built-in camera and sensors to identify the user's ID. The device then uses facial recognition technology to identify the user's ID and reads the user's unique settings and history data from its built-in memory. This makes it possible to provide information that takes into account the user's preferences and past usage history.

[1431] input:

[1432] User's facial image, biometric data

[1433] Data processing / calculation:

[1434] It uses a facial recognition algorithm to identify the user ID and loads corresponding settings and history data.

[1435] output:

[1436] User ID, user settings and history data

[1437] Specific behavior:

[1438] The device uses a facial recognition sensor to capture the user's facial image and verify their identity by comparing it with an internal database.

[1439] Step 2: Eye tracking

[1440] The sensor tracks the user's eye movements in real time and identifies the position of the gaze. The device uses an infrared sensor to detect the user's gaze position and updates the data in real time, thereby identifying the position where the gaze is fixed.

[1441] input:

[1442] User eye position data

[1443] Data processing / calculation:

[1444] Uses an eye-tracking algorithm to determine the user's gaze position.

[1445] output:

[1446] Gaze position data

[1447] Specific behavior:

[1448] The device's infrared sensor detects the user's eye movements, analyzes the data, and displays the gaze position in real time.

[1449] Step 3: Collecting gaze data and capturing images

[1450] When a user's gaze is fixed at a specific location, the device automatically captures an image of the object at that location using its camera, and this image data is collected in real time and sent to an external server.

[1451] input:

[1452] Gaze position data

[1453] Data processing / calculation:

[1454] Controls the camera based on gaze position to capture images of the target object

[1455] output:

[1456] Captured image data

[1457] Specific behavior:

[1458] The camera automatically focuses on the position where the gaze is fixed and captures an image of the object.

[1459] Step 4: Sending image data

[1460] The captured image data is sent to an external server using a secure communication protocol. The device sends the image data to the server using HTTPS.

[1461] input:

[1462] Captured image data

[1463] Data processing / calculation:

[1464] Data encryption and transmission

[1465] output:

[1466] Image data sent to the server

[1467] Specific behavior:

[1468] The device encrypts the data and sends it securely to the server using the HTTPS protocol.

[1469] Step 5: Image analysis

[1470] The server analyzes the received image data and extracts information about the object in the line of sight. Using a generative AI model, it analyzes detailed information about the object in the line of sight.

[1471] input:

[1472] Received image data

[1473] Data processing / calculation:

[1474] Extract information using image recognition algorithms

[1475] output:

[1476] Analyzed object information

[1477] Specific behavior:

[1478] The server runs a generative AI model to extract detailed information and related data about the object.

[1479] Step 6: Emotion Recognition

[1480] The device analyzes the user's biometric data (heart rate, facial expressions, etc.) collected by the device, and the emotion engine identifies the user's emotional state. The emotion engine analyzes the user's biometric data in real time to determine the user's emotional state.

[1481] input:

[1482] User biometric data

[1483] Data processing / calculation:

[1484] Identify emotional states using emotion recognition algorithms

[1485] output:

[1486] The user's emotional state

[1487] Specific behavior:

[1488] The device analyzes data from a heart rate monitor and facial recognition sensors to determine whether the user is relaxed or tired.

[1489] Step 7: Personalized information generation

[1490] The server then uses the analyzed information to generate personalized information based on the user's history, preferences, and emotional state, as determined by an emotion engine. For example, if the user is looking at a statue at a tourist spot, the server can provide historical background about the statue and recommend sightseeing routes.

[1491] input:

[1492] Analyzed object information, user emotional state, historical data and settings

[1493] Data processing / calculation:

[1494] Data Integration and Information Generation

[1495] output:

[1496] Personalized Information

[1497] Specific behavior:

[1498] The server integrates this data to generate customized information. The generative AI model is given a prompt: "Please tell me more about the statue I'm looking at. I'd also like to know its historical background and its creator.", and the required data is generated.

[1499] Step 8: Provide information

[1500] The device provides the generated information to the user, either visually displayed on the smart glasses display or as audio feedback.

[1501] input:

[1502] Personalized Information

[1503] Data processing / calculation:

[1504] Display format conversion

[1505] output:

[1506] Visual or audio information provided to the user

[1507] Specific behavior:

[1508] The device displays text about the statue's historical background and simultaneously provides audio feedback explaining the statue's creator.

[1509] (Application example 2)

[1510] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1511] In current factory work, workers must manually search or refer to manuals to obtain information about individual parts and tools. This procedure is inefficient and increases the burden on workers. In addition, there is a lack of a system to detect workers' stress and fatigue in real time and provide appropriate breaks and advice. As a result, work efficiency decreases and there is a risk of work errors and safety issues.

[1512] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a wearable device equipped with an optical device and a detection device for tracking the user's gaze, an gaze tracking device for identifying the user's gaze position based on data acquired by the optical device and the detection device, means for capturing an image based on the gaze position identified by the gaze tracking device, means for transmitting the captured image to an external installation device, means for the external installation device to analyze information based on the captured image, means for personalizing the analyzed information and providing it to the user, an emotion recognition device for identifying the user's emotional state, and means for providing advice and work improvement suggestions to the user based on the emotional state identified by the emotion recognition device. This enables the worker to obtain information on the parts and tools in front of the worker's gaze in real time and to receive appropriate breaks and advice according to the worker's emotional state, thereby improving work efficiency and ensuring safety.

[1513] An "optical device" is a device that handles light, such as a camera or lens, to detect the user's line of sight.

[1514] A "detection device" is a sensor or device that acquires gaze data or biometric data.

[1515] A "wearable device" is a digital device that can be worn by the user, examples of which include smart glasses and head-mounted displays.

[1516] An "eye tracking device" is a system that tracks the position of a user's gaze in real time.

[1517] "Means for capturing" refers to a function for acquiring an image at a specified gaze position.

[1518] An "external device" is a server or cloud service that receives and analyzes data sent from a wearable device.

[1519] "Means for analyzing information" refers to the algorithms or software that analyze the captured image data and extract information about the object in the user's line of sight.

[1520] "Means for providing personalized information" refers to a function that customizes analyzed information based on the user's history and settings, and provides it in the most optimal form for the user.

[1521] An "emotion recognition device" refers to a sensor or analysis device used to detect a user's emotional state.

[1522] The "means for providing advice and work improvement suggestions" is a function that presents optimal breaks and suggestions for improving work procedures based on the user's emotional state detected by the emotion recognition device.

[1523] An embodiment of the present invention will be described. This system uses a head-mounted display (HMD) as a wearable device worn by a factory worker. The main elements and their operations are shown below.

[1524] Hardware and software used

[1525] 1. Wearable device (HMD): A device equipped with optical and detection devices to track the user's gaze.

[1526] 2. Eye tracking device: A system that uses the dlib library and other tools to track the user's gaze position in real time.

[1527] 3. External device (server): This device receives the captured image data and analyzes the information. TensorFlow is used for this analysis.

[1528] 4. Emotion Recognition Device: Used to analyze camera footage and identify the user's emotional state. Here, we use an emotion recognition model built with TensorFlow.

[1529] Data handling and calculations

[1530] 1. Gaze Tracking: The HMD camera captures the user's face and uses the dlib library to detect facial landmarks. The gaze direction is calculated based on these landmarks.

[1531] 2. Image capture and transmission: Images of the object are automatically captured based on the direction of gaze and transmitted to an external server, where they are analyzed in real time.

[1532] 3. Analysis and provision of information: The server analyzes the received image data using TensorFlow and extracts information about the parts and tools in front of the user's gaze. This information is then customized based on the user's history and settings and displayed on the HMD.

[1533] 4. Identifying and responding to emotional states: The HMD is equipped with an emotion recognition device that analyzes the user's facial expressions, heart rate, etc. to identify their emotional state. For example, if the user is feeling stressed, the system will suggest taking a break.

[1534] Specific examples

[1535] When a factory worker looks at a part of the robot arm they are operating, the HMD visually displays the manual and work procedures related to that part. Also, if the HMD detects that the worker is under stress, it will display instructions to take a break.

[1536] Example prompts for generative AI models

[1537] "Factory workers wear head-mounted displays and work while tracking their gaze. Generate an application program that displays information about parts and tools in front of their eyes in real time, analyzes the worker's emotional state, and suggests taking a break if the worker feels fatigued or stressed."

[1538] In this way, the system provides real-time information on the line of sight of factory workers and gives appropriate advice according to their emotional state, thereby improving work efficiency and ensuring safety.

[1539] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1540] Step 1:

[1541] The user wears a wearable device (HMD). The HMD has built-in optical and detection devices, which enable gaze tracking and emotion recognition. The input is the user's facial expression and gaze data, and the output is the capture of this data.

[1542] Step 2:

[1543] The device uses an eye-tracking device to track the user's gaze in real time. The dlib library is used to calculate the user's gaze direction from the captured facial landmark information. The input is the captured facial image data, and the output is the calculated gaze direction.

[1544] Step 3:

[1545] The gaze tracking device captures an image of the object in front of the gaze. The captured image is sent to an external device (server). The input is the calculated gaze direction and image data of the gaze destination, and the output is data sent to the server.

[1546] Step 4:

[1547] The server receives images sent from an external source and uses TensorFlow to analyze information about parts and tools in the line of sight. The analyzed information is personalized based on the user's history and settings. The input is the captured image data, and the output is the analyzed information about the line of sight.

[1548] Step 5:

[1549] The server sends the analyzed information to the HMD and displays it visually. The user can check detailed information about parts and tools in real time through the HMD. The input is the analyzed information, and the output is the information displayed on the HMD.

[1550] Step 6:

[1551] The device uses an emotion recognition device to analyze biometric data such as the user's facial expressions and heart rate to identify their emotional state. An emotion recognition model built with TensorFlow is used. The input is the user's biometric data, and the output is the identified emotional state.

[1552] Step 7:

[1553] The server sends appropriate advice and work improvement suggestions to the HMD based on the emotional state identified by the emotion recognition device. For example, if the user is recognized as being in a stressful state, a message suggesting a break will be displayed. The input is the identified emotional state, and the output is the advice displayed on the HMD.

[1554] Step 8:

[1555] The user performs the work according to the information and advice displayed through the HMD, which improves work efficiency and ensures safety. The input is the information displayed by the HMD, and the output is the user's work behavior.

[1556] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1557] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1558] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1559] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1560] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1561] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1562] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1563] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1564] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1565] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1566] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1567] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1568] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1569] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1570] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1571] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1572] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1573] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1574] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1575] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1576] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1577] The following is further disclosed regarding the above embodiment.

[1578] (Claim 1)

[1579] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[1580] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[1581] means for capturing an image based on the gaze position identified by the gaze tracking means;

[1582] means for transmitting the captured image to an external server;

[1583] means for the external server to analyze information based on the captured image;

[1584] A means for personalizing the analyzed information and providing it to the user;

[1585] A system including:

[1586] (Claim 2)

[1587] 10. The system of claim 1, further comprising an eye-gaze tracking means for identifying the position of the user's gaze in real time, and providing the information identified by the eye-gaze tracking means to the user in real time.

[1588] (Claim 3)

[1589] 10. The system according to claim 1, further comprising means for providing the analyzed information to a user as audio feedback, the system being adapted to accommodate visually impaired users.

[1590] "Example 1"

[1591] (Claim 1)

[1592] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[1593] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[1594] means for capturing an image based on the gaze position identified by the gaze tracking means;

[1595] means for transmitting the captured image to an external server;

[1596] means for the external server to analyze information based on the captured image using advanced image recognition algorithms;

[1597] means for providing the analyzed information in a personalized manner according to the user's history and settings;

[1598] A system including:

[1599] (Claim 2)

[1600] 10. The system of claim 1, further comprising an eye-gaze tracking means for identifying the position of the user's gaze in real time, and providing the information identified by the eye-gaze tracking means to the user in real time.

[1601] (Claim 3)

[1602] 10. The system according to claim 1, further comprising means for providing the analyzed information to a user as audio feedback, the system being adapted to accommodate visually impaired users.

[1603] "Application Example 1"

[1604] (Claim 1)

[1605] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[1606] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[1607] means for capturing an image based on the gaze position identified by the gaze tracking means;

[1608] means for transmitting the captured image to an external server;

[1609] means for the external server to analyze information based on the captured image;

[1610] A means for personalizing the analyzed information and providing it to the user;

[1611] A means for providing real-time information on products that users are looking at in physical stores through a display device of smart glasses;

[1612] A system including:

[1613] (Claim 2)

[1614] 10. The system of claim 1, further comprising an eye-gaze tracking means for identifying the position of the user's gaze in real time, and providing the information identified by the eye-gaze tracking means to the user in real time.

[1615] (Claim 3)

[1616] 10. The system according to claim 1, further comprising means for providing the analyzed information to a user as audio feedback, the system being adapted to accommodate visually impaired users.

[1617] "Example 2: Combining Emotion Engines"

[1618] (Claim 1)

[1619] a wearable device equipped with a camera and sensors for tracking a user's gaze;

[1620] an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor;

[1621] means for capturing an image based on the gaze position identified by the gaze tracking means;

[1622] means for transmitting the captured image to an external server;

[1623] means for the external server to analyze information based on the captured image;

[1624] A means for personalizing the analyzed information and providing it to the user;

[1625] an emotion recognition means for identifying an emotional state of a user based on biometric data;

[1626] a means for personalizing information provided based on the user's emotional state;

[1627] a means for conveying the provided information to the user as a display or audio feedback;

[1628] A system including:

[1629] (Claim 2)

[1630] 10. The system of claim 1, further comprising an eye-gaze tracking means for identifying the position of the user's gaze in real time, and providing the information identified by the eye-gaze tracking means to the user in real time.

[1631] (Claim 3)

[1632] 10. The system of claim 1, further comprising a biometric data analysis means for analyzing the user's emotional state in real time, and a means for providing the analyzed information to the user as audio feedback, and capable of accommodating visually impaired persons.

[1633] "Application example 2 when combining emotion engines"

[1634] (Claim 1)

[1635] a wearable device having an optical device and a detection device for tracking a user's gaze;

[1636] an eye tracking device that identifies the position of the user's gaze based on data acquired by the optical device and the detection device;

[1637] means for capturing an image based on the gaze position determined by the gaze tracking device;

[1638] means for transmitting the captured image to an external installation device;

[1639] means for the external installation device to analyze information based on the captured image;

[1640] A means for personalizing the analyzed information and providing it to the user;

[1641] an emotion recognition device for identifying an emotional state of a user;

[1642] means for providing advice or suggestions for work improvement to the user based on the emotional state identified by the emotion recognition device;

[1643] A system including:

[1644] (Claim 2)

[1645] 10. The system of claim 1, further comprising an eye-tracking device for determining the position of the user's gaze in real time, and providing the user with information determined by the eye-tracking device in real time.

[1646] (Claim 3)

[1647] 10. The system according to claim 1, further comprising means for providing the analyzed information to a user as audio feedback, the system being adapted to accommodate visually impaired users. [Explanation of symbols]

[1648] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a wearable device equipped with a camera and sensors for tracking a user's gaze; an eye tracking unit for identifying the position of the user's gaze based on data acquired by the camera and the sensor; means for capturing an image based on the gaze position identified by the gaze tracking means; means for transmitting the captured image to an external server; means for the external server to analyze information based on the captured image; A means for personalizing the analyzed information and providing it to the user; A system including:

2. The system according to claim 1 , further comprising an eye-gaze tracking means for determining the position of the user's gaze in real time, and providing the information determined by the eye-gaze tracking means to the user in real time.

3. The system according to claim 1, further comprising means for providing the analyzed information to the user as audio feedback, which is also suitable for visually impaired people.

Images