System

Abstract

A system is provided.SOLUTION: A system comprising: means for identifying an object viewed by a user using a pupil tracking device; means for generating a method of using the identified object as audio guidance using a generative model; means for monitoring an action of the user using a plurality of sensors and detecting an abnormal action; and means for transmitting an emergency notification to a pre-designated emergency contact when the abnormal action is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As the population ages, ensuring the safety and comfort of elderly people's daily lives has become a major social issue. In particular, accidents caused by misuse or forgetting to take medication, and improper use of tools are occurring frequently. Furthermore, with the increase in elderly people living alone and elderly care by other elderly people, there is a need for rapid response to accidents and emergencies that occur when elderly people are alone. However, conventional technologies have had difficulty in comprehensively resolving these issues. The present invention aims to solve these issues efficiently and effectively. [Means for solving the problem]

[0005] The present invention is a system that uses a pupil tracking device to identify an object viewed by a user and generates audio guidance on how to use the identified object using a generative model. It also has the function of monitoring the user's movements using multiple sensors and detecting abnormal movements. Furthermore, if abnormal movements are detected, an emergency notification is automatically sent to a pre-designated emergency contact. In this way, the present invention provides an environment where elderly people can live safely at home and reduces the anxiety of their families and caregivers.

[0006] Specifically, a pupil tracking device and a camera are combined to capture an object seen by the user, and the data is analyzed using a generative model to provide voice guidance on how to use the object. The generative model also has the function of converting the identified object's usage instructions into voice guidance using natural language processing. The elderly's movements are continuously monitored through multiple sensors, and if an abnormality is detected, an emergency notification is sent immediately, enabling a rapid response. With this configuration, the present invention can significantly improve the safety and convenience of the elderly in their daily lives.

[0007] A "pupil tracking device" is a device that tracks the movement of a user's pupils in real time and identifies their direction and focus.

[0008] A "generative model" is an artificial intelligence model that generates a specific output (e.g., audio guidance) based on input data.

[0009] A "sensor" is a device that detects physical changes, converts them into electrical signals, and outputs them.

[0010] An "object" is a specific object used daily, such as a medicine or tool, that is visible to the user.

[0011] "Audio guidance" is a system that provides instructions and explanations by voice.

[0012] "Abnormal behavior" refers to movements that deviate from normal behavioral patterns, such as falls or sudden physical changes.

[0013] "Emergency notification" is a function that quickly sends warnings and information to pre-designated contacts when an abnormal situation occurs.

[0014] "Natural language processing" is the technology for understanding, generating, and responding to human language, specifically the process of converting text data into speech.

[0015] "Capture" is the act of acquiring an image or video using a device such as a camera.

[0016] Analysis is the process of examining, understanding, and generating usable output from input data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye tracker, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[0039] Eye tracking and object recognition

[0040] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which then generates voice guidance on how to use the object.

[0041] Voice guidance using generative models

[0042] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[0043] Sensor function monitors abnormal behavior

[0044] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0045] Abnormal behavior detection and emergency notification

[0046] If abnormal behavior is detected, for example if an elderly person falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[0047] Specific examples

[0048] Example 1: Medication

[0049] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt, such as "Take one pill of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[0050] Example 2: Detecting abnormal behavior

[0051] Take the example of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and sends additional alerts if the user does not respond.

[0052] The present invention combines eye tracking, generative models, and sensor technology to make the daily lives of seniors safer and more comfortable, while reducing anxiety for their families and caregivers.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[0056] Step 2:

[0057] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[0058] Step 3:

[0059] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[0060] Step 4:

[0061] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[0062] Step 5:

[0063] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[0064] Step 6:

[0065] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[0066] Step 7:

[0067] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[0068] Step 8:

[0069] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[0070] Step 9:

[0071] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[0072] Step 10:

[0073] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[0074] Step 11:

[0075] If the user does not respond in a critical situation, the server will repeatedly send additional emergency alerts to prompt the necessary response.

[0076] Through the above steps, this system comprehensively supports the daily lives of elderly people and provides an environment in which they can live safely and comfortably.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] To support the safe and comfortable daily lives of the elderly, advanced technologies are needed for gaze tracking, object recognition, health monitoring, and other functions. However, conventional systems have difficulty integrating these functions and have limitations in real-time monitoring and anomaly detection. As a result, they are unable to completely alleviate the anxiety of the elderly and their families, and are also difficult to respond to emergency situations that require a rapid response. This invention aims to solve these problems and make the daily lives of the elderly safer and more comfortable.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes a means for identifying an object the user is looking at using an eye-tracking device, a means for generating audio guidance on how to use the identified object using a generative model, and a means for monitoring the user's motion and health status using multiple sensors and detecting abnormal motion. This enables integrated gaze tracking, object recognition, and real-time monitoring. Furthermore, if an abnormality is detected, an emergency notification can be sent immediately, enabling a rapid response and improving the safety and quality of life of the elderly.

[0082] An "eye tracking device" is a device that identifies the position and movement of a user's eyes and tracks the direction of their gaze in real time.

[0083] A "generative model" is a machine learning model that analyzes received data and generates voice guidance.

[0084] A "photography device" is a camera or image capture device that captures an image of an object viewed by a user.

[0085] A "voice synthesizer" is a device that reproduces voice guidance created by a generative model as actual voice.

[0086] A "sensor" is a device used to measure and monitor a user's activity or health. Examples include motion sensors, heart rate sensors, and blood pressure sensors.

[0087] A "profile" is a collection of data that includes personal information and settings about a user.

[0088] "Initialization" is a preparatory procedure for starting the operation of a system or device.

[0089] An "emergency notification" is a warning message sent to a pre-designated emergency contact when abnormal operation is detected.

[0090] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye-tracking device, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[0091] Eye tracking and object recognition

[0092] When a user puts on the smart glasses, the device automatically initializes itself. The smart glasses are equipped with an eye-tracking device and a camera. To identify the object the user is looking at in real time, the eye-tracking device measures the position and movement of the eyes, and the camera captures the image of the object being viewed. For example, if the user is looking at a medicine bottle, the device captures that information and sends it to a server.

[0093] Voice guidance using generative models

[0094] The server hosts a generative model (e.g., GPT-4 (registered trademark)) that analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as, "Take one tablet of this medicine after breakfast." The generated voice guidance is then sent to the device and notified to the user via the device's voice synthesizer.

[0095] Sensor function monitors abnormal behavior

[0096] The device is equipped with multiple sensors that continuously monitor the elderly person's activity and health status, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's activity and health status in real time, and if any abnormal activity or situation is detected, the data is immediately sent to the server.

[0097] Abnormal behavior detection and emergency notification

[0098] If an abnormal movement is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling a prompt response even when the elderly are alone.

[0099] Specific examples

[0100] Example 1: Medication

[0101] Suppose a user is wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye-tracking device and a camera, and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then notifies the user with the voice prompt.

[0102] Example 2: Detecting abnormal behavior

[0103] Consider the case of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and also sends additional alerts if the user does not respond.

[0104] Example prompt sentence:

[0105] "Please explain how eye tracking and generative models can be used to support elderly people in taking medication."

[0106] "Please explain how emergency notifications are sent if an elderly person falls, including how the specific sensors work."

[0107] In this way, the combination of eye-tracking technology, generative models, and sensor technology can make the daily lives of older people safer and more comfortable, and reduce anxiety for their families and caregivers.

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1: Initialize the device

[0110] The smart glasses as a terminal perform initialization. This includes checking the operation of the eye tracking device, the camera, and various sensors. It loads the user's profile and reads saved health data and settings. Specifically, the terminal recognizes the user's face and checks the status of each sensor. The input data includes the user's profile information and execution confirmation data from the sensors, and the output data is the initialization completion status.

[0111] Step 2: Eye tracking and object recognition

[0112] To monitor the user's daily activities, the device keeps its eye-tracking device and camera enabled at all times. When the user gazes at a specific object, the device identifies the direction of the user's gaze and captures a photo of that object. Specifically, the eye-tracking device captures the movement of the user's pupils, and the camera captures a video of the recognized object. The input data is the user's gaze information and the captured video data, and the output data is information about the recognized object, which is sent to the server.

[0113] Step 3: Send data

[0114] The device sends data of recognized objects to the server, including captured video data and gaze tracking data. Specifically, the device collects data and sends it to the server using a secure communication protocol. The input data is the captured video and gaze data, and the output data is the status of completion of transmission to the server.

[0115] Step 4: Analysis and Guidance Generation

[0116] The server inputs the received data into a generative model to generate appropriate guidance. The generative model uses natural language processing technology to create voice guidance that provides appropriate instructions to the user. For example, if it recognizes a medicine bottle, it generates instructions such as "Take one tablet of this medicine after breakfast." The input data is gaze and video data, and the output data is the generated voice guidance.

[0117] Step 5: Provide audio guidance

[0118] The terminal that receives the voice guidance generated by the server uses a voice synthesizer to convey the content to the user. The input data is the generated voice guidance, and the output data is the voice instructions that the user hears. Specifically, the terminal receives the voice guidance data, and the voice synthesizer plays it back.

[0119] Step 6: Detecting Abnormal Behavior

[0120] The device continuously monitors the user's movement and health status using motion sensors, heart rate sensors, and blood pressure sensors. If an abnormality is detected, the data is immediately sent to the server. The input data is the movement and health data collected by the sensors, and the output data is an alert of the abnormality. Specifically, if the user falls, the acceleration sensor and other sensors detect the abnormality and send the information to the server.

[0121] Step 7: Emergency Notification

[0122] The server receives data on detected abnormal behavior and sends a notification to emergency contacts. The notification includes location information and real-time health data. The input data is the abnormality detection notification data, and the output data is the notification content sent to emergency contacts. Specifically, the server analyzes the abnormal data and sends a message to family members or medical institutions such as, "The user has fallen. The location is ____."

[0123] (Application example 1)

[0124] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In modern society, safety and health management, especially for the elderly, are important issues. In the course of elderly people's daily lives, prompt responses are required to abnormal behavior, such as medication errors or falls. However, current systems have limitations in identifying objects visually recognized by elderly people and in responding quickly to emergencies. Furthermore, generating voice guidance requires manual input, which is cumbersome for elderly people. To address these issues, the present invention aims to automate the identification of objects visually recognized by elderly people and the detection of abnormal behavior, and provide appropriate voice guidance and emergency notifications in real time.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0127] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's behavior using multiple sensors and detecting abnormal behavior, means for sending an emergency notification to a pre-specified emergency contact when abnormal behavior is detected, and means for converting the captured object into audio guidance based on natural language processing. This makes it possible to improve the safety and comfort of elderly people's daily lives and reduce the anxiety of their families and caregivers.

[0128] 1. "Pupil tracking device" means a device that detects the position and movement of the user's pupils.

[0129] 2. "User" refers to a person who uses this system.

[0130] 3. "Object Identification" means the act of using an eye tracker and camera to identify an object that is viewed by the user.

[0131] 4. A "generative model" is a program or system that uses AI or machine learning algorithms to generate useful output from input data.

[0132] 5. "Voice guidance" is a system that uses voice synthesis technology to provide specific information as voice.

[0133] 6. "Multiple sensors" refers to multiple sensor devices used to measure various data, such as a motion sensor, a heart rate sensor, and a blood pressure sensor.

[0134] 7. "Movement monitoring" refers to the act of continuously observing a user's physical movements and condition using sensors.

[0135] 8. "Abnormal behavior" refers to behavior that deviates from normal or healthy behavior, such as falls by elderly people.

[0136] 9. "Emergency Notification" means a notification sent immediately to an emergency contact when abnormal activity is detected.

[0137] 10. "Emergency Contact" means a contact (such as a family member or medical institution) designated to receive notification in the event of an emergency.

[0138] 11. "Camera" means a photographic device for capturing images.

[0139] 12. "Capture" means the act of collecting and recording image or video data.

[0140] 13. "Natural language processing" is a general term for technologies that allow computers to understand and generate human language.

[0141] The present invention is a system for making the daily lives of elderly people safe and comfortable. This system is realized by integrating various hardware and software components, such as a pupil tracking device, a generative model, and multiple sensors. The specific configuration and operating procedures for implementing the present invention are described in detail below.

[0142] Eye tracking and object recognition

[0143] The smart glasses used as a terminal are equipped with an eye tracker and a camera. This eye tracker allows the user (elderly person) to identify the object they are looking at in real time. The camera captures image data of the object being viewed and sends it to a server. On the server side, the image data is analyzed using a generative model, and audio guidance on how to use the identified object is generated.

[0144] Voice guidance using generative models

[0145] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the generative model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the user.

[0146] Sensor function monitors abnormal behavior

[0147] The device is equipped with multiple sensors that continuously monitor the user's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0148] Abnormal behavior detection and emergency notification

[0149] If abnormal behavior is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[0150] Hardware and software used

[0151] The system uses the following hardware and software:

[0152] Pupil tracker: A device that detects the position and movement of the pupil.

[0153] Camera: A device that captures image data of objects in view.

[0154] Movement, heart rate, and blood pressure sensors: Sensors for monitoring the movement and health of seniors.

[0155] Server: Hosts the generative model, analyzes data, and generates audio guidance.

[0156] Generative model: A program or system that uses AI or machine learning algorithms to generate useful output from input data.

[0157] Natural Language Processing (NLP) technology: Technology that enables computers to understand and generate human language.

[0158] Specific examples

[0159] Example 1: Medication

[0160] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[0161] Example 2: Detecting abnormal behavior

[0162] Consider the case of a user at home. The device's built-in sensors monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and if the user does not respond, it sends an additional alert.

[0163] Prompt Sentence Examples

[0164] "The identified object is a 'medicine bottle'. Please generate the following audio prompt: 'Take one tablet of this medicine after breakfast.'"

[0165] As described above, the present invention is a system that combines a pupil tracking device, a generative model, and multiple sensors to make the daily lives of elderly people safe and comfortable.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] The camera in the smart glasses worn by the device captures the image of the user's field of vision in real time. The input is the image data of the user's field of vision, and the output is the image data. The camera provides high-resolution images, which are sent to a processor for use in the next step.

[0169] Step 2:

[0170] The device's built-in eye tracking device tracks the position of the user's eyes and overlays them on the camera image to identify the object the user is looking at. The input is the user's eye data (pupil position and movement), and the output is the coordinate data of the object the user is looking at. The position of the identified object is clarified based on the eye position information.

[0171] Step 3:

[0172] The device sends image data of the identified object to the server. The input is the image data of the identified object, and the output is the data transmission to the server. The data is transmitted at high speed and is optimized to minimize latency.

[0173] Step 4:

[0174] The server inputs the received image data into a generative model to recognize the object. The input is the image data of the identified object, and the output is the object name. The generative model uses a pre-trained AI algorithm to classify the object and obtain a result. For example, the image data is recognized as a medicine bottle.

[0175] Step 5:

[0176] The server converts the recognized object's usage instructions into voice guidance using natural language processing technology. The input is the object name and data on its usage, and the output is text data for the voice guidance. The generative model prepares the voice guidance, saying, "Take one tablet of this medicine after breakfast."

[0177] Step 6:

[0178] The server sends the text data of the voice guidance to the terminal. The input is the text data of the voice guidance, and the output is the data transmission to the terminal. The data is sent in real time, providing quick guidance to the elderly.

[0179] Step 7:

[0180] The device uses voice synthesis technology to play back the voice guidance. The input is text data received from the server, and the output is voice guidance. In the case of smart glasses, the voice is transmitted to the elderly using a built-in speaker.

[0181] Step 8:

[0182] Multiple sensors built into the device monitor the user's movements and health. The input is real-time data from the sensors (e.g., heart rate, movements), and the output is the monitoring results. The sensor data is continuously collected and checked for any abnormalities.

[0183] Step 9:

[0184] When a device detects abnormal behavior (e.g., a fall), the data is immediately sent to the server. The input is sensor data of the abnormal behavior, and the output is emergency data sent to the server. The data is sent at high speed, allowing for a rapid response to emergencies.

[0185] Step 10:

[0186] The server receives data on abnormal behavior and sends an emergency notification to pre-designated emergency contacts. The input is data on abnormal behavior, and the output is a notification to the emergency contact (including location information and health data). An email or SMS is sent to the emergency contact, enabling a prompt response.

[0187] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0188] This invention is a system for making the daily lives of the elderly safe and comfortable. This system combines an eye tracker, a generative model, multiple sensors, and an emotion engine to comprehensively support the daily lives of the elderly.

[0189] Eye tracking and object recognition

[0190] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which generates voice guidance on how to use the object.

[0191] Voice guidance using generative models

[0192] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[0193] Sensor function monitors abnormal behavior

[0194] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0195] Emotion recognition by emotion engine

[0196] The device uses a microphone and camera to analyze the elderly person's tone of voice and facial expressions in real time, thereby recognizing their emotional state (e.g., anxiety, sadness, joy). This emotional data is sent to a server, and a generative model generates appropriate voice guidance based on the elderly person's emotional state.

[0197] Abnormal behavior detection and emergency notification

[0198] If abnormal behavior is detected, for example, if an elderly person falls, the server receives the information and uses the emotion engine data to create a more appropriate emergency notification. The emotion engine data, along with the abnormal behavior information, is included in the emergency notification and sent to pre-designated emergency contacts (family members or medical institutions).

[0199] Specific examples

[0200] Example 1: Medication

[0201] Take the example of a user wearing smart glasses. When the user looks at a pill bottle, the device uses an eye tracker and camera to capture that information and sends it to the server. The server uses a generative model to generate a voice prompt such as, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays this voice prompt to the user. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice, and if the user feels anxious, it provides additional voice prompts such as, "Are you worried?"

[0202] Example 2: Detecting abnormal behavior

[0203] Take the example of a user at home. An IoT sensor installed on the device monitors the user's movements. If the user falls, the device detects abnormal movements and sends the data to the server. The server receives the data, and if the emotion engine analyzes the data and determines that the user is feeling fear or anxiety, it adds this information to the emergency notification. To ensure that the situation is understood, the emergency notification is sent to pre-registered family members and medical institutions.

[0204] The present invention combines eye tracking technology, generative models, sensor technology, and an emotion engine to support the daily lives of elderly people while taking into account their emotional state, providing a safe and comfortable environment.

[0205] The processing flow will be explained below.

[0206] Step 1:

[0207] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[0208] Step 2:

[0209] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[0210] Step 3:

[0211] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[0212] Step 4:

[0213] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[0214] Step 5:

[0215] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[0216] Step 6:

[0217] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[0218] Step 7:

[0219] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[0220] Step 8:

[0221] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[0222] Step 9:

[0223] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[0224] Step 10:

[0225] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[0226] Step 11:

[0227] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotional state (e.g., anxiety, sadness, joy).

[0228] Step 12:

[0229] The server adjusts the content of the voice guidance based on the emotion engine data. For example, if the user is feeling anxious, it generates a message that provides additional reassurance.

[0230] Step 13:

[0231] The server combines abnormal behavior data with data from the emotion engine to dynamically adjust the content of emergency notifications and inform relevant parties of the situation.

[0232] Step 14:

[0233] The device periodically sends the emotion engine's analysis results to the server, and the generative model provides instructions optimized for the user's situation.

[0234] Through the above steps, the present invention comprehensively supports the daily life of the elderly and provides a safe and comfortable living environment taking into account their emotional state.

[0235] Example 2

[0236] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0237] There is a growing need for a system that can comprehensively monitor and support the visual, behavioral, and emotional states of the elderly in their daily lives. Conventional systems have the technology to monitor individual states (visual, behavioral, and emotional), but it is difficult to comprehensively monitor and immediately respond appropriately. The purpose of this invention is to solve these problems and provide the elderly with a safe and comfortable daily life.

[0238] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0239] In this invention, the server

[0240] means for identifying an object viewed by a user using a pupil tracking device;

[0241] means for generating audio guidance on how to use the object identified using the generative model;

[0242] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[0243] a means for analyzing the user's emotional state using an emotion engine and providing an appropriate response;

[0244] This will enable comprehensive monitoring of the daily lives of elderly people and provide timely and appropriate support.

[0245] A "pupil tracking device" is a device that detects the movement of a user's pupils in real time and acquires gaze data.

[0246] A "generative model" is an artificial intelligence technique that analyzes input data and produces output in a specified format.

[0247] "Voice guidance" is a message that conveys specific information to the user by voice.

[0248] A "sensor" is a device that detects physical phenomena (movement, heart rate, blood pressure, etc.) in real time and acquires them as data.

[0249] "Abnormal movement" refers to a movement or state that deviates from the user's normal movement (e.g., falling).

[0250] An "emergency notification" is an emergency message sent to pre-designated contacts when abnormal operation is detected.

[0251] The "emotion engine" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[0252] "User" refers to a person who uses this system.

[0253] MODE FOR CARRYING OUT THE INVENTION

[0254] The present invention is a comprehensive support system for the daily lives of elderly people, which aims to make users' daily lives safer and more comfortable by combining an eye tracker, a generative model, multiple sensors, and an emotion engine. The main hardware and software components for implementing the present invention are described below.

[0255] Hardware Configuration

[0256] 1. Device (Smart Glasses):

[0257] Eye tracking device: A device that captures the user's gaze in real time and acquires that data.

[0258] Camera: A device that captures an object corresponding to the user's line of sight and generates image data.

[0259] Sensors: Equipped with motion sensors, heart rate sensors, blood pressure sensors, etc., it monitors the user's movements and health status in real time.

[0260] Microphone: A device that captures the tone of a user's voice.

[0261] Speaker: A device that conveys the generated audio guidance to the user.

[0262] 2. Server:

[0263] Generative model: A model that analyzes gaze data and image data and generates voice guidance using natural language processing technology.

[0264] Emotion engine: Technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[0265] Software Configuration

[0266] 1. Data analysis software: Installed on the server, it analyzes the data sent from the pupil tracker and camera.

[0267] 2. Speech synthesis software: Converts the text data output by the generative model into speech data.

[0268] 3. Communication software: Software for sending and receiving data between the terminal and the server.

[0269] Specific system processing examples

[0270] 1. Medication Use:

[0271] The user puts on the smart glasses and looks at the pill bottle.

[0272] The device's eye tracking device and camera capture this information, obtaining gaze data and image data.

[0273] The terminal transmits the data to the server.

[0274] The server's generative model generates voice guidance such as "Take one tablet of this medicine after breakfast."

[0275] The server transmits the guidance to the terminal, and the terminal's speaker delivers the voice guidance to the user.

[0276] The emotion engine analyzes the user's facial expressions and tone of voice, and if they feel anxious, it provides additional voice guidance such as, "Are you worried about something?"

[0277] 2. Abnormal behavior detection:

[0278] The user falls while at home.

[0279] A motion sensor installed in the device detects falls.

[0280] The terminal transmits the data to the server.

[0281] The server determines that the user is feeling fear based on the abnormal behavior data and the analysis results of the emotion engine, and creates an emergency notification.

[0282] Emergency notifications will be sent to pre-designated family members and medical institutions.

[0283] Prompt Sentence Examples

[0284] 1. Example prompts for medication use:

[0285] The user is looking at a bottle of medicine. The information is captured by the eye tracker and camera, and analyzed by the server to generate a voice prompt such as, "Take one tablet of this medicine after breakfast." If the user is feeling anxious, the system provides additional guidance such as, "Are you worried about something?"

[0286] 2. Example prompt for detecting abnormal behavior:

[0287] If a user falls while at home, the motion sensor detects the abnormality and sends this information to a server, which then sends an emergency notification to family members or medical institutions, including anxiety data analyzed by an emotion engine.

[0288] As described above, the present invention combines an eye tracking device, a generative model, sensor technology, and an emotion engine to comprehensively support the daily lives of the elderly and provide a safe and comfortable environment.

[0289] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0290] System program processing steps

[0291] Specific processing steps for medication use

[0292] Step 1:

[0293] A user wears the smart glasses and looks at a medicine bottle, while gaze data is captured in real time by a pupil tracker.

[0294] Input: User's gaze

[0295] Output: Gaze data

[0296] Step 2:

[0297] The device's camera captures an image of the object (medicine bottle) that corresponds to the user's line of sight.

[0298] Input: Gaze data

[0299] Output: Image data

[0300] Step 3:

[0301] The device transmits the captured gaze data and image data to a server via Wi-Fi or mobile networks.

[0302] Input: Gaze data, image data

[0303] Output: Send data to the server

[0304] Step 4:

[0305] The server analyzes the received gaze data and image data, and a generative model identifies the medicine bottle based on the specific image data and gaze data.

[0306] Input: Gaze data, image data

[0307] Output: Identified object (medicine bottle)

[0308] Step 5:

[0309] The server's generative model analyzes how to use the identified object and uses natural language processing technology to generate voice guidance such as, "Take one tablet of this medicine after breakfast."

[0310] Input: Identified object

[0311] Output: Voice guidance (text format)

[0312] Step 6:

[0313] The server transmits the generated voice guidance to the terminal.

[0314] Input: Voice guidance (text format)

[0315] Output: Send voice guidance to the device

[0316] Step 7:

[0317] The device converts the received voice guidance into audio data and plays it back through the speaker, providing the user with audio instructions on how to use the device.

[0318] Input: Voice guidance (text format)

[0319] Output: Voice guidance (voice data)

[0320] Step 8:

[0321] The device's emotion engine analyzes the user's facial expressions and tone of voice in real time, and if they are feeling anxious, it generates additional voice guidance such as, "Are you worried about something?"

[0322] Input: User's facial expression data, tone of voice

[0323] Output: Additional voice guidance

[0324] Step 9:

[0325] Additional audio guidance generated by the terminal is played back to give the user a sense of security.

[0326] Input: Additional voice guidance

[0327] Output: Additional voice guidance playback

[0328] Specific processing steps for detecting abnormal behavior

[0329] Step 1:

[0330] The user falls while at home.

[0331] Input: User action

[0332] Step 2:

[0333] The motion sensor installed in the device detects falls, for example, detecting abnormal movements from sudden changes in gravitational acceleration.

[0334] Input: Operation data

[0335] Output: Abnormal operation data

[0336] Step 3:

[0337] The terminal transmits the abnormal operation data to the server.

[0338] Input: Abnormal behavior data

[0339] Output: Send abnormal operation data to the server

[0340] Step 4:

[0341] The server analyzes the received abnormal operation data and confirms the abnormal operation.

[0342] Input: Abnormal behavior data

[0343] Output: Analysis results (confirmation of abnormal behavior)

[0344] Step 5:

[0345] The server uses an emotion engine to analyze whether the user feels fear or anxiety when they fall.

[0346] Input: User's facial expression data, tone of voice

[0347] Output: Emotion data (fear, anxiety)

[0348] Step 6:

[0349] The server generates an emergency notification based on the abnormal motion data and emotion data, such as "The user has fallen and is currently experiencing fear."

[0350] Input: Abnormal behavior data, emotion data

[0351] Output: Urgent Notification

[0352] Step 7:

[0353] The server then sends the emergency notification to pre-designated family members or medical institutions.

[0354] Input: Emergency Notification

[0355] Output: Send emergency notification

[0356] As mentioned above, each processing step is explained in detail with specific operations and data flows. The overall system effectively combines eye tracking technology, generative models, sensor technology, and an emotion engine to help the elderly live a safe and comfortable life.

[0357] (Application example 2)

[0358] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0359] In recent years, technology to support the lives of the elderly has been attracting attention. However, when using food delivery services, many difficulties arise, particularly when selecting menus and receiving food. For this reason, there is a need for a system that allows elderly people to use food delivery services safely and comfortably. Furthermore, the current situation is that meal suggestions tailored to health conditions and support when receiving deliveries are not being provided sufficiently. To solve these issues, a comprehensive system is needed that allows elderly people to use food delivery services with peace of mind.

[0360] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0361] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's movements using multiple sensors and detecting abnormal movements, means for sending an emergency notification to a pre-designated emergency contact when abnormal movements are detected, means for supporting the elderly with menu selection and meal suggestions based on their health condition when using a food delivery service using audio guidance, and means for measuring the height and distance when a delivery arrives and supporting the receiving operation. This allows the elderly to use food delivery services with peace of mind, select appropriate meals based on their health condition, and receive their delivery safely.

[0362] An "eye tracker" is a device that tracks a user's gaze and identifies the object they are looking at.

[0363] "Generative model" refers to an algorithm that generates guidance in natural language based on input data.

[0364] A "sensor" is a device that detects physical movements and environmental conditions and outputs them as digital data.

[0365] "Abnormal movement" refers to movement that differs from the user's normal behavioral pattern, and includes, for example, falling.

[0366] An "emergency notification" is an alert sent to pre-registered contacts when an unexpected situation occurs.

[0367] "Food delivery service" refers to a service that delivers food or meals to a location specified by the customer.

[0368] "Menu selection" refers to the act of selecting what to order from the dishes and foods offered by a food delivery service.

[0369] "Meal suggestions tailored to your health condition" refers to recommending appropriate meals taking into consideration the user's health condition and preferences.

[0370] "Delivery arrival" refers to the moment when the ordered food or meal is delivered to the location specified by the user.

[0371] "Measuring height and distance" means measuring the surrounding conditions to ensure that elderly people can receive deliveries safely.

[0372] This invention is a system that enables elderly people to use food delivery services safely and comfortably, and is realized by combining an eye tracking device, a generative model, sensors, and an emotion engine. The system program is implemented using the following hardware and software.

[0373] Hardware

[0374] Smart glasses: Equipped with a camera, eye tracker, microphone, and speaker.

[0375] Server: Used to analyze data and generate audio guidance.

[0376] software

[0377] OpenCV: A library for image processing.

[0378] TENSORFLOW(R) & Keras: Implementation and working of pupil tracking model.

[0379] Transformers: A library for sentiment analysis.

[0380] HTTP Requests: A library for communicating with the server.

[0381] Data processing and calculation

[0382] The server first processes the video frames captured by the smart glasses using OpenCV and identifies the eye position using an eye tracking model. This allows it to identify the menu or object the elderly person is looking at. The generative model then uses natural language processing to convert the identified object's usage instructions and appropriate meal suggestions into voice guidance, which is then provided to the elderly. Multiple sensors are also used to monitor the elderly's movements and health status, and if abnormal behavior is detected, an emergency notification is sent to a pre-designated emergency contact.

[0383] Specific examples

[0384] Menu selection assistance

[0385] For example, consider a situation where an elderly person is wearing smart glasses and browsing a food delivery app. If the eye tracking device detects that the user's gaze is fixed on a particular menu item (e.g., pancakes), a voice guidance such as "This dish is high in calories. We recommend you change it to a salad" is generated and provided to the user.

[0386] Delivery reception assistance

[0387] When the delivery person arrives, sensors installed in the smart glasses measure the location and distance and provide guidance such as "The door is one meter away. Please walk slowly" to avoid the risk of the elderly person falling.

[0388] Prompt Sentence Examples

[0389] "When an elderly person looks at the menu of a food delivery app, generate meal suggestions tailored to their health condition."

[0390] The present invention allows elderly people to use food delivery services with peace of mind, allowing them to select appropriate meals according to their health condition and receive deliveries safely, thereby improving the quality of life of elderly people.

[0391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0392] Step 1:

[0393] The camera in the smart glasses captures the elderly person's field of vision.

[0394] Input: Elderly person's visual field

[0395] Output: Captured video frames

[0396] The device uses a camera mounted on the smart glasses to capture the elderly person's field of vision in real time, and these video frames are used as input for the subsequent eye tracking process.

[0397] Step 2:

[0398] An eye tracker identifies the location of the elderly person's eyes.

[0399] Input: Captured video frames

[0400] Output: Eye coordinate data

[0401] The device processes the captured video frames using the OpenCV library and identifies the location of the elderly person's eyes using an eye tracking model (TensorFlow and Keras). The obtained eye coordinate data is used to identify the object the user is looking at.

[0402] Step 3:

[0403] The generative model identifies the objects the elderly person sees and generates audio guidance.

[0404] Input: pupil coordinate data, captured video frame

[0405] Output: Voice guidance data

[0406] The server analyzes the pupil coordinate data and the captured video frames to identify the object the user is looking at. It then inputs a prompt sentence into a generative model based on the identified object to generate appropriate voice guidance. The generative model then uses natural language processing technology to generate voice guidance data and transmit it to the device.

[0407] Step 4:

[0408] The device plays audio guidance.

[0409] Input: Voice guidance data

[0410] Output: Voice guidance

[0411] The terminal plays the voice guidance data received from the server and provides voice guidance to the elderly, such as "This dish is high in calories. We recommend changing it to a salad."

[0412] Step 5:

[0413] Sensors monitor the movements and health of the elderly.

[0414] Input: Sensor data (heart rate, blood pressure, movement, etc.)

[0415] Output: Monitoring data

[0416] The device uses multiple sensors to continuously monitor the elderly person's heart rate, blood pressure, and movements, and collects this data, which is used to detect abnormal movements.

[0417] Step 6:

[0418] If abnormal behavior is detected, an emergency notification will be sent.

[0419] Input: Monitoring data

[0420] Output: Urgent Notification

[0421] The server analyzes the monitoring data sent from the device and generates an emergency notification if any abnormal behavior is detected, and sends it to pre-designated emergency contacts (such as family members or medical institutions). This notification may also include information about the elderly person's health and emotional state.

[0422] Step 7:

[0423] When a delivery arrives, the system measures the height and distance to assist with the receiving process.

[0424] Input: Sensor data (distance measurement)

[0425] Output: Voice guidance

[0426] When a delivery arrives, the device uses sensors to measure the height and distance, and provides voice guidance such as "The distance to the door is one meter. Please walk slowly" to ensure that the elderly person can receive the delivery safely.

[0427] This will allow elderly people to use food delivery services with peace of mind, make appropriate meal choices, and receive their deliveries safely.

[0428] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0429] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0430] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0431] [Second embodiment]

[0432] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0433] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0434] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0435] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0436] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0437] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0438] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0439] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0440] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0441] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0442] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0443] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0444] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye tracker, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[0445] Eye tracking and object recognition

[0446] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which then generates voice guidance on how to use the object.

[0447] Voice guidance using generative models

[0448] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[0449] Sensor function monitors abnormal behavior

[0450] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0451] Abnormal behavior detection and emergency notification

[0452] If abnormal behavior is detected, for example if an elderly person falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[0453] Specific examples

[0454] Example 1: Medication

[0455] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt, such as "Take one pill of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[0456] Example 2: Detecting abnormal behavior

[0457] Take the example of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and sends additional alerts if the user does not respond.

[0458] The present invention combines eye tracking, generative models, and sensor technology to make the daily lives of seniors safer and more comfortable, while reducing anxiety for their families and caregivers.

[0459] The processing flow will be explained below.

[0460] Step 1:

[0461] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[0462] Step 2:

[0463] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[0464] Step 3:

[0465] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[0466] Step 4:

[0467] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[0468] Step 5:

[0469] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[0470] Step 6:

[0471] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[0472] Step 7:

[0473] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[0474] Step 8:

[0475] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[0476] Step 9:

[0477] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[0478] Step 10:

[0479] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[0480] Step 11:

[0481] If the user does not respond in a critical situation, the server will repeatedly send additional emergency alerts to prompt the necessary response.

[0482] Through the above steps, this system comprehensively supports the daily lives of elderly people and provides an environment in which they can live safely and comfortably.

[0483] Example 1

[0484] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0485] To support the safe and comfortable daily lives of the elderly, advanced technologies are needed for gaze tracking, object recognition, health monitoring, and other functions. However, conventional systems have difficulty integrating these functions and have limitations in real-time monitoring and anomaly detection. As a result, they are unable to completely alleviate the anxiety of the elderly and their families, and are also difficult to respond to emergency situations that require a rapid response. This invention aims to solve these problems and make the daily lives of the elderly safer and more comfortable.

[0486] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0487] In this invention, the server includes a means for identifying an object the user is looking at using an eye-tracking device, a means for generating audio guidance on how to use the identified object using a generative model, and a means for monitoring the user's motion and health status using multiple sensors and detecting abnormal motion. This enables integrated gaze tracking, object recognition, and real-time monitoring. Furthermore, if an abnormality is detected, an emergency notification can be sent immediately, enabling a rapid response and improving the safety and quality of life of the elderly.

[0488] An "eye tracking device" is a device that identifies the position and movement of a user's eyes and tracks the direction of their gaze in real time.

[0489] A "generative model" is a machine learning model that analyzes received data and generates voice guidance.

[0490] A "photography device" is a camera or image capture device that captures an image of an object viewed by a user.

[0491] A "voice synthesizer" is a device that reproduces voice guidance created by a generative model as actual voice.

[0492] A "sensor" is a device used to measure and monitor a user's activity or health. Examples include motion sensors, heart rate sensors, and blood pressure sensors.

[0493] A "profile" is a collection of data that includes personal information and settings about a user.

[0494] "Initialization" is a preparatory procedure for starting the operation of a system or device.

[0495] An "emergency notification" is a warning message sent to a pre-designated emergency contact when abnormal operation is detected.

[0496] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye-tracking device, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[0497] Eye tracking and object recognition

[0498] When a user puts on the smart glasses, the device automatically initializes itself. The smart glasses are equipped with an eye-tracking device and a camera. To identify the object the user is looking at in real time, the eye-tracking device measures the position and movement of the eyes, and the camera captures the image of the object being viewed. For example, if the user is looking at a medicine bottle, the device captures that information and sends it to a server.

[0499] Voice guidance using generative models

[0500] The server hosts a generative model (e.g., GPT-4) that analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, it generates guidance such as, "Take one tablet of this medicine after breakfast." The generated voice guidance is then sent to the device and notified to the user via the device's voice synthesizer.

[0501] Sensor function monitors abnormal behavior

[0502] The device is equipped with multiple sensors that continuously monitor the elderly person's activity and health status, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's activity and health status in real time, and if any abnormal activity or situation is detected, the data is immediately sent to the server.

[0503] Abnormal behavior detection and emergency notification

[0504] If an abnormal movement is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling a prompt response even when the elderly are alone.

[0505] Specific examples

[0506] Example 1: Medication

[0507] Suppose a user is wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye-tracking device and a camera, and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then notifies the user with the voice prompt.

[0508] Example 2: Detecting abnormal behavior

[0509] Consider the case of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and also sends additional alerts if the user does not respond.

[0510] Example prompt sentence:

[0511] "Please explain how eye tracking and generative models can be used to support elderly people in taking medication."

[0512] "Please explain how emergency notifications are sent if an elderly person falls, including how the specific sensors work."

[0513] In this way, the combination of eye-tracking technology, generative models, and sensor technology can make the daily lives of older people safer and more comfortable, and reduce anxiety for their families and caregivers.

[0514] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0515] Step 1: Initialize the device

[0516] The smart glasses as a terminal perform initialization. This includes checking the operation of the eye tracking device, the camera, and various sensors. It loads the user's profile and reads saved health data and settings. Specifically, the terminal recognizes the user's face and checks the status of each sensor. The input data includes the user's profile information and execution confirmation data from the sensors, and the output data is the initialization completion status.

[0517] Step 2: Eye tracking and object recognition

[0518] To monitor the user's daily activities, the device keeps its eye-tracking device and camera enabled at all times. When the user gazes at a specific object, the device identifies the direction of the user's gaze and captures a photo of that object. Specifically, the eye-tracking device captures the movement of the user's pupils, and the camera captures a video of the recognized object. The input data is the user's gaze information and the captured video data, and the output data is information about the recognized object, which is sent to the server.

[0519] Step 3: Send data

[0520] The device sends data of recognized objects to the server, including captured video data and gaze tracking data. Specifically, the device collects data and sends it to the server using a secure communication protocol. The input data is the captured video and gaze data, and the output data is the status of completion of transmission to the server.

[0521] Step 4: Analysis and Guidance Generation

[0522] The server inputs the received data into a generative model to generate appropriate guidance. The generative model uses natural language processing technology to create voice guidance that provides appropriate instructions to the user. For example, if it recognizes a medicine bottle, it generates instructions such as "Take one tablet of this medicine after breakfast." The input data is gaze and video data, and the output data is the generated voice guidance.

[0523] Step 5: Provide audio guidance

[0524] The terminal that receives the voice guidance generated by the server uses a voice synthesizer to convey the content to the user. The input data is the generated voice guidance, and the output data is the voice instructions that the user hears. Specifically, the terminal receives the voice guidance data, and the voice synthesizer plays it back.

[0525] Step 6: Detecting Abnormal Behavior

[0526] The device continuously monitors the user's movement and health status using motion sensors, heart rate sensors, and blood pressure sensors. If an abnormality is detected, the data is immediately sent to the server. The input data is the movement and health data collected by the sensors, and the output data is an alert of the abnormality. Specifically, if the user falls, the acceleration sensor and other sensors detect the abnormality and send the information to the server.

[0527] Step 7: Emergency Notification

[0528] The server receives data on detected abnormal behavior and sends a notification to emergency contacts. The notification includes location information and real-time health data. The input data is the abnormality detection notification data, and the output data is the notification content sent to emergency contacts. Specifically, the server analyzes the abnormal data and sends a message to family members or medical institutions such as, "The user has fallen. The location is ____."

[0529] (Application example 1)

[0530] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0531] In modern society, safety and health management, especially for the elderly, are important issues. In the course of elderly people's daily lives, prompt responses are required to abnormal behavior, such as medication errors or falls. However, current systems have limitations in identifying objects visually recognized by elderly people and in responding quickly to emergencies. Furthermore, generating voice guidance requires manual input, which is cumbersome for elderly people. To address these issues, the present invention aims to automate the identification of objects visually recognized by elderly people and the detection of abnormal behavior, and provide appropriate voice guidance and emergency notifications in real time.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0533] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's behavior using multiple sensors and detecting abnormal behavior, means for sending an emergency notification to a pre-specified emergency contact when abnormal behavior is detected, and means for converting the captured object into audio guidance based on natural language processing. This makes it possible to improve the safety and comfort of elderly people's daily lives and reduce the anxiety of their families and caregivers.

[0534] 1. "Pupil tracking device" means a device that detects the position and movement of the user's pupils.

[0535] 2. "User" refers to a person who uses this system.

[0536] 3. "Object Identification" means the act of using an eye tracker and camera to identify an object that is viewed by the user.

[0537] 4. A "generative model" is a program or system that uses AI or machine learning algorithms to generate useful output from input data.

[0538] 5. "Voice guidance" is a system that uses voice synthesis technology to provide specific information as voice.

[0539] 6. "Multiple sensors" refers to multiple sensor devices used to measure various data, such as a motion sensor, a heart rate sensor, and a blood pressure sensor.

[0540] 7. "Movement monitoring" refers to the act of continuously observing a user's physical movements and condition using sensors.

[0541] 8. "Abnormal behavior" refers to behavior that deviates from normal or healthy behavior, such as falls by elderly people.

[0542] 9. "Emergency Notification" means a notification sent immediately to an emergency contact when abnormal activity is detected.

[0543] 10. "Emergency Contact" means a contact (such as a family member or medical institution) designated to receive notification in the event of an emergency.

[0544] 11. "Camera" means a photographic device for capturing images.

[0545] 12. "Capture" means the act of collecting and recording image or video data.

[0546] 13. "Natural language processing" is a general term for technologies that allow computers to understand and generate human language.

[0547] The present invention is a system for making the daily lives of elderly people safe and comfortable. This system is realized by integrating various hardware and software components, such as a pupil tracking device, a generative model, and multiple sensors. The specific configuration and operating procedures for implementing the present invention are described in detail below.

[0548] Eye tracking and object recognition

[0549] The smart glasses used as a terminal are equipped with an eye tracker and a camera. This eye tracker allows the user (elderly person) to identify the object they are looking at in real time. The camera captures image data of the object being viewed and sends it to a server. On the server side, the image data is analyzed using a generative model, and audio guidance on how to use the identified object is generated.

[0550] Voice guidance using generative models

[0551] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the generative model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the user.

[0552] Sensor function monitors abnormal behavior

[0553] The device is equipped with multiple sensors that continuously monitor the user's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0554] Abnormal behavior detection and emergency notification

[0555] If abnormal behavior is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[0556] Hardware and software used

[0557] The system uses the following hardware and software:

[0558] Pupil tracker: A device that detects the position and movement of the pupil.

[0559] Camera: A device that captures image data of objects in view.

[0560] Movement, heart rate, and blood pressure sensors: Sensors for monitoring the movement and health of seniors.

[0561] Server: Hosts the generative model, analyzes data, and generates audio guidance.

[0562] Generative model: A program or system that uses AI or machine learning algorithms to generate useful output from input data.

[0563] Natural Language Processing (NLP) technology: Technology that enables computers to understand and generate human language.

[0564] Specific examples

[0565] Example 1: Medication

[0566] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[0567] Example 2: Detecting abnormal behavior

[0568] Consider the case of a user at home. The device's built-in sensors monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and if the user does not respond, it sends an additional alert.

[0569] Prompt Sentence Examples

[0570] "The identified object is a 'medicine bottle'. Please generate the following audio prompt: 'Take one tablet of this medicine after breakfast.'"

[0571] As described above, the present invention is a system that combines a pupil tracking device, a generative model, and multiple sensors to make the daily lives of elderly people safe and comfortable.

[0572] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0573] Step 1:

[0574] The camera in the smart glasses worn by the device captures the image of the user's field of vision in real time. The input is the image data of the user's field of vision, and the output is the image data. The camera provides high-resolution images, which are sent to a processor for use in the next step.

[0575] Step 2:

[0576] The device's built-in eye tracking device tracks the position of the user's eyes and overlays them on the camera image to identify the object the user is looking at. The input is the user's eye data (pupil position and movement), and the output is the coordinate data of the object the user is looking at. The position of the identified object is clarified based on the eye position information.

[0577] Step 3:

[0578] The device sends image data of the identified object to the server. The input is the image data of the identified object, and the output is the data transmission to the server. The data is transmitted at high speed and is optimized to minimize latency.

[0579] Step 4:

[0580] The server inputs the received image data into a generative model to recognize the object. The input is the image data of the identified object, and the output is the object name. The generative model uses a pre-trained AI algorithm to classify the object and obtain a result. For example, the image data is recognized as a medicine bottle.

[0581] Step 5:

[0582] The server converts the recognized object's usage instructions into voice guidance using natural language processing technology. The input is the object name and data on its usage, and the output is text data for the voice guidance. The generative model prepares the voice guidance, saying, "Take one tablet of this medicine after breakfast."

[0583] Step 6:

[0584] The server sends the text data of the voice guidance to the terminal. The input is the text data of the voice guidance, and the output is the data transmission to the terminal. The data is sent in real time, providing quick guidance to the elderly.

[0585] Step 7:

[0586] The device uses voice synthesis technology to play back the voice guidance. The input is text data received from the server, and the output is voice guidance. In the case of smart glasses, the voice is transmitted to the elderly using a built-in speaker.

[0587] Step 8:

[0588] Multiple sensors built into the device monitor the user's movements and health. The input is real-time data from the sensors (e.g., heart rate, movements), and the output is the monitoring results. The sensor data is continuously collected and checked for any abnormalities.

[0589] Step 9:

[0590] When a device detects abnormal behavior (e.g., a fall), the data is immediately sent to the server. The input is sensor data of the abnormal behavior, and the output is emergency data sent to the server. The data is sent at high speed, allowing for a rapid response to emergencies.

[0591] Step 10:

[0592] The server receives data on abnormal behavior and sends an emergency notification to pre-designated emergency contacts. The input is data on abnormal behavior, and the output is a notification to the emergency contact (including location information and health data). An email or SMS is sent to the emergency contact, enabling a prompt response.

[0593] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0594] This invention is a system for making the daily lives of the elderly safe and comfortable. This system combines an eye tracker, a generative model, multiple sensors, and an emotion engine to comprehensively support the daily lives of the elderly.

[0595] Eye tracking and object recognition

[0596] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which generates voice guidance on how to use the object.

[0597] Voice guidance using generative models

[0598] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[0599] Sensor function monitors abnormal behavior

[0600] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0601] Emotion recognition by emotion engine

[0602] The device uses a microphone and camera to analyze the elderly person's tone of voice and facial expressions in real time, thereby recognizing their emotional state (e.g., anxiety, sadness, joy). This emotional data is sent to a server, and a generative model generates appropriate voice guidance based on the elderly person's emotional state.

[0603] Abnormal behavior detection and emergency notification

[0604] If abnormal behavior is detected, for example, if an elderly person falls, the server receives the information and uses the emotion engine data to create a more appropriate emergency notification. The emotion engine data, along with the abnormal behavior information, is included in the emergency notification and sent to pre-designated emergency contacts (family members or medical institutions).

[0605] Specific examples

[0606] Example 1: Medication

[0607] Take the example of a user wearing smart glasses. When the user looks at a pill bottle, the device uses an eye tracker and camera to capture that information and sends it to the server. The server uses a generative model to generate a voice prompt such as, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays this voice prompt to the user. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice, and if the user feels anxious, it provides additional voice prompts such as, "Are you worried?"

[0608] Example 2: Detecting abnormal behavior

[0609] Take the example of a user at home. An IoT sensor installed on the device monitors the user's movements. If the user falls, the device detects abnormal movements and sends the data to the server. The server receives the data, and if the emotion engine analyzes the data and determines that the user is feeling fear or anxiety, it adds this information to the emergency notification. To ensure that the situation is understood, the emergency notification is sent to pre-registered family members and medical institutions.

[0610] The present invention combines eye tracking technology, generative models, sensor technology, and an emotion engine to support the daily lives of elderly people while taking into account their emotional state, providing a safe and comfortable environment.

[0611] The processing flow will be explained below.

[0612] Step 1:

[0613] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[0614] Step 2:

[0615] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[0616] Step 3:

[0617] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[0618] Step 4:

[0619] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[0620] Step 5:

[0621] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[0622] Step 6:

[0623] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[0624] Step 7:

[0625] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[0626] Step 8:

[0627] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[0628] Step 9:

[0629] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[0630] Step 10:

[0631] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[0632] Step 11:

[0633] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotional state (e.g., anxiety, sadness, joy).

[0634] Step 12:

[0635] The server adjusts the content of the voice guidance based on the emotion engine data. For example, if the user is feeling anxious, it generates a message that provides additional reassurance.

[0636] Step 13:

[0637] The server combines abnormal behavior data with data from the emotion engine to dynamically adjust the content of emergency notifications and inform relevant parties of the situation.

[0638] Step 14:

[0639] The device periodically sends the emotion engine's analysis results to the server, and the generative model provides instructions optimized for the user's situation.

[0640] Through the above steps, the present invention comprehensively supports the daily life of the elderly and provides a safe and comfortable living environment taking into account their emotional state.

[0641] Example 2

[0642] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] There is a growing need for a system that can comprehensively monitor and support the visual, behavioral, and emotional states of the elderly in their daily lives. Conventional systems have the technology to monitor individual states (visual, behavioral, and emotional), but it is difficult to comprehensively monitor and immediately respond appropriately. The purpose of this invention is to solve these problems and provide the elderly with a safe and comfortable daily life.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0645] In this invention, the server

[0646] means for identifying an object viewed by a user using a pupil tracking device;

[0647] means for generating audio guidance on how to use the object identified using the generative model;

[0648] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[0649] a means for analyzing the user's emotional state using an emotion engine and providing an appropriate response;

[0650] This will enable comprehensive monitoring of the daily lives of elderly people and provide timely and appropriate support.

[0651] A "pupil tracking device" is a device that detects the movement of a user's pupils in real time and acquires gaze data.

[0652] A "generative model" is an artificial intelligence technique that analyzes input data and produces output in a specified format.

[0653] "Voice guidance" is a message that conveys specific information to the user by voice.

[0654] A "sensor" is a device that detects physical phenomena (movement, heart rate, blood pressure, etc.) in real time and acquires them as data.

[0655] "Abnormal movement" refers to a movement or state that deviates from the user's normal movement (e.g., falling).

[0656] An "emergency notification" is an emergency message sent to pre-designated contacts when abnormal operation is detected.

[0657] The "emotion engine" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[0658] "User" refers to a person who uses this system.

[0659] MODE FOR CARRYING OUT THE INVENTION

[0660] The present invention is a comprehensive support system for the daily lives of elderly people, which aims to make users' daily lives safer and more comfortable by combining an eye tracker, a generative model, multiple sensors, and an emotion engine. The main hardware and software components for implementing the present invention are described below.

[0661] Hardware Configuration

[0662] 1. Device (Smart Glasses):

[0663] Eye tracking device: A device that captures the user's gaze in real time and acquires that data.

[0664] Camera: A device that captures an object corresponding to the user's line of sight and generates image data.

[0665] Sensors: Equipped with motion sensors, heart rate sensors, blood pressure sensors, etc., it monitors the user's movements and health status in real time.

[0666] Microphone: A device that captures the tone of a user's voice.

[0667] Speaker: A device that conveys the generated audio guidance to the user.

[0668] 2. Server:

[0669] Generative model: A model that analyzes gaze data and image data and generates voice guidance using natural language processing technology.

[0670] Emotion engine: Technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[0671] Software Configuration

[0672] 1. Data analysis software: Installed on the server, it analyzes the data sent from the pupil tracker and camera.

[0673] 2. Speech synthesis software: Converts the text data output by the generative model into speech data.

[0674] 3. Communication software: Software for sending and receiving data between the terminal and the server.

[0675] Specific system processing examples

[0676] 1. Medication Use:

[0677] The user puts on the smart glasses and looks at the pill bottle.

[0678] The device's eye tracking device and camera capture this information, obtaining gaze data and image data.

[0679] The terminal transmits the data to the server.

[0680] The server's generative model generates voice guidance such as "Take one tablet of this medicine after breakfast."

[0681] The server transmits the guidance to the terminal, and the terminal's speaker delivers the voice guidance to the user.

[0682] The emotion engine analyzes the user's facial expressions and tone of voice, and if they feel anxious, it provides additional voice guidance such as, "Are you worried about something?"

[0683] 2. Abnormal behavior detection:

[0684] The user falls while at home.

[0685] A motion sensor installed in the device detects falls.

[0686] The terminal transmits the data to the server.

[0687] The server determines that the user is feeling fear based on the abnormal behavior data and the analysis results of the emotion engine, and creates an emergency notification.

[0688] Emergency notifications will be sent to pre-designated family members and medical institutions.

[0689] Prompt Sentence Examples

[0690] 1. Example prompts for medication use:

[0691] The user is looking at a bottle of medicine. The information is captured by the eye tracker and camera, and analyzed by the server to generate a voice prompt such as, "Take one tablet of this medicine after breakfast." If the user is feeling anxious, the system provides additional guidance such as, "Are you worried about something?"

[0692] 2. Example prompt for detecting abnormal behavior:

[0693] If a user falls while at home, the motion sensor detects the abnormality and sends this information to a server, which then sends an emergency notification to family members or medical institutions, including anxiety data analyzed by an emotion engine.

[0694] As described above, the present invention combines an eye tracking device, a generative model, sensor technology, and an emotion engine to comprehensively support the daily lives of the elderly and provide a safe and comfortable environment.

[0695] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0696] System program processing steps

[0697] Specific processing steps for medication use

[0698] Step 1:

[0699] A user wears the smart glasses and looks at a medicine bottle, while gaze data is captured in real time by a pupil tracker.

[0700] Input: User's gaze

[0701] Output: Gaze data

[0702] Step 2:

[0703] The device's camera captures an image of the object (medicine bottle) that corresponds to the user's line of sight.

[0704] Input: Gaze data

[0705] Output: Image data

[0706] Step 3:

[0707] The device transmits the captured gaze data and image data to a server via Wi-Fi or mobile networks.

[0708] Input: Gaze data, image data

[0709] Output: Send data to the server

[0710] Step 4:

[0711] The server analyzes the received gaze data and image data, and a generative model identifies the medicine bottle based on the specific image data and gaze data.

[0712] Input: Gaze data, image data

[0713] Output: Identified object (medicine bottle)

[0714] Step 5:

[0715] The server's generative model analyzes how to use the identified object and uses natural language processing technology to generate voice guidance such as, "Take one tablet of this medicine after breakfast."

[0716] Input: Identified object

[0717] Output: Voice guidance (text format)

[0718] Step 6:

[0719] The server transmits the generated voice guidance to the terminal.

[0720] Input: Voice guidance (text format)

[0721] Output: Send voice guidance to the device

[0722] Step 7:

[0723] The device converts the received voice guidance into audio data and plays it back through the speaker, providing the user with audio instructions on how to use the device.

[0724] Input: Voice guidance (text format)

[0725] Output: Voice guidance (voice data)

[0726] Step 8:

[0727] The device's emotion engine analyzes the user's facial expressions and tone of voice in real time, and if they are feeling anxious, it generates additional voice guidance such as, "Are you worried about something?"

[0728] Input: User's facial expression data, tone of voice

[0729] Output: Additional voice guidance

[0730] Step 9:

[0731] Additional audio guidance generated by the terminal is played back to give the user a sense of security.

[0732] Input: Additional voice guidance

[0733] Output: Additional voice guidance playback

[0734] Specific processing steps for detecting abnormal behavior

[0735] Step 1:

[0736] The user falls while at home.

[0737] Input: User action

[0738] Step 2:

[0739] The motion sensor installed in the device detects falls, for example, detecting abnormal movements from sudden changes in gravitational acceleration.

[0740] Input: Operation data

[0741] Output: Abnormal operation data

[0742] Step 3:

[0743] The terminal transmits the abnormal operation data to the server.

[0744] Input: Abnormal behavior data

[0745] Output: Send abnormal operation data to the server

[0746] Step 4:

[0747] The server analyzes the received abnormal operation data and confirms the abnormal operation.

[0748] Input: Abnormal behavior data

[0749] Output: Analysis results (confirmation of abnormal behavior)

[0750] Step 5:

[0751] The server uses an emotion engine to analyze whether the user feels fear or anxiety when they fall.

[0752] Input: User's facial expression data, tone of voice

[0753] Output: Emotion data (fear, anxiety)

[0754] Step 6:

[0755] The server generates an emergency notification based on the abnormal motion data and emotion data, such as "The user has fallen and is currently experiencing fear."

[0756] Input: Abnormal behavior data, emotion data

[0757] Output: Urgent Notification

[0758] Step 7:

[0759] The server then sends the emergency notification to pre-designated family members or medical institutions.

[0760] Input: Emergency Notification

[0761] Output: Send emergency notification

[0762] As mentioned above, each processing step is explained in detail with specific operations and data flows. The overall system effectively combines eye tracking technology, generative models, sensor technology, and an emotion engine to help the elderly live a safe and comfortable life.

[0763] (Application example 2)

[0764] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0765] In recent years, technology to support the lives of the elderly has been attracting attention. However, when using food delivery services, many difficulties arise, particularly when selecting menus and receiving food. For this reason, there is a need for a system that allows elderly people to use food delivery services safely and comfortably. Furthermore, the current situation is that meal suggestions tailored to health conditions and support when receiving deliveries are not being provided sufficiently. To solve these issues, a comprehensive system is needed that allows elderly people to use food delivery services with peace of mind.

[0766] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0767] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's movements using multiple sensors and detecting abnormal movements, means for sending an emergency notification to a pre-designated emergency contact when abnormal movements are detected, means for supporting the elderly with menu selection and meal suggestions based on their health condition when using a food delivery service using audio guidance, and means for measuring the height and distance when a delivery arrives and supporting the receiving operation. This allows the elderly to use food delivery services with peace of mind, select appropriate meals based on their health condition, and receive their delivery safely.

[0768] An "eye tracker" is a device that tracks a user's gaze and identifies the object they are looking at.

[0769] "Generative model" refers to an algorithm that generates guidance in natural language based on input data.

[0770] A "sensor" is a device that detects physical movements and environmental conditions and outputs them as digital data.

[0771] "Abnormal movement" refers to movement that differs from the user's normal behavioral pattern, and includes, for example, falling.

[0772] An "emergency notification" is an alert sent to pre-registered contacts when an unexpected situation occurs.

[0773] "Food delivery service" refers to a service that delivers food or meals to a location specified by the customer.

[0774] "Menu selection" refers to the act of selecting what to order from the dishes and foods offered by a food delivery service.

[0775] "Meal suggestions tailored to your health condition" refers to recommending appropriate meals taking into consideration the user's health condition and preferences.

[0776] "Delivery arrival" refers to the moment when the ordered food or meal is delivered to the location specified by the user.

[0777] "Measuring height and distance" means measuring the surrounding conditions to ensure that elderly people can receive deliveries safely.

[0778] This invention is a system that enables elderly people to use food delivery services safely and comfortably, and is realized by combining an eye tracking device, a generative model, sensors, and an emotion engine. The system program is implemented using the following hardware and software.

[0779] Hardware

[0780] Smart glasses: Equipped with a camera, eye tracker, microphone, and speaker.

[0781] Server: Used to analyze data and generate audio guidance.

[0782] software

[0783] OpenCV: A library for image processing.

[0784] TensorFlow & Keras: Implementation and operation of pupil tracking model.

[0785] Transformers: A library for sentiment analysis.

[0786] HTTP Requests: A library for communicating with the server.

[0787] Data processing and calculation

[0788] The server first processes the video frames captured by the smart glasses using OpenCV and identifies the eye position using an eye tracking model. This allows it to identify the menu or object the elderly person is looking at. The generative model then uses natural language processing to convert the identified object's usage instructions and appropriate meal suggestions into voice guidance, which is then provided to the elderly. Multiple sensors are also used to monitor the elderly's movements and health status, and if abnormal behavior is detected, an emergency notification is sent to a pre-designated emergency contact.

[0789] Specific examples

[0790] Menu selection assistance

[0791] For example, consider a situation where an elderly person is wearing smart glasses and browsing a food delivery app. If the eye tracking device detects that the user's gaze is fixed on a particular menu item (e.g., pancakes), a voice guidance such as "This dish is high in calories. We recommend you change it to a salad" is generated and provided to the user.

[0792] Delivery reception assistance

[0793] When the delivery person arrives, sensors installed in the smart glasses measure the location and distance and provide guidance such as "The door is one meter away. Please walk slowly" to avoid the risk of the elderly person falling.

[0794] Prompt Sentence Examples

[0795] "When an elderly person looks at the menu of a food delivery app, generate meal suggestions tailored to their health condition."

[0796] The present invention allows elderly people to use food delivery services with peace of mind, allowing them to select appropriate meals according to their health condition and receive deliveries safely, thereby improving the quality of life of elderly people.

[0797] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0798] Step 1:

[0799] The camera in the smart glasses captures the elderly person's field of vision.

[0800] Input: Elderly person's visual field

[0801] Output: Captured video frames

[0802] The device uses a camera mounted on the smart glasses to capture the elderly person's field of vision in real time, and these video frames are used as input for the subsequent eye tracking process.

[0803] Step 2:

[0804] An eye tracker identifies the location of the elderly person's eyes.

[0805] Input: Captured video frames

[0806] Output: Eye coordinate data

[0807] The device processes the captured video frames using the OpenCV library and identifies the location of the elderly person's eyes using an eye tracking model (TensorFlow and Keras). The obtained eye coordinate data is used to identify the object the user is looking at.

[0808] Step 3:

[0809] The generative model identifies the objects the elderly person sees and generates audio guidance.

[0810] Input: pupil coordinate data, captured video frame

[0811] Output: Voice guidance data

[0812] The server analyzes the pupil coordinate data and the captured video frames to identify the object the user is looking at. It then inputs a prompt sentence into a generative model based on the identified object to generate appropriate voice guidance. The generative model then uses natural language processing technology to generate voice guidance data and transmit it to the device.

[0813] Step 4:

[0814] The device plays audio guidance.

[0815] Input: Voice guidance data

[0816] Output: Voice guidance

[0817] The terminal plays the voice guidance data received from the server and provides voice guidance to the elderly, such as "This dish is high in calories. We recommend changing it to a salad."

[0818] Step 5:

[0819] Sensors monitor the movements and health of the elderly.

[0820] Input: Sensor data (heart rate, blood pressure, movement, etc.)

[0821] Output: Monitoring data

[0822] The device uses multiple sensors to continuously monitor the elderly person's heart rate, blood pressure, and movements, and collects this data, which is used to detect abnormal movements.

[0823] Step 6:

[0824] If abnormal behavior is detected, an emergency notification will be sent.

[0825] Input: Monitoring data

[0826] Output: Urgent Notification

[0827] The server analyzes the monitoring data sent from the device and generates an emergency notification if any abnormal behavior is detected, and sends it to pre-designated emergency contacts (such as family members or medical institutions). This notification may also include information about the elderly person's health and emotional state.

[0828] Step 7:

[0829] When a delivery arrives, the system measures the height and distance to assist with the receiving process.

[0830] Input: Sensor data (distance measurement)

[0831] Output: Voice guidance

[0832] When a delivery arrives, the device uses sensors to measure the height and distance, and provides voice guidance such as "The distance to the door is one meter. Please walk slowly" to ensure that the elderly person can receive the delivery safely.

[0833] This will allow elderly people to use food delivery services with peace of mind, make appropriate meal choices, and receive their deliveries safely.

[0834] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0835] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0836] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0837] [Third embodiment]

[0838] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0839] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0840] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0841] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0842] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0843] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0844] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0845] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0846] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0847] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0848] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0849] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0850] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye tracker, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[0851] Eye tracking and object recognition

[0852] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which then generates voice guidance on how to use the object.

[0853] Voice guidance using generative models

[0854] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[0855] Sensor function monitors abnormal behavior

[0856] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0857] Abnormal behavior detection and emergency notification

[0858] If abnormal behavior is detected, for example if an elderly person falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[0859] Specific examples

[0860] Example 1: Medication

[0861] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt, such as "Take one pill of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[0862] Example 2: Detecting abnormal behavior

[0863] Take the example of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and sends additional alerts if the user does not respond.

[0864] The present invention combines eye tracking, generative models, and sensor technology to make the daily lives of seniors safer and more comfortable, while reducing anxiety for their families and caregivers.

[0865] The processing flow will be explained below.

[0866] Step 1:

[0867] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[0868] Step 2:

[0869] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[0870] Step 3:

[0871] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[0872] Step 4:

[0873] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[0874] Step 5:

[0875] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[0876] Step 6:

[0877] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[0878] Step 7:

[0879] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[0880] Step 8:

[0881] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[0882] Step 9:

[0883] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[0884] Step 10:

[0885] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[0886] Step 11:

[0887] If the user does not respond in a critical situation, the server will repeatedly send additional emergency alerts to prompt the necessary response.

[0888] Through the above steps, this system comprehensively supports the daily lives of elderly people and provides an environment in which they can live safely and comfortably.

[0889] Example 1

[0890] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0891] To support the safe and comfortable daily lives of the elderly, advanced technologies are needed for gaze tracking, object recognition, health monitoring, and other functions. However, conventional systems have difficulty integrating these functions and have limitations in real-time monitoring and anomaly detection. As a result, they are unable to completely alleviate the anxiety of the elderly and their families, and are also difficult to respond to emergency situations that require a rapid response. This invention aims to solve these problems and make the daily lives of the elderly safer and more comfortable.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0893] In this invention, the server includes a means for identifying an object the user is looking at using an eye-tracking device, a means for generating audio guidance on how to use the identified object using a generative model, and a means for monitoring the user's motion and health status using multiple sensors and detecting abnormal motion. This enables integrated gaze tracking, object recognition, and real-time monitoring. Furthermore, if an abnormality is detected, an emergency notification can be sent immediately, enabling a rapid response and improving the safety and quality of life of the elderly.

[0894] An "eye tracking device" is a device that identifies the position and movement of a user's eyes and tracks the direction of their gaze in real time.

[0895] A "generative model" is a machine learning model that analyzes received data and generates voice guidance.

[0896] A "photography device" is a camera or image capture device that captures an image of an object viewed by a user.

[0897] A "voice synthesizer" is a device that reproduces voice guidance created by a generative model as actual voice.

[0898] A "sensor" is a device used to measure and monitor a user's activity or health. Examples include motion sensors, heart rate sensors, and blood pressure sensors.

[0899] A "profile" is a collection of data that includes personal information and settings about a user.

[0900] "Initialization" is a preparatory procedure for starting the operation of a system or device.

[0901] An "emergency notification" is a warning message sent to a pre-designated emergency contact when abnormal operation is detected.

[0902] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye-tracking device, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[0903] Eye tracking and object recognition

[0904] When a user puts on the smart glasses, the device automatically initializes itself. The smart glasses are equipped with an eye-tracking device and a camera. To identify the object the user is looking at in real time, the eye-tracking device measures the position and movement of the eyes, and the camera captures the image of the object being viewed. For example, if the user is looking at a medicine bottle, the device captures that information and sends it to a server.

[0905] Voice guidance using generative models

[0906] The server hosts a generative model (e.g., GPT-4) that analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, it generates guidance such as, "Take one tablet of this medicine after breakfast." The generated voice guidance is then sent to the device and notified to the user via the device's voice synthesizer.

[0907] Sensor function monitors abnormal behavior

[0908] The device is equipped with multiple sensors that continuously monitor the elderly person's activity and health status, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's activity and health status in real time, and if any abnormal activity or situation is detected, the data is immediately sent to the server.

[0909] Abnormal behavior detection and emergency notification

[0910] If an abnormal movement is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling a prompt response even when the elderly are alone.

[0911] Specific examples

[0912] Example 1: Medication

[0913] Suppose a user is wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye-tracking device and a camera, and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then notifies the user with the voice prompt.

[0914] Example 2: Detecting abnormal behavior

[0915] Consider the case of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and also sends additional alerts if the user does not respond.

[0916] Example prompt sentence:

[0917] "Please explain how eye tracking and generative models can be used to support elderly people in taking medication."

[0918] "Please explain how emergency notifications are sent if an elderly person falls, including how the specific sensors work."

[0919] In this way, the combination of eye-tracking technology, generative models, and sensor technology can make the daily lives of older people safer and more comfortable, and reduce anxiety for their families and caregivers.

[0920] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0921] Step 1: Initialize the device

[0922] The smart glasses as a terminal perform initialization. This includes checking the operation of the eye tracking device, the camera, and various sensors. It loads the user's profile and reads saved health data and settings. Specifically, the terminal recognizes the user's face and checks the status of each sensor. The input data includes the user's profile information and execution confirmation data from the sensors, and the output data is the initialization completion status.

[0923] Step 2: Eye tracking and object recognition

[0924] To monitor the user's daily activities, the device keeps its eye-tracking device and camera enabled at all times. When the user gazes at a specific object, the device identifies the direction of the user's gaze and captures a photo of that object. Specifically, the eye-tracking device captures the movement of the user's pupils, and the camera captures a video of the recognized object. The input data is the user's gaze information and the captured video data, and the output data is information about the recognized object, which is sent to the server.

[0925] Step 3: Send data

[0926] The device sends data of recognized objects to the server, including captured video data and gaze tracking data. Specifically, the device collects data and sends it to the server using a secure communication protocol. The input data is the captured video and gaze data, and the output data is the status of completion of transmission to the server.

[0927] Step 4: Analysis and Guidance Generation

[0928] The server inputs the received data into a generative model to generate appropriate guidance. The generative model uses natural language processing technology to create voice guidance that provides appropriate instructions to the user. For example, if it recognizes a medicine bottle, it generates instructions such as "Take one tablet of this medicine after breakfast." The input data is gaze and video data, and the output data is the generated voice guidance.

[0929] Step 5: Provide audio guidance

[0930] The terminal that receives the voice guidance generated by the server uses a voice synthesizer to convey the content to the user. The input data is the generated voice guidance, and the output data is the voice instructions that the user hears. Specifically, the terminal receives the voice guidance data, and the voice synthesizer plays it back.

[0931] Step 6: Detecting Abnormal Behavior

[0932] The device continuously monitors the user's movement and health status using motion sensors, heart rate sensors, and blood pressure sensors. If an abnormality is detected, the data is immediately sent to the server. The input data is the movement and health data collected by the sensors, and the output data is an alert of the abnormality. Specifically, if the user falls, the acceleration sensor and other sensors detect the abnormality and send the information to the server.

[0933] Step 7: Emergency Notification

[0934] The server receives data on detected abnormal behavior and sends a notification to emergency contacts. The notification includes location information and real-time health data. The input data is the abnormality detection notification data, and the output data is the notification content sent to emergency contacts. Specifically, the server analyzes the abnormal data and sends a message to family members or medical institutions such as, "The user has fallen. The location is ____."

[0935] (Application example 1)

[0936] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] In modern society, safety and health management, especially for the elderly, are important issues. In the course of elderly people's daily lives, prompt responses are required to abnormal behavior, such as medication errors or falls. However, current systems have limitations in identifying objects visually recognized by elderly people and in responding quickly to emergencies. Furthermore, generating voice guidance requires manual input, which is cumbersome for elderly people. To address these issues, the present invention aims to automate the identification of objects visually recognized by elderly people and the detection of abnormal behavior, and provide appropriate voice guidance and emergency notifications in real time.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0939] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's behavior using multiple sensors and detecting abnormal behavior, means for sending an emergency notification to a pre-specified emergency contact when abnormal behavior is detected, and means for converting the captured object into audio guidance based on natural language processing. This makes it possible to improve the safety and comfort of elderly people's daily lives and reduce the anxiety of their families and caregivers.

[0940] 1. "Pupil tracking device" means a device that detects the position and movement of the user's pupils.

[0941] 2. "User" refers to a person who uses this system.

[0942] 3. "Object Identification" means the act of using an eye tracker and camera to identify an object that is viewed by the user.

[0943] 4. A "generative model" is a program or system that uses AI or machine learning algorithms to generate useful output from input data.

[0944] 5. "Voice guidance" is a system that uses voice synthesis technology to provide specific information as voice.

[0945] 6. "Multiple sensors" refers to multiple sensor devices used to measure various data, such as a motion sensor, a heart rate sensor, and a blood pressure sensor.

[0946] 7. "Movement monitoring" refers to the act of continuously observing a user's physical movements and condition using sensors.

[0947] 8. "Abnormal behavior" refers to behavior that deviates from normal or healthy behavior, such as falls by elderly people.

[0948] 9. "Emergency Notification" means a notification sent immediately to an emergency contact when abnormal activity is detected.

[0949] 10. "Emergency Contact" means a contact (such as a family member or medical institution) designated to receive notification in the event of an emergency.

[0950] 11. "Camera" means a photographic device for capturing images.

[0951] 12. "Capture" means the act of collecting and recording image or video data.

[0952] 13. "Natural language processing" is a general term for technologies that allow computers to understand and generate human language.

[0953] The present invention is a system for making the daily lives of elderly people safe and comfortable. This system is realized by integrating various hardware and software components, such as a pupil tracking device, a generative model, and multiple sensors. The specific configuration and operating procedures for implementing the present invention are described in detail below.

[0954] Eye tracking and object recognition

[0955] The smart glasses used as a terminal are equipped with an eye tracker and a camera. This eye tracker allows the user (elderly person) to identify the object they are looking at in real time. The camera captures image data of the object being viewed and sends it to a server. On the server side, the image data is analyzed using a generative model, and audio guidance on how to use the identified object is generated.

[0956] Voice guidance using generative models

[0957] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the generative model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the user.

[0958] Sensor function monitors abnormal behavior

[0959] The device is equipped with multiple sensors that continuously monitor the user's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[0960] Abnormal behavior detection and emergency notification

[0961] If abnormal behavior is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[0962] Hardware and software used

[0963] The system uses the following hardware and software:

[0964] Pupil tracker: A device that detects the position and movement of the pupil.

[0965] Camera: A device that captures image data of objects in view.

[0966] Movement, heart rate, and blood pressure sensors: Sensors for monitoring the movement and health of seniors.

[0967] Server: Hosts the generative model, analyzes data, and generates audio guidance.

[0968] Generative model: A program or system that uses AI or machine learning algorithms to generate useful output from input data.

[0969] Natural Language Processing (NLP) technology: Technology that enables computers to understand and generate human language.

[0970] Specific examples

[0971] Example 1: Medication

[0972] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[0973] Example 2: Detecting abnormal behavior

[0974] Consider the case of a user at home. The device's built-in sensors monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and if the user does not respond, it sends an additional alert.

[0975] Prompt Sentence Examples

[0976] "The identified object is a 'medicine bottle'. Please generate the following audio prompt: 'Take one tablet of this medicine after breakfast.'"

[0977] As described above, the present invention is a system that combines a pupil tracking device, a generative model, and multiple sensors to make the daily lives of elderly people safe and comfortable.

[0978] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0979] Step 1:

[0980] The camera in the smart glasses worn by the device captures the image of the user's field of vision in real time. The input is the image data of the user's field of vision, and the output is the image data. The camera provides high-resolution images, which are sent to a processor for use in the next step.

[0981] Step 2:

[0982] The device's built-in eye tracking device tracks the position of the user's eyes and overlays them on the camera image to identify the object the user is looking at. The input is the user's eye data (pupil position and movement), and the output is the coordinate data of the object the user is looking at. The position of the identified object is clarified based on the eye position information.

[0983] Step 3:

[0984] The device sends image data of the identified object to the server. The input is the image data of the identified object, and the output is the data transmission to the server. The data is transmitted at high speed and is optimized to minimize latency.

[0985] Step 4:

[0986] The server inputs the received image data into a generative model to recognize the object. The input is the image data of the identified object, and the output is the object name. The generative model uses a pre-trained AI algorithm to classify the object and obtain a result. For example, the image data is recognized as a medicine bottle.

[0987] Step 5:

[0988] The server converts the recognized object's usage instructions into voice guidance using natural language processing technology. The input is the object name and data on its usage, and the output is text data for the voice guidance. The generative model prepares the voice guidance, saying, "Take one tablet of this medicine after breakfast."

[0989] Step 6:

[0990] The server sends the text data of the voice guidance to the terminal. The input is the text data of the voice guidance, and the output is the data transmission to the terminal. The data is sent in real time, providing quick guidance to the elderly.

[0991] Step 7:

[0992] The device uses voice synthesis technology to play back the voice guidance. The input is text data received from the server, and the output is voice guidance. In the case of smart glasses, the voice is transmitted to the elderly using a built-in speaker.

[0993] Step 8:

[0994] Multiple sensors built into the device monitor the user's movements and health. The input is real-time data from the sensors (e.g., heart rate, movements), and the output is the monitoring results. The sensor data is continuously collected and checked for any abnormalities.

[0995] Step 9:

[0996] When a device detects abnormal behavior (e.g., a fall), the data is immediately sent to the server. The input is sensor data of the abnormal behavior, and the output is emergency data sent to the server. The data is sent at high speed, allowing for a rapid response to emergencies.

[0997] Step 10:

[0998] The server receives data on abnormal behavior and sends an emergency notification to pre-designated emergency contacts. The input is data on abnormal behavior, and the output is a notification to the emergency contact (including location information and health data). An email or SMS is sent to the emergency contact, enabling a prompt response.

[0999] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1000] This invention is a system for making the daily lives of the elderly safe and comfortable. This system combines an eye tracker, a generative model, multiple sensors, and an emotion engine to comprehensively support the daily lives of the elderly.

[1001] Eye tracking and object recognition

[1002] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which generates voice guidance on how to use the object.

[1003] Voice guidance using generative models

[1004] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[1005] Sensor function monitors abnormal behavior

[1006] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[1007] Emotion recognition by emotion engine

[1008] The device uses a microphone and camera to analyze the elderly person's tone of voice and facial expressions in real time, thereby recognizing their emotional state (e.g., anxiety, sadness, joy). This emotional data is sent to a server, and a generative model generates appropriate voice guidance based on the elderly person's emotional state.

[1009] Abnormal behavior detection and emergency notification

[1010] If abnormal behavior is detected, for example, if an elderly person falls, the server receives the information and uses the emotion engine data to create a more appropriate emergency notification. The emotion engine data, along with the abnormal behavior information, is included in the emergency notification and sent to pre-designated emergency contacts (family members or medical institutions).

[1011] Specific examples

[1012] Example 1: Medication

[1013] Take the example of a user wearing smart glasses. When the user looks at a pill bottle, the device uses an eye tracker and camera to capture that information and sends it to the server. The server uses a generative model to generate a voice prompt such as, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays this voice prompt to the user. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice, and if the user feels anxious, it provides additional voice prompts such as, "Are you worried?"

[1014] Example 2: Detecting abnormal behavior

[1015] Take the example of a user at home. An IoT sensor installed on the device monitors the user's movements. If the user falls, the device detects abnormal movements and sends the data to the server. The server receives the data, and if the emotion engine analyzes the data and determines that the user is feeling fear or anxiety, it adds this information to the emergency notification. To ensure that the situation is understood, the emergency notification is sent to pre-registered family members and medical institutions.

[1016] The present invention combines eye tracking technology, generative models, sensor technology, and an emotion engine to support the daily lives of elderly people while taking into account their emotional state, providing a safe and comfortable environment.

[1017] The processing flow will be explained below.

[1018] Step 1:

[1019] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[1020] Step 2:

[1021] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[1022] Step 3:

[1023] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[1024] Step 4:

[1025] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[1026] Step 5:

[1027] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[1028] Step 6:

[1029] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[1030] Step 7:

[1031] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[1032] Step 8:

[1033] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[1034] Step 9:

[1035] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[1036] Step 10:

[1037] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[1038] Step 11:

[1039] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotional state (e.g., anxiety, sadness, joy).

[1040] Step 12:

[1041] The server adjusts the content of the voice guidance based on the emotion engine data. For example, if the user is feeling anxious, it generates a message that provides additional reassurance.

[1042] Step 13:

[1043] The server combines abnormal behavior data with data from the emotion engine to dynamically adjust the content of emergency notifications and inform relevant parties of the situation.

[1044] Step 14:

[1045] The device periodically sends the emotion engine's analysis results to the server, and the generative model provides instructions optimized for the user's situation.

[1046] Through the above steps, the present invention comprehensively supports the daily life of the elderly and provides a safe and comfortable living environment taking into account their emotional state.

[1047] Example 2

[1048] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1049] There is a growing need for a system that can comprehensively monitor and support the visual, behavioral, and emotional states of the elderly in their daily lives. Conventional systems have the technology to monitor individual states (visual, behavioral, and emotional), but it is difficult to comprehensively monitor and immediately respond appropriately. The purpose of this invention is to solve these problems and provide the elderly with a safe and comfortable daily life.

[1050] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1051] In this invention, the server

[1052] means for identifying an object viewed by a user using a pupil tracking device;

[1053] means for generating audio guidance on how to use the object identified using the generative model;

[1054] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[1055] a means for analyzing the user's emotional state using an emotion engine and providing an appropriate response;

[1056] This will enable comprehensive monitoring of the daily lives of elderly people and provide timely and appropriate support.

[1057] A "pupil tracking device" is a device that detects the movement of a user's pupils in real time and acquires gaze data.

[1058] A "generative model" is an artificial intelligence technique that analyzes input data and produces output in a specified format.

[1059] "Voice guidance" is a message that conveys specific information to the user by voice.

[1060] A "sensor" is a device that detects physical phenomena (movement, heart rate, blood pressure, etc.) in real time and acquires them as data.

[1061] "Abnormal movement" refers to a movement or state that deviates from the user's normal movement (e.g., falling).

[1062] An "emergency notification" is an emergency message sent to pre-designated contacts when abnormal operation is detected.

[1063] The "emotion engine" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[1064] "User" refers to a person who uses this system.

[1065] MODE FOR CARRYING OUT THE INVENTION

[1066] The present invention is a comprehensive support system for the daily lives of elderly people, which aims to make users' daily lives safer and more comfortable by combining an eye tracker, a generative model, multiple sensors, and an emotion engine. The main hardware and software components for implementing the present invention are described below.

[1067] Hardware Configuration

[1068] 1. Device (Smart Glasses):

[1069] Eye tracking device: A device that captures the user's gaze in real time and acquires that data.

[1070] Camera: A device that captures an object corresponding to the user's line of sight and generates image data.

[1071] Sensors: Equipped with motion sensors, heart rate sensors, blood pressure sensors, etc., it monitors the user's movements and health status in real time.

[1072] Microphone: A device that captures the tone of a user's voice.

[1073] Speaker: A device that conveys the generated audio guidance to the user.

[1074] 2. Server:

[1075] Generative model: A model that analyzes gaze data and image data and generates voice guidance using natural language processing technology.

[1076] Emotion engine: Technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[1077] Software Configuration

[1078] 1. Data analysis software: Installed on the server, it analyzes the data sent from the pupil tracker and camera.

[1079] 2. Speech synthesis software: Converts the text data output by the generative model into speech data.

[1080] 3. Communication software: Software for sending and receiving data between the terminal and the server.

[1081] Specific system processing examples

[1082] 1. Medication Use:

[1083] The user puts on the smart glasses and looks at the pill bottle.

[1084] The device's eye tracking device and camera capture this information, obtaining gaze data and image data.

[1085] The terminal transmits the data to the server.

[1086] The server's generative model generates voice guidance such as "Take one tablet of this medicine after breakfast."

[1087] The server transmits the guidance to the terminal, and the terminal's speaker delivers the voice guidance to the user.

[1088] The emotion engine analyzes the user's facial expressions and tone of voice, and if they feel anxious, it provides additional voice guidance such as, "Are you worried about something?"

[1089] 2. Abnormal behavior detection:

[1090] The user falls while at home.

[1091] A motion sensor installed in the device detects falls.

[1092] The terminal transmits the data to the server.

[1093] The server determines that the user is feeling fear based on the abnormal behavior data and the analysis results of the emotion engine, and creates an emergency notification.

[1094] Emergency notifications will be sent to pre-designated family members and medical institutions.

[1095] Prompt Sentence Examples

[1096] 1. Example prompts for medication use:

[1097] The user is looking at a bottle of medicine. The information is captured by the eye tracker and camera, and analyzed by the server to generate a voice prompt such as, "Take one tablet of this medicine after breakfast." If the user is feeling anxious, the system provides additional guidance such as, "Are you worried about something?"

[1098] 2. Example prompt for detecting abnormal behavior:

[1099] If a user falls while at home, the motion sensor detects the abnormality and sends this information to a server, which then sends an emergency notification to family members or medical institutions, including anxiety data analyzed by an emotion engine.

[1100] As described above, the present invention combines an eye tracking device, a generative model, sensor technology, and an emotion engine to comprehensively support the daily lives of the elderly and provide a safe and comfortable environment.

[1101] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1102] System program processing steps

[1103] Specific processing steps for medication use

[1104] Step 1:

[1105] A user wears the smart glasses and looks at a medicine bottle, while gaze data is captured in real time by a pupil tracker.

[1106] Input: User's gaze

[1107] Output: Gaze data

[1108] Step 2:

[1109] The device's camera captures an image of the object (medicine bottle) that corresponds to the user's line of sight.

[1110] Input: Gaze data

[1111] Output: Image data

[1112] Step 3:

[1113] The device transmits the captured gaze data and image data to a server via Wi-Fi or mobile networks.

[1114] Input: Gaze data, image data

[1115] Output: Send data to the server

[1116] Step 4:

[1117] The server analyzes the received gaze data and image data, and a generative model identifies the medicine bottle based on the specific image data and gaze data.

[1118] Input: Gaze data, image data

[1119] Output: Identified object (medicine bottle)

[1120] Step 5:

[1121] The server's generative model analyzes how to use the identified object and uses natural language processing technology to generate voice guidance such as, "Take one tablet of this medicine after breakfast."

[1122] Input: Identified object

[1123] Output: Voice guidance (text format)

[1124] Step 6:

[1125] The server transmits the generated voice guidance to the terminal.

[1126] Input: Voice guidance (text format)

[1127] Output: Send voice guidance to the device

[1128] Step 7:

[1129] The device converts the received voice guidance into audio data and plays it back through the speaker, providing the user with audio instructions on how to use the device.

[1130] Input: Voice guidance (text format)

[1131] Output: Voice guidance (voice data)

[1132] Step 8:

[1133] The device's emotion engine analyzes the user's facial expressions and tone of voice in real time, and if they are feeling anxious, it generates additional voice guidance such as, "Are you worried about something?"

[1134] Input: User's facial expression data, tone of voice

[1135] Output: Additional voice guidance

[1136] Step 9:

[1137] Additional audio guidance generated by the terminal is played back to give the user a sense of security.

[1138] Input: Additional voice guidance

[1139] Output: Additional voice guidance playback

[1140] Specific processing steps for detecting abnormal behavior

[1141] Step 1:

[1142] The user falls while at home.

[1143] Input: User action

[1144] Step 2:

[1145] The motion sensor installed in the device detects falls, for example, detecting abnormal movements from sudden changes in gravitational acceleration.

[1146] Input: Operation data

[1147] Output: Abnormal operation data

[1148] Step 3:

[1149] The terminal transmits the abnormal operation data to the server.

[1150] Input: Abnormal behavior data

[1151] Output: Send abnormal operation data to the server

[1152] Step 4:

[1153] The server analyzes the received abnormal operation data and confirms the abnormal operation.

[1154] Input: Abnormal behavior data

[1155] Output: Analysis results (confirmation of abnormal behavior)

[1156] Step 5:

[1157] The server uses an emotion engine to analyze whether the user feels fear or anxiety when they fall.

[1158] Input: User's facial expression data, tone of voice

[1159] Output: Emotion data (fear, anxiety)

[1160] Step 6:

[1161] The server generates an emergency notification based on the abnormal motion data and emotion data, such as "The user has fallen and is currently experiencing fear."

[1162] Input: Abnormal behavior data, emotion data

[1163] Output: Urgent Notification

[1164] Step 7:

[1165] The server then sends the emergency notification to pre-designated family members or medical institutions.

[1166] Input: Emergency Notification

[1167] Output: Send emergency notification

[1168] As mentioned above, each processing step is explained in detail with specific operations and data flows. The overall system effectively combines eye tracking technology, generative models, sensor technology, and an emotion engine to help the elderly live a safe and comfortable life.

[1169] (Application example 2)

[1170] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1171] In recent years, technology to support the lives of the elderly has been attracting attention. However, when using food delivery services, many difficulties arise, particularly when selecting menus and receiving food. For this reason, there is a need for a system that allows elderly people to use food delivery services safely and comfortably. Furthermore, the current situation is that meal suggestions tailored to health conditions and support when receiving deliveries are not being provided sufficiently. To solve these issues, a comprehensive system is needed that allows elderly people to use food delivery services with peace of mind.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1173] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's movements using multiple sensors and detecting abnormal movements, means for sending an emergency notification to a pre-designated emergency contact when abnormal movements are detected, means for supporting the elderly with menu selection and meal suggestions based on their health condition when using a food delivery service using audio guidance, and means for measuring the height and distance when a delivery arrives and supporting the receiving operation. This allows the elderly to use food delivery services with peace of mind, select appropriate meals based on their health condition, and receive their delivery safely.

[1174] An "eye tracker" is a device that tracks a user's gaze and identifies the object they are looking at.

[1175] "Generative model" refers to an algorithm that generates guidance in natural language based on input data.

[1176] A "sensor" is a device that detects physical movements and environmental conditions and outputs them as digital data.

[1177] "Abnormal movement" refers to movement that differs from the user's normal behavioral pattern, and includes, for example, falling.

[1178] An "emergency notification" is an alert sent to pre-registered contacts when an unexpected situation occurs.

[1179] "Food delivery service" refers to a service that delivers food or meals to a location specified by the customer.

[1180] "Menu selection" refers to the act of selecting what to order from the dishes and foods offered by a food delivery service.

[1181] "Meal suggestions tailored to your health condition" refers to recommending appropriate meals taking into consideration the user's health condition and preferences.

[1182] "Delivery arrival" refers to the moment when the ordered food or meal is delivered to the location specified by the user.

[1183] "Measuring height and distance" means measuring the surrounding conditions to ensure that elderly people can receive deliveries safely.

[1184] This invention is a system that enables elderly people to use food delivery services safely and comfortably, and is realized by combining an eye tracking device, a generative model, sensors, and an emotion engine. The system program is implemented using the following hardware and software.

[1185] Hardware

[1186] Smart glasses: Equipped with a camera, eye tracker, microphone, and speaker.

[1187] Server: Used to analyze data and generate audio guidance.

[1188] software

[1189] OpenCV: A library for image processing.

[1190] TensorFlow & Keras: Implementation and operation of pupil tracking model.

[1191] Transformers: A library for sentiment analysis.

[1192] HTTP Requests: A library for communicating with the server.

[1193] Data processing and calculation

[1194] The server first processes the video frames captured by the smart glasses using OpenCV and identifies the eye position using an eye tracking model. This allows it to identify the menu or object the elderly person is looking at. The generative model then uses natural language processing to convert the identified object's usage instructions and appropriate meal suggestions into voice guidance, which is then provided to the elderly. Multiple sensors are also used to monitor the elderly's movements and health status, and if abnormal behavior is detected, an emergency notification is sent to a pre-designated emergency contact.

[1195] Specific examples

[1196] Menu selection assistance

[1197] For example, consider a situation where an elderly person is wearing smart glasses and browsing a food delivery app. If the eye tracking device detects that the user's gaze is fixed on a particular menu item (e.g., pancakes), a voice guidance such as "This dish is high in calories. We recommend you change it to a salad" is generated and provided to the user.

[1198] Delivery reception assistance

[1199] When the delivery person arrives, sensors installed in the smart glasses measure the location and distance and provide guidance such as "The door is one meter away. Please walk slowly" to avoid the risk of the elderly person falling.

[1200] Prompt Sentence Examples

[1201] "When an elderly person looks at the menu of a food delivery app, generate meal suggestions tailored to their health condition."

[1202] The present invention allows elderly people to use food delivery services with peace of mind, allowing them to select appropriate meals according to their health condition and receive deliveries safely, thereby improving the quality of life of elderly people.

[1203] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1204] Step 1:

[1205] The camera in the smart glasses captures the elderly person's field of vision.

[1206] Input: Elderly person's visual field

[1207] Output: Captured video frames

[1208] The device uses a camera mounted on the smart glasses to capture the elderly person's field of vision in real time, and these video frames are used as input for the subsequent eye tracking process.

[1209] Step 2:

[1210] An eye tracker identifies the location of the elderly person's eyes.

[1211] Input: Captured video frames

[1212] Output: Eye coordinate data

[1213] The device processes the captured video frames using the OpenCV library and identifies the location of the elderly person's eyes using an eye tracking model (TensorFlow and Keras). The obtained eye coordinate data is used to identify the object the user is looking at.

[1214] Step 3:

[1215] The generative model identifies the objects the elderly person sees and generates audio guidance.

[1216] Input: pupil coordinate data, captured video frame

[1217] Output: Voice guidance data

[1218] The server analyzes the pupil coordinate data and the captured video frames to identify the object the user is looking at. It then inputs a prompt sentence into a generative model based on the identified object to generate appropriate voice guidance. The generative model then uses natural language processing technology to generate voice guidance data and transmit it to the device.

[1219] Step 4:

[1220] The device plays audio guidance.

[1221] Input: Voice guidance data

[1222] Output: Voice guidance

[1223] The terminal plays the voice guidance data received from the server and provides voice guidance to the elderly, such as "This dish is high in calories. We recommend changing it to a salad."

[1224] Step 5:

[1225] Sensors monitor the movements and health of the elderly.

[1226] Input: Sensor data (heart rate, blood pressure, movement, etc.)

[1227] Output: Monitoring data

[1228] The device uses multiple sensors to continuously monitor the elderly person's heart rate, blood pressure, and movements, and collects this data, which is used to detect abnormal movements.

[1229] Step 6:

[1230] If abnormal behavior is detected, an emergency notification will be sent.

[1231] Input: Monitoring data

[1232] Output: Urgent Notification

[1233] The server analyzes the monitoring data sent from the device and generates an emergency notification if any abnormal behavior is detected, and sends it to pre-designated emergency contacts (such as family members or medical institutions). This notification may also include information about the elderly person's health and emotional state.

[1234] Step 7:

[1235] When a delivery arrives, the system measures the height and distance to assist with the receiving process.

[1236] Input: Sensor data (distance measurement)

[1237] Output: Voice guidance

[1238] When a delivery arrives, the device uses sensors to measure the height and distance, and provides voice guidance such as "The distance to the door is one meter. Please walk slowly" to ensure that the elderly person can receive the delivery safely.

[1239] This will allow elderly people to use food delivery services with peace of mind, make appropriate meal choices, and receive their deliveries safely.

[1240] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1241] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1242] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1243] [Fourth embodiment]

[1244] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1245] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1246] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1247] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1248] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1249] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1250] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1251] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1252] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1253] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1254] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1255] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1256] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1257] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye tracker, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[1258] Eye tracking and object recognition

[1259] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which then generates voice guidance on how to use the object.

[1260] Voice guidance using generative models

[1261] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[1262] Sensor function monitors abnormal behavior

[1263] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[1264] Abnormal behavior detection and emergency notification

[1265] If abnormal behavior is detected, for example if an elderly person falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[1266] Specific examples

[1267] Example 1: Medication

[1268] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt, such as "Take one pill of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[1269] Example 2: Detecting abnormal behavior

[1270] Take the example of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and sends additional alerts if the user does not respond.

[1271] The present invention combines eye tracking, generative models, and sensor technology to make the daily lives of seniors safer and more comfortable, while reducing anxiety for their families and caregivers.

[1272] The processing flow will be explained below.

[1273] Step 1:

[1274] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[1275] Step 2:

[1276] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[1277] Step 3:

[1278] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[1279] Step 4:

[1280] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[1281] Step 5:

[1282] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[1283] Step 6:

[1284] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[1285] Step 7:

[1286] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[1287] Step 8:

[1288] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[1289] Step 9:

[1290] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[1291] Step 10:

[1292] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[1293] Step 11:

[1294] If the user does not respond in a critical situation, the server will repeatedly send additional emergency alerts to prompt the necessary response.

[1295] Through the above steps, this system comprehensively supports the daily lives of elderly people and provides an environment in which they can live safely and comfortably.

[1296] Example 1

[1297] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1298] To support the safe and comfortable daily lives of the elderly, advanced technologies are needed for gaze tracking, object recognition, health monitoring, and other functions. However, conventional systems have difficulty integrating these functions and have limitations in real-time monitoring and anomaly detection. As a result, they are unable to completely alleviate the anxiety of the elderly and their families, and are also difficult to respond to emergency situations that require a rapid response. This invention aims to solve these problems and make the daily lives of the elderly safer and more comfortable.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1300] In this invention, the server includes a means for identifying an object the user is looking at using an eye-tracking device, a means for generating audio guidance on how to use the identified object using a generative model, and a means for monitoring the user's motion and health status using multiple sensors and detecting abnormal motion. This enables integrated gaze tracking, object recognition, and real-time monitoring. Furthermore, if an abnormality is detected, an emergency notification can be sent immediately, enabling a rapid response and improving the safety and quality of life of the elderly.

[1301] An "eye tracking device" is a device that identifies the position and movement of a user's eyes and tracks the direction of their gaze in real time.

[1302] A "generative model" is a machine learning model that analyzes received data and generates voice guidance.

[1303] A "photography device" is a camera or image capture device that captures an image of an object viewed by a user.

[1304] A "voice synthesizer" is a device that reproduces voice guidance created by a generative model as actual voice.

[1305] A "sensor" is a device used to measure and monitor a user's activity or health. Examples include motion sensors, heart rate sensors, and blood pressure sensors.

[1306] A "profile" is a collection of data that includes personal information and settings about a user.

[1307] "Initialization" is a preparatory procedure for starting the operation of a system or device.

[1308] An "emergency notification" is a warning message sent to a pre-designated emergency contact when abnormal operation is detected.

[1309] This invention is a system for making the daily lives of elderly people safe and comfortable. This system uses an eye-tracking device, a generative model, and multiple sensors to comprehensively support the daily lives of elderly people.

[1310] Eye tracking and object recognition

[1311] When a user puts on the smart glasses, the device automatically initializes itself. The smart glasses are equipped with an eye-tracking device and a camera. To identify the object the user is looking at in real time, the eye-tracking device measures the position and movement of the eyes, and the camera captures the image of the object being viewed. For example, if the user is looking at a medicine bottle, the device captures that information and sends it to a server.

[1312] Voice guidance using generative models

[1313] The server hosts a generative model (e.g., GPT-4) that analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, it generates guidance such as, "Take one tablet of this medicine after breakfast." The generated voice guidance is then sent to the device and notified to the user via the device's voice synthesizer.

[1314] Sensor function monitors abnormal behavior

[1315] The device is equipped with multiple sensors that continuously monitor the elderly person's activity and health status, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's activity and health status in real time, and if any abnormal activity or situation is detected, the data is immediately sent to the server.

[1316] Abnormal behavior detection and emergency notification

[1317] If an abnormal movement is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling a prompt response even when the elderly are alone.

[1318] Specific examples

[1319] Example 1: Medication

[1320] Suppose a user is wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye-tracking device and a camera, and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then notifies the user with the voice prompt.

[1321] Example 2: Detecting abnormal behavior

[1322] Consider the case of a user at home. IoT sensors on the device monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and also sends additional alerts if the user does not respond.

[1323] Example prompt sentence:

[1324] "Please explain how eye tracking and generative models can be used to support elderly people in taking medication."

[1325] "Please explain how emergency notifications are sent if an elderly person falls, including how the specific sensors work."

[1326] In this way, the combination of eye-tracking technology, generative models, and sensor technology can make the daily lives of older people safer and more comfortable, and reduce anxiety for their families and caregivers.

[1327] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1328] Step 1: Initialize the device

[1329] The smart glasses as a terminal perform initialization. This includes checking the operation of the eye tracking device, the camera, and various sensors. It loads the user's profile and reads saved health data and settings. Specifically, the terminal recognizes the user's face and checks the status of each sensor. The input data includes the user's profile information and execution confirmation data from the sensors, and the output data is the initialization completion status.

[1330] Step 2: Eye tracking and object recognition

[1331] To monitor the user's daily activities, the device keeps its eye-tracking device and camera enabled at all times. When the user gazes at a specific object, the device identifies the direction of the user's gaze and captures a photo of that object. Specifically, the eye-tracking device captures the movement of the user's pupils, and the camera captures a video of the recognized object. The input data is the user's gaze information and the captured video data, and the output data is information about the recognized object, which is sent to the server.

[1332] Step 3: Send data

[1333] The device sends data of recognized objects to the server, including captured video data and gaze tracking data. Specifically, the device collects data and sends it to the server using a secure communication protocol. The input data is the captured video and gaze data, and the output data is the status of completion of transmission to the server.

[1334] Step 4: Analysis and Guidance Generation

[1335] The server inputs the received data into a generative model to generate appropriate guidance. The generative model uses natural language processing technology to create voice guidance that provides appropriate instructions to the user. For example, if it recognizes a medicine bottle, it generates instructions such as "Take one tablet of this medicine after breakfast." The input data is gaze and video data, and the output data is the generated voice guidance.

[1336] Step 5: Provide audio guidance

[1337] The terminal that receives the voice guidance generated by the server uses a voice synthesizer to convey the content to the user. The input data is the generated voice guidance, and the output data is the voice instructions that the user hears. Specifically, the terminal receives the voice guidance data, and the voice synthesizer plays it back.

[1338] Step 6: Detecting Abnormal Behavior

[1339] The device continuously monitors the user's movement and health status using motion sensors, heart rate sensors, and blood pressure sensors. If an abnormality is detected, the data is immediately sent to the server. The input data is the movement and health data collected by the sensors, and the output data is an alert of the abnormality. Specifically, if the user falls, the acceleration sensor and other sensors detect the abnormality and send the information to the server.

[1340] Step 7: Emergency Notification

[1341] The server receives data on detected abnormal behavior and sends a notification to emergency contacts. The notification includes location information and real-time health data. The input data is the abnormality detection notification data, and the output data is the notification content sent to emergency contacts. Specifically, the server analyzes the abnormal data and sends a message to family members or medical institutions such as, "The user has fallen. The location is ____."

[1342] (Application example 1)

[1343] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1344] In modern society, safety and health management, especially for the elderly, are important issues. In the course of elderly people's daily lives, prompt responses are required to abnormal behavior, such as medication errors or falls. However, current systems have limitations in identifying objects visually recognized by elderly people and in responding quickly to emergencies. Furthermore, generating voice guidance requires manual input, which is cumbersome for elderly people. To address these issues, the present invention aims to automate the identification of objects visually recognized by elderly people and the detection of abnormal behavior, and provide appropriate voice guidance and emergency notifications in real time.

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1346] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's behavior using multiple sensors and detecting abnormal behavior, means for sending an emergency notification to a pre-specified emergency contact when abnormal behavior is detected, and means for converting the captured object into audio guidance based on natural language processing. This makes it possible to improve the safety and comfort of elderly people's daily lives and reduce the anxiety of their families and caregivers.

[1347] 1. "Pupil tracking device" means a device that detects the position and movement of the user's pupils.

[1348] 2. "User" refers to a person who uses this system.

[1349] 3. "Object Identification" means the act of using an eye tracker and camera to identify an object that is viewed by the user.

[1350] 4. A "generative model" is a program or system that uses AI or machine learning algorithms to generate useful output from input data.

[1351] 5. "Voice guidance" is a system that uses voice synthesis technology to provide specific information as voice.

[1352] 6. "Multiple sensors" refers to multiple sensor devices used to measure various data, such as a motion sensor, a heart rate sensor, and a blood pressure sensor.

[1353] 7. "Movement monitoring" refers to the act of continuously observing a user's physical movements and condition using sensors.

[1354] 8. "Abnormal behavior" refers to behavior that deviates from normal or healthy behavior, such as falls by elderly people.

[1355] 9. "Emergency Notification" means a notification sent immediately to an emergency contact when abnormal activity is detected.

[1356] 10. "Emergency Contact" means a contact (such as a family member or medical institution) designated to receive notification in the event of an emergency.

[1357] 11. "Camera" means a photographic device for capturing images.

[1358] 12. "Capture" means the act of collecting and recording image or video data.

[1359] 13. "Natural language processing" is a general term for technologies that allow computers to understand and generate human language.

[1360] The present invention is a system for making the daily lives of elderly people safe and comfortable. This system is realized by integrating various hardware and software components, such as a pupil tracking device, a generative model, and multiple sensors. The specific configuration and operating procedures for implementing the present invention are described in detail below.

[1361] Eye tracking and object recognition

[1362] The smart glasses used as a terminal are equipped with an eye tracker and a camera. This eye tracker allows the user (elderly person) to identify the object they are looking at in real time. The camera captures image data of the object being viewed and sends it to a server. On the server side, the image data is analyzed using a generative model, and audio guidance on how to use the identified object is generated.

[1363] Voice guidance using generative models

[1364] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This generative model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the generative model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the user.

[1365] Sensor function monitors abnormal behavior

[1366] The device is equipped with multiple sensors that continuously monitor the user's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the user's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[1367] Abnormal behavior detection and emergency notification

[1368] If abnormal behavior is detected, for example if the user falls, the server receives the information and sends an emergency notification to pre-designated emergency contacts (family members or medical institutions). This notification includes location information and real-time health data (heart rate, blood pressure, etc.), enabling prompt response even when the elderly person is alone.

[1369] Hardware and software used

[1370] The system uses the following hardware and software:

[1371] Pupil tracker: A device that detects the position and movement of the pupil.

[1372] Camera: A device that captures image data of objects in view.

[1373] Movement, heart rate, and blood pressure sensors: Sensors for monitoring the movement and health of seniors.

[1374] Server: Hosts the generative model, analyzes data, and generates audio guidance.

[1375] Generative model: A program or system that uses AI or machine learning algorithms to generate useful output from input data.

[1376] Natural Language Processing (NLP) technology: Technology that enables computers to understand and generate human language.

[1377] Specific examples

[1378] Example 1: Medication

[1379] For example, consider a user wearing smart glasses. When the user looks at a pill bottle, the device captures that information using an eye tracker and a camera and sends it to the server. The server uses a generative model to generate a voice prompt saying, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays that prompt to the user.

[1380] Example 2: Detecting abnormal behavior

[1381] Consider the case of a user at home. The device's built-in sensors monitor the user's movements. If the user falls, the device detects the abnormal movement and sends the data to a server. The server then sends an emergency notification to family members or medical institutions, and if the user does not respond, it sends an additional alert.

[1382] Prompt Sentence Examples

[1383] "The identified object is a 'medicine bottle'. Please generate the following audio prompt: 'Take one tablet of this medicine after breakfast.'"

[1384] As described above, the present invention is a system that combines a pupil tracking device, a generative model, and multiple sensors to make the daily lives of elderly people safe and comfortable.

[1385] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1386] Step 1:

[1387] The camera in the smart glasses worn by the device captures the image of the user's field of vision in real time. The input is the image data of the user's field of vision, and the output is the image data. The camera provides high-resolution images, which are sent to a processor for use in the next step.

[1388] Step 2:

[1389] The device's built-in eye tracking device tracks the position of the user's eyes and overlays them on the camera image to identify the object the user is looking at. The input is the user's eye data (pupil position and movement), and the output is the coordinate data of the object the user is looking at. The position of the identified object is clarified based on the eye position information.

[1390] Step 3:

[1391] The device sends image data of the identified object to the server. The input is the image data of the identified object, and the output is the data transmission to the server. The data is transmitted at high speed and is optimized to minimize latency.

[1392] Step 4:

[1393] The server inputs the received image data into a generative model to recognize the object. The input is the image data of the identified object, and the output is the object name. The generative model uses a pre-trained AI algorithm to classify the object and obtain a result. For example, the image data is recognized as a medicine bottle.

[1394] Step 5:

[1395] The server converts the recognized object's usage instructions into voice guidance using natural language processing technology. The input is the object name and data on its usage, and the output is text data for the voice guidance. The generative model prepares the voice guidance, saying, "Take one tablet of this medicine after breakfast."

[1396] Step 6:

[1397] The server sends the text data of the voice guidance to the terminal. The input is the text data of the voice guidance, and the output is the data transmission to the terminal. The data is sent in real time, providing quick guidance to the elderly.

[1398] Step 7:

[1399] The device uses voice synthesis technology to play back the voice guidance. The input is text data received from the server, and the output is voice guidance. In the case of smart glasses, the voice is transmitted to the elderly using a built-in speaker.

[1400] Step 8:

[1401] Multiple sensors built into the device monitor the user's movements and health. The input is real-time data from the sensors (e.g., heart rate, movements), and the output is the monitoring results. The sensor data is continuously collected and checked for any abnormalities.

[1402] Step 9:

[1403] When a device detects abnormal behavior (e.g., a fall), the data is immediately sent to the server. The input is sensor data of the abnormal behavior, and the output is emergency data sent to the server. The data is sent at high speed, allowing for a rapid response to emergencies.

[1404] Step 10:

[1405] The server receives data on abnormal behavior and sends an emergency notification to pre-designated emergency contacts. The input is data on abnormal behavior, and the output is a notification to the emergency contact (including location information and health data). An email or SMS is sent to the emergency contact, enabling a prompt response.

[1406] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1407] This invention is a system for making the daily lives of the elderly safe and comfortable. This system combines an eye tracker, a generative model, multiple sensors, and an emotion engine to comprehensively support the daily lives of the elderly.

[1408] Eye tracking and object recognition

[1409] The smart glasses used as the device are equipped with an eye tracker and a camera, which allows them to identify the object the elderly person is looking at in real time. For example, if the elderly person is looking at a medicine bottle, the device captures that information and sends it to the server. This data is analyzed by a server-side generative model, which generates voice guidance on how to use the object.

[1410] Voice guidance using generative models

[1411] A generative model is hosted on the server and analyzes how to use the object based on the captured data. This model uses natural language processing technology to convert the analysis results into voice guidance. For example, if a medicine bottle is identified, the model generates guidance such as "Take one tablet of this medicine after breakfast." The voice guidance is then sent to the device, where the instructions are played aloud to the elderly person.

[1412] Sensor function monitors abnormal behavior

[1413] The device is equipped with multiple sensors that continuously monitor the elderly's movements, including a motion sensor, heart rate sensor, and blood pressure sensor. These sensors monitor the elderly's movements and health condition in real time, and if any abnormal movements or conditions are detected, the data is immediately sent to the server.

[1414] Emotion recognition by emotion engine

[1415] The device uses a microphone and camera to analyze the elderly person's tone of voice and facial expressions in real time, thereby recognizing their emotional state (e.g., anxiety, sadness, joy). This emotional data is sent to a server, and a generative model generates appropriate voice guidance based on the elderly person's emotional state.

[1416] Abnormal behavior detection and emergency notification

[1417] If abnormal behavior is detected, for example, if an elderly person falls, the server receives the information and uses the emotion engine data to create a more appropriate emergency notification. The emotion engine data, along with the abnormal behavior information, is included in the emergency notification and sent to pre-designated emergency contacts (family members or medical institutions).

[1418] Specific examples

[1419] Example 1: Medication

[1420] Take the example of a user wearing smart glasses. When the user looks at a pill bottle, the device uses an eye tracker and camera to capture that information and sends it to the server. The server uses a generative model to generate a voice prompt such as, "Take one tablet of this medicine after breakfast," and sends it to the device. The device then relays this voice prompt to the user. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice, and if the user feels anxious, it provides additional voice prompts such as, "Are you worried?"

[1421] Example 2: Detecting abnormal behavior

[1422] Take the example of a user at home. An IoT sensor installed on the device monitors the user's movements. If the user falls, the device detects abnormal movements and sends the data to the server. The server receives the data, and if the emotion engine analyzes the data and determines that the user is feeling fear or anxiety, it adds this information to the emergency notification. To ensure that the situation is understood, the emergency notification is sent to pre-registered family members and medical institutions.

[1423] The present invention combines eye tracking technology, generative models, sensor technology, and an emotion engine to support the daily lives of elderly people while taking into account their emotional state, providing a safe and comfortable environment.

[1424] The processing flow will be explained below.

[1425] Step 1:

[1426] The user puts on the smart glasses and starts up the device. The device detects the user's face and performs facial recognition. This facial recognition data is sent to the server.

[1427] Step 2:

[1428] The server compares the received facial recognition data with an existing database to authenticate the user, and if authentication is successful, sends the result back to the device.

[1429] Step 3:

[1430] The device will activate the eye tracker and calibrate the user's eye movements in real time, and once calibration is complete, it will begin tracking the eye movements.

[1431] Step 4:

[1432] When a user looks at an object, such as a medicine bottle or tool, the device's built-in camera captures an image of the object, and this image data and eye tracking data are sent to a server.

[1433] Step 5:

[1434] The server analyzes the received video data and eye tracking data to identify the object the user is looking at. Information about the identified object is input into a generative model, and text data on how to use it is generated.

[1435] Step 6:

[1436] The server converts the generated text data into audio guidance and creates an audio file, which is then sent to the terminal.

[1437] Step 7:

[1438] The device plays the received audio file and provides the user with voice guidance such as, "Take one tablet of this medicine after breakfast."

[1439] Step 8:

[1440] Sensors on the device continuously monitor the user's movements, including motion sensors, heart rate sensors, and blood pressure sensors.

[1441] Step 9:

[1442] If the device detects any abnormal behavior, such as a fall or a sudden change in heart rate, it immediately sends the data to the server.

[1443] Step 10:

[1444] The server receives abnormal behavior data and automatically sends emergency notifications to pre-designated emergency contacts (family members or medical institutions), including location information and real-time health data.

[1445] Step 11:

[1446] The emotion engine analyzes the user's tone of voice and facial expressions in real time to recognize the user's emotional state (e.g., anxiety, sadness, joy).

[1447] Step 12:

[1448] The server adjusts the content of the voice guidance based on the emotion engine data. For example, if the user is feeling anxious, it generates a message that provides additional reassurance.

[1449] Step 13:

[1450] The server combines abnormal behavior data with data from the emotion engine to dynamically adjust the content of emergency notifications and inform relevant parties of the situation.

[1451] Step 14:

[1452] The device periodically sends the emotion engine's analysis results to the server, and the generative model provides instructions optimized for the user's situation.

[1453] Through the above steps, the present invention comprehensively supports the daily life of the elderly and provides a safe and comfortable living environment taking into account their emotional state.

[1454] Example 2

[1455] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1456] There is a growing need for a system that can comprehensively monitor and support the visual, behavioral, and emotional states of the elderly in their daily lives. Conventional systems have the technology to monitor individual states (visual, behavioral, and emotional), but it is difficult to comprehensively monitor and immediately respond appropriately. The purpose of this invention is to solve these problems and provide the elderly with a safe and comfortable daily life.

[1457] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1458] In this invention, the server

[1459] means for identifying an object viewed by a user using a pupil tracking device;

[1460] means for generating audio guidance on how to use the object identified using the generative model;

[1461] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[1462] a means for analyzing the user's emotional state using an emotion engine and providing an appropriate response;

[1463] This will enable comprehensive monitoring of the daily lives of elderly people and provide timely and appropriate support.

[1464] A "pupil tracking device" is a device that detects the movement of a user's pupils in real time and acquires gaze data.

[1465] A "generative model" is an artificial intelligence technique that analyzes input data and produces output in a specified format.

[1466] "Voice guidance" is a message that conveys specific information to the user by voice.

[1467] A "sensor" is a device that detects physical phenomena (movement, heart rate, blood pressure, etc.) in real time and acquires them as data.

[1468] "Abnormal movement" refers to a movement or state that deviates from the user's normal movement (e.g., falling).

[1469] An "emergency notification" is an emergency message sent to pre-designated contacts when abnormal operation is detected.

[1470] The "emotion engine" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[1471] "User" refers to a person who uses this system.

[1472] MODE FOR CARRYING OUT THE INVENTION

[1473] The present invention is a comprehensive support system for the daily lives of elderly people, which aims to make users' daily lives safer and more comfortable by combining an eye tracker, a generative model, multiple sensors, and an emotion engine. The main hardware and software components for implementing the present invention are described below.

[1474] Hardware Configuration

[1475] 1. Device (Smart Glasses):

[1476] Eye tracking device: A device that captures the user's gaze in real time and acquires that data.

[1477] Camera: A device that captures an object corresponding to the user's line of sight and generates image data.

[1478] Sensors: Equipped with motion sensors, heart rate sensors, blood pressure sensors, etc., it monitors the user's movements and health status in real time.

[1479] Microphone: A device that captures the tone of a user's voice.

[1480] Speaker: A device that conveys the generated audio guidance to the user.

[1481] 2. Server:

[1482] Generative model: A model that analyzes gaze data and image data and generates voice guidance using natural language processing technology.

[1483] Emotion engine: Technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[1484] Software Configuration

[1485] 1. Data analysis software: Installed on the server, it analyzes the data sent from the pupil tracker and camera.

[1486] 2. Speech synthesis software: Converts the text data output by the generative model into speech data.

[1487] 3. Communication software: Software for sending and receiving data between the terminal and the server.

[1488] Specific system processing examples

[1489] 1. Medication Use:

[1490] The user puts on the smart glasses and looks at the pill bottle.

[1491] The device's eye tracking device and camera capture this information, obtaining gaze data and image data.

[1492] The terminal transmits the data to the server.

[1493] The server's generative model generates voice guidance such as "Take one tablet of this medicine after breakfast."

[1494] The server transmits the guidance to the terminal, and the terminal's speaker delivers the voice guidance to the user.

[1495] The emotion engine analyzes the user's facial expressions and tone of voice, and if they feel anxious, it provides additional voice guidance such as, "Are you worried about something?"

[1496] 2. Abnormal behavior detection:

[1497] The user falls while at home.

[1498] A motion sensor installed in the device detects falls.

[1499] The terminal transmits the data to the server.

[1500] The server determines that the user is feeling fear based on the abnormal behavior data and the analysis results of the emotion engine, and creates an emergency notification.

[1501] Emergency notifications will be sent to pre-designated family members and medical institutions.

[1502] Prompt Sentence Examples

[1503] 1. Example prompts for medication use:

[1504] The user is looking at a bottle of medicine. The information is captured by the eye tracker and camera, and analyzed by the server to generate a voice prompt such as, "Take one tablet of this medicine after breakfast." If the user is feeling anxious, the system provides additional guidance such as, "Are you worried about something?"

[1505] 2. Example prompt for detecting abnormal behavior:

[1506] If a user falls while at home, the motion sensor detects the abnormality and sends this information to a server, which then sends an emergency notification to family members or medical institutions, including anxiety data analyzed by an emotion engine.

[1507] As described above, the present invention combines an eye tracking device, a generative model, sensor technology, and an emotion engine to comprehensively support the daily lives of the elderly and provide a safe and comfortable environment.

[1508] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1509] System program processing steps

[1510] Specific processing steps for medication use

[1511] Step 1:

[1512] A user wears the smart glasses and looks at a medicine bottle, while gaze data is captured in real time by a pupil tracker.

[1513] Input: User's gaze

[1514] Output: Gaze data

[1515] Step 2:

[1516] The device's camera captures an image of the object (medicine bottle) that corresponds to the user's line of sight.

[1517] Input: Gaze data

[1518] Output: Image data

[1519] Step 3:

[1520] The device transmits the captured gaze data and image data to a server via Wi-Fi or mobile networks.

[1521] Input: Gaze data, image data

[1522] Output: Send data to the server

[1523] Step 4:

[1524] The server analyzes the received gaze data and image data, and a generative model identifies the medicine bottle based on the specific image data and gaze data.

[1525] Input: Gaze data, image data

[1526] Output: Identified object (medicine bottle)

[1527] Step 5:

[1528] The server's generative model analyzes how to use the identified object and uses natural language processing technology to generate voice guidance such as, "Take one tablet of this medicine after breakfast."

[1529] Input: Identified object

[1530] Output: Voice guidance (text format)

[1531] Step 6:

[1532] The server transmits the generated voice guidance to the terminal.

[1533] Input: Voice guidance (text format)

[1534] Output: Send voice guidance to the device

[1535] Step 7:

[1536] The device converts the received voice guidance into audio data and plays it back through the speaker, providing the user with audio instructions on how to use the device.

[1537] Input: Voice guidance (text format)

[1538] Output: Voice guidance (voice data)

[1539] Step 8:

[1540] The device's emotion engine analyzes the user's facial expressions and tone of voice in real time, and if they are feeling anxious, it generates additional voice guidance such as, "Are you worried about something?"

[1541] Input: User's facial expression data, tone of voice

[1542] Output: Additional voice guidance

[1543] Step 9:

[1544] Additional audio guidance generated by the terminal is played back to give the user a sense of security.

[1545] Input: Additional voice guidance

[1546] Output: Additional voice guidance playback

[1547] Specific processing steps for detecting abnormal behavior

[1548] Step 1:

[1549] The user falls while at home.

[1550] Input: User action

[1551] Step 2:

[1552] The motion sensor installed in the device detects falls, for example, detecting abnormal movements from sudden changes in gravitational acceleration.

[1553] Input: Operation data

[1554] Output: Abnormal operation data

[1555] Step 3:

[1556] The terminal transmits the abnormal operation data to the server.

[1557] Input: Abnormal behavior data

[1558] Output: Send abnormal operation data to the server

[1559] Step 4:

[1560] The server analyzes the received abnormal operation data and confirms the abnormal operation.

[1561] Input: Abnormal behavior data

[1562] Output: Analysis results (confirmation of abnormal behavior)

[1563] Step 5:

[1564] The server uses an emotion engine to analyze whether the user feels fear or anxiety when they fall.

[1565] Input: User's facial expression data, tone of voice

[1566] Output: Emotion data (fear, anxiety)

[1567] Step 6:

[1568] The server generates an emergency notification based on the abnormal motion data and emotion data, such as "The user has fallen and is currently experiencing fear."

[1569] Input: Abnormal behavior data, emotion data

[1570] Output: Urgent Notification

[1571] Step 7:

[1572] The server then sends the emergency notification to pre-designated family members or medical institutions.

[1573] Input: Emergency Notification

[1574] Output: Send emergency notification

[1575] As mentioned above, each processing step is explained in detail with specific operations and data flows. The overall system effectively combines eye tracking technology, generative models, sensor technology, and an emotion engine to help the elderly live a safe and comfortable life.

[1576] (Application example 2)

[1577] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1578] In recent years, technology to support the lives of the elderly has been attracting attention. However, when using food delivery services, many difficulties arise, particularly when selecting menus and receiving food. For this reason, there is a need for a system that allows elderly people to use food delivery services safely and comfortably. Furthermore, the current situation is that meal suggestions tailored to health conditions and support when receiving deliveries are not being provided sufficiently. To solve these issues, a comprehensive system is needed that allows elderly people to use food delivery services with peace of mind.

[1579] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1580] In this invention, the server includes means for identifying an object viewed by a user using a pupil tracking device, means for generating audio guidance on how to use the identified object using a generative model, means for monitoring the user's movements using multiple sensors and detecting abnormal movements, means for sending an emergency notification to a pre-designated emergency contact when abnormal movements are detected, means for supporting the elderly with menu selection and meal suggestions based on their health condition when using a food delivery service using audio guidance, and means for measuring the height and distance when a delivery arrives and supporting the receiving operation. This allows the elderly to use food delivery services with peace of mind, select appropriate meals based on their health condition, and receive their delivery safely.

[1581] An "eye tracker" is a device that tracks a user's gaze and identifies the object they are looking at.

[1582] "Generative model" refers to an algorithm that generates guidance in natural language based on input data.

[1583] A "sensor" is a device that detects physical movements and environmental conditions and outputs them as digital data.

[1584] "Abnormal movement" refers to movement that differs from the user's normal behavioral pattern, and includes, for example, falling.

[1585] An "emergency notification" is an alert sent to pre-registered contacts when an unexpected situation occurs.

[1586] "Food delivery service" refers to a service that delivers food or meals to a location specified by the customer.

[1587] "Menu selection" refers to the act of selecting what to order from the dishes and foods offered by a food delivery service.

[1588] "Meal suggestions tailored to your health condition" refers to recommending appropriate meals taking into consideration the user's health condition and preferences.

[1589] "Delivery arrival" refers to the moment when the ordered food or meal is delivered to the location specified by the user.

[1590] "Measuring height and distance" means measuring the surrounding conditions to ensure that elderly people can receive deliveries safely.

[1591] This invention is a system that enables elderly people to use food delivery services safely and comfortably, and is realized by combining an eye tracking device, a generative model, sensors, and an emotion engine. The system program is implemented using the following hardware and software.

[1592] Hardware

[1593] Smart glasses: Equipped with a camera, eye tracker, microphone, and speaker.

[1594] Server: Used to analyze data and generate audio guidance.

[1595] software

[1596] OpenCV: A library for image processing.

[1597] TensorFlow & Keras: Implementation and operation of pupil tracking model.

[1598] Transformers: A library for sentiment analysis.

[1599] HTTP Requests: A library for communicating with the server.

[1600] Data processing and calculation

[1601] The server first processes the video frames captured by the smart glasses using OpenCV and identifies the eye position using an eye tracking model. This allows it to identify the menu or object the elderly person is looking at. The generative model then uses natural language processing to convert the identified object's usage instructions and appropriate meal suggestions into voice guidance, which is then provided to the elderly. Multiple sensors are also used to monitor the elderly's movements and health status, and if abnormal behavior is detected, an emergency notification is sent to a pre-designated emergency contact.

[1602] Specific examples

[1603] Menu selection assistance

[1604] For example, consider a situation where an elderly person is wearing smart glasses and browsing a food delivery app. If the eye tracking device detects that the user's gaze is fixed on a particular menu item (e.g., pancakes), a voice guidance such as "This dish is high in calories. We recommend you change it to a salad" is generated and provided to the user.

[1605] Delivery reception assistance

[1606] When the delivery person arrives, sensors installed in the smart glasses measure the location and distance and provide guidance such as "The door is one meter away. Please walk slowly" to avoid the risk of the elderly person falling.

[1607] Prompt Sentence Examples

[1608] "When an elderly person looks at the menu of a food delivery app, generate meal suggestions tailored to their health condition."

[1609] The present invention allows elderly people to use food delivery services with peace of mind, allowing them to select appropriate meals according to their health condition and receive deliveries safely, thereby improving the quality of life of elderly people.

[1610] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1611] Step 1:

[1612] The camera in the smart glasses captures the elderly person's field of vision.

[1613] Input: Elderly person's visual field

[1614] Output: Captured video frames

[1615] The device uses a camera mounted on the smart glasses to capture the elderly person's field of vision in real time, and these video frames are used as input for the subsequent eye tracking process.

[1616] Step 2:

[1617] An eye tracker identifies the location of the elderly person's eyes.

[1618] Input: Captured video frames

[1619] Output: Eye coordinate data

[1620] The device processes the captured video frames using the OpenCV library and identifies the location of the elderly person's eyes using an eye tracking model (TensorFlow and Keras). The obtained eye coordinate data is used to identify the object the user is looking at.

[1621] Step 3:

[1622] The generative model identifies the objects the elderly person sees and generates audio guidance.

[1623] Input: pupil coordinate data, captured video frame

[1624] Output: Voice guidance data

[1625] The server analyzes the pupil coordinate data and the captured video frames to identify the object the user is looking at. It then inputs a prompt sentence into a generative model based on the identified object to generate appropriate voice guidance. The generative model then uses natural language processing technology to generate voice guidance data and transmit it to the device.

[1626] Step 4:

[1627] The device plays audio guidance.

[1628] Input: Voice guidance data

[1629] Output: Voice guidance

[1630] The terminal plays the voice guidance data received from the server and provides voice guidance to the elderly, such as "This dish is high in calories. We recommend changing it to a salad."

[1631] Step 5:

[1632] Sensors monitor the movements and health of the elderly.

[1633] Input: Sensor data (heart rate, blood pressure, movement, etc.)

[1634] Output: Monitoring data

[1635] The device uses multiple sensors to continuously monitor the elderly person's heart rate, blood pressure, and movements, and collects this data, which is used to detect abnormal movements.

[1636] Step 6:

[1637] If abnormal behavior is detected, an emergency notification will be sent.

[1638] Input: Monitoring data

[1639] Output: Urgent Notification

[1640] The server analyzes the monitoring data sent from the device and generates an emergency notification if any abnormal behavior is detected, and sends it to pre-designated emergency contacts (such as family members or medical institutions). This notification may also include information about the elderly person's health and emotional state.

[1641] Step 7:

[1642] When a delivery arrives, the system measures the height and distance to assist with the receiving process.

[1643] Input: Sensor data (distance measurement)

[1644] Output: Voice guidance

[1645] When a delivery arrives, the device uses sensors to measure the height and distance, and provides voice guidance such as "The distance to the door is one meter. Please walk slowly" to ensure that the elderly person can receive the delivery safely.

[1646] This will allow elderly people to use food delivery services with peace of mind, make appropriate meal choices, and receive their deliveries safely.

[1647] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1648] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1649] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1650] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1651] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1652] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1653] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1654] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1655] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1656] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1657] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1658] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1659] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1660] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1661] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1662] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1663] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1664] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1665] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1666] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1667] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1668] The following is further disclosed regarding the above embodiment.

[1669] (Claim 1)

[1670] means for identifying an object viewed by a user using a pupil tracking device;

[1671] means for generating audio guidance on how to use the object identified using the generative model;

[1672] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[1673] means for sending an emergency notification to a pre-designated emergency contact when abnormal operation is detected;

[1674] A system including:

[1675] (Claim 2)

[1676] 10. The system of claim 1, further comprising means for combining a pupil tracker and a camera to capture an object viewed by a user.

[1677] (Claim 3)

[1678] 10. The system of claim 1, wherein the generative model includes means for converting the usage of the identified object using natural language processing when generating the audio guidance.

[1679] "Example 1"

[1680] (Claim 1)

[1681] A means for identifying an object being viewed by a user using an eye tracking device;

[1682] A means for generating audio guidance on how to use the object identified using the generative model;

[1683] A means for monitoring the user's motion and health status using a plurality of sensors and detecting abnormal motion;

[1684] means for sending an emergency notification to a pre-designated emergency contact when abnormal operation is detected;

[1685] means for loading a user profile and initializing the device;

[1686] a means for notifying a user of the guidance analyzed by the generative model through a speech synthesizer;

[1687] A system including:

[1688] (Claim 2)

[1689] 10. The system of claim 1, further comprising means for combining an eye-tracking device and a camera to capture what the user is looking at.

[1690] (Claim 3)

[1691] 10. The system of claim 1, wherein the generative model includes means for converting the usage of the identified object using natural language processing when generating the audio guidance.

[1692] "Application Example 1"

[1693] (Claim 1)

[1694] means for identifying an object viewed by a user using a pupil tracking device;

[1695] means for generating audio guidance on how to use the object identified using the generative model;

[1696] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[1697] means for sending an emergency notification to a pre-designated emergency contact when abnormal operation is detected;

[1698] means for converting the captured object into audio guidance based on natural language processing;

[1699] A system including:

[1700] (Claim 2)

[1701] 10. The system of claim 1, further comprising means for combining a pupil tracker and a camera to capture an object viewed by a user.

[1702] (Claim 3)

[1703] 10. The system of claim 1, wherein the generative model includes means for converting the usage of the identified object using natural language processing when generating the audio guidance.

[1704] "Example 2: Combining Emotion Engines"

[1705] (Claim 1)

[1706] means for identifying an object viewed by a user using a pupil tracking device;

[1707] means for generating audio guidance on how to use the object identified using the generative model;

[1708] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[1709] means for sending an emergency notification to a pre-designated emergency contact when abnormal operation is detected;

[1710] a means for analyzing the user's emotional state using an emotion engine and providing an appropriate response;

[1711] A system including:

[1712] (Claim 2)

[1713] 10. The system of claim 1, further comprising means for combining a pupil tracker and a camera to capture an object viewed by a user.

[1714] (Claim 3)

[1715] 10. The system of claim 1, wherein the generative model includes means for converting the usage of the identified object using natural language processing when generating the audio guidance.

[1716] "Application example 2 when combining emotion engines"

[1717] (Claim 1)

[1718] means for identifying an object viewed by a user using a pupil tracking device;

[1719] means for generating audio guidance on how to use the object identified using the generative model;

[1720] a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior;

[1721] means for sending an emergency notification to a pre-designated emergency contact when abnormal operation is detected;

[1722] When elderly people use food delivery services, audio guidance is provided to help them select menus and suggest meals that suit their health condition.

[1723] A means to measure the height and distance when the delivery arrives and support the receiving operation,

[1724] A system including:

[1725] (Claim 2)

[1726] 10. The system of claim 1, further comprising means for combining a pupil tracker and a camera to capture an object viewed by a user.

[1727] (Claim 3)

[1728] 10. The system of claim 1, wherein the generative model includes means for converting the usage of the identified object using natural language processing when generating the audio guidance. [Explanation of symbols]

[1729] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for identifying an object viewed by a user using a pupil tracking device; means for generating audio guidance on how to use the object identified using the generative model; a means for monitoring user behavior using a plurality of sensors and detecting abnormal behavior; means for sending an emergency notification to a pre-designated emergency contact when abnormal operation is detected; A system including:

2. 10. The system of claim 1, further comprising means for combining a pupil tracker and a camera to capture an object viewed by a user.

3. The system of claim 1 , wherein the generative model includes means for converting the usage of the identified object using natural language processing when generating the audio guidance.

Images