system

Abstract

Provide a system. [Solution] sensor means for collecting eye data of a user; a transmitting means for transmitting the collected data to a server; analysis means for analyzing the received data; a notification means for notifying a user of an appropriate action based on the analysis result; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the widespread use of digital devices has led to an increase in eye strain and dry eyes due to prolonged use. In particular, advances in technologies such as augmented reality (AR) and virtual reality (VR) have made eye strain more common. Under these circumstances, there is a growing need for methods to effectively monitor and maintain eye health. Current methods require users to consciously care for their eyes, making it difficult to take appropriate rest times and prevent eye strain. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. The system includes a sensor means for collecting data on a user's eyes, a transmission means for transmitting the collected data to a server, an analysis means for analyzing the received data, and a notification means for informing the user of appropriate actions based on the analysis results. In particular, the system measures the number of blinks of the user's eyes and determines eye fatigue or dryness based on the data, thereby providing a notification in real time encouraging the user to take a break. This allows the user to autonomously take care of their eyes and effectively prevent eye strain. Furthermore, since the server receives and analyzes data in real time, rapid feedback is possible.

[0006] The "sensor means" is a device or mechanism for detecting the user's eye movements, blinks, line of sight, and other information.

[0007] The "transmission means" is a device or mechanism having a communication function for transmitting collected data to an external device such as a server.

[0008] "Analysis Means" means the algorithms and associated computer systems used to process the received data and evaluate and analyze the condition of the user's eyes.

[0009] The "notification means" is a device or mechanism for notifying the user of an action based on the analysis results, and may use methods such as sound, visual display, or vibration.

[0010] A "user" is an individual who wears the smart glasses and receives notifications and suggestions from the system.

[0011] "Collecting" refers to the act of acquiring data on the user's eyes using a sensor means.

[0012] "Sending" refers to the act of transferring collected data to a server over a network.

[0013] "Analyzing" refers to the act of processing the data received by the server using an algorithm to evaluate the user's blink rate and eye condition.

[0014] "Notifying" refers to the act of providing information to the user suggesting eye breaks or other actions based on the analysis results.

[0015] "Real-time" means that data is collected, transmitted, analyzed, and communicated in near real time, providing feedback to users with no delay. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these. The program processing of this system will be explained in detail below.

[0038] Data collection

[0039] Device (smart glasses)

[0040] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if the user blinks 20 times per minute, that information will be collected.

[0041] Data transmission

[0042] Device (smart glasses)

[0043] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. This process allows the data to reach the server in real time, enabling rapid analysis.

[0044] Data reception and analysis

[0045] server

[0046] The server has an open receiving port to receive data sent from the smart glasses. The received data is stored in a database and then analyzed by AI. This analysis includes the number of times the user blinks, their eye movements, and eye opening and closing patterns. For example, if the blinking frequency over the past five minutes is lower than average, it is determined that the eyes are dry.

[0047] Deciding on an action

[0048] server

[0049] The server-side AI calculates the appropriate action based on the analysis results. For example, if dry eyes are detected, a suggestion to "take a two-minute break" is generated. This suggestion is formatted as a notification message and sent to the smart glasses.

[0050] notification

[0051] Device (smart glasses)

[0052] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. The notification can be done through audio, visual alerts, or vibration. For example, the user may hear an audio notification saying, "Please rest your eyes."

[0053] Specific examples

[0054] Suppose a user is wearing smart glasses and working on a computer. At this time, the smart glasses' sensors monitor the number of times the user blinks and send the data to a server every second. The server analyzes the received data and detects that the user is only blinking five times per minute. The AI ​​suspects dry eyes and decides to take an action such as "take a two-minute break." The smart glasses receive this notification and play a voice message to the user saying, "Please rest your eyes." This series of processes allows the user to autonomously take a break from their eyes and prevent eye strain.

[0055] This system performs all steps in real time, from data collection to analysis and notification, allowing users to effectively manage their eye health and prevent eye strain caused by prolonged use of digital devices.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] Device (smart glasses)

[0059] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[0060] Step 2:

[0061] Device (smart glasses)

[0062] The collected data is temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data is stored in the buffer.

[0063] Step 3:

[0064] Device (smart glasses)

[0065] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[0066] Step 4:

[0067] server

[0068] The server waits for data sent from the smart glasses via a receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data for the past minute is stored in the database.

[0069] Step 5:

[0070] server

[0071] The AI ​​algorithm on the server retrieves data from the database and analyzes it in real time. The analysis detects cases where the blink rate is lower than average or where eye movements are unnatural. For example, the AI ​​may detect that a user is blinking less than half the normal rate.

[0072] Step 6:

[0073] server

[0074] Based on the analysis results, the AI ​​algorithm generates the optimal action suggestion for the user. For example, if it determines that the user's eyes are dry, it will decide on an action suggesting a two-minute break. This suggestion is generated as a notification message in JSON format.

[0075] Step 7:

[0076] server

[0077] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Please rest your eyes" is sent.

[0078] Step 8:

[0079] Device (smart glasses)

[0080] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is given to the user. For example, a voice notification saying "Please rest your eyes" is played from the smart glasses.

[0081] Step 9:

[0082] User

[0083] The user receives a notification from the smart glasses and performs a suggested action (e.g., take a two-minute break), allowing the user to rest their eyes at the appropriate time and prevent eye strain.

[0084] In this way, each step works in conjunction with the others, allowing the user to autonomously manage their eye health in real time.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] This solution solves the problem of eye strain and dry eyes caused by prolonged use of digital devices, which reduces user productivity, and also addresses the lack of a system that monitors the user's eye condition in real time and advises them on appropriate break times.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes a means for receiving and analyzing data in real time, a means for storing the received data in a database, and a means for analyzing the data using a generative AI model, thereby enabling the user's eye data to be analyzed in real time and the appropriate action to be taken to be taken.

[0090] A "sensor means" is a device or mechanism that collects data about a user's eyes.

[0091] "Transmitting means" refers to a device or protocol for transmitting collected data to a server.

[0092] "Analysis means" refers to a device or software for analyzing received data.

[0093] The "notification means" is a device or function for notifying the user of appropriate actions based on the analysis results.

[0094] "Server means" refers to the function or configuration of a server that receives and analyzes data in real time.

[0095] The "camera" is a photographic device that detects and records the user's eye movements and blinks.

[0096] An "infrared sensor" is a sensor that detects whether the eyes are open or closed and the direction of gaze.

[0097] A "database" is a storage device or system for storing received data.

[0098] A "generative AI model" is an artificial intelligence algorithm used to analyze received data.

[0099] The "means for measuring the number of blinks" is a device or function that quantitatively measures the number of blinks of the user's eyes.

[0100] "Real-time" is a concept that refers to the time frame in which data is collected, transmitted, received, analyzed, and notified without delay.

[0101] This invention is a system for managing a user's eye health and preventing eye strain. In a specific embodiment, this system includes a visual data collection device (smart glasses) worn by the user, a server for storing and analyzing data, and communication means for connecting these.

[0102] Hardware and Software Usage

[0103] Device (smart glasses)

[0104] Smart glasses are equipped with a built-in camera and infrared sensors to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if a user blinks 20 times in one minute, this information will be collected immediately.

[0105] communication means

[0106] The collected visual data is sent to a server using wireless communication technologies such as Bluetooth or Wi-Fi. This data is packetized and sent at regular intervals (e.g., every second). The smart glasses temporarily store the data in a buffer memory before transmitting it and confirm that the transmission was successful.

[0107] server

[0108] The server opens a specific port to receive data from the smart glasses, which is then stored in a database and analyzed using a generative AI model.

[0109] Data processing and calculation

[0110] The server inputs the stored visual data into a generative AI model for analysis. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye open / close status. For example, if the blink rate over the past five minutes is lower than average, the server can suspect dry eyes.

[0111] Based on the analysis results, the server-side AI determines the appropriate action for the user (e.g., "Take a two-minute break," "Put in eye drops," etc.) This suggestion is formatted as a notification message and sent to the smart glasses in real time.

[0112] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. This notification can be audio, visual alert, or vibration. For example, a voice notification saying "Please rest your eyes" can be played through the smart glasses' speaker.

[0113] Specific examples

[0114] Consider a case where a user wears smart glasses while working on a computer. The smart glasses' sensors monitor the number of times the user blinks and send this data to a server every second. The server analyzes the received data and detects, for example, that the user is only blinking five times per minute. In this case, the server's AI suspects dry eyes and decides to take the action of "taking a two-minute break." The smart glasses receive this notification and play a voice notification to the user saying, "Please rest your eyes." This allows the user to autonomously take appropriate breaks and prevent eye strain.

[0115] Prompt Sentence Examples

[0116] Below is an example of a prompt sentence to input to the generative AI model.

[0117] "If a user blinks 50 times in an hour, I want the AI ​​model to analyze whether that number is high or low compared to the average and suggest an action."

[0118] This prompt allows you to simulate the process in which the AI ​​model suggests appropriate actions based on the results of data collection and analysis.

[0119] Based on the above specific embodiments, the present invention can manage the eye health of users and effectively prevent eye strain caused by using digital devices for long periods of time.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1: Data collection

[0122] Device (smart glasses)

[0123] Smart glasses activate cameras and infrared sensors to detect the user's eye movements. While the user is using a digital device, these sensors detect blinking, eye movement, and eye opening / closing status in real time. For example, if the user blinks 20 times in one minute, that data will be collected.

[0124] Input: User's eye movements (blinks, gaze, eye open / closed state)

[0125] Output: Collected visual data (e.g., blink count per minute)

[0126] Step 2: Send data

[0127] Device (smart glasses)

[0128] The collected visual data is packetized at regular intervals (e.g., every second) and sent to a server using Bluetooth or Wi-Fi. The smart glasses temporarily store the data in a buffer memory and confirm that the transmission was successful. This process ensures that the data reaches the server without interruption.

[0129] Input: Collected visual data

[0130] Output: Packetized data

[0131] Step 3: Receiving and storing data

[0132] server

[0133] The server opens a specific port to receive data sent from the smart glasses. The received data is stored in a database in real time and used for later analysis. For example, JSON format data is parsed and stored in a database.

[0134] Input: Packetized data

[0135] Output: Data stored in the database

[0136] Step 4: Data analysis

[0137] server

[0138] The server uses a generative AI model to analyze the visual data stored in the database. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye opening / closing status. For example, if the blink frequency is lower than average based on the past five minutes of data, the server will output a result suggesting possible dry eyes.

[0139] Input: Visual data retrieved from a database

[0140] Output: Analysis results (e.g., possibility of dry eyes)

[0141] Step 5: Decide on an action

[0142] server

[0143] The server-side AI then uses the analysis results to suggest appropriate actions, such as "take a two-minute break" or "put in eye drops." This suggestion is formatted as a notification message and sent to the smart glasses.

[0144] Input: Analysis results

[0145] Output: Notification message

[0146] Step 6: Notification

[0147] Device (smart glasses)

[0148] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be delivered using audio, visual alerts, or vibrations. For example, a speaker on the smart glasses might say, "Please rest your eyes."

[0149] Input: Notification message

[0150] Output: Notification to the user (e.g., audio notification)

[0151] These steps allow the system to manage the user's eye health in real time and prevent eye strain.

[0152] (Application example 1)

[0153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Modern factories are becoming increasingly automated, with many processes now being carried out by robots, but visual monitoring by workers remains important. In particular, it is necessary to quickly detect abnormalities in the robots and changes in the work environment and take appropriate action. However, this places a heavy burden on workers, making it difficult to monitor efficiently. Furthermore, managing the workers' own health is also important, and it is necessary to prevent eye fatigue caused by long hours of work.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0156] In this invention, the server includes a means for monitoring the work environment and equipment status, a means for analyzing collected data, and a means for notifying workers based on the analysis results. This allows for real-time analysis of gaze data transmitted from the smart glasses worn by workers, enabling the detection of abnormalities and the prompt provision of appropriate instructions. Furthermore, by simultaneously monitoring the fatigue and dryness of workers' eyes and encouraging them to take breaks at appropriate times, it is possible to achieve both work efficiency and health management.

[0157] "Sensor means" refers to a device for collecting data on the user's eyes.

[0158] The "transmitting means" is a communication device for transmitting collected data to a server.

[0159] "Analysis means" is a system for analyzing received data.

[0160] The "notification means" is a device for notifying the user of appropriate actions based on the analysis results.

[0161] "Work environment" refers to the location and conditions in which a user performs work, and includes factors related to safety and efficiency.

[0162] "Equipment status" refers to the operating status and normal operation of devices and robots used in work environments such as factories.

[0163] "Gaze data" is information about the movement of the user's eyes, and is data that indicates the direction and concentration of the gaze.

[0164] A "warning" is a notification or signal that notifies the user when an abnormality or danger occurs.

[0165] "Instructions" are information that informs workers of the actions and procedures they should take.

[0166] "Real-time" refers to data processing and communication occurring immediately and without delay.

[0167] "Abnormal" refers to a state in which something deviates from normal conditions or behavior and causes a problem.

[0168] "Analysis results" are information and judgments obtained as a result of analysis based on collected data.

[0169] "Rest prompt" is an action that notifies the user to take a rest at an appropriate time.

[0170] To implement this invention, we will build the following system, which consists of smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these.

[0171] Data collection

[0172] Device (smart glasses):

[0173] Smart glasses are equipped with sensors that detect the user's gaze data, blink rate, and eye movement. These sensors use cameras and infrared sensors to collect data in real time. For example, when a user focuses their gaze on a specific device, that gaze data is recorded.

[0174] Data transmission

[0175] Device (smart glasses):

[0176] The collected data is sent to the server every second using wireless communication technologies such as Bluetooth or Wi-Fi, allowing the data to be delivered to the server in real time.

[0177] Data reception and analysis

[0178] server:

[0179] The server receives data sent from the smart glasses via a TCP / UDP port. The received data is stored in a NoSQL database (e.g., MongoDB) and then analyzed by AI. AI frameworks such as TENSORFLOW (registered trademark) and PyTorch are used for the analysis. For example, anomalies in specific devices can be detected from gaze data, and an abnormality warning can be generated based on the results.

[0180] Deciding on an action

[0181] server:

[0182] The server-side AI calculates appropriate actions based on the analysis results. For example, if an abnormality is detected based on gaze data, an instruction such as "Please inspect this robot" is immediately generated.

[0183] notification

[0184] Device (smart glasses):

[0185] The smart glasses receive notifications from the server and warn workers using audio, visual, or vibration alerts. For example, if an abnormality is detected, a voice notification will sound saying, "Anomaly detected. Please inspect."

[0186] Specific examples

[0187] Let's say Worker A is wearing smart glasses at a factory. When Worker A focuses his gaze on a specific robot for a long period of time, that gaze data is sent to a server in real time. The server analyzes the gaze data and detects that there may be an abnormality in that specific robot. An instruction to "Please inspect this robot" is immediately sent to the smart glasses, and Worker A is notified by voice.

[0188] Prompt Sentence Examples

[0189] prompt:

[0190] "Design an application that allows factory workers wearing smart glasses to collect gaze data in real time and then have it analyzed on a server to provide important information about the status of robots and the work environment. This includes the smart glasses sensors, data transmission, server-side analysis, and notification processes."

[0191] This system allows workers to safely and efficiently monitor their work environment, enabling early detection of abnormalities and prompt response.It also allows for simultaneous management of workers' eye health, achieving both work efficiency and health management.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] Data collection

[0195] The camera and infrared sensor installed in the device (smart glasses) collect the user's gaze data, blink frequency, and eye movement. The sensor captures data every 100 milliseconds and measures the gaze direction and concentration. The input is the user's gaze and blink behavior, and the output is the captured gaze data.

[0196] Step 2:

[0197] Data transmission

[0198] The device (smart glasses) transmits the collected gaze data to the server via Wi-Fi or Bluetooth. This transmission occurs every second, and the data is delivered to the server in real time. The input is the gaze data collected in step 1, and the output is the data transmitted to the server.

[0199] Step 3:

[0200] Data reception

[0201] The server receives gaze data sent from the smart glasses using a TCP / UDP port. The received data is temporarily cached for subsequent data processing. The input is the data sent from the device, and the output is the cache of the received data.

[0202] Step 4:

[0203] Data analysis

[0204] The server stores the received gaze data in a NoSQL database (e.g., MongoDB) and then analyzes the data using an AI framework (e.g., TensorFlow, PyTorch). The analysis detects abnormalities in specific devices and changes in the work environment from gaze movements. The input is the received gaze data, and the output is the analysis results.

[0205] Step 5:

[0206] Deciding on an action

[0207] The server calculates appropriate actions based on the results of the AI ​​analysis and notifies the worker. For example, if an abnormality is detected in the robot, it generates an instruction such as "Please inspect this robot." The input is the analysis result from step 4, and the output is the generated instruction.

[0208] Step 6:

[0209] notification

[0210] The device (smart glasses) receives the notification sent from the server and notifies the worker using a voice notification, visual alert, or vibration notification. For example, when an abnormality is detected, a voice notification saying "Anomaly detected, please inspect" is played. The input is the notification content from step 5, and the output is the notification to the user.

[0211] The processing flow and specific operations at each step make it possible to use smart glasses to monitor work in real time and respond to abnormalities.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means for connecting these components. The program processing of this system is explained in detail below.

[0214] Data collection

[0215] Device (smart glasses)

[0216] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect the number of blinks, gaze movements, and eye opening and closing. In addition, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate the user's emotional state (e.g., stress, joy, surprise). For example, if a user blinks 20 times per minute and feels stressed at the same time, this information will be collected and recorded.

[0217] Data transmission

[0218] Device (smart glasses)

[0219] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The collected data includes both information about eye movements and emotion data from the emotion engine. For example, eye blink data and stress level data are sent to the server simultaneously.

[0220] Data reception and analysis

[0221] server

[0222] The server waits for composite data sent from the smart glasses via a receiving port and retrieves the received data. The retrieved data is stored in a database and then analyzed by AI. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, if the blink rate over the past five minutes has decreased and a high stress level is detected at the same time, it is determined that the eyes are dry.

[0223] Deciding on an action

[0224] server

[0225] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0226] notification

[0227] Device (smart glasses)

[0228] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, the user may be told to "rest their eyes" while additional advice such as "Your stress level is high, so take a deep breath" is displayed.

[0229] Specific examples

[0230] Suppose a user wears smart glasses and works at a desk for a long period of time. At this time, the smart glasses' sensors monitor the number of blinks and eye movements, while the emotion engine simultaneously analyzes facial expressions and voice tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smart glasses: "Please rest your eyes" and "Take deep breaths to reduce stress." This series of processes allows users to maintain eye health and manage stress.

[0231] By integrating the monitoring and management of eye health and emotional state, the system can provide users with more accurate and appropriate advice, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[0232] The processing flow will be explained below.

[0233] Step 1:

[0234] Device (smart glasses)

[0235] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[0236] Step 2:

[0237] Device (smart glasses)

[0238] The smart glasses' emotion engine analyzes the user's facial expressions and vocal tone, allowing it to assess the user's emotions (e.g., stress, joy, surprise) in real time. For example, it can detect high stress levels from the user's facial expressions.

[0239] Step 3:

[0240] Device (smart glasses)

[0241] The collected eye data and emotion data are temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data and emotion data are stored in the buffer.

[0242] Step 4:

[0243] Device (smart glasses)

[0244] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[0245] Step 5:

[0246] server

[0247] The server waits for the composite data sent from the smart glasses via the receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data and emotion data for the past minute are stored in the database.

[0248] Step 6:

[0249] server

[0250] The server's AI algorithm retrieves data from the database and analyzes it in real time. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, the AI ​​detects that the user's blink rate is less than half the normal rate and that their stress level is very high.

[0251] Step 7:

[0252] server

[0253] Based on the analysis results, the AI ​​algorithm generates optimal action suggestions for the user. For example, if dry eyes and high stress are detected at the same time, the algorithm will suggest taking a two-minute break and taking deep breaths to reduce stress. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0254] Step 8:

[0255] server

[0256] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Rest your eyes" is sent with additional advice such as "Take deep breaths to reduce stress."

[0257] Step 9:

[0258] Device (smart glasses)

[0259] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is sent to the user. For example, the smart glasses may provide an audio notification saying "Please rest your eyes" and a visual notification saying "Your stress level is high, so please take a deep breath."

[0260] Step 10:

[0261] User

[0262] Users receive notifications from their smart glasses and take suggested actions (e.g., take a two-minute break and take a deep breath), allowing them to rest their eyes at the right time, reduce stress, and prevent eye strain.

[0263] In this way, each step works in tandem, allowing users to autonomously manage their eye health and emotional state in real time.

[0264] Example 2

[0265] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0266] Conventional eye health management systems simply monitor the condition of the eyes and are unable to provide appropriate advice that takes into account the user's emotional state and stress level. Furthermore, these systems struggle to provide real-time analysis and action suggestions, limiting their effectiveness in situations where a rapid response is required. This has left the problems of eye fatigue, dryness, and stress unresolved, especially in environments where digital devices are used for long periods of time.

[0267] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0268] In this invention, the server includes a sensor means for collecting information on the user's eye movements, blinks, and gaze, an emotion analysis means for analyzing the user's facial expressions and voice tone to evaluate their emotional state, a transmission means for transmitting the collected data to the server, a data storage means for storing the received data, an analysis means for analyzing the received data using AI, and a means for generating and notifying the user of optimal action suggestions based on the analysis results. This makes it possible to comprehensively monitor not only the user's eye health but also their emotional state, enabling more accurate and prompt responses.

[0269] "Sensor means" refers to devices and technologies that collect information on a user's eye movements, blinks, and gaze in real time.

[0270] "Emotion analysis means" refers to technology and devices that analyze a user's facial expressions and tone of voice to assess the user's emotional state.

[0271] "Transmission means" refers to the communication technology and devices used to transmit collected data to the server.

[0272] "Data Storage Means" refers to a data storage system for long-term archiving of received data.

[0273] "Analysis means" refers to the technology and devices that use AI technology to analyze received data and determine the user's status and necessary actions.

[0274] The "action suggestion means" refers to a technology and device that generates optimal action suggestions for the user based on the analysis results and notifies the user of them.

[0275] "Notification means" refers to devices and technologies that notify the user of the generated action suggestions through sound, visual display, vibration, etc.

[0276] "User's eye movements" refers to eye movements such as the user moving their line of sight or blinking.

[0277] The "number of blinks" refers to the number of times the user blinks within a certain period of time.

[0278] "Gaze information" refers to data indicating the direction of the user's gaze and the object on which the user is focused.

[0279] "Emotional state" refers to the emotional state that a user is feeling, such as stress, joy, surprise, etc.

[0280] "AI technology" refers to technology that uses artificial intelligence to analyze data and derive optimal results, such as suggesting actions.

[0281] MODE FOR CARRYING OUT THE INVENTION

[0282] This invention is a system that comprehensively monitors a user's eye health and emotional state and provides appropriate action suggestions in real time. This system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means connecting these components.

[0283] Hardware and Software Used

[0284] Device (smart glasses)

[0285] The smart glasses include the following hardware:

[0286] camera

[0287] Infrared sensor

[0288] microphone

[0289] Bluetooth and Wi-Fi modules

[0290] These hardware devices are used to collect information on the user's eye movements, blinks, gaze, facial expressions, and vocal tones, and use Bluetooth or Wi-Fi as communication methods to transmit the data to a server in real time.

[0291] server

[0292] The following software and AI models are implemented on the server side:

[0293] Database system (e.g., MySQL (registered trademark))

[0294] AI analysis engine (e.g. TensorFlow, PyTorch)

[0295] Sentiment Analysis Engine

[0296] This software receives, stores, and analyzes data sent from the device and uses it to generate optimal action suggestions based on the user's state.

[0297] Specific examples of processing

[0298] Let's take the example of a user wearing smart glasses and working at a desk for a long period of time. At this time, the smart glasses' sensors (camera and infrared sensor) monitor the user's blink rate and eye movements. At the same time, the emotion engine analyzes the user's facial expressions and voice tone to evaluate their stress level. This data is sent from the smart glasses to a server via Wi-Fi every second.

[0299] The server receives the data and immediately stores it in a database. The stored data is then analyzed by an AI analysis engine to generate detailed information about the user's eye movements and emotional state. For example, if the analysis detects "fewer than five blinks in the past minute" and "high stress level," the server-side AI generates notification messages in JSON format with suggested actions, such as "rest your eyes" and "take deep breaths to reduce stress." These notifications are sent to the smart glasses in real time and notify the user via audio, visual display, or vibration.

[0300] Examples of prompt statements

[0301] Below are some example prompts to input to a generative AI model:

[0302] "Describe the design of a system that monitors a user's eye health and emotional state in real time and suggests appropriate actions. The system includes smart glasses, a server, and an emotion engine."

[0303] As described above, the system of the present invention comprehensively monitors eye health and emotional state and can suggest optimal actions in real time, thereby effectively reducing eye strain and stress caused by prolonged use of digital devices.

[0304] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0305] Step 1:

[0306] Data collection

[0307] Subject: Device (Smart Glasses)

[0308] Input: User's eye movements, blinks, gaze information, facial expressions, and voice tones

[0309] Processing: The smart glasses' built-in camera and infrared sensors collect real-time information on the user's eye movements, blinks, and gaze. The emotion engine analyzes the user's facial expressions and vocal tone to assess their emotional state.

[0310] Output: Collected blink count, gaze data, emotional state data

[0311] Step 2:

[0312] Data transmission

[0313] Subject: Device (Smart Glasses)

[0314] Input: Collected blink count, gaze data, emotional state data

[0315] Processing: The smart glasses send the collected data to the server at regular intervals (every second). Bluetooth or Wi-Fi is used as the communication method.

[0316] Output: Number of blinks, gaze data, and emotion data sent to the server

[0317] Step 3:

[0318] Data reception and storage

[0319] Subject: Server

[0320] Input: Number of blinks, gaze data, emotion data sent from smart glasses

[0321] Processing: The server waits for data on the receiving port and receives the data sent from the smart glasses. After receiving the data, it stores it in a database system (e.g., MySQL).

[0322] Output: Number of blinks, gaze data, and emotion data stored in a database

[0323] Step 4:

[0324] Data analysis

[0325] Subject: Server

[0326] Input: Number of blinks, gaze data, emotion data stored in the database

[0327] Processing: Analyze the stored data using an AI analysis engine (e.g., TensorFlow, PyTorch). The analysis includes blink count, eye movement, eye opening and closing patterns, and emotional data. Detect cases where the blink count is below average or where the emotional data indicates high stress.

[0328] Output: Analysis results (e.g., decreased blinking rate and high stress levels in the past 5 minutes)

[0329] Step 5:

[0330] Generate action suggestions

[0331] Subject: Server

[0332] Input: Analysis results (e.g., decreased blink rate and high stress levels over the past 5 minutes)

[0333] Processing: Based on the analysis results, generate optimal action suggestions for the user. For example, generate suggestions such as "Rest your eyes" and "Take deep breaths to reduce stress." The suggestions are generated as notification messages in JSON format.

[0334] Output: Action proposal notification message (JSON format)

[0335] Step 6:

[0336] proposal notification

[0337] Subject: Device (Smart Glasses)

[0338] Input: Notification message of suggested action sent from the server

[0339] Processing: The smart glasses receive the notification sent from the server. The notification is communicated to the user through audio, visual display, or vibration. For example, the smart glasses may notify the user by audio, "Please rest your eyes," and show "Take deep breaths to reduce stress" on the display.

[0340] Output: Notification of suggested actions to the user (audio, visual display, vibration)

[0341] Through these steps, the system can monitor the user's eye health and emotional state in real time and provide appropriate behavioral suggestions.

[0342] (Application example 2)

[0343] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] In today's world, prolonged use of digital devices can cause eye fatigue and increased stress when users shop comfortably in virtual stores. This can lead to disadvantages such as a poor shopping experience and reduced decision-making ability. This invention aims to reduce eye fatigue and stress and provide a comfortable shopping environment by analyzing the user's eye health and emotional state in real time and providing appropriate advice.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0346] In this invention, the server includes a sensor means for collecting data on the user's eyes, a transmission means for transmitting the collected data to the server, an analysis means for analyzing the received data, a notification means for informing the user of appropriate actions based on the analysis results, a means for analyzing the user's emotional state, a means for detecting shopping stress based on the emotional state, and a means for suggesting actions to the user to reduce shopping stress. This makes it possible to comprehensively evaluate the user's eye health and emotional state and achieve stress reduction in real time.

[0347] "Sensor means" refers to a device that collects data on the user's eyes, including sensors that detect gaze and blinking.

[0348] The "transmission means" is a function for transmitting collected data to a server using wireless communication or the like.

[0349] "Analysis means" refers to a program or algorithm that analyzes the data sent to the server and evaluates the user's eye health and emotional state.

[0350] The "notification means" is a function for notifying the user of appropriate actions based on the analysis results, and includes notifications by voice, visual display, vibration, etc.

[0351] The "means for analyzing emotional state" is a device or software that determines and analyzes the emotional state of the user from facial expressions and tone of voice.

[0352] The "means for detecting shopping stress" is a means for detecting stress from the user's emotional state and gaze data, etc., and evaluating the quality of the shopping experience.

[0353] The "means for suggesting an action" is a device or software that suggests appropriate actions (e.g., taking a break or a relaxation method) to the user based on the analysis results and evaluation results.

[0354] "Receiving in real time" means that the server receives the data almost simultaneously with the time it is sent.

[0355] "Generating in real time" means that the server immediately analyzes the data it receives and generates the analysis results.

[0356] "Real-time notification" means instantly notifying the user of the analysis results. It also includes comprehensively evaluating the user's eye health and emotional state and providing appropriate advice in real time.

[0357] The system that realizes this application example mainly consists of the following components: smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and a means of communication that connects these components.

[0358] Data collection

[0359] Device (smart glasses)

[0360] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect blink counts, gaze movements, and eye opening and closing. In addition, an emotion engine analyzes the user's facial expressions and vocal tone to assess the user's emotional state (e.g., stress, joy, surprise).

[0361] Data transmission

[0362] Device (smart glasses)

[0363] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. The collected data includes both information about eye movements and emotion data from the emotion engine.

[0364] Data reception and analysis

[0365] server

[0366] The server listens for and receives composite data sent from the smart glasses via a receiving port. The data is then stored in a database and analyzed by AI. The analysis includes the user's eye blink rate, eye movements, and eye opening / closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress.

[0367] Deciding on an action

[0368] server

[0369] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0370] notification

[0371] Device (smart glasses)

[0372] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, a voice notification saying "Please rest your eyes" may be followed by additional visual advice such as "Your stress level is high, so take a deep breath."

[0373] Specific examples

[0374] Suppose a user is shopping in a virtual store. Comparing products for a long time can cause eye strain and stress. The smartglasses' sensors monitor blinking and eye movements, while the emotion engine analyzes facial expressions and vocal tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smartglasses: "Please rest your eyes" and "Take a deep breath to reduce stress."

[0375] Prompt Sentence Examples

[0376] "Generate a data collection method to detect whether users are experiencing stress in a virtual store."

[0377] "Please propose an algorithm that derives appropriate advice from a user's gaze information and emotional data."

[0378] The system provides users with more accurate and appropriate advice by comprehensively monitoring and managing their eye health and emotional state, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[0379] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0380] Step 1:

[0381] Data collection

[0382] The device (smart glasses) is equipped with multiple sensors. These sensors collect information on the user's blink count, eye movement, and eye opening / closing in real time. In parallel with this, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate their emotional state (e.g., stress, joy, surprise). The input of the data collection is the eye movement and facial expression detected by the sensors, and the output is the blink count, eye movement data, and emotion evaluation data.

[0383] Step 2:

[0384] Data transmission

[0385] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The input for data transmission is the number of blinks, gaze data, and emotion data collected by the device, and the output is the transmission of this data to the server.

[0386] Step 3:

[0387] Data reception and storage

[0388] The server waits for the composite data sent from the smart glasses at the receiving port and acquires the received data. The acquired data is stored in a database. The input of the data reception and storage is the blink count, gaze data, and emotion data sent to the server, and the output is that these data are stored in the database.

[0389] Step 4:

[0390] Data analysis

[0391] The server-side analysis uses data stored in the database to analyze the user's eye health and emotional state. An AI algorithm is used for the analysis, evaluating blink count, eye movement, eye opening and closing patterns, emotional data, and other factors. In particular, it detects cases where the blink count is lower than average or emotional data indicates high stress. It also compares the results with past data. The input for the data analysis is all data stored in the database, and the output is the analysis results.

[0392] Step 5:

[0393] Generate action suggestions

[0394] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected simultaneously, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format. The input for action suggestion generation is the analysis results, and the output is the action suggestion message.

[0395] Step 6:

[0396] notification

[0397] The device (smart glasses) receives the notification sent from the server and prompts the user to take appropriate action. The notification is given by voice, visual display, or vibration. For example, a voice notification saying "Please rest your eyes" may be given, while additional advice such as "Your stress level is high, so take a deep breath" is displayed. The input of the notification is the action suggestion message, and its output is the notification to the user.

[0398] Through these steps, the system can comprehensively monitor and manage the user's eye health and emotional state, and provide accurate and appropriate advice to the user, allowing the user to enjoy a comfortable shopping experience in the virtual store.

[0399] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0400] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0401] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0402] [Second embodiment]

[0403] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0404] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0405] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0406] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0407] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0408] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0409] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0410] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0411] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0412] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0413] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0414] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0415] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these. The program processing of this system will be explained in detail below.

[0416] Data collection

[0417] Device (smart glasses)

[0418] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if the user blinks 20 times per minute, that information will be collected.

[0419] Data transmission

[0420] Device (smart glasses)

[0421] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. This process allows the data to reach the server in real time, enabling rapid analysis.

[0422] Data reception and analysis

[0423] server

[0424] The server has an open receiving port to receive data sent from the smart glasses. The received data is stored in a database and then analyzed by AI. This analysis includes the number of times the user blinks, their eye movements, and eye opening and closing patterns. For example, if the blinking frequency over the past five minutes is lower than average, it is determined that the eyes are dry.

[0425] Deciding on an action

[0426] server

[0427] The server-side AI calculates the appropriate action based on the analysis results. For example, if dry eyes are detected, a suggestion to "take a two-minute break" is generated. This suggestion is formatted as a notification message and sent to the smart glasses.

[0428] notification

[0429] Device (smart glasses)

[0430] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. The notification can be done through audio, visual alerts, or vibration. For example, the user may hear an audio notification saying, "Please rest your eyes."

[0431] Specific examples

[0432] Suppose a user is wearing smart glasses and working on a computer. At this time, the smart glasses' sensors monitor the number of times the user blinks and send the data to a server every second. The server analyzes the received data and detects that the user is only blinking five times per minute. The AI ​​suspects dry eyes and decides to take an action such as "take a two-minute break." The smart glasses receive this notification and play a voice message to the user saying, "Please rest your eyes." This series of processes allows the user to autonomously take a break from their eyes and prevent eye strain.

[0433] This system performs all steps in real time, from data collection to analysis and notification, allowing users to effectively manage their eye health and prevent eye strain caused by prolonged use of digital devices.

[0434] The processing flow will be explained below.

[0435] Step 1:

[0436] Device (smart glasses)

[0437] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[0438] Step 2:

[0439] Device (smart glasses)

[0440] The collected data is temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data is stored in the buffer.

[0441] Step 3:

[0442] Device (smart glasses)

[0443] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[0444] Step 4:

[0445] server

[0446] The server waits for data sent from the smart glasses via a receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data for the past minute is stored in the database.

[0447] Step 5:

[0448] server

[0449] The AI ​​algorithm on the server retrieves data from the database and analyzes it in real time. The analysis detects cases where the blink rate is lower than average or where eye movements are unnatural. For example, the AI ​​may detect that a user is blinking less than half the normal rate.

[0450] Step 6:

[0451] server

[0452] Based on the analysis results, the AI ​​algorithm generates the optimal action suggestion for the user. For example, if it determines that the user's eyes are dry, it will decide on an action suggesting a two-minute break. This suggestion is generated as a notification message in JSON format.

[0453] Step 7:

[0454] server

[0455] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Please rest your eyes" is sent.

[0456] Step 8:

[0457] Device (smart glasses)

[0458] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is given to the user. For example, a voice notification saying "Please rest your eyes" is played from the smart glasses.

[0459] Step 9:

[0460] User

[0461] The user receives a notification from the smart glasses and performs a suggested action (e.g., take a two-minute break), allowing the user to rest their eyes at the appropriate time and prevent eye strain.

[0462] In this way, each step works in conjunction with the others, allowing the user to autonomously manage their eye health in real time.

[0463] Example 1

[0464] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0465] This solution solves the problem of eye strain and dry eyes caused by prolonged use of digital devices, which reduces user productivity, and also addresses the lack of a system that monitors the user's eye condition in real time and advises them on appropriate break times.

[0466] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0467] In this invention, the server includes a means for receiving and analyzing data in real time, a means for storing the received data in a database, and a means for analyzing the data using a generative AI model, thereby enabling the user's eye data to be analyzed in real time and the appropriate action to be taken to be taken.

[0468] A "sensor means" is a device or mechanism that collects data about a user's eyes.

[0469] "Transmitting means" refers to a device or protocol for transmitting collected data to a server.

[0470] "Analysis means" refers to a device or software for analyzing received data.

[0471] The "notification means" is a device or function for notifying the user of appropriate actions based on the analysis results.

[0472] "Server means" refers to the function or configuration of a server that receives and analyzes data in real time.

[0473] The "camera" is a photographic device that detects and records the user's eye movements and blinks.

[0474] An "infrared sensor" is a sensor that detects whether the eyes are open or closed and the direction of gaze.

[0475] A "database" is a storage device or system for storing received data.

[0476] A "generative AI model" is an artificial intelligence algorithm used to analyze received data.

[0477] The "means for measuring the number of blinks" is a device or function that quantitatively measures the number of blinks of the user's eyes.

[0478] "Real-time" is a concept that refers to the time frame in which data is collected, transmitted, received, analyzed, and notified without delay.

[0479] This invention is a system for managing a user's eye health and preventing eye strain. In a specific embodiment, this system includes a visual data collection device (smart glasses) worn by the user, a server for storing and analyzing data, and communication means for connecting these.

[0480] Hardware and Software Usage

[0481] Device (smart glasses)

[0482] Smart glasses are equipped with a built-in camera and infrared sensors to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if a user blinks 20 times in one minute, this information will be collected immediately.

[0483] communication means

[0484] The collected visual data is sent to a server using wireless communication technologies such as Bluetooth or Wi-Fi. This data is packetized and sent at regular intervals (e.g., every second). The smart glasses temporarily store the data in a buffer memory before transmitting it and confirm that the transmission was successful.

[0485] server

[0486] The server opens a specific port to receive data from the smart glasses, which is then stored in a database and analyzed using a generative AI model.

[0487] Data processing and calculation

[0488] The server inputs the stored visual data into a generative AI model for analysis. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye open / close status. For example, if the blink rate over the past five minutes is lower than average, the server can suspect dry eyes.

[0489] Based on the analysis results, the server-side AI determines the appropriate action for the user (e.g., "Take a two-minute break," "Put in eye drops," etc.) This suggestion is formatted as a notification message and sent to the smart glasses in real time.

[0490] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. This notification can be audio, visual alert, or vibration. For example, a voice notification saying "Please rest your eyes" can be played through the smart glasses' speaker.

[0491] Specific examples

[0492] Consider a case where a user wears smart glasses while working on a computer. The smart glasses' sensors monitor the number of times the user blinks and send this data to a server every second. The server analyzes the received data and detects, for example, that the user is only blinking five times per minute. In this case, the server's AI suspects dry eyes and decides to take the action of "taking a two-minute break." The smart glasses receive this notification and play a voice notification to the user saying, "Please rest your eyes." This allows the user to autonomously take appropriate breaks and prevent eye strain.

[0493] Prompt Sentence Examples

[0494] Below is an example of a prompt sentence to input to the generative AI model.

[0495] "If a user blinks 50 times in an hour, I want the AI ​​model to analyze whether that number is high or low compared to the average and suggest an action."

[0496] This prompt allows you to simulate the process in which the AI ​​model suggests appropriate actions based on the results of data collection and analysis.

[0497] Based on the above specific embodiments, the present invention can manage the eye health of users and effectively prevent eye strain caused by using digital devices for long periods of time.

[0498] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0499] Step 1: Data collection

[0500] Device (smart glasses)

[0501] Smart glasses activate cameras and infrared sensors to detect the user's eye movements. While the user is using a digital device, these sensors detect blinking, eye movement, and eye opening / closing status in real time. For example, if the user blinks 20 times in one minute, that data will be collected.

[0502] Input: User's eye movements (blinks, gaze, eye open / closed state)

[0503] Output: Collected visual data (e.g., blink count per minute)

[0504] Step 2: Send data

[0505] Device (smart glasses)

[0506] The collected visual data is packetized at regular intervals (e.g., every second) and sent to a server using Bluetooth or Wi-Fi. The smart glasses temporarily store the data in a buffer memory and confirm that the transmission was successful. This process ensures that the data reaches the server without interruption.

[0507] Input: Collected visual data

[0508] Output: Packetized data

[0509] Step 3: Receiving and storing data

[0510] server

[0511] The server opens a specific port to receive data sent from the smart glasses. The received data is stored in a database in real time and used for later analysis. For example, JSON format data is parsed and stored in a database.

[0512] Input: Packetized data

[0513] Output: Data stored in the database

[0514] Step 4: Data analysis

[0515] server

[0516] The server uses a generative AI model to analyze the visual data stored in the database. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye opening / closing status. For example, if the blink frequency is lower than average based on the past five minutes of data, the server will output a result suggesting possible dry eyes.

[0517] Input: Visual data retrieved from a database

[0518] Output: Analysis results (e.g., possibility of dry eyes)

[0519] Step 5: Decide on an action

[0520] server

[0521] The server-side AI then uses the analysis results to suggest appropriate actions, such as "take a two-minute break" or "put in eye drops." This suggestion is formatted as a notification message and sent to the smart glasses.

[0522] Input: Analysis results

[0523] Output: Notification message

[0524] Step 6: Notification

[0525] Device (smart glasses)

[0526] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be delivered using audio, visual alerts, or vibrations. For example, a speaker on the smart glasses might say, "Please rest your eyes."

[0527] Input: Notification message

[0528] Output: Notification to the user (e.g., audio notification)

[0529] These steps allow the system to manage the user's eye health in real time and prevent eye strain.

[0530] (Application example 1)

[0531] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0532] Modern factories are becoming increasingly automated, with many processes now being carried out by robots, but visual monitoring by workers remains important. In particular, it is necessary to quickly detect abnormalities in the robots and changes in the work environment and take appropriate action. However, this places a heavy burden on workers, making it difficult to monitor efficiently. Furthermore, managing the workers' own health is also important, and it is necessary to prevent eye fatigue caused by long hours of work.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0534] In this invention, the server includes a means for monitoring the work environment and equipment status, a means for analyzing collected data, and a means for notifying workers based on the analysis results. This allows for real-time analysis of gaze data transmitted from the smart glasses worn by workers, enabling the detection of abnormalities and the prompt provision of appropriate instructions. Furthermore, by simultaneously monitoring the fatigue and dryness of workers' eyes and encouraging them to take breaks at appropriate times, it is possible to achieve both work efficiency and health management.

[0535] "Sensor means" refers to a device for collecting data on the user's eyes.

[0536] The "transmitting means" is a communication device for transmitting collected data to a server.

[0537] "Analysis means" is a system for analyzing received data.

[0538] The "notification means" is a device for notifying the user of appropriate actions based on the analysis results.

[0539] "Work environment" refers to the location and conditions in which a user performs work, and includes factors related to safety and efficiency.

[0540] "Equipment status" refers to the operating status and normal operation of devices and robots used in work environments such as factories.

[0541] "Gaze data" is information about the movement of the user's eyes, and is data that indicates the direction and concentration of the gaze.

[0542] A "warning" is a notification or signal that notifies the user when an abnormality or danger occurs.

[0543] "Instructions" are information that informs workers of the actions and procedures they should take.

[0544] "Real-time" refers to data processing and communication occurring immediately and without delay.

[0545] "Abnormal" refers to a state in which something deviates from normal conditions or behavior and causes a problem.

[0546] "Analysis results" are information and judgments obtained as a result of analysis based on collected data.

[0547] "Rest prompt" is an action that notifies the user to take a rest at an appropriate time.

[0548] To implement this invention, we will build the following system, which consists of smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these.

[0549] Data collection

[0550] Device (smart glasses):

[0551] Smart glasses are equipped with sensors that detect the user's gaze data, blink rate, and eye movement. These sensors use cameras and infrared sensors to collect data in real time. For example, when a user focuses their gaze on a specific device, that gaze data is recorded.

[0552] Data transmission

[0553] Device (smart glasses):

[0554] The collected data is sent to the server every second using wireless communication technologies such as Bluetooth or Wi-Fi, allowing the data to be delivered to the server in real time.

[0555] Data reception and analysis

[0556] server:

[0557] The server receives data sent from the smart glasses via a TCP / UDP port. The received data is stored in a NoSQL database (e.g., MongoDB) and then analyzed using AI. AI frameworks such as TensorFlow and PyTorch are used for the analysis. For example, anomalies in specific devices can be detected from gaze data, and an abnormality warning can be generated based on the results.

[0558] Deciding on an action

[0559] server:

[0560] The server-side AI calculates appropriate actions based on the analysis results. For example, if an abnormality is detected based on gaze data, an instruction such as "Please inspect this robot" is immediately generated.

[0561] notification

[0562] Device (smart glasses):

[0563] The smart glasses receive notifications from the server and warn workers using audio, visual, or vibration alerts. For example, if an abnormality is detected, a voice notification will sound saying, "Anomaly detected. Please inspect."

[0564] Specific examples

[0565] Let's say Worker A is wearing smart glasses at a factory. When Worker A focuses his gaze on a specific robot for a long period of time, that gaze data is sent to a server in real time. The server analyzes the gaze data and detects that there may be an abnormality in that specific robot. An instruction to "Please inspect this robot" is immediately sent to the smart glasses, and Worker A is notified by voice.

[0566] Prompt Sentence Examples

[0567] prompt:

[0568] "Design an application that allows factory workers wearing smart glasses to collect gaze data in real time and then have it analyzed on a server to provide important information about the status of robots and the work environment. This includes the smart glasses sensors, data transmission, server-side analysis, and notification processes."

[0569] This system allows workers to safely and efficiently monitor their work environment, enabling early detection of abnormalities and prompt response.It also allows for simultaneous management of workers' eye health, achieving both work efficiency and health management.

[0570] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0571] Step 1:

[0572] Data collection

[0573] The camera and infrared sensor installed in the device (smart glasses) collect the user's gaze data, blink frequency, and eye movement. The sensor captures data every 100 milliseconds and measures the gaze direction and concentration. The input is the user's gaze and blink behavior, and the output is the captured gaze data.

[0574] Step 2:

[0575] Data transmission

[0576] The device (smart glasses) transmits the collected gaze data to the server via Wi-Fi or Bluetooth. This transmission occurs every second, and the data is delivered to the server in real time. The input is the gaze data collected in step 1, and the output is the data transmitted to the server.

[0577] Step 3:

[0578] Data reception

[0579] The server receives gaze data sent from the smart glasses using a TCP / UDP port. The received data is temporarily cached for subsequent data processing. The input is the data sent from the device, and the output is the cache of the received data.

[0580] Step 4:

[0581] Data analysis

[0582] The server stores the received gaze data in a NoSQL database (e.g., MongoDB) and then analyzes the data using an AI framework (e.g., TensorFlow, PyTorch). The analysis detects abnormalities in specific devices and changes in the work environment from gaze movements. The input is the received gaze data, and the output is the analysis results.

[0583] Step 5:

[0584] Deciding on an action

[0585] The server calculates appropriate actions based on the results of the AI ​​analysis and notifies the worker. For example, if an abnormality is detected in the robot, it generates an instruction such as "Please inspect this robot." The input is the analysis result from step 4, and the output is the generated instruction.

[0586] Step 6:

[0587] notification

[0588] The device (smart glasses) receives the notification sent from the server and notifies the worker using a voice notification, visual alert, or vibration notification. For example, when an abnormality is detected, a voice notification saying "Anomaly detected, please inspect" is played. The input is the notification content from step 5, and the output is the notification to the user.

[0589] The processing flow and specific operations at each step make it possible to use smart glasses to monitor work in real time and respond to abnormalities.

[0590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0591] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means for connecting these components. The program processing of this system is explained in detail below.

[0592] Data collection

[0593] Device (smart glasses)

[0594] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect the number of blinks, gaze movements, and eye opening and closing. In addition, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate the user's emotional state (e.g., stress, joy, surprise). For example, if a user blinks 20 times per minute and feels stressed at the same time, this information will be collected and recorded.

[0595] Data transmission

[0596] Device (smart glasses)

[0597] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The collected data includes both information about eye movements and emotion data from the emotion engine. For example, eye blink data and stress level data are sent to the server simultaneously.

[0598] Data reception and analysis

[0599] server

[0600] The server waits for composite data sent from the smart glasses via a receiving port and retrieves the received data. The retrieved data is stored in a database and then analyzed by AI. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, if the blink rate over the past five minutes has decreased and a high stress level is detected at the same time, it is determined that the eyes are dry.

[0601] Deciding on an action

[0602] server

[0603] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0604] notification

[0605] Device (smart glasses)

[0606] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, the user may be told to "rest their eyes" while additional advice such as "Your stress level is high, so take a deep breath" is displayed.

[0607] Specific examples

[0608] Suppose a user wears smart glasses and works at a desk for a long period of time. At this time, the smart glasses' sensors monitor the number of blinks and eye movements, while the emotion engine simultaneously analyzes facial expressions and voice tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smart glasses: "Please rest your eyes" and "Take deep breaths to reduce stress." This series of processes allows users to maintain eye health and manage stress.

[0609] By integrating the monitoring and management of eye health and emotional state, the system can provide users with more accurate and appropriate advice, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[0610] The processing flow will be explained below.

[0611] Step 1:

[0612] Device (smart glasses)

[0613] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[0614] Step 2:

[0615] Device (smart glasses)

[0616] The smart glasses' emotion engine analyzes the user's facial expressions and vocal tone, allowing it to assess the user's emotions (e.g., stress, joy, surprise) in real time. For example, it can detect high stress levels from the user's facial expressions.

[0617] Step 3:

[0618] Device (smart glasses)

[0619] The collected eye data and emotion data are temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data and emotion data are stored in the buffer.

[0620] Step 4:

[0621] Device (smart glasses)

[0622] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[0623] Step 5:

[0624] server

[0625] The server waits for the composite data sent from the smart glasses via the receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data and emotion data for the past minute are stored in the database.

[0626] Step 6:

[0627] server

[0628] The server's AI algorithm retrieves data from the database and analyzes it in real time. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, the AI ​​detects that the user's blink rate is less than half the normal rate and that their stress level is very high.

[0629] Step 7:

[0630] server

[0631] Based on the analysis results, the AI ​​algorithm generates optimal action suggestions for the user. For example, if dry eyes and high stress are detected at the same time, the algorithm will suggest taking a two-minute break and taking deep breaths to reduce stress. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0632] Step 8:

[0633] server

[0634] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Rest your eyes" is sent with additional advice such as "Take deep breaths to reduce stress."

[0635] Step 9:

[0636] Device (smart glasses)

[0637] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is sent to the user. For example, the smart glasses may provide an audio notification saying "Please rest your eyes" and a visual notification saying "Your stress level is high, so please take a deep breath."

[0638] Step 10:

[0639] User

[0640] Users receive notifications from their smart glasses and take suggested actions (e.g., take a two-minute break and take a deep breath), allowing them to rest their eyes at the right time, reduce stress, and prevent eye strain.

[0641] In this way, each step works in tandem, allowing users to autonomously manage their eye health and emotional state in real time.

[0642] Example 2

[0643] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0644] Conventional eye health management systems simply monitor the condition of the eyes and are unable to provide appropriate advice that takes into account the user's emotional state and stress level. Furthermore, these systems struggle to provide real-time analysis and action suggestions, limiting their effectiveness in situations where a rapid response is required. This has left the problems of eye fatigue, dryness, and stress unresolved, especially in environments where digital devices are used for long periods of time.

[0645] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0646] In this invention, the server includes a sensor means for collecting information on the user's eye movements, blinks, and gaze, an emotion analysis means for analyzing the user's facial expressions and voice tone to evaluate their emotional state, a transmission means for transmitting the collected data to the server, a data storage means for storing the received data, an analysis means for analyzing the received data using AI, and a means for generating and notifying the user of optimal action suggestions based on the analysis results. This makes it possible to comprehensively monitor not only the user's eye health but also their emotional state, enabling more accurate and prompt responses.

[0647] "Sensor means" refers to devices and technologies that collect information on a user's eye movements, blinks, and gaze in real time.

[0648] "Emotion analysis means" refers to technology and devices that analyze a user's facial expressions and tone of voice to assess the user's emotional state.

[0649] "Transmission means" refers to the communication technology and devices used to transmit collected data to the server.

[0650] "Data Storage Means" refers to a data storage system for long-term archiving of received data.

[0651] "Analysis means" refers to the technology and devices that use AI technology to analyze received data and determine the user's status and necessary actions.

[0652] The "action suggestion means" refers to a technology and device that generates optimal action suggestions for the user based on the analysis results and notifies the user of them.

[0653] "Notification means" refers to devices and technologies that notify the user of the generated action suggestions through sound, visual display, vibration, etc.

[0654] "User's eye movements" refers to eye movements such as the user moving their line of sight or blinking.

[0655] The "number of blinks" refers to the number of times the user blinks within a certain period of time.

[0656] "Gaze information" refers to data indicating the direction of the user's gaze and the object on which the user is focused.

[0657] "Emotional state" refers to the emotional state that a user is feeling, such as stress, joy, surprise, etc.

[0658] "AI technology" refers to technology that uses artificial intelligence to analyze data and derive optimal results, such as suggesting actions.

[0659] MODE FOR CARRYING OUT THE INVENTION

[0660] This invention is a system that comprehensively monitors a user's eye health and emotional state and provides appropriate action suggestions in real time. This system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means connecting these components.

[0661] Hardware and Software Used

[0662] Device (smart glasses)

[0663] The smart glasses include the following hardware:

[0664] camera

[0665] Infrared sensor

[0666] microphone

[0667] Bluetooth and Wi-Fi modules

[0668] These hardware devices are used to collect information on the user's eye movements, blinks, gaze, facial expressions, and vocal tones, and use Bluetooth or Wi-Fi as communication methods to transmit the data to a server in real time.

[0669] server

[0670] The following software and AI models are implemented on the server side:

[0671] Database system (e.g. MySQL)

[0672] AI analysis engine (e.g. TensorFlow, PyTorch)

[0673] Sentiment Analysis Engine

[0674] This software receives, stores, and analyzes data sent from the device and uses it to generate optimal action suggestions based on the user's state.

[0675] Specific examples of processing

[0676] Let's take the example of a user wearing smart glasses and working at a desk for a long period of time. At this time, the smart glasses' sensors (camera and infrared sensor) monitor the user's blink rate and eye movements. At the same time, the emotion engine analyzes the user's facial expressions and voice tone to evaluate their stress level. This data is sent from the smart glasses to a server via Wi-Fi every second.

[0677] The server receives the data and immediately stores it in a database. The stored data is then analyzed by an AI analysis engine to generate detailed information about the user's eye movements and emotional state. For example, if the analysis detects "fewer than five blinks in the past minute" and "high stress level," the server-side AI generates notification messages in JSON format with suggested actions, such as "rest your eyes" and "take deep breaths to reduce stress." These notifications are sent to the smart glasses in real time and notify the user via audio, visual display, or vibration.

[0678] Examples of prompt statements

[0679] Below are some example prompts to input to a generative AI model:

[0680] "Describe the design of a system that monitors a user's eye health and emotional state in real time and suggests appropriate actions. The system includes smart glasses, a server, and an emotion engine."

[0681] As described above, the system of the present invention comprehensively monitors eye health and emotional state and can suggest optimal actions in real time, thereby effectively reducing eye strain and stress caused by prolonged use of digital devices.

[0682] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0683] Step 1:

[0684] Data collection

[0685] Subject: Device (Smart Glasses)

[0686] Input: User's eye movements, blinks, gaze information, facial expressions, and voice tones

[0687] Processing: The smart glasses' built-in camera and infrared sensors collect real-time information on the user's eye movements, blinks, and gaze. The emotion engine analyzes the user's facial expressions and vocal tone to assess their emotional state.

[0688] Output: Collected blink count, gaze data, emotional state data

[0689] Step 2:

[0690] Data transmission

[0691] Subject: Device (Smart Glasses)

[0692] Input: Collected blink count, gaze data, emotional state data

[0693] Processing: The smart glasses send the collected data to the server at regular intervals (every second). Bluetooth or Wi-Fi is used as the communication method.

[0694] Output: Number of blinks, gaze data, and emotion data sent to the server

[0695] Step 3:

[0696] Data reception and storage

[0697] Subject: Server

[0698] Input: Number of blinks, gaze data, emotion data sent from smart glasses

[0699] Processing: The server waits for data on the receiving port and receives the data sent from the smart glasses. After receiving the data, it stores it in a database system (e.g., MySQL).

[0700] Output: Number of blinks, gaze data, and emotion data stored in a database

[0701] Step 4:

[0702] Data analysis

[0703] Subject: Server

[0704] Input: Number of blinks, gaze data, emotion data stored in the database

[0705] Processing: Analyze the stored data using an AI analysis engine (e.g., TensorFlow, PyTorch). The analysis includes blink count, eye movement, eye opening and closing patterns, and emotional data. Detect cases where the blink count is below average or where the emotional data indicates high stress.

[0706] Output: Analysis results (e.g., decreased blinking rate and high stress levels in the past 5 minutes)

[0707] Step 5:

[0708] Generate action suggestions

[0709] Subject: Server

[0710] Input: Analysis results (e.g., decreased blink rate and high stress levels over the past 5 minutes)

[0711] Processing: Based on the analysis results, generate optimal action suggestions for the user. For example, generate suggestions such as "Rest your eyes" and "Take deep breaths to reduce stress." The suggestions are generated as notification messages in JSON format.

[0712] Output: Action proposal notification message (JSON format)

[0713] Step 6:

[0714] proposal notification

[0715] Subject: Device (Smart Glasses)

[0716] Input: Notification message of suggested action sent from the server

[0717] Processing: The smart glasses receive the notification sent from the server. The notification is communicated to the user through audio, visual display, or vibration. For example, the smart glasses may notify the user by audio, "Please rest your eyes," and show "Take deep breaths to reduce stress" on the display.

[0718] Output: Notification of suggested actions to the user (audio, visual display, vibration)

[0719] Through these steps, the system can monitor the user's eye health and emotional state in real time and provide appropriate behavioral suggestions.

[0720] (Application example 2)

[0721] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] In today's world, prolonged use of digital devices can cause eye fatigue and increased stress when users shop comfortably in virtual stores. This can lead to disadvantages such as a poor shopping experience and reduced decision-making ability. This invention aims to reduce eye fatigue and stress and provide a comfortable shopping environment by analyzing the user's eye health and emotional state in real time and providing appropriate advice.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0724] In this invention, the server includes a sensor means for collecting data on the user's eyes, a transmission means for transmitting the collected data to the server, an analysis means for analyzing the received data, a notification means for informing the user of appropriate actions based on the analysis results, a means for analyzing the user's emotional state, a means for detecting shopping stress based on the emotional state, and a means for suggesting actions to the user to reduce shopping stress. This makes it possible to comprehensively evaluate the user's eye health and emotional state and achieve stress reduction in real time.

[0725] "Sensor means" refers to a device that collects data on the user's eyes, including sensors that detect gaze and blinking.

[0726] The "transmission means" is a function for transmitting collected data to a server using wireless communication or the like.

[0727] "Analysis means" refers to a program or algorithm that analyzes the data sent to the server and evaluates the user's eye health and emotional state.

[0728] The "notification means" is a function for notifying the user of appropriate actions based on the analysis results, and includes notifications by voice, visual display, vibration, etc.

[0729] The "means for analyzing emotional state" is a device or software that determines and analyzes the emotional state of the user from facial expressions and tone of voice.

[0730] The "means for detecting shopping stress" is a means for detecting stress from the user's emotional state and gaze data, etc., and evaluating the quality of the shopping experience.

[0731] The "means for suggesting an action" is a device or software that suggests appropriate actions (e.g., taking a break or a relaxation method) to the user based on the analysis results and evaluation results.

[0732] "Receiving in real time" means that the server receives the data almost simultaneously with the time it is sent.

[0733] "Generating in real time" means that the server immediately analyzes the data it receives and generates the analysis results.

[0734] "Real-time notification" means instantly notifying the user of the analysis results. It also includes comprehensively evaluating the user's eye health and emotional state and providing appropriate advice in real time.

[0735] The system that realizes this application example mainly consists of the following components: smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and a means of communication that connects these components.

[0736] Data collection

[0737] Device (smart glasses)

[0738] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect blink counts, gaze movements, and eye opening and closing. In addition, an emotion engine analyzes the user's facial expressions and vocal tone to assess the user's emotional state (e.g., stress, joy, surprise).

[0739] Data transmission

[0740] Device (smart glasses)

[0741] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. The collected data includes both information about eye movements and emotion data from the emotion engine.

[0742] Data reception and analysis

[0743] server

[0744] The server listens for and receives composite data sent from the smart glasses via a receiving port. The data is then stored in a database and analyzed by AI. The analysis includes the user's eye blink rate, eye movements, and eye opening / closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress.

[0745] Deciding on an action

[0746] server

[0747] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0748] notification

[0749] Device (smart glasses)

[0750] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, a voice notification saying "Please rest your eyes" may be followed by additional visual advice such as "Your stress level is high, so take a deep breath."

[0751] Specific examples

[0752] Suppose a user is shopping in a virtual store. Comparing products for a long time can cause eye strain and stress. The smartglasses' sensors monitor blinking and eye movements, while the emotion engine analyzes facial expressions and vocal tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smartglasses: "Please rest your eyes" and "Take a deep breath to reduce stress."

[0753] Prompt Sentence Examples

[0754] "Generate a data collection method to detect whether users are experiencing stress in a virtual store."

[0755] "Please propose an algorithm that derives appropriate advice from a user's gaze information and emotional data."

[0756] The system provides users with more accurate and appropriate advice by comprehensively monitoring and managing their eye health and emotional state, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[0757] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0758] Step 1:

[0759] Data collection

[0760] The device (smart glasses) is equipped with multiple sensors. These sensors collect information on the user's blink count, eye movement, and eye opening / closing in real time. In parallel with this, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate their emotional state (e.g., stress, joy, surprise). The input of the data collection is the eye movement and facial expression detected by the sensors, and the output is the blink count, eye movement data, and emotion evaluation data.

[0761] Step 2:

[0762] Data transmission

[0763] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The input for data transmission is the number of blinks, gaze data, and emotion data collected by the device, and the output is the transmission of this data to the server.

[0764] Step 3:

[0765] Data reception and storage

[0766] The server waits for the composite data sent from the smart glasses at the receiving port and acquires the received data. The acquired data is stored in a database. The input of the data reception and storage is the blink count, gaze data, and emotion data sent to the server, and the output is that these data are stored in the database.

[0767] Step 4:

[0768] Data analysis

[0769] The server-side analysis uses data stored in the database to analyze the user's eye health and emotional state. An AI algorithm is used for the analysis, evaluating blink count, eye movement, eye opening and closing patterns, emotional data, and other factors. In particular, it detects cases where the blink count is lower than average or emotional data indicates high stress. It also compares the results with past data. The input for the data analysis is all data stored in the database, and the output is the analysis results.

[0770] Step 5:

[0771] Generate action suggestions

[0772] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected simultaneously, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format. The input for action suggestion generation is the analysis results, and the output is the action suggestion message.

[0773] Step 6:

[0774] notification

[0775] The device (smart glasses) receives the notification sent from the server and prompts the user to take appropriate action. The notification is given by voice, visual display, or vibration. For example, a voice notification saying "Please rest your eyes" may be given, while additional advice such as "Your stress level is high, so take a deep breath" is displayed. The input of the notification is the action suggestion message, and its output is the notification to the user.

[0776] Through these steps, the system can comprehensively monitor and manage the user's eye health and emotional state, and provide accurate and appropriate advice to the user, allowing the user to enjoy a comfortable shopping experience in the virtual store.

[0777] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0778] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0779] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0780] [Third embodiment]

[0781] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0782] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0783] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0784] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0785] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0786] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0787] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0788] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0789] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0790] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0791] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0792] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0793] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these. The program processing of this system will be explained in detail below.

[0794] Data collection

[0795] Device (smart glasses)

[0796] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if the user blinks 20 times per minute, that information will be collected.

[0797] Data transmission

[0798] Device (smart glasses)

[0799] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. This process allows the data to reach the server in real time, enabling rapid analysis.

[0800] Data reception and analysis

[0801] server

[0802] The server has an open receiving port to receive data sent from the smart glasses. The received data is stored in a database and then analyzed by AI. This analysis includes the number of times the user blinks, their eye movements, and eye opening and closing patterns. For example, if the blinking frequency over the past five minutes is lower than average, it is determined that the eyes are dry.

[0803] Deciding on an action

[0804] server

[0805] The server-side AI calculates the appropriate action based on the analysis results. For example, if dry eyes are detected, a suggestion to "take a two-minute break" is generated. This suggestion is formatted as a notification message and sent to the smart glasses.

[0806] notification

[0807] Device (smart glasses)

[0808] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. The notification can be done through audio, visual alerts, or vibration. For example, the user may hear an audio notification saying, "Please rest your eyes."

[0809] Specific examples

[0810] Suppose a user is wearing smart glasses and working on a computer. At this time, the smart glasses' sensors monitor the number of times the user blinks and send the data to a server every second. The server analyzes the received data and detects that the user is only blinking five times per minute. The AI ​​suspects dry eyes and decides to take an action such as "take a two-minute break." The smart glasses receive this notification and play a voice message to the user saying, "Please rest your eyes." This series of processes allows the user to autonomously take a break from their eyes and prevent eye strain.

[0811] This system performs all steps in real time, from data collection to analysis and notification, allowing users to effectively manage their eye health and prevent eye strain caused by prolonged use of digital devices.

[0812] The processing flow will be explained below.

[0813] Step 1:

[0814] Device (smart glasses)

[0815] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[0816] Step 2:

[0817] Device (smart glasses)

[0818] The collected data is temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data is stored in the buffer.

[0819] Step 3:

[0820] Device (smart glasses)

[0821] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[0822] Step 4:

[0823] server

[0824] The server waits for data sent from the smart glasses via a receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data for the past minute is stored in the database.

[0825] Step 5:

[0826] server

[0827] The AI ​​algorithm on the server retrieves data from the database and analyzes it in real time. The analysis detects cases where the blink rate is lower than average or where eye movements are unnatural. For example, the AI ​​may detect that a user is blinking less than half the normal rate.

[0828] Step 6:

[0829] server

[0830] Based on the analysis results, the AI ​​algorithm generates the optimal action suggestion for the user. For example, if it determines that the user's eyes are dry, it will decide on an action suggesting a two-minute break. This suggestion is generated as a notification message in JSON format.

[0831] Step 7:

[0832] server

[0833] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Please rest your eyes" is sent.

[0834] Step 8:

[0835] Device (smart glasses)

[0836] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is given to the user. For example, a voice notification saying "Please rest your eyes" is played from the smart glasses.

[0837] Step 9:

[0838] User

[0839] The user receives a notification from the smart glasses and performs a suggested action (e.g., take a two-minute break), allowing the user to rest their eyes at the appropriate time and prevent eye strain.

[0840] In this way, each step works in conjunction with the others, allowing the user to autonomously manage their eye health in real time.

[0841] Example 1

[0842] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0843] This solution solves the problem of eye strain and dry eyes caused by prolonged use of digital devices, which reduces user productivity, and also addresses the lack of a system that monitors the user's eye condition in real time and advises them on appropriate break times.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0845] In this invention, the server includes a means for receiving and analyzing data in real time, a means for storing the received data in a database, and a means for analyzing the data using a generative AI model, thereby enabling the user's eye data to be analyzed in real time and the appropriate action to be taken to be taken.

[0846] A "sensor means" is a device or mechanism that collects data about a user's eyes.

[0847] "Transmitting means" refers to a device or protocol for transmitting collected data to a server.

[0848] "Analysis means" refers to a device or software for analyzing received data.

[0849] The "notification means" is a device or function for notifying the user of appropriate actions based on the analysis results.

[0850] "Server means" refers to the function or configuration of a server that receives and analyzes data in real time.

[0851] The "camera" is a photographic device that detects and records the user's eye movements and blinks.

[0852] An "infrared sensor" is a sensor that detects whether the eyes are open or closed and the direction of gaze.

[0853] A "database" is a storage device or system for storing received data.

[0854] A "generative AI model" is an artificial intelligence algorithm used to analyze received data.

[0855] The "means for measuring the number of blinks" is a device or function that quantitatively measures the number of blinks of the user's eyes.

[0856] "Real-time" is a concept that refers to the time frame in which data is collected, transmitted, received, analyzed, and notified without delay.

[0857] This invention is a system for managing a user's eye health and preventing eye strain. In a specific embodiment, this system includes a visual data collection device (smart glasses) worn by the user, a server for storing and analyzing data, and communication means for connecting these.

[0858] Hardware and Software Usage

[0859] Device (smart glasses)

[0860] Smart glasses are equipped with a built-in camera and infrared sensors to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if a user blinks 20 times in one minute, this information will be collected immediately.

[0861] communication means

[0862] The collected visual data is sent to a server using wireless communication technologies such as Bluetooth or Wi-Fi. This data is packetized and sent at regular intervals (e.g., every second). The smart glasses temporarily store the data in a buffer memory before transmitting it and confirm that the transmission was successful.

[0863] server

[0864] The server opens a specific port to receive data from the smart glasses, which is then stored in a database and analyzed using a generative AI model.

[0865] Data processing and calculation

[0866] The server inputs the stored visual data into a generative AI model for analysis. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye open / close status. For example, if the blink rate over the past five minutes is lower than average, the server can suspect dry eyes.

[0867] Based on the analysis results, the server-side AI determines the appropriate action for the user (e.g., "Take a two-minute break," "Put in eye drops," etc.) This suggestion is formatted as a notification message and sent to the smart glasses in real time.

[0868] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. This notification can be audio, visual alert, or vibration. For example, a voice notification saying "Please rest your eyes" can be played through the smart glasses' speaker.

[0869] Specific examples

[0870] Consider a case where a user wears smart glasses while working on a computer. The smart glasses' sensors monitor the number of times the user blinks and send this data to a server every second. The server analyzes the received data and detects, for example, that the user is only blinking five times per minute. In this case, the server's AI suspects dry eyes and decides to take the action of "taking a two-minute break." The smart glasses receive this notification and play a voice notification to the user saying, "Please rest your eyes." This allows the user to autonomously take appropriate breaks and prevent eye strain.

[0871] Prompt Sentence Examples

[0872] Below is an example of a prompt sentence to input to the generative AI model.

[0873] "If a user blinks 50 times in an hour, I want the AI ​​model to analyze whether that number is high or low compared to the average and suggest an action."

[0874] This prompt allows you to simulate the process in which the AI ​​model suggests appropriate actions based on the results of data collection and analysis.

[0875] Based on the above specific embodiments, the present invention can manage the eye health of users and effectively prevent eye strain caused by using digital devices for long periods of time.

[0876] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0877] Step 1: Data collection

[0878] Device (smart glasses)

[0879] Smart glasses activate cameras and infrared sensors to detect the user's eye movements. While the user is using a digital device, these sensors detect blinking, eye movement, and eye opening / closing status in real time. For example, if the user blinks 20 times in one minute, that data will be collected.

[0880] Input: User's eye movements (blinks, gaze, eye open / closed state)

[0881] Output: Collected visual data (e.g., blink count per minute)

[0882] Step 2: Send data

[0883] Device (smart glasses)

[0884] The collected visual data is packetized at regular intervals (e.g., every second) and sent to a server using Bluetooth or Wi-Fi. The smart glasses temporarily store the data in a buffer memory and confirm that the transmission was successful. This process ensures that the data reaches the server without interruption.

[0885] Input: Collected visual data

[0886] Output: Packetized data

[0887] Step 3: Receiving and storing data

[0888] server

[0889] The server opens a specific port to receive data sent from the smart glasses. The received data is stored in a database in real time and used for later analysis. For example, JSON format data is parsed and stored in a database.

[0890] Input: Packetized data

[0891] Output: Data stored in the database

[0892] Step 4: Data analysis

[0893] server

[0894] The server uses a generative AI model to analyze the visual data stored in the database. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye opening / closing status. For example, if the blink frequency is lower than average based on the past five minutes of data, the server will output a result suggesting possible dry eyes.

[0895] Input: Visual data retrieved from a database

[0896] Output: Analysis results (e.g., possibility of dry eyes)

[0897] Step 5: Decide on an action

[0898] server

[0899] The server-side AI then uses the analysis results to suggest appropriate actions, such as "take a two-minute break" or "put in eye drops." This suggestion is formatted as a notification message and sent to the smart glasses.

[0900] Input: Analysis results

[0901] Output: Notification message

[0902] Step 6: Notification

[0903] Device (smart glasses)

[0904] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be delivered using audio, visual alerts, or vibrations. For example, a speaker on the smart glasses might say, "Please rest your eyes."

[0905] Input: Notification message

[0906] Output: Notification to the user (e.g., audio notification)

[0907] These steps allow the system to manage the user's eye health in real time and prevent eye strain.

[0908] (Application example 1)

[0909] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0910] Modern factories are becoming increasingly automated, with many processes now being carried out by robots, but visual monitoring by workers remains important. In particular, it is necessary to quickly detect abnormalities in the robots and changes in the work environment and take appropriate action. However, this places a heavy burden on workers, making it difficult to monitor efficiently. Furthermore, managing the workers' own health is also important, and it is necessary to prevent eye fatigue caused by long hours of work.

[0911] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0912] In this invention, the server includes a means for monitoring the work environment and equipment status, a means for analyzing collected data, and a means for notifying workers based on the analysis results. This allows for real-time analysis of gaze data transmitted from the smart glasses worn by workers, enabling the detection of abnormalities and the prompt provision of appropriate instructions. Furthermore, by simultaneously monitoring the fatigue and dryness of workers' eyes and encouraging them to take breaks at appropriate times, it is possible to achieve both work efficiency and health management.

[0913] "Sensor means" refers to a device for collecting data on the user's eyes.

[0914] The "transmitting means" is a communication device for transmitting collected data to a server.

[0915] "Analysis means" is a system for analyzing received data.

[0916] The "notification means" is a device for notifying the user of appropriate actions based on the analysis results.

[0917] "Work environment" refers to the location and conditions in which a user performs work, and includes factors related to safety and efficiency.

[0918] "Equipment status" refers to the operating status and normal operation of devices and robots used in work environments such as factories.

[0919] "Gaze data" is information about the movement of the user's eyes, and is data that indicates the direction and concentration of the gaze.

[0920] A "warning" is a notification or signal that notifies the user when an abnormality or danger occurs.

[0921] "Instructions" are information that informs workers of the actions and procedures they should take.

[0922] "Real-time" refers to data processing and communication occurring immediately and without delay.

[0923] "Abnormal" refers to a state in which something deviates from normal conditions or behavior and causes a problem.

[0924] "Analysis results" are information and judgments obtained as a result of analysis based on collected data.

[0925] "Rest prompt" is an action that notifies the user to take a rest at an appropriate time.

[0926] To implement this invention, we will build the following system, which consists of smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these.

[0927] Data collection

[0928] Device (smart glasses):

[0929] Smart glasses are equipped with sensors that detect the user's gaze data, blink rate, and eye movement. These sensors use cameras and infrared sensors to collect data in real time. For example, when a user focuses their gaze on a specific device, that gaze data is recorded.

[0930] Data transmission

[0931] Device (smart glasses):

[0932] The collected data is sent to the server every second using wireless communication technologies such as Bluetooth or Wi-Fi, allowing the data to be delivered to the server in real time.

[0933] Data reception and analysis

[0934] server:

[0935] The server receives data sent from the smart glasses via a TCP / UDP port. The received data is stored in a NoSQL database (e.g., MongoDB) and then analyzed using AI. AI frameworks such as TensorFlow and PyTorch are used for the analysis. For example, anomalies in specific devices can be detected from gaze data, and an abnormality warning can be generated based on the results.

[0936] Deciding on an action

[0937] server:

[0938] The server-side AI calculates appropriate actions based on the analysis results. For example, if an abnormality is detected based on gaze data, an instruction such as "Please inspect this robot" is immediately generated.

[0939] notification

[0940] Device (smart glasses):

[0941] The smart glasses receive notifications from the server and warn workers using audio, visual, or vibration alerts. For example, if an abnormality is detected, a voice notification will sound saying, "Anomaly detected. Please inspect."

[0942] Specific examples

[0943] Let's say Worker A is wearing smart glasses at a factory. When Worker A focuses his gaze on a specific robot for a long period of time, that gaze data is sent to a server in real time. The server analyzes the gaze data and detects that there may be an abnormality in that specific robot. An instruction to "Please inspect this robot" is immediately sent to the smart glasses, and Worker A is notified by voice.

[0944] Prompt Sentence Examples

[0945] prompt:

[0946] "Design an application that allows factory workers wearing smart glasses to collect gaze data in real time and then have it analyzed on a server to provide important information about the status of robots and the work environment. This includes the smart glasses sensors, data transmission, server-side analysis, and notification processes."

[0947] This system allows workers to safely and efficiently monitor their work environment, enabling early detection of abnormalities and prompt response.It also allows for simultaneous management of workers' eye health, achieving both work efficiency and health management.

[0948] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0949] Step 1:

[0950] Data collection

[0951] The camera and infrared sensor installed in the device (smart glasses) collect the user's gaze data, blink frequency, and eye movement. The sensor captures data every 100 milliseconds and measures the gaze direction and concentration. The input is the user's gaze and blink behavior, and the output is the captured gaze data.

[0952] Step 2:

[0953] Data transmission

[0954] The device (smart glasses) transmits the collected gaze data to the server via Wi-Fi or Bluetooth. This transmission occurs every second, and the data is delivered to the server in real time. The input is the gaze data collected in step 1, and the output is the data transmitted to the server.

[0955] Step 3:

[0956] Data reception

[0957] The server receives gaze data sent from the smart glasses using a TCP / UDP port. The received data is temporarily cached for subsequent data processing. The input is the data sent from the device, and the output is the cache of the received data.

[0958] Step 4:

[0959] Data analysis

[0960] The server stores the received gaze data in a NoSQL database (e.g., MongoDB) and then analyzes the data using an AI framework (e.g., TensorFlow, PyTorch). The analysis detects abnormalities in specific devices and changes in the work environment from gaze movements. The input is the received gaze data, and the output is the analysis results.

[0961] Step 5:

[0962] Deciding on an action

[0963] The server calculates appropriate actions based on the results of the AI ​​analysis and notifies the worker. For example, if an abnormality is detected in the robot, it generates an instruction such as "Please inspect this robot." The input is the analysis result from step 4, and the output is the generated instruction.

[0964] Step 6:

[0965] notification

[0966] The device (smart glasses) receives the notification sent from the server and notifies the worker using a voice notification, visual alert, or vibration notification. For example, when an abnormality is detected, a voice notification saying "Anomaly detected, please inspect" is played. The input is the notification content from step 5, and the output is the notification to the user.

[0967] The processing flow and specific operations at each step make it possible to use smart glasses to monitor work in real time and respond to abnormalities.

[0968] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0969] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means for connecting these components. The program processing of this system is explained in detail below.

[0970] Data collection

[0971] Device (smart glasses)

[0972] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect the number of blinks, gaze movements, and eye opening and closing. In addition, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate the user's emotional state (e.g., stress, joy, surprise). For example, if a user blinks 20 times per minute and feels stressed at the same time, this information will be collected and recorded.

[0973] Data transmission

[0974] Device (smart glasses)

[0975] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The collected data includes both information about eye movements and emotion data from the emotion engine. For example, eye blink data and stress level data are sent to the server simultaneously.

[0976] Data reception and analysis

[0977] server

[0978] The server waits for composite data sent from the smart glasses via a receiving port and retrieves the received data. The retrieved data is stored in a database and then analyzed by AI. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, if the blink rate over the past five minutes has decreased and a high stress level is detected at the same time, it is determined that the eyes are dry.

[0979] Deciding on an action

[0980] server

[0981] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[0982] notification

[0983] Device (smart glasses)

[0984] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, the user may be told to "rest their eyes" while additional advice such as "Your stress level is high, so take a deep breath" is displayed.

[0985] Specific examples

[0986] Suppose a user wears smart glasses and works at a desk for a long period of time. At this time, the smart glasses' sensors monitor the number of blinks and eye movements, while the emotion engine simultaneously analyzes facial expressions and voice tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smart glasses: "Please rest your eyes" and "Take deep breaths to reduce stress." This series of processes allows users to maintain eye health and manage stress.

[0987] By integrating the monitoring and management of eye health and emotional state, the system can provide users with more accurate and appropriate advice, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[0988] The processing flow will be explained below.

[0989] Step 1:

[0990] Device (smart glasses)

[0991] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[0992] Step 2:

[0993] Device (smart glasses)

[0994] The smart glasses' emotion engine analyzes the user's facial expressions and vocal tone, allowing it to assess the user's emotions (e.g., stress, joy, surprise) in real time. For example, it can detect high stress levels from the user's facial expressions.

[0995] Step 3:

[0996] Device (smart glasses)

[0997] The collected eye data and emotion data are temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data and emotion data are stored in the buffer.

[0998] Step 4:

[0999] Device (smart glasses)

[1000] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[1001] Step 5:

[1002] server

[1003] The server waits for the composite data sent from the smart glasses via the receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data and emotion data for the past minute are stored in the database.

[1004] Step 6:

[1005] server

[1006] The server's AI algorithm retrieves data from the database and analyzes it in real time. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, the AI ​​detects that the user's blink rate is less than half the normal rate and that their stress level is very high.

[1007] Step 7:

[1008] server

[1009] Based on the analysis results, the AI ​​algorithm generates optimal action suggestions for the user. For example, if dry eyes and high stress are detected at the same time, the algorithm will suggest taking a two-minute break and taking deep breaths to reduce stress. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[1010] Step 8:

[1011] server

[1012] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Rest your eyes" is sent with additional advice such as "Take deep breaths to reduce stress."

[1013] Step 9:

[1014] Device (smart glasses)

[1015] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is sent to the user. For example, the smart glasses may provide an audio notification saying "Please rest your eyes" and a visual notification saying "Your stress level is high, so please take a deep breath."

[1016] Step 10:

[1017] User

[1018] Users receive notifications from their smart glasses and take suggested actions (e.g., take a two-minute break and take a deep breath), allowing them to rest their eyes at the right time, reduce stress, and prevent eye strain.

[1019] In this way, each step works in tandem, allowing users to autonomously manage their eye health and emotional state in real time.

[1020] Example 2

[1021] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1022] Conventional eye health management systems simply monitor the condition of the eyes and are unable to provide appropriate advice that takes into account the user's emotional state and stress level. Furthermore, these systems struggle to provide real-time analysis and action suggestions, limiting their effectiveness in situations where a rapid response is required. This has left the problems of eye fatigue, dryness, and stress unresolved, especially in environments where digital devices are used for long periods of time.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1024] In this invention, the server includes a sensor means for collecting information on the user's eye movements, blinks, and gaze, an emotion analysis means for analyzing the user's facial expressions and voice tone to evaluate their emotional state, a transmission means for transmitting the collected data to the server, a data storage means for storing the received data, an analysis means for analyzing the received data using AI, and a means for generating and notifying the user of optimal action suggestions based on the analysis results. This makes it possible to comprehensively monitor not only the user's eye health but also their emotional state, enabling more accurate and prompt responses.

[1025] "Sensor means" refers to devices and technologies that collect information on a user's eye movements, blinks, and gaze in real time.

[1026] "Emotion analysis means" refers to technology and devices that analyze a user's facial expressions and tone of voice to assess the user's emotional state.

[1027] "Transmission means" refers to the communication technology and devices used to transmit collected data to the server.

[1028] "Data Storage Means" refers to a data storage system for long-term archiving of received data.

[1029] "Analysis means" refers to the technology and devices that use AI technology to analyze received data and determine the user's status and necessary actions.

[1030] The "action suggestion means" refers to a technology and device that generates optimal action suggestions for the user based on the analysis results and notifies the user of them.

[1031] "Notification means" refers to devices and technologies that notify the user of the generated action suggestions through sound, visual display, vibration, etc.

[1032] "User's eye movements" refers to eye movements such as the user moving their line of sight or blinking.

[1033] The "number of blinks" refers to the number of times the user blinks within a certain period of time.

[1034] "Gaze information" refers to data indicating the direction of the user's gaze and the object on which the user is focused.

[1035] "Emotional state" refers to the emotional state that a user is feeling, such as stress, joy, surprise, etc.

[1036] "AI technology" refers to technology that uses artificial intelligence to analyze data and derive optimal results, such as suggesting actions.

[1037] MODE FOR CARRYING OUT THE INVENTION

[1038] This invention is a system that comprehensively monitors a user's eye health and emotional state and provides appropriate action suggestions in real time. This system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means connecting these components.

[1039] Hardware and Software Used

[1040] Device (smart glasses)

[1041] The smart glasses include the following hardware:

[1042] camera

[1043] Infrared sensor

[1044] microphone

[1045] Bluetooth and Wi-Fi modules

[1046] These hardware devices are used to collect information on the user's eye movements, blinks, gaze, facial expressions, and vocal tones, and use Bluetooth or Wi-Fi as communication methods to transmit the data to a server in real time.

[1047] server

[1048] The following software and AI models are implemented on the server side:

[1049] Database system (e.g. MySQL)

[1050] AI analysis engine (e.g. TensorFlow, PyTorch)

[1051] Sentiment Analysis Engine

[1052] This software receives, stores, and analyzes data sent from the device and uses it to generate optimal action suggestions based on the user's state.

[1053] Specific examples of processing

[1054] Let's take the example of a user wearing smart glasses and working at a desk for a long period of time. At this time, the smart glasses' sensors (camera and infrared sensor) monitor the user's blink rate and eye movements. At the same time, the emotion engine analyzes the user's facial expressions and voice tone to evaluate their stress level. This data is sent from the smart glasses to a server via Wi-Fi every second.

[1055] The server receives the data and immediately stores it in a database. The stored data is then analyzed by an AI analysis engine to generate detailed information about the user's eye movements and emotional state. For example, if the analysis detects "fewer than five blinks in the past minute" and "high stress level," the server-side AI generates notification messages in JSON format with suggested actions, such as "rest your eyes" and "take deep breaths to reduce stress." These notifications are sent to the smart glasses in real time and notify the user via audio, visual display, or vibration.

[1056] Examples of prompt statements

[1057] Below are some example prompts to input to a generative AI model:

[1058] "Describe the design of a system that monitors a user's eye health and emotional state in real time and suggests appropriate actions. The system includes smart glasses, a server, and an emotion engine."

[1059] As described above, the system of the present invention comprehensively monitors eye health and emotional state and can suggest optimal actions in real time, thereby effectively reducing eye strain and stress caused by prolonged use of digital devices.

[1060] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1061] Step 1:

[1062] Data collection

[1063] Subject: Device (Smart Glasses)

[1064] Input: User's eye movements, blinks, gaze information, facial expressions, and voice tones

[1065] Processing: The smart glasses' built-in camera and infrared sensors collect real-time information on the user's eye movements, blinks, and gaze. The emotion engine analyzes the user's facial expressions and vocal tone to assess their emotional state.

[1066] Output: Collected blink count, gaze data, emotional state data

[1067] Step 2:

[1068] Data transmission

[1069] Subject: Device (Smart Glasses)

[1070] Input: Collected blink count, gaze data, emotional state data

[1071] Processing: The smart glasses send the collected data to the server at regular intervals (every second). Bluetooth or Wi-Fi is used as the communication method.

[1072] Output: Number of blinks, gaze data, and emotion data sent to the server

[1073] Step 3:

[1074] Data reception and storage

[1075] Subject: Server

[1076] Input: Number of blinks, gaze data, emotion data sent from smart glasses

[1077] Processing: The server waits for data on the receiving port and receives the data sent from the smart glasses. After receiving the data, it stores it in a database system (e.g., MySQL).

[1078] Output: Number of blinks, gaze data, and emotion data stored in a database

[1079] Step 4:

[1080] Data analysis

[1081] Subject: Server

[1082] Input: Number of blinks, gaze data, emotion data stored in the database

[1083] Processing: Analyze the stored data using an AI analysis engine (e.g., TensorFlow, PyTorch). The analysis includes blink count, eye movement, eye opening and closing patterns, and emotional data. Detect cases where the blink count is below average or where the emotional data indicates high stress.

[1084] Output: Analysis results (e.g., decreased blinking rate and high stress levels in the past 5 minutes)

[1085] Step 5:

[1086] Generate action suggestions

[1087] Subject: Server

[1088] Input: Analysis results (e.g., decreased blink rate and high stress levels over the past 5 minutes)

[1089] Processing: Based on the analysis results, generate optimal action suggestions for the user. For example, generate suggestions such as "Rest your eyes" and "Take deep breaths to reduce stress." The suggestions are generated as notification messages in JSON format.

[1090] Output: Action proposal notification message (JSON format)

[1091] Step 6:

[1092] proposal notification

[1093] Subject: Device (Smart Glasses)

[1094] Input: Notification message of suggested action sent from the server

[1095] Processing: The smart glasses receive the notification sent from the server. The notification is communicated to the user through audio, visual display, or vibration. For example, the smart glasses may notify the user by audio, "Please rest your eyes," and show "Take deep breaths to reduce stress" on the display.

[1096] Output: Notification of suggested actions to the user (audio, visual display, vibration)

[1097] Through these steps, the system can monitor the user's eye health and emotional state in real time and provide appropriate behavioral suggestions.

[1098] (Application example 2)

[1099] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1100] In today's world, prolonged use of digital devices can cause eye fatigue and increased stress when users shop comfortably in virtual stores. This can lead to disadvantages such as a poor shopping experience and reduced decision-making ability. This invention aims to reduce eye fatigue and stress and provide a comfortable shopping environment by analyzing the user's eye health and emotional state in real time and providing appropriate advice.

[1101] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1102] In this invention, the server includes a sensor means for collecting data on the user's eyes, a transmission means for transmitting the collected data to the server, an analysis means for analyzing the received data, a notification means for informing the user of appropriate actions based on the analysis results, a means for analyzing the user's emotional state, a means for detecting shopping stress based on the emotional state, and a means for suggesting actions to the user to reduce shopping stress. This makes it possible to comprehensively evaluate the user's eye health and emotional state and achieve stress reduction in real time.

[1103] "Sensor means" refers to a device that collects data on the user's eyes, including sensors that detect gaze and blinking.

[1104] The "transmission means" is a function for transmitting collected data to a server using wireless communication or the like.

[1105] "Analysis means" refers to a program or algorithm that analyzes the data sent to the server and evaluates the user's eye health and emotional state.

[1106] The "notification means" is a function for notifying the user of appropriate actions based on the analysis results, and includes notifications by voice, visual display, vibration, etc.

[1107] The "means for analyzing emotional state" is a device or software that determines and analyzes the emotional state of the user from facial expressions and tone of voice.

[1108] The "means for detecting shopping stress" is a means for detecting stress from the user's emotional state and gaze data, etc., and evaluating the quality of the shopping experience.

[1109] The "means for suggesting an action" is a device or software that suggests appropriate actions (e.g., taking a break or a relaxation method) to the user based on the analysis results and evaluation results.

[1110] "Receiving in real time" means that the server receives the data almost simultaneously with the time it is sent.

[1111] "Generating in real time" means that the server immediately analyzes the data it receives and generates the analysis results.

[1112] "Real-time notification" means instantly notifying the user of the analysis results. It also includes comprehensively evaluating the user's eye health and emotional state and providing appropriate advice in real time.

[1113] The system that realizes this application example mainly consists of the following components: smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and a means of communication that connects these components.

[1114] Data collection

[1115] Device (smart glasses)

[1116] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect blink counts, gaze movements, and eye opening and closing. In addition, an emotion engine analyzes the user's facial expressions and vocal tone to assess the user's emotional state (e.g., stress, joy, surprise).

[1117] Data transmission

[1118] Device (smart glasses)

[1119] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. The collected data includes both information about eye movements and emotion data from the emotion engine.

[1120] Data reception and analysis

[1121] server

[1122] The server listens for and receives composite data sent from the smart glasses via a receiving port. The data is then stored in a database and analyzed by AI. The analysis includes the user's eye blink rate, eye movements, and eye opening / closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress.

[1123] Deciding on an action

[1124] server

[1125] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[1126] notification

[1127] Device (smart glasses)

[1128] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, a voice notification saying "Please rest your eyes" may be followed by additional visual advice such as "Your stress level is high, so take a deep breath."

[1129] Specific examples

[1130] Suppose a user is shopping in a virtual store. Comparing products for a long time can cause eye strain and stress. The smartglasses' sensors monitor blinking and eye movements, while the emotion engine analyzes facial expressions and vocal tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smartglasses: "Please rest your eyes" and "Take a deep breath to reduce stress."

[1131] Prompt Sentence Examples

[1132] "Generate a data collection method to detect whether users are experiencing stress in a virtual store."

[1133] "Please propose an algorithm that derives appropriate advice from a user's gaze information and emotional data."

[1134] The system provides users with more accurate and appropriate advice by comprehensively monitoring and managing their eye health and emotional state, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[1135] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1136] Step 1:

[1137] Data collection

[1138] The device (smart glasses) is equipped with multiple sensors. These sensors collect information on the user's blink count, eye movement, and eye opening / closing in real time. In parallel with this, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate their emotional state (e.g., stress, joy, surprise). The input of the data collection is the eye movement and facial expression detected by the sensors, and the output is the blink count, eye movement data, and emotion evaluation data.

[1139] Step 2:

[1140] Data transmission

[1141] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The input for data transmission is the number of blinks, gaze data, and emotion data collected by the device, and the output is the transmission of this data to the server.

[1142] Step 3:

[1143] Data reception and storage

[1144] The server waits for the composite data sent from the smart glasses at the receiving port and acquires the received data. The acquired data is stored in a database. The input of the data reception and storage is the blink count, gaze data, and emotion data sent to the server, and the output is that these data are stored in the database.

[1145] Step 4:

[1146] Data analysis

[1147] The server-side analysis uses data stored in the database to analyze the user's eye health and emotional state. An AI algorithm is used for the analysis, evaluating blink count, eye movement, eye opening and closing patterns, emotional data, and other factors. In particular, it detects cases where the blink count is lower than average or emotional data indicates high stress. It also compares the results with past data. The input for the data analysis is all data stored in the database, and the output is the analysis results.

[1148] Step 5:

[1149] Generate action suggestions

[1150] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected simultaneously, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format. The input for action suggestion generation is the analysis results, and the output is the action suggestion message.

[1151] Step 6:

[1152] notification

[1153] The device (smart glasses) receives the notification sent from the server and prompts the user to take appropriate action. The notification is given by voice, visual display, or vibration. For example, a voice notification saying "Please rest your eyes" may be given, while additional advice such as "Your stress level is high, so take a deep breath" is displayed. The input of the notification is the action suggestion message, and its output is the notification to the user.

[1154] Through these steps, the system can comprehensively monitor and manage the user's eye health and emotional state, and provide accurate and appropriate advice to the user, allowing the user to enjoy a comfortable shopping experience in the virtual store.

[1155] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1156] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1157] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1158] [Fourth embodiment]

[1159] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1160] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1161] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1162] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1163] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1164] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1165] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1166] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1167] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1168] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1169] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1170] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1171] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1172] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these. The program processing of this system will be explained in detail below.

[1173] Data collection

[1174] Device (smart glasses)

[1175] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if the user blinks 20 times per minute, that information will be collected.

[1176] Data transmission

[1177] Device (smart glasses)

[1178] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. This process allows the data to reach the server in real time, enabling rapid analysis.

[1179] Data reception and analysis

[1180] server

[1181] The server has an open receiving port to receive data sent from the smart glasses. The received data is stored in a database and then analyzed by AI. This analysis includes the number of times the user blinks, their eye movements, and eye opening and closing patterns. For example, if the blinking frequency over the past five minutes is lower than average, it is determined that the eyes are dry.

[1182] Deciding on an action

[1183] server

[1184] The server-side AI calculates the appropriate action based on the analysis results. For example, if dry eyes are detected, a suggestion to "take a two-minute break" is generated. This suggestion is formatted as a notification message and sent to the smart glasses.

[1185] notification

[1186] Device (smart glasses)

[1187] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. The notification can be done through audio, visual alerts, or vibration. For example, the user may hear an audio notification saying, "Please rest your eyes."

[1188] Specific examples

[1189] Suppose a user is wearing smart glasses and working on a computer. At this time, the smart glasses' sensors monitor the number of times the user blinks and send the data to a server every second. The server analyzes the received data and detects that the user is only blinking five times per minute. The AI ​​suspects dry eyes and decides to take an action such as "take a two-minute break." The smart glasses receive this notification and play a voice message to the user saying, "Please rest your eyes." This series of processes allows the user to autonomously take a break from their eyes and prevent eye strain.

[1190] This system performs all steps in real time, from data collection to analysis and notification, allowing users to effectively manage their eye health and prevent eye strain caused by prolonged use of digital devices.

[1191] The processing flow will be explained below.

[1192] Step 1:

[1193] Device (smart glasses)

[1194] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[1195] Step 2:

[1196] Device (smart glasses)

[1197] The collected data is temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data is stored in the buffer.

[1198] Step 3:

[1199] Device (smart glasses)

[1200] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[1201] Step 4:

[1202] server

[1203] The server waits for data sent from the smart glasses via a receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data for the past minute is stored in the database.

[1204] Step 5:

[1205] server

[1206] The AI ​​algorithm on the server retrieves data from the database and analyzes it in real time. The analysis detects cases where the blink rate is lower than average or where eye movements are unnatural. For example, the AI ​​may detect that a user is blinking less than half the normal rate.

[1207] Step 6:

[1208] server

[1209] Based on the analysis results, the AI ​​algorithm generates the optimal action suggestion for the user. For example, if it determines that the user's eyes are dry, it will decide on an action suggesting a two-minute break. This suggestion is generated as a notification message in JSON format.

[1210] Step 7:

[1211] server

[1212] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Please rest your eyes" is sent.

[1213] Step 8:

[1214] Device (smart glasses)

[1215] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is given to the user. For example, a voice notification saying "Please rest your eyes" is played from the smart glasses.

[1216] Step 9:

[1217] User

[1218] The user receives a notification from the smart glasses and performs a suggested action (e.g., take a two-minute break), allowing the user to rest their eyes at the appropriate time and prevent eye strain.

[1219] In this way, each step works in conjunction with the others, allowing the user to autonomously manage their eye health in real time.

[1220] Example 1

[1221] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] This solution solves the problem of eye strain and dry eyes caused by prolonged use of digital devices, which reduces user productivity, and also addresses the lack of a system that monitors the user's eye condition in real time and advises them on appropriate break times.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1224] In this invention, the server includes a means for receiving and analyzing data in real time, a means for storing the received data in a database, and a means for analyzing the data using a generative AI model, thereby enabling the user's eye data to be analyzed in real time and the appropriate action to be taken to be taken.

[1225] A "sensor means" is a device or mechanism that collects data about a user's eyes.

[1226] "Transmitting means" refers to a device or protocol for transmitting collected data to a server.

[1227] "Analysis means" refers to a device or software for analyzing received data.

[1228] The "notification means" is a device or function for notifying the user of appropriate actions based on the analysis results.

[1229] "Server means" refers to the function or configuration of a server that receives and analyzes data in real time.

[1230] The "camera" is a photographic device that detects and records the user's eye movements and blinks.

[1231] An "infrared sensor" is a sensor that detects whether the eyes are open or closed and the direction of gaze.

[1232] A "database" is a storage device or system for storing received data.

[1233] A "generative AI model" is an artificial intelligence algorithm used to analyze received data.

[1234] The "means for measuring the number of blinks" is a device or function that quantitatively measures the number of blinks of the user's eyes.

[1235] "Real-time" is a concept that refers to the time frame in which data is collected, transmitted, received, analyzed, and notified without delay.

[1236] This invention is a system for managing a user's eye health and preventing eye strain. In a specific embodiment, this system includes a visual data collection device (smart glasses) worn by the user, a server for storing and analyzing data, and communication means for connecting these.

[1237] Hardware and Software Usage

[1238] Device (smart glasses)

[1239] Smart glasses are equipped with a built-in camera and infrared sensors to detect the user's eye movements. These sensors collect visual data in real time, such as the user's blinking, eye movement, and eye opening and closing. For example, if a user blinks 20 times in one minute, this information will be collected immediately.

[1240] communication means

[1241] The collected visual data is sent to a server using wireless communication technologies such as Bluetooth or Wi-Fi. This data is packetized and sent at regular intervals (e.g., every second). The smart glasses temporarily store the data in a buffer memory before transmitting it and confirm that the transmission was successful.

[1242] server

[1243] The server opens a specific port to receive data from the smart glasses, which is then stored in a database and analyzed using a generative AI model.

[1244] Data processing and calculation

[1245] The server inputs the stored visual data into a generative AI model for analysis. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye open / close status. For example, if the blink rate over the past five minutes is lower than average, the server can suspect dry eyes.

[1246] Based on the analysis results, the server-side AI determines the appropriate action for the user (e.g., "Take a two-minute break," "Put in eye drops," etc.) This suggestion is formatted as a notification message and sent to the smart glasses in real time.

[1247] The smart glasses receive the notification sent from the server and prompt the user to take appropriate action. This notification can be audio, visual alert, or vibration. For example, a voice notification saying "Please rest your eyes" can be played through the smart glasses' speaker.

[1248] Specific examples

[1249] Consider a case where a user wears smart glasses while working on a computer. The smart glasses' sensors monitor the number of times the user blinks and send this data to a server every second. The server analyzes the received data and detects, for example, that the user is only blinking five times per minute. In this case, the server's AI suspects dry eyes and decides to take the action of "taking a two-minute break." The smart glasses receive this notification and play a voice notification to the user saying, "Please rest your eyes." This allows the user to autonomously take appropriate breaks and prevent eye strain.

[1250] Prompt Sentence Examples

[1251] Below is an example of a prompt sentence to input to the generative AI model.

[1252] "If a user blinks 50 times in an hour, I want the AI ​​model to analyze whether that number is high or low compared to the average and suggest an action."

[1253] This prompt allows you to simulate the process in which the AI ​​model suggests appropriate actions based on the results of data collection and analysis.

[1254] Based on the above specific embodiments, the present invention can manage the eye health of users and effectively prevent eye strain caused by using digital devices for long periods of time.

[1255] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1256] Step 1: Data collection

[1257] Device (smart glasses)

[1258] Smart glasses activate cameras and infrared sensors to detect the user's eye movements. While the user is using a digital device, these sensors detect blinking, eye movement, and eye opening / closing status in real time. For example, if the user blinks 20 times in one minute, that data will be collected.

[1259] Input: User's eye movements (blinks, gaze, eye open / closed state)

[1260] Output: Collected visual data (e.g., blink count per minute)

[1261] Step 2: Send data

[1262] Device (smart glasses)

[1263] The collected visual data is packetized at regular intervals (e.g., every second) and sent to a server using Bluetooth or Wi-Fi. The smart glasses temporarily store the data in a buffer memory and confirm that the transmission was successful. This process ensures that the data reaches the server without interruption.

[1264] Input: Collected visual data

[1265] Output: Packetized data

[1266] Step 3: Receiving and storing data

[1267] server

[1268] The server opens a specific port to receive data sent from the smart glasses. The received data is stored in a database in real time and used for later analysis. For example, JSON format data is parsed and stored in a database.

[1269] Input: Packetized data

[1270] Output: Data stored in the database

[1271] Step 4: Data analysis

[1272] server

[1273] The server uses a generative AI model to analyze the visual data stored in the database. The analysis process evaluates the user's blink frequency, eye movement patterns, and eye opening / closing status. For example, if the blink frequency is lower than average based on the past five minutes of data, the server will output a result suggesting possible dry eyes.

[1274] Input: Visual data retrieved from a database

[1275] Output: Analysis results (e.g., possibility of dry eyes)

[1276] Step 5: Decide on an action

[1277] server

[1278] The server-side AI then uses the analysis results to suggest appropriate actions, such as "take a two-minute break" or "put in eye drops." This suggestion is formatted as a notification message and sent to the smart glasses.

[1279] Input: Analysis results

[1280] Output: Notification message

[1281] Step 6: Notification

[1282] Device (smart glasses)

[1283] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be delivered using audio, visual alerts, or vibrations. For example, a speaker on the smart glasses might say, "Please rest your eyes."

[1284] Input: Notification message

[1285] Output: Notification to the user (e.g., audio notification)

[1286] These steps allow the system to manage the user's eye health in real time and prevent eye strain.

[1287] (Application example 1)

[1288] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] Modern factories are becoming increasingly automated, with many processes now being carried out by robots, but visual monitoring by workers remains important. In particular, it is necessary to quickly detect abnormalities in the robots and changes in the work environment and take appropriate action. However, this places a heavy burden on workers, making it difficult to monitor efficiently. Furthermore, managing the workers' own health is also important, and it is necessary to prevent eye fatigue caused by long hours of work.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1291] In this invention, the server includes a means for monitoring the work environment and equipment status, a means for analyzing collected data, and a means for notifying workers based on the analysis results. This allows for real-time analysis of gaze data transmitted from the smart glasses worn by workers, enabling the detection of abnormalities and the prompt provision of appropriate instructions. Furthermore, by simultaneously monitoring the fatigue and dryness of workers' eyes and encouraging them to take breaks at appropriate times, it is possible to achieve both work efficiency and health management.

[1292] "Sensor means" refers to a device for collecting data on the user's eyes.

[1293] The "transmitting means" is a communication device for transmitting collected data to a server.

[1294] "Analysis means" is a system for analyzing received data.

[1295] The "notification means" is a device for notifying the user of appropriate actions based on the analysis results.

[1296] "Work environment" refers to the location and conditions in which a user performs work, and includes factors related to safety and efficiency.

[1297] "Equipment status" refers to the operating status and normal operation of devices and robots used in work environments such as factories.

[1298] "Gaze data" is information about the movement of the user's eyes, and is data that indicates the direction and concentration of the gaze.

[1299] A "warning" is a notification or signal that notifies the user when an abnormality or danger occurs.

[1300] "Instructions" are information that informs workers of the actions and procedures they should take.

[1301] "Real-time" refers to data processing and communication occurring immediately and without delay.

[1302] "Abnormal" refers to a state in which something deviates from normal conditions or behavior and causes a problem.

[1303] "Analysis results" are information and judgments obtained as a result of analysis based on collected data.

[1304] "Rest prompt" is an action that notifies the user to take a rest at an appropriate time.

[1305] To implement this invention, we will build the following system, which consists of smart glasses worn by the user, a server for storing and analyzing data, and a communication means for connecting these.

[1306] Data collection

[1307] Device (smart glasses):

[1308] Smart glasses are equipped with sensors that detect the user's gaze data, blink rate, and eye movement. These sensors use cameras and infrared sensors to collect data in real time. For example, when a user focuses their gaze on a specific device, that gaze data is recorded.

[1309] Data transmission

[1310] Device (smart glasses):

[1311] The collected data is sent to the server every second using wireless communication technologies such as Bluetooth or Wi-Fi, allowing the data to be delivered to the server in real time.

[1312] Data reception and analysis

[1313] server:

[1314] The server receives data sent from the smart glasses via a TCP / UDP port. The received data is stored in a NoSQL database (e.g., MongoDB) and then analyzed using AI. AI frameworks such as TensorFlow and PyTorch are used for the analysis. For example, anomalies in specific devices can be detected from gaze data, and an abnormality warning can be generated based on the results.

[1315] Deciding on an action

[1316] server:

[1317] The server-side AI calculates appropriate actions based on the analysis results. For example, if an abnormality is detected based on gaze data, an instruction such as "Please inspect this robot" is immediately generated.

[1318] notification

[1319] Device (smart glasses):

[1320] The smart glasses receive notifications from the server and warn workers using audio, visual, or vibration alerts. For example, if an abnormality is detected, a voice notification will sound saying, "Anomaly detected. Please inspect."

[1321] Specific examples

[1322] Let's say Worker A is wearing smart glasses at a factory. When Worker A focuses his gaze on a specific robot for a long period of time, that gaze data is sent to a server in real time. The server analyzes the gaze data and detects that there may be an abnormality in that specific robot. An instruction to "Please inspect this robot" is immediately sent to the smart glasses, and Worker A is notified by voice.

[1323] Prompt Sentence Examples

[1324] prompt:

[1325] "Design an application that allows factory workers wearing smart glasses to collect gaze data in real time and then have it analyzed on a server to provide important information about the status of robots and the work environment. This includes the smart glasses sensors, data transmission, server-side analysis, and notification processes."

[1326] This system allows workers to safely and efficiently monitor their work environment, enabling early detection of abnormalities and prompt response.It also allows for simultaneous management of workers' eye health, achieving both work efficiency and health management.

[1327] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1328] Step 1:

[1329] Data collection

[1330] The camera and infrared sensor installed in the device (smart glasses) collect the user's gaze data, blink frequency, and eye movement. The sensor captures data every 100 milliseconds and measures the gaze direction and concentration. The input is the user's gaze and blink behavior, and the output is the captured gaze data.

[1331] Step 2:

[1332] Data transmission

[1333] The device (smart glasses) transmits the collected gaze data to the server via Wi-Fi or Bluetooth. This transmission occurs every second, and the data is delivered to the server in real time. The input is the gaze data collected in step 1, and the output is the data transmitted to the server.

[1334] Step 3:

[1335] Data reception

[1336] The server receives gaze data sent from the smart glasses using a TCP / UDP port. The received data is temporarily cached for subsequent data processing. The input is the data sent from the device, and the output is the cache of the received data.

[1337] Step 4:

[1338] Data analysis

[1339] The server stores the received gaze data in a NoSQL database (e.g., MongoDB) and then analyzes the data using an AI framework (e.g., TensorFlow, PyTorch). The analysis detects abnormalities in specific devices and changes in the work environment from gaze movements. The input is the received gaze data, and the output is the analysis results.

[1340] Step 5:

[1341] Deciding on an action

[1342] The server calculates appropriate actions based on the results of the AI ​​analysis and notifies the worker. For example, if an abnormality is detected in the robot, it generates an instruction such as "Please inspect this robot." The input is the analysis result from step 4, and the output is the generated instruction.

[1343] Step 6:

[1344] notification

[1345] The device (smart glasses) receives the notification sent from the server and notifies the worker using a voice notification, visual alert, or vibration notification. For example, when an abnormality is detected, a voice notification saying "Anomaly detected, please inspect" is played. The input is the notification content from step 5, and the output is the notification to the user.

[1346] The processing flow and specific operations at each step make it possible to use smart glasses to monitor work in real time and respond to abnormalities.

[1347] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1348] As a specific embodiment, this system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means for connecting these components. The program processing of this system is explained in detail below.

[1349] Data collection

[1350] Device (smart glasses)

[1351] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect the number of blinks, gaze movements, and eye opening and closing. In addition, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate the user's emotional state (e.g., stress, joy, surprise). For example, if a user blinks 20 times per minute and feels stressed at the same time, this information will be collected and recorded.

[1352] Data transmission

[1353] Device (smart glasses)

[1354] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The collected data includes both information about eye movements and emotion data from the emotion engine. For example, eye blink data and stress level data are sent to the server simultaneously.

[1355] Data reception and analysis

[1356] server

[1357] The server waits for composite data sent from the smart glasses via a receiving port and retrieves the received data. The retrieved data is stored in a database and then analyzed by AI. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, if the blink rate over the past five minutes has decreased and a high stress level is detected at the same time, it is determined that the eyes are dry.

[1358] Deciding on an action

[1359] server

[1360] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[1361] notification

[1362] Device (smart glasses)

[1363] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, the user may be told to "rest their eyes" while additional advice such as "Your stress level is high, so take a deep breath" is displayed.

[1364] Specific examples

[1365] Suppose a user wears smart glasses and works at a desk for a long period of time. At this time, the smart glasses' sensors monitor the number of blinks and eye movements, while the emotion engine simultaneously analyzes facial expressions and voice tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smart glasses: "Please rest your eyes" and "Take deep breaths to reduce stress." This series of processes allows users to maintain eye health and manage stress.

[1366] By integrating the monitoring and management of eye health and emotional state, the system can provide users with more accurate and appropriate advice, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[1367] The processing flow will be explained below.

[1368] Step 1:

[1369] Device (smart glasses)

[1370] The sensors in the smart glasses detect the user's eye movements in real time, including blink frequency, eye movement, and eye open / closed status. For example, the sensors record the user blinking twice per second.

[1371] Step 2:

[1372] Device (smart glasses)

[1373] The smart glasses' emotion engine analyzes the user's facial expressions and vocal tone, allowing it to assess the user's emotions (e.g., stress, joy, surprise) in real time. For example, it can detect high stress levels from the user's facial expressions.

[1374] Step 3:

[1375] Device (smart glasses)

[1376] The collected eye data and emotion data are temporarily stored in a buffer. The buffer holds the latest data for a certain period of time and updates its contents every second. For example, the most recent 5 seconds of blink data and emotion data are stored in the buffer.

[1377] Step 4:

[1378] Device (smart glasses)

[1379] The data accumulated in the buffer is sent to the server at regular intervals (e.g., every second). This is done using wireless communication methods such as Wi-Fi or Bluetooth. The data is sent to the server in an appropriate format (e.g., JSON format).

[1380] Step 5:

[1381] server

[1382] The server waits for the composite data sent from the smart glasses via the receiving port and acquires the received data. The acquired data is stored in a database. For example, blink data and emotion data for the past minute are stored in the database.

[1383] Step 6:

[1384] server

[1385] The server's AI algorithm retrieves data from the database and analyzes it in real time. The analysis includes the user's blink rate, eye movements, eye opening and closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress. For example, the AI ​​detects that the user's blink rate is less than half the normal rate and that their stress level is very high.

[1386] Step 7:

[1387] server

[1388] Based on the analysis results, the AI ​​algorithm generates optimal action suggestions for the user. For example, if dry eyes and high stress are detected at the same time, the algorithm will suggest taking a two-minute break and taking deep breaths to reduce stress. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[1389] Step 8:

[1390] server

[1391] The generated notification message is sent to the smart glasses using a secure communication protocol such as HTTP or HTTPS. For example, a suggestion message such as "Rest your eyes" is sent with additional advice such as "Take deep breaths to reduce stress."

[1392] Step 9:

[1393] Device (smart glasses)

[1394] The smart glasses receive the notification message sent from the server. The received message is analyzed and an appropriate notification is sent to the user. For example, the smart glasses may provide an audio notification saying "Please rest your eyes" and a visual notification saying "Your stress level is high, so please take a deep breath."

[1395] Step 10:

[1396] User

[1397] Users receive notifications from their smart glasses and take suggested actions (e.g., take a two-minute break and take a deep breath), allowing them to rest their eyes at the right time, reduce stress, and prevent eye strain.

[1398] In this way, each step works in tandem, allowing users to autonomously manage their eye health and emotional state in real time.

[1399] Example 2

[1400] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1401] Conventional eye health management systems simply monitor the condition of the eyes and are unable to provide appropriate advice that takes into account the user's emotional state and stress level. Furthermore, these systems struggle to provide real-time analysis and action suggestions, limiting their effectiveness in situations where a rapid response is required. This has left the problems of eye fatigue, dryness, and stress unresolved, especially in environments where digital devices are used for long periods of time.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1403] In this invention, the server includes a sensor means for collecting information on the user's eye movements, blinks, and gaze, an emotion analysis means for analyzing the user's facial expressions and voice tone to evaluate their emotional state, a transmission means for transmitting the collected data to the server, a data storage means for storing the received data, an analysis means for analyzing the received data using AI, and a means for generating and notifying the user of optimal action suggestions based on the analysis results. This makes it possible to comprehensively monitor not only the user's eye health but also their emotional state, enabling more accurate and prompt responses.

[1404] "Sensor means" refers to devices and technologies that collect information on a user's eye movements, blinks, and gaze in real time.

[1405] "Emotion analysis means" refers to technology and devices that analyze a user's facial expressions and tone of voice to assess the user's emotional state.

[1406] "Transmission means" refers to the communication technology and devices used to transmit collected data to the server.

[1407] "Data Storage Means" refers to a data storage system for long-term archiving of received data.

[1408] "Analysis means" refers to the technology and devices that use AI technology to analyze received data and determine the user's status and necessary actions.

[1409] The "action suggestion means" refers to a technology and device that generates optimal action suggestions for the user based on the analysis results and notifies the user of them.

[1410] "Notification means" refers to devices and technologies that notify the user of the generated action suggestions through sound, visual display, vibration, etc.

[1411] "User's eye movements" refers to eye movements such as the user moving their line of sight or blinking.

[1412] The "number of blinks" refers to the number of times the user blinks within a certain period of time.

[1413] "Gaze information" refers to data indicating the direction of the user's gaze and the object on which the user is focused.

[1414] "Emotional state" refers to the emotional state that a user is feeling, such as stress, joy, surprise, etc.

[1415] "AI technology" refers to technology that uses artificial intelligence to analyze data and derive optimal results, such as suggesting actions.

[1416] MODE FOR CARRYING OUT THE INVENTION

[1417] This invention is a system that comprehensively monitors a user's eye health and emotional state and provides appropriate action suggestions in real time. This system includes smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and communication means connecting these components.

[1418] Hardware and Software Used

[1419] Device (smart glasses)

[1420] The smart glasses include the following hardware:

[1421] camera

[1422] Infrared sensor

[1423] microphone

[1424] Bluetooth and Wi-Fi modules

[1425] These hardware devices are used to collect information on the user's eye movements, blinks, gaze, facial expressions, and vocal tones, and use Bluetooth or Wi-Fi as communication methods to transmit the data to a server in real time.

[1426] server

[1427] The following software and AI models are implemented on the server side:

[1428] Database system (e.g. MySQL)

[1429] AI analysis engine (e.g. TensorFlow, PyTorch)

[1430] Sentiment Analysis Engine

[1431] This software receives, stores, and analyzes data sent from the device and uses it to generate optimal action suggestions based on the user's state.

[1432] Specific examples of processing

[1433] Let's take the example of a user wearing smart glasses and working at a desk for a long period of time. At this time, the smart glasses' sensors (camera and infrared sensor) monitor the user's blink rate and eye movements. At the same time, the emotion engine analyzes the user's facial expressions and voice tone to evaluate their stress level. This data is sent from the smart glasses to a server via Wi-Fi every second.

[1434] The server receives the data and immediately stores it in a database. The stored data is then analyzed by an AI analysis engine to generate detailed information about the user's eye movements and emotional state. For example, if the analysis detects "fewer than five blinks in the past minute" and "high stress level," the server-side AI generates notification messages in JSON format with suggested actions, such as "rest your eyes" and "take deep breaths to reduce stress." These notifications are sent to the smart glasses in real time and notify the user via audio, visual display, or vibration.

[1435] Examples of prompt statements

[1436] Below are some example prompts to input to a generative AI model:

[1437] "Describe the design of a system that monitors a user's eye health and emotional state in real time and suggests appropriate actions. The system includes smart glasses, a server, and an emotion engine."

[1438] As described above, the system of the present invention comprehensively monitors eye health and emotional state and can suggest optimal actions in real time, thereby effectively reducing eye strain and stress caused by prolonged use of digital devices.

[1439] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1440] Step 1:

[1441] Data collection

[1442] Subject: Device (Smart Glasses)

[1443] Input: User's eye movements, blinks, gaze information, facial expressions, and voice tones

[1444] Processing: The smart glasses' built-in camera and infrared sensors collect real-time information on the user's eye movements, blinks, and gaze. The emotion engine analyzes the user's facial expressions and vocal tone to assess their emotional state.

[1445] Output: Collected blink count, gaze data, emotional state data

[1446] Step 2:

[1447] Data transmission

[1448] Subject: Device (Smart Glasses)

[1449] Input: Collected blink count, gaze data, emotional state data

[1450] Processing: The smart glasses send the collected data to the server at regular intervals (every second). Bluetooth or Wi-Fi is used as the communication method.

[1451] Output: Number of blinks, gaze data, and emotion data sent to the server

[1452] Step 3:

[1453] Data reception and storage

[1454] Subject: Server

[1455] Input: Number of blinks, gaze data, emotion data sent from smart glasses

[1456] Processing: The server waits for data on the receiving port and receives the data sent from the smart glasses. After receiving the data, it stores it in a database system (e.g., MySQL).

[1457] Output: Number of blinks, gaze data, and emotion data stored in a database

[1458] Step 4:

[1459] Data analysis

[1460] Subject: Server

[1461] Input: Number of blinks, gaze data, emotion data stored in the database

[1462] Processing: Analyze the stored data using an AI analysis engine (e.g., TensorFlow, PyTorch). The analysis includes blink count, eye movement, eye opening and closing patterns, and emotional data. Detect cases where the blink count is below average or where the emotional data indicates high stress.

[1463] Output: Analysis results (e.g., decreased blinking rate and high stress levels in the past 5 minutes)

[1464] Step 5:

[1465] Generate action suggestions

[1466] Subject: Server

[1467] Input: Analysis results (e.g., decreased blink rate and high stress levels over the past 5 minutes)

[1468] Processing: Based on the analysis results, generate optimal action suggestions for the user. For example, generate suggestions such as "Rest your eyes" and "Take deep breaths to reduce stress." The suggestions are generated as notification messages in JSON format.

[1469] Output: Action proposal notification message (JSON format)

[1470] Step 6:

[1471] proposal notification

[1472] Subject: Device (Smart Glasses)

[1473] Input: Notification message of suggested action sent from the server

[1474] Processing: The smart glasses receive the notification sent from the server. The notification is communicated to the user through audio, visual display, or vibration. For example, the smart glasses may notify the user by audio, "Please rest your eyes," and show "Take deep breaths to reduce stress" on the display.

[1475] Output: Notification of suggested actions to the user (audio, visual display, vibration)

[1476] Through these steps, the system can monitor the user's eye health and emotional state in real time and provide appropriate behavioral suggestions.

[1477] (Application example 2)

[1478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1479] In today's world, prolonged use of digital devices can cause eye fatigue and increased stress when users shop comfortably in virtual stores. This can lead to disadvantages such as a poor shopping experience and reduced decision-making ability. This invention aims to reduce eye fatigue and stress and provide a comfortable shopping environment by analyzing the user's eye health and emotional state in real time and providing appropriate advice.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1481] In this invention, the server includes a sensor means for collecting data on the user's eyes, a transmission means for transmitting the collected data to the server, an analysis means for analyzing the received data, a notification means for informing the user of appropriate actions based on the analysis results, a means for analyzing the user's emotional state, a means for detecting shopping stress based on the emotional state, and a means for suggesting actions to the user to reduce shopping stress. This makes it possible to comprehensively evaluate the user's eye health and emotional state and achieve stress reduction in real time.

[1482] "Sensor means" refers to a device that collects data on the user's eyes, including sensors that detect gaze and blinking.

[1483] The "transmission means" is a function for transmitting collected data to a server using wireless communication or the like.

[1484] "Analysis means" refers to a program or algorithm that analyzes the data sent to the server and evaluates the user's eye health and emotional state.

[1485] The "notification means" is a function for notifying the user of appropriate actions based on the analysis results, and includes notifications by voice, visual display, vibration, etc.

[1486] The "means for analyzing emotional state" is a device or software that determines and analyzes the emotional state of the user from facial expressions and tone of voice.

[1487] The "means for detecting shopping stress" is a means for detecting stress from the user's emotional state and gaze data, etc., and evaluating the quality of the shopping experience.

[1488] The "means for suggesting an action" is a device or software that suggests appropriate actions (e.g., taking a break or a relaxation method) to the user based on the analysis results and evaluation results.

[1489] "Receiving in real time" means that the server receives the data almost simultaneously with the time it is sent.

[1490] "Generating in real time" means that the server immediately analyzes the data it receives and generates the analysis results.

[1491] "Real-time notification" means instantly notifying the user of the analysis results. It also includes comprehensively evaluating the user's eye health and emotional state and providing appropriate advice in real time.

[1492] The system that realizes this application example mainly consists of the following components: smart glasses worn by the user, a server for storing and analyzing data, an emotion engine, and a means of communication that connects these components.

[1493] Data collection

[1494] Device (smart glasses)

[1495] Smart glasses are equipped with multiple sensors (e.g., cameras and infrared sensors) to collect real-time information on the user's eye movements, blinks, and gaze. These sensors detect blink counts, gaze movements, and eye opening and closing. In addition, an emotion engine analyzes the user's facial expressions and vocal tone to assess the user's emotional state (e.g., stress, joy, surprise).

[1496] Data transmission

[1497] Device (smart glasses)

[1498] The collected data is sent to a server at regular intervals (e.g., every second) using wireless communication technologies such as Bluetooth or Wi-Fi. The collected data includes both information about eye movements and emotion data from the emotion engine.

[1499] Data reception and analysis

[1500] server

[1501] The server listens for and receives composite data sent from the smart glasses via a receiving port. The data is then stored in a database and analyzed by AI. The analysis includes the user's eye blink rate, eye movements, and eye opening / closing patterns, as well as emotional data from the emotion engine. In particular, it detects cases where the blink rate is lower than average or where the emotional data indicates high stress.

[1502] Deciding on an action

[1503] server

[1504] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected at the same time, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format and sent to the smart glasses.

[1505] notification

[1506] Device (smart glasses)

[1507] The smart glasses receive notifications sent from the server and prompt the user to take appropriate action. Notifications can be audio, visual, or vibration. For example, a voice notification saying "Please rest your eyes" may be followed by additional visual advice such as "Your stress level is high, so take a deep breath."

[1508] Specific examples

[1509] Suppose a user is shopping in a virtual store. Comparing products for a long time can cause eye strain and stress. The smartglasses' sensors monitor blinking and eye movements, while the emotion engine analyzes facial expressions and vocal tones to assess stress levels. This data is sent to a server every second. The server analyzes the data, and if it detects a blink rate of less than five times per minute and a high stress level, it sends a double notification to the user via the smartglasses: "Please rest your eyes" and "Take a deep breath to reduce stress."

[1510] Prompt Sentence Examples

[1511] "Generate a data collection method to detect whether users are experiencing stress in a virtual store."

[1512] "Please propose an algorithm that derives appropriate advice from a user's gaze information and emotional data."

[1513] The system provides users with more accurate and appropriate advice by comprehensively monitoring and managing their eye health and emotional state, thereby simultaneously reducing eye strain and stress caused by prolonged use of digital devices.

[1514] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1515] Step 1:

[1516] Data collection

[1517] The device (smart glasses) is equipped with multiple sensors. These sensors collect information on the user's blink count, eye movement, and eye opening / closing in real time. In parallel with this, the emotion engine analyzes the user's facial expressions and vocal tone to evaluate their emotional state (e.g., stress, joy, surprise). The input of the data collection is the eye movement and facial expression detected by the sensors, and the output is the blink count, eye movement data, and emotion evaluation data.

[1518] Step 2:

[1519] Data transmission

[1520] The collected data is sent to a server at regular intervals (e.g., every second). Wireless communication technologies such as Bluetooth and Wi-Fi are used for transmission. The input for data transmission is the number of blinks, gaze data, and emotion data collected by the device, and the output is the transmission of this data to the server.

[1521] Step 3:

[1522] Data reception and storage

[1523] The server waits for the composite data sent from the smart glasses at the receiving port and acquires the received data. The acquired data is stored in a database. The input of the data reception and storage is the blink count, gaze data, and emotion data sent to the server, and the output is that these data are stored in the database.

[1524] Step 4:

[1525] Data analysis

[1526] The server-side analysis uses data stored in the database to analyze the user's eye health and emotional state. An AI algorithm is used for the analysis, evaluating blink count, eye movement, eye opening and closing patterns, emotional data, and other factors. In particular, it detects cases where the blink count is lower than average or emotional data indicates high stress. It also compares the results with past data. The input for the data analysis is all data stored in the database, and the output is the analysis results.

[1527] Step 5:

[1528] Generate action suggestions

[1529] The server-side AI generates optimal action suggestions for the user based on the analysis results. For example, if dry eyes and high stress are detected simultaneously, a suggestion to take a two-minute break will be generated. This suggestion is generated as a notification message in JSON format. The input for action suggestion generation is the analysis results, and the output is the action suggestion message.

[1530] Step 6:

[1531] notification

[1532] The device (smart glasses) receives the notification sent from the server and prompts the user to take appropriate action. The notification is given by voice, visual display, or vibration. For example, a voice notification saying "Please rest your eyes" may be given, while additional advice such as "Your stress level is high, so take a deep breath" is displayed. The input of the notification is the action suggestion message, and its output is the notification to the user.

[1533] Through these steps, the system can comprehensively monitor and manage the user's eye health and emotional state, and provide accurate and appropriate advice to the user, allowing the user to enjoy a comfortable shopping experience in the virtual store.

[1534] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1535] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1536] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1537] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1538] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1539] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1540] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1541] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1542] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1543] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1544] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1545] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1546] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1547] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1548] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1549] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1550] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1551] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1552] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1553] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1554] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1555] The following is further disclosed regarding the above embodiment.

[1556] (Claim 1)

[1557] sensor means for collecting eye data of a user;

[1558] a transmitting means for transmitting the collected data to a server;

[1559] analysis means for analyzing the received data;

[1560] a notification means for notifying a user of an appropriate action based on the analysis result;

[1561] A system including:

[1562] (Claim 2)

[1563] means for measuring the blink rate of the user's eyes;

[1564] means for the sensor means to continuously monitor the blink rate;

[1565] a means for determining eye fatigue or dryness based on the number of blinks by the analyzing means;

[1566] means for informing the user to take a break based on the determination result;

[1567] 10. The system of claim 1, comprising:

[1568] (Claim 3)

[1569] The server receives the data in real time;

[1570] The analysis means generates analysis results in real time,

[1571] The notification means notifies the user in real time.

[1572] 10. The system of claim 1.

[1573] "Example 1"

[1574] (Claim 1)

[1575] sensor means for collecting eye data of a user;

[1576] a transmitting means for transmitting the collected data to a server;

[1577] analysis means for analyzing the received data;

[1578] a notification means for notifying a user of an appropriate action based on the analysis result;

[1579] a server means for receiving and analyzing data in real time;

[1580] A system including:

[1581] (Claim 2)

[1582] means for measuring the blink rate of the user's eyes;

[1583] means for the sensor means to continuously monitor the blink rate;

[1584] a means for determining eye fatigue or dryness based on the number of blinks by the analyzing means;

[1585] means for informing the user to take a break based on the determination result;

[1586] A means of detecting blinks and eye movements using cameras and infrared sensors,

[1587] 10. The system of claim 1, comprising:

[1588] (Claim 3)

[1589] The server receives the data in real time;

[1590] The analysis means generates analysis results in real time,

[1591] The notification means notifies the user in real time.

[1592] The received data is stored in a database.

[1593] including means for analyzing the data with a generative AI model;

[1594] 10. The system of claim 1.

[1595] "Application Example 1"

[1596] (Claim 1)

[1597] sensor means for collecting eye data of a user;

[1598] a transmitting means for transmitting the collected data to a server;

[1599] analysis means for analyzing the received data;

[1600] a notification means for notifying a user of an appropriate action based on the analysis result;

[1601] A means of monitoring the working environment and equipment conditions;

[1602] a means of providing warnings and instructions to workers;

[1603] A system including:

[1604] (Claim 2)

[1605] means for measuring the blink rate of the user's eyes;

[1606] means for the sensor means to continuously monitor the blink rate;

[1607] a means for determining eye fatigue or dryness based on the number of blinks by the analyzing means;

[1608] means for informing the user to take a break based on the determination result;

[1609] A means for analyzing the gaze data of the worker;

[1610] A means of immediate notification if an abnormality is detected in a specific piece of equipment or work area,

[1611] 10. The system of claim 1, comprising:

[1612] (Claim 3)

[1613] The server receives the data in real time;

[1614] The analysis means generates analysis results in real time,

[1615] The notification means notifies the user in real time.

[1616] It is characterized by analyzing and notifying abnormalities in specific work areas in real time.

[1617] 10. The system of claim 1.

[1618] "Example 2: Combining Emotion Engines"

[1619] (Claim 1)

[1620] A sensor means for collecting information on the user's eye movements, blinks, and gaze;

[1621] emotion analysis means for analyzing a user's facial expression and / or vocal tone to assess the user's emotional state;

[1622] a transmitting means for transmitting the collected data to a server;

[1623] a data storage means for storing the received data;

[1624] An analysis method for analyzing received data using AI;

[1625] A means for generating and notifying an optimal action proposal for the user based on the analysis result;

[1626] A system including:

[1627] (Claim 2)

[1628] means for measuring the blink rate of the user's eyes;

[1629] means for the sensor means to continuously monitor the blink rate;

[1630] means for determining eye fatigue or dryness based on the blink count and data from the emotion analysis means;

[1631] means for informing the user to take a break based on the determination result;

[1632] 10. The system of claim 1, comprising:

[1633] (Claim 3)

[1634] The server receives the data in real time;

[1635] The analysis means generates analysis results in real time,

[1636] The notification means notifies the user in real time.

[1637] 10. The system of claim 1.

[1638] "Application example 2 when combining emotion engines"

[1639] (Claim 1)

[1640] sensor means for collecting eye data of a user;

[1641] a transmitting means for transmitting the collected data to a server;

[1642] analysis means for analyzing the received data;

[1643] a notification means for notifying a user of an appropriate action based on the analysis result;

[1644] means for analyzing the emotional state of a user;

[1645] means for detecting shopping stress based on emotional state;

[1646] A means for suggesting actions to the user to reduce shopping stress;

[1647] A system including:

[1648] (Claim 2)

[1649] means for measuring the blink rate of the user's eyes;

[1650] means for the sensor means to continuously monitor the blink rate;

[1651] a means for determining eye fatigue or dryness based on the number of blinks by the analyzing means;

[1652] means for informing the user to take a break based on the determination result;

[1653] a transmitting means for transmitting the user's emotion data to a server;

[1654] a means for comprehensively evaluating the eye health state and emotional state of a user based on emotion data and the number of blinks;

[1655] a means for suggesting a method for reducing stress in virtual shopping to the user based on the evaluation result;

[1656] 10. The system of claim 1, comprising:

[1657] (Claim 3)

[1658] The server receives the data in real time;

[1659] The analysis means generates analysis results in real time,

[1660] The notification means notifies the user in real time.

[1661] The evaluation means analyzes the user's eye health and emotional state in real time;

[1662] The system is characterized by suggesting stress reduction methods in real time when a user's stress level increases.

[1663] 10. The system of claim 1. [Explanation of symbols]

[1664] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. sensor means for collecting eye data of a user; a transmitting means for transmitting the collected data to a server; analysis means for analyzing the received data; a notification means for notifying a user of an appropriate action based on the analysis result; A system including:

2. means for measuring the blink rate of the user's eyes; means for the sensor means to continuously monitor the blink rate; a means for determining eye fatigue or dryness based on the number of blinks by the analyzing means; means for informing the user to take a break based on the determination result; The system of claim 1 , comprising:

3. The server receives the data in real time; The analysis means generates analysis results in real time, The notification means notifies the user in real time. The system of claim 1 .

Images