system

Abstract

We provide the system. [Solution] A terminal that collects the user's biometric information in real time, A communication means for transmitting biometric information collected by the aforementioned terminal to a server, An analytical means for analyzing the aforementioned biological information and evaluating the health status, A generation means for generating advice that recommends specific actions to the user based on the aforementioned health status assessment, A transmission means for sending the generated advice to the user's terminal, A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, it is important to continuously monitor an individual's health status and perform appropriate health management in real time based on it. However, in conventional systems, it has been difficult to collect data in real time, analyze it, and provide more suitable specific behavioral advice. In addition, the mechanism for providing feedback on the effect of the generated advice and continuously improving the accuracy of the system based on it has not been sufficient. As a result, there has been a problem that specific behavioral proposals for improving and preventing the user's health are not effectively carried out.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a terminal that collects a user's biometric information in real time and transmits that information to a server. The server receives and analyzes this biometric information and evaluates the health status of each user. Based on the analysis results, a generative artificial intelligence generates specific health advice and transmits that advice back to the user's terminal. Furthermore, the system is constructed to improve the accuracy of future advice by receiving user feedback and allowing the server to learn the analysis means. Specifically, the system includes the following means.

[0006] 1. A device that collects users' biometric information in real time.

[0007] 2. Communication means for transmitting biometric information collected by the terminal to a server.

[0008] 3. Analytical means for analyzing the aforementioned biological information and evaluating the health status.

[0009] 4. A generation means for generating advice that recommends specific actions to the user based on the assessment of their health condition.

[0010] 5. A transmission means for sending the generated advice to the user's terminal.

[0011] 6. A learning means that receives user feedback based on the advice and improves the accuracy of the analysis means.

[0012] This allows users to constantly monitor their health status and receive support to take appropriate actions at any given time.

[0013] A "terminal" is a device worn by a user that collects biometric information in real time.

[0014] "Biometric information" refers to data about the user's physical condition, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0015] "Communication means" refers to a combination of a communication protocol and hardware for transmitting biometric information collected by a terminal to a server.

[0016] "Server" refers to a computer system that receives biometric information transmitted from a terminal and performs analysis and advice generation.

[0017] "Analysis means" refers to algorithms and software for analyzing biometric information received by a server using statistical methods and evaluating the health status of a user.

[0018] "Abnormal value" refers to a value that statistically deviates from the normal range among a user's biometric information.

[0019] "Generation means" refers to artificial intelligence or other generation algorithms for generating specific action advice based on the evaluation of the health status obtained by the analysis means.

[0020] "Transmission means" refers to a combination of a communication protocol and hardware for transmitting advice generated by a server back to a user's terminal.

[0021] "Feedback" refers to information that a user inputs the results and feelings after following the provided advice into a terminal and transmits to a server.

[0022] "Learning means" refers to a method and system for updating and improving the algorithms of the analysis means and the artificial intelligence model of the generation means based on the received user feedback.

Brief Explanation of Drawings

[0023] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3]It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example ② when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0024] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0025] First, the language used in the following description will be explained.

[0026] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0027] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0028] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0029] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0031] [First Embodiment]

[0032] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0033] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0036] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0039] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0043] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0044] This invention is implemented using a terminal that collects a user's biometric information in real time and transmits that information to a server, through the following six steps:

[0045] 1. Data Collection

[0046] Device: A wearable device worn by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. For example, a heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes.

[0047] 2. Data transmission

[0048] Terminal: Encrypts collected biometric information and transmits it to the server at regular intervals. Wireless communication methods such as Bluetooth and Wi-Fi are used for communication.

[0049] 3. Data reception and storage

[0050] Server: Receives biometric information transmitted from terminals and stores it in the database. This data is necessary for analysis, so it is registered with a timestamp when saved.

[0051] 4. Data Analysis

[0052] Server: To analyze biometric information stored in the database, it uses statistical methods to detect anomalies and evaluate the user's health status. This analysis employs techniques that combine historical and current data.

[0053] 5. Generating personalized advice

[0054] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT) to generate customized health advice. The generated advice is based on the user's past behavior and specific biometric information for that day.

[0055] 6. Sending advice and collecting feedback

[0056] Server: Sends generated advice to the user's terminal. The user receives the advice and acts accordingly. They also use their terminal to input feedback on the advice and send it to the server.

[0057] User: Follow the advice provided and provide feedback on the results and your impressions via your device.

[0058] Specific examples

[0059] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[0060] Sensors: Equipped with heart rate sensors, blood pressure sensors, etc., to measure biometric information in real time.

[0061] Communication module: Communicates with the server via Wi-Fi or Bluetooth.

[0062] At 9:00 AM, users begin wearing the device. The device measures heart rate every second and blood pressure every 30 minutes. The collected data is encrypted and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is analyzed in real time using Python's pandas library.

[0063] At 9:10 AM, the server detected that the user's heart rate was higher than normal. Based on this, the generative artificial intelligence generated the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sent this advice to the device, and the user received a notification on their device.

[0064] The user follows the advice and tries deep breathing. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and performed deep breathing, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[0065] This entire process allows users to constantly monitor their health status and receive real-time guidance on appropriate actions. Furthermore, the system can utilize user feedback to continuously improve the accuracy of its advice.

[0066] The following describes the processing flow.

[0067] Step 1:

[0068] Device: The user wears the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. An accelerometer detects the user's movements and records steps and activity levels.

[0069] Step 2:

[0070] Terminal: Collected biometric information is batch-processed at regular intervals (e.g., every 10 minutes), encrypted, and sent to the server. Bluetooth or Wi-Fi is used as the communication method, and SSL / TLS is applied to maintain data confidentiality.

[0071] Step 3:

[0072] Server: Receives encrypted data sent from terminals and decrypts it. The received data is stored in a database (e.g., MariaDB) with a timestamp. The data format is JSON or CSV.

[0073] Step 4:

[0074] Server: The stored data is analyzed using the Python pandas library. In particular, statistical methods (e.g., values ​​exceeding the mean ± 2 standard deviations) are used to detect abnormal heart rate and blood pressure. The analysis results are recorded in the database as an assessment of each user's health status.

[0075] Step 5:

[0076] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT-4®) to generate specific advice for the user. The advice is personalized according to the user's past behavioral history and current health status.

[0077] Step 6:

[0078] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol. After transmission, a log indicating success is recorded in the database.

[0079] Step 7:

[0080] Terminal: Decodes advice received from the server and notifies the user. Notification methods include push notifications on smartphones and displays on the terminal's screen. For example, it might display a message such as, "Your heart rate is high, please take a break."

[0081] Step 8:

[0082] User: Receives advice notifications from their device and acts accordingly. Later, they input their impressions and the effectiveness of the advice into a feedback form and send it from their device to the server.

[0083] Step 9:

[0084] Server: Receives feedback submitted by users and stores it in the database. The feedback is used to improve future analysis and advice generation.

[0085] Step 10:

[0086] Server: Analyzes received feedback and improves the algorithms of the analysis and generation methods. For example, it trains the model with successful advice and the conditions under which it provides it, thereby improving the accuracy of future advice.

[0087] (Example 1)

[0088] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Conventional health management systems struggle to collect users' biometric information in real time and provide accurate advice based on that information. Furthermore, they lack mechanisms to improve the accuracy of advice by incorporating user feedback into the system. Therefore, there is a challenge in continuously and appropriately managing users' health status.

[0090] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0091] In this invention, the server includes means for receiving and storing the user's biometric information in a database, means for analyzing the biometric information stored in the database and evaluating the user's health status, means for generating advice that recommends specific actions to the user based on the evaluation of the health status, means for transmitting the generated advice to the user's terminal, and learning means for receiving user feedback based on the advice and improving the accuracy of the analysis and generation means. This makes it possible to provide appropriate and personalized advice based on the real-time collection and analysis of the user's biometric information. Furthermore, by utilizing user feedback, the accuracy of the system can be improved, enabling continuous and accurate health management.

[0092] "User biometric information" refers to physiological data that indicates the user's health and activity level, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0093] A "terminal" refers to a device, such as a wearable device or smartphone worn by a user, that collects biometric information and transmits it to a server.

[0094] "Communication methods" refer to protocols and technologies for transmitting biometric information from a terminal to a server, specifically referring to wireless communication technologies such as Bluetooth and Wi-Fi.

[0095] "Receiving and storage means" refers to a system in which a server receives biometric information transmitted from a terminal and stores that data in a database.

[0096] A "database" is a data storage system that accumulates biometric information and retrieves and analyzes it as needed.

[0097] "Analysis means" refers to a system that analyzes biometric information stored in a database using statistical methods and other analytical methods to evaluate the user's health status.

[0098] A "generation method" is a system for generating advice that recommends appropriate actions to the user based on the analysis results, and it uses artificial intelligence (AI) models, etc.

[0099] "Transmission means" refers to the protocols and technologies used to send the generated advice to the user's terminal.

[0100] A "learning tool" is a system that receives feedback from users and improves the accuracy of the analysis and generation tools.

[0101] "Feedback" refers to information that users report to the server via their device, including actions taken in response to the advice provided, the results, and their impressions.

[0102] This invention is implemented using a terminal that collects the user's biometric information in real time and transmits that information to a server. This makes it possible to appropriately evaluate the user's health status and provide personalized advice.

[0103] Hardware and software to be used

[0104] hardware

[0105] Device: A wearable device worn by the user (e.g., fitness band or smartwatch). This device is equipped with a heart rate sensor and blood pressure sensor to collect biometric information in real time. It is equipped with either Bluetooth or Wi-Fi as a communication module.

[0106] Server: A computer system used for receiving, storing, and analyzing data. This server includes a database (e.g., MariaDB) and analysis software (e.g., Python, pandas library).

[0107] software

[0108] Database: Biometric information will be stored using a relational database such as MariaDB.

[0109] Analysis method: Biological information is analyzed using statistical methods with Python and the pandas library.

[0110] Generation method: Generative artificial intelligence (e.g., GPT model) is used to generate personalized advice for the user based on the analysis results.

[0111] Specific Examples of the System

[0112] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[0113] Sensors: Equipped with heart rate and blood pressure sensors, it measures biometric information in real time.

[0114] Communication module: Communicates with the server using Wi-Fi or Bluetooth.

[0115] At 9:00 AM, when the user puts on the device, it begins measuring heart rate every second and blood pressure every 30 minutes. The collected data is encrypted by the device and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is then analyzed in real time using Python's pandas library.

[0116] At 9:10 AM, the server detects that the user's heart rate is higher than normal. Based on this, the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sends this advice to the device, and the user receives a notification from the device.

[0117] The user follows the advice and attempts to take deep breaths. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and took deep breaths, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[0118] Example of a prompt

[0119] "Please describe a program that uses a wearable device to measure heart rate and blood pressure daily."

[0120] "Please provide a specific example of a system in which a user collects biometric information from a wearable device and transmits it to a server."

[0121] Please describe a system that has real-time data analysis and health advice generation capabilities, and provide specific examples.

[0122] Thus, the present invention realizes a system that collects a user's biometric information in real time and generates and provides personalized advice based on analysis. This system allows users to continuously monitor their health status and take appropriate actions. Furthermore, by incorporating feedback into the system, the accuracy of the generated advice can be improved.

[0123] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0124] Step 1:

[0125] Data collection

[0126] Device: The user wears a wearable device. The device uses built-in heart rate and blood pressure sensors to collect biometric information such as heart rate and blood pressure in real time. Specifically, the heart rate sensor measures heart rate every second, and the blood pressure sensor measures blood pressure every 30 minutes.

[0127] Input: User's real-time biometric information

[0128] Output: Collected biometric data (heart rate, blood pressure, etc.)

[0129] Step 2:

[0130] Data transmission

[0131] Device: Encrypts collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used for communication.

[0132] Input: Collected biometric data

[0133] Data processing: Encryption of biometric information

[0134] Output: Encrypted biometric data

[0135] Step 3:

[0136] Data reception and storage

[0137] Server: Receives encrypted data sent from the terminal and decrypts it. Stores the decrypted data, along with a timestamp, in a database such as MariaDB.

[0138] Input: Encrypted biometric data

[0139] Data processing: Decrypting data and adding timestamps.

[0140] Output: Biometric data stored in the database

[0141] Step 4:

[0142] Data Analysis

[0143] Server: Analyzes biometric information stored in a database in real time using Python's pandas library. Statistical methods are used to detect anomalies and assess the user's health status. For example, it issues a warning if the heart rate exceeds the normal range.

[0144] Input: Biometric data stored in the database

[0145] Data processing: Anomaly detection using statistical analysis

[0146] Output: Analysis results (evaluation of health status)

[0147] Step 5:

[0148] Personalized advice generation

[0149] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT model) to generate personalized health advice for the user. For example, it might create specific advice such as, "Your heart rate is high, so take some deep breaths and relax."

[0150] Input: Analysis results (health status assessment)

[0151] Data processing: AI-generated advice

[0152] Output: Generated health advice

[0153] Step 6:

[0154] Sending advice and collecting feedback

[0155] Server: Sends generated advice to the user's terminal. The user acts according to the advice and sends the results back to the server as feedback from the terminal. The server receives this feedback and uses it to improve the accuracy of the analysis and generation methods.

[0156] Input: Generated health advice, user feedback

[0157] Data processing: Advice distribution and feedback analysis

[0158] Output: Delivered advice, feedback data

[0159] As a specific example, if a user puts on the device at 9:00 AM, the following actions will occur:

[0160] 9:00: The device starts measuring heart rate every second and blood pressure every 30 minutes.

[0161] 9:10: The device collects biometric information for 10 minutes and sends encrypted data to the server.

[0162] 9:11: The server receives the data and saves it to MariaDB. Then, it is analyzed using Python's pandas library.

[0163] 9:12: The server detects an abnormal heart rate, and the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking some deep breaths or a short walk."

[0164] 9:13: The server sends the generated advice to the terminal.

[0165] 9:14: The user follows the advice and takes deep breaths. As a result, their heart rate stabilizes.

[0166] 9:15: The user sends feedback to the server via their device. The message reads, "I followed the advice and took deep breaths, and my heart rate stabilized."

[0167] 9:16: The server receives feedback and incorporates it into future advice generation.

[0168] This allows users to receive appropriate health advice in real time and improve the system's accuracy based on their feedback.

[0169] (Application Example 1)

[0170] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0171] In today's brick-and-mortar stores, there is a growing demand to enhance the customer experience by providing personalized services to each individual customer. However, the technology to quickly and accurately assess customers' health conditions and stress levels and provide appropriate advice based on that information is still insufficient. Furthermore, there is a lack of mechanisms to improve the accuracy of services by incorporating this feedback.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0173] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information collected by the terminal to the server, and analysis means that analyzes the biometric information and evaluates the health status. This makes it possible to grasp the customer's health status in real time and provide personalized services and product recommendations based on that information.

[0174] "User biometric information" refers to data that indicates the user's health status, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0175] A "terminal" refers to a device worn by the user, such as a wearable device or a smartphone, through which biometric information is collected.

[0176] "Communication methods" refer to wireless communication technologies used to transmit collected biometric information to a server. Specifically, this includes technologies such as Bluetooth and Wi-Fi.

[0177] A "server" is a computer system that receives and stores collected biometric information, and is a device that performs data analysis and generates advice.

[0178] "Analysis means" refers to algorithms and software used by the server to analyze biometric information and evaluate the user's health status. This includes statistical methods and anomaly detection techniques.

[0179] The "generation method" refers to a technology that generates advice recommending specific actions to the user based on the analysis results, and artificial intelligence is used for this purpose.

[0180] "Transmission method" refers to the technology used to send the generated advice to the user's terminal. This primarily involves communication via the internet.

[0181] "Learning methods" refer to techniques for receiving user feedback and improving the accuracy of analysis and generation methods. Machine learning is applied to these methods.

[0182] "Service delivery means" refers to technologies that provide personalized services and product recommendations within physical stores based on the user's biometric information.

[0183] This invention is a system that collects users' biometric information in real time and provides personalized services within stores. Specific embodiments for implementing this invention are described below.

[0184] Hardware and software to be used

[0185] Device: Wearable devices and smartphones worn by the user.

[0186] Communication methods: Bluetooth and Wi-Fi

[0187] Server: A computer system that receives, stores, analyzes, and generates advice from data.

[0188] Database: A system for storing biometric information (e.g., MariaDB)

[0189] Analysis software: Python libraries used for data analysis (e.g., pandas, scikit-learn)

[0190] Generative AI models: Artificial intelligence models that generate advice (e.g., GPT)

[0191] Communication protocol: HTTPS

[0192] Process Overview

[0193] The server receives biometric information transmitted from the terminal and stores it in a database. This data is registered with a timestamp, making it possible to analyze past and present data together. Statistical methods are used in the analysis to detect outliers. Based on the analysis results, the user's health status is evaluated, and a generative AI model generates appropriate advice based on this evaluation. The generated advice is sent to the user's terminal for the user to receive. User feedback is also received and used to improve the accuracy of the analysis and generation methods.

[0194] Specific example

[0195] For example, suppose a customer comes into a physical store wearing a wearable device. This device is equipped with heart rate sensors, blood pressure sensors, etc., and measures biometric information in real time. The measured data is transmitted to a smartphone via Bluetooth, and then from the smartphone to a server via Wi-Fi.

[0196] The server receives data in real time and stores it in a database. The stored data is analyzed in real time using the Python pandas library. For example, if a user's heart rate is higher than normal, the server detects this information. Using a generative AI model (e.g., GPT), it generates advice such as, "Your heart rate is high, try deep breathing or a short walk to relax." This improves the customer experience.

[0197] The generated advice is sent to the user's smartphone, and the user receives a notification on their device. The user then provides feedback, such as, "I followed the advice and took deep breaths, and my heart rate stabilized." This feedback is sent back to the server and used as data to improve the system's analysis and generation accuracy.

[0198] Example of a prompt

[0199] Here are some examples of prompt statements to input into the generated AI model.

[0200] The user's current heart rate is 120 bpm at 1:45 PM. This heart rate is higher than normal, suggesting the user is likely experiencing stress. What advice would you offer in this situation?

[0201] With the above configuration, this invention makes it possible to provide personalized services in physical stores by utilizing the user's biometric information, thereby improving the customer experience.

[0202] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0203] Step 1:

[0204] The device collects the user's biometric information. Specifically, it uses heart rate and blood pressure sensors to measure heart rate every second and blood pressure at regular intervals (e.g., every 30 minutes). The input is the user's real-time heart rate and blood pressure data, and the output is this biometric data. This data is temporarily stored inside the device.

[0205] Step 2:

[0206] The device encrypts the collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method. The input is the biometric data collected in step 1, and the output is the encrypted biometric data sent to the server. Specifically, the data is protected using an encryption algorithm and transmitted via a wireless communication protocol.

[0207] Step 3:

[0208] The server receives biometric information transmitted from the terminal and stores it in a database. The software used in this step includes a database management system (e.g., MariaDB). The input is encrypted biometric data, and the output is the data stored in the database. The server registers the data with a timestamp.

[0209] Step 4:

[0210] The server uses Python libraries (e.g., pandas, scikit-learn) to analyze stored biometric data. The input is biometric data from a database, and the output is the analysis results. Specifically, it performs data preprocessing (e.g., normalization and cleansing) and uses statistical methods to detect outliers.

[0211] Step 5:

[0212] The server uses a generative AI model (e.g., GPT) to generate personalized advice based on the analysis results. The input is the analysis results and prompt text obtained in step 4, and the output is the generated advice. Specifically, the prompt text is input to the generative AI model based on the analysis results, and the generated text is retrieved.

[0213] Example of a prompt:

[0214] "The user's current heart rate is 120 at 1:45 PM. This heart rate is higher than normal, so we've determined that they are likely experiencing stress. What advice would you offer in this situation?"

[0215] Step 6:

[0216] The server sends the generated advice to the user's terminal. The input is the generated advice, and the output is the advice displayed on the user's terminal. Specifically, the server sends the advice using a communication method (e.g., HTTPS).

[0217] Step 7:

[0218] The user acts based on the advice received and inputs the results and their impressions as feedback into the device. The input is the user's feedback data, and the output is the feedback data sent from the device to the server.

[0219] Step 8:

[0220] The server receives feedback from users and uses it to improve the accuracy of the analysis and generation methods. The input is feedback data sent by the user, and the output is the updated analysis algorithm and generation model. Specifically, the model is updated using a machine learning algorithm.

[0221] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0222] This invention is a system that collects a user's biometric information in real time, uses an emotion engine to recognize the user's emotions based on that information, and provides specific behavioral advice based on the user's health and emotional state. The components of this invention are as follows:

[0223] 1. Data Collection

[0224] Device: The user wears a wearable device that collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. At the same time, the device is equipped with a camera and microphone to collect the user's facial expressions and voice data.

[0225] Example: Heart rate is measured every second, and facial expression information is saved as a photograph at regular intervals. Audio data is collected during active conversation.

[0226] 2. Data transmission

[0227] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[0228] 3. Data reception and storage

[0229] Server: Receives and decrypts encrypted data sent from the terminal. Received data is stored in the database with a timestamp. Biometric information is stored in JSON format, and media data (facial expressions, voice) is stored in an appropriate format.

[0230] 4. Data Analysis

[0231] Server: Uses statistical methods to analyze biometric information and detect anomalies and patterns. In addition, it operates an emotion engine that estimates the user's emotions using facial recognition algorithms and voice analysis algorithms.

[0232] Example: Stress levels are assessed by analyzing heart rate, and facial recognition software is used to classify the user's emotional state from their face into categories such as "joy," "anger," and "sadness."

[0233] 5. Emotion recognition

[0234] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used to estimate the emotional state.

[0235] Example: The system recognizes a user's smiling face as "joy" and detects "stress" from their tone of voice during conversation.

[0236] 6. Generating personalized advice

[0237] Server: Based on analysis results and emotion recognition results, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. The generated advice is customized according to the user's current health status and emotions.

[0238] Example: Generates advice such as, "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[0239] 7. Sending advice and collecting feedback

[0240] Server: Sends the generated advice to the user's terminal, encrypts it, and transmits it using a communication protocol. After transmission, it maintains a success log.

[0241] Terminal: Decodes advice received from the server and notifies the user. The terminal displays the advice to the user using the smartphone's notification function and display.

[0242] User: Receives advice and acts accordingly. Later, sends feedback on the effectiveness of the advice from the device to the server.

[0243] Example: Enter feedback such as, "I took a deep breath. My heart rate has stabilized."

[0244] 8. Processing and Learning from Feedback

[0245] Server: Receives user feedback and stores it in the database. Analyzes the feedback content and uses it to improve the accuracy of future analyses and advice generation.

[0246] Through the above process, users are constantly aware of their health and emotional state and receive real-time recommendations for specific actions accordingly. Furthermore, the system learns from user feedback, continuously improving the accuracy of subsequent advice.

[0247] The following describes the processing flow.

[0248] Step 1:

[0249] Device: The user puts on the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. Simultaneously, a camera module collects facial expression information, and a microphone collects audio data.

[0250] Step 2:

[0251] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method.

[0252] Step 3:

[0253] Server: Receives and decrypts encrypted data sent from the terminal. The received data is stored in the database with a timestamp. Biometric information is stored in JSON format, while facial expression and voice data are stored in appropriate formats (e.g., JPEG, WAV).

[0254] Step 4:

[0255] Server: To analyze biometric information, statistical analysis is performed using Python's pandas library and other libraries. Anomalies and specific health risks are detected and flagged if applicable. In addition, facial recognition algorithms are run using OpenCV and TENSORFLOW® to estimate emotional states. For audio data, an audio analysis algorithm (e.g., LibROSA) is used.

[0256] Step 5:

[0257] Server: The emotion engine integrates facial expression information and voice data to estimate the user's emotional state. For example, if a smiling facial expression and a calm tone of voice are detected, the emotional state is determined to be "joyful."

[0258] Step 6:

[0259] Server: Based on emotional and health status, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. If health is good and emotions are "joyful," advice recommending continued care is generated. Conversely, if stress or a high heart rate is detected, advice recommending relaxation or rest is generated.

[0260] Step 7:

[0261] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol (e.g., HTTPS). After transmission, a log indicating success is recorded in the database.

[0262] Step 8:

[0263] Terminal: Decodes advice received from the server and notifies the user. Specific action instructions are displayed to the user using the smartphone's notification function or the terminal's display.

[0264] For example, it might display: "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[0265] Step 9:

[0266] User: Follow the advice provided. For example, take deep breaths or go for a short walk. Enter the results and your thoughts into the feedback form and send it from your device to the server.

[0267] Step 10:

[0268] Server: Receives feedback submitted by users and stores it in a database. Analyzes the content of the feedback and improves the algorithms of the analysis and generation methods. For example, successful advice and the conditions under which it was given are incorporated into the model as training data to improve the accuracy of future advice.

[0269] (Example 2)

[0270] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0271] In recent years, there has been a growing demand for technologies that can understand users' health and emotional states in real time and recommend specific actions based on that information. However, existing systems merely collect biometric information and have low accuracy in using that information to provide optimal advice to users. Furthermore, the accuracy of data analysis and emotional state estimation is insufficient, often resulting in ineffective advice. Moreover, the lack of functionality to incorporate user feedback makes continuous system improvement difficult. Therefore, the challenge is to develop a system that can accurately analyze users' biometric information and emotional states and provide personalized advice based on that analysis.

[0272] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0273] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information, facial expression information, and voice data collected by the terminal to the server, analysis means that analyzes the biometric information, facial expression information, and voice data and evaluates the user's health and emotional state, generation means that generates advice recommending specific actions to the user based on the evaluation of the health and emotional state, transmission means that transmits the generated advice to the user's terminal, and learning means that receives user feedback based on the advice and improves the accuracy of the analysis means. This enables accurate analysis of the user's biometric information and emotional state and provides personalized advice, thereby enabling health management and emotional care for the user.

[0274] A "terminal" is a device that a user can carry with them and that collects biometric information, facial expression information, and voice data in real time.

[0275] "Communication methods" refer to technologies that use wireless communication to transmit data collected by a terminal to a server, and include Wi-Fi and Bluetooth.

[0276] "Biometric information" refers to data collected to assess a user's health status, and includes heart rate, blood pressure, steps taken, sleep patterns, and more.

[0277] "Facial expression information" refers to data collected to analyze a user's facial expressions, and includes image data recorded by a camera.

[0278] "Audio data" refers to data collected to analyze user speech and conversations, and includes audio files recorded by a microphone.

[0279] A "server" is a central management device that receives, stores, and analyzes data transmitted from terminals.

[0280] The "analysis means" is a technology for analyzing the collected data and evaluating the user's health status and emotional state. It includes statistical methods and machine learning algorithms.

[0281] The "generation means" is a technology for generating specific action advice for the user based on the analysis results. It includes a generation AI model.

[0282] The "transmission means" is a technology for transmitting the generated advice to the user's terminal.

[0283] The "learning means" is a technology for receiving feedback from the user and improving the accuracy of the analysis means. It includes a process of re-learning including machine learning.

[0284] The "emotional state" is an index indicating the user's psychological state, which is estimated by analyzing expression information and voice data.

[0285] The "personalized advice" is specific action recommendations customized based on the user's current health status and emotional state.

[0286] Mode for Carrying Out the Invention

[0287] The present invention is a system that collects the user's biological information, expression information, and voice data in real time, analyzes them to evaluate the health status and emotional state, and provides personalized advice to the user. This system includes the following components.

[0288] 1. Data collection

[0289] Terminal: The wearable terminal carried by the user collects biological information such as heart rate, blood pressure, number of steps, and sleep pattern in real time. Furthermore, expression information and voice data are also collected simultaneously using the camera and microphone built into the terminal.

[0290] Hardware to use: Common wearable devices (e.g., smartwatches, fitness trackers)

[0291] Software used: OS (e.g., iOS, Android®)

[0292] Specific example:

[0293] The user wears a smartwatch, and a heart rate sensor measures their heart rate every second. The camera takes a picture of their face every 10 seconds, and the microphone collects voice data during conversations.

[0294] 2. Data transmission

[0295] Device: The wearable device encrypts the collected data and sends it to the server at regular intervals. It uses Wi-Fi or Bluetooth.

[0296] Hardware used: Wireless communication module

[0297] Software used: Communication protocol (e.g., HTTPS, Bluetooth protocol)

[0298] Specific example:

[0299] The device checks for a Wi-Fi connection, and once the connection is confirmed, it sends encrypted data to the server using a secure protocol.

[0300] 3. Data reception and storage

[0301] Server: The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database with a timestamp.

[0302] Hardware used: High-performance servers, database systems

[0303] Software to be used: Database management system (e.g., MySQL (registered trademark), PostgreSQL)

[0304] Specific example:

[0305] The server receives data and decrypts it using a dedicated encryption key. Then, it stores biometric information such as heart rate and step count in JSON format and stores facial expression and voice data in appropriate media formats.

[0306] 4. Data analysis

[0307] Server: The server analyzes biometric information using statistical methods to detect outliers and patterns. It also estimates the user's emotional state using facial expression recognition algorithms and voice analysis algorithms.

[0308] Software to be used: Statistical analysis tools (e.g., Python, Scikit-learn), facial expression recognition software (e.g., OpenCV)

[0309] Specific example:

[0310] The server performs time series analysis on heart rate data and generates an alert when an abnormal heart rate is detected. Additionally, it classifies emotions from facial expression images using OpenCV and analyzes voice data using Scikit-learn.

[0311] 5. Emotion recognition

[0312] Server: The emotion engine analyzes the collected facial expression information and voice data to estimate the user's emotional state. A machine learning model (e.g., convolutional neural network) is used for the estimation.

[0313] Software to be used: Machine learning platform (e.g., TensorFlow, Keras)

[0314] Specific example:

[0315] The server uses TensorFlow to apply a convolutional neural network (CNN) model to analyze facial images and classify emotions. It also applies a natural language processing (NLP) model to audio data to assess stress levels.

[0316] 6. Generating personalized advice

[0317] Server: Based on the analysis results and emotion recognition results, it generates personalized advice using generative artificial intelligence (e.g., generative AI model).

[0318] Software used: Generative AI model (e.g., GPT-4)

[0319] Specific example:

[0320] The server inputs prompt messages into GPT-4 and generates advice tailored to the user's current health status and emotions.

[0321] Example of a prompt:

[0322] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[0323] 7. Sending advice and collecting feedback

[0324] Server, terminal, and user: Generated advice is sent to the user's terminal, and the user acts based on that advice. Later, the user sends feedback from the terminal to the server.

[0325] Hardware used: Smartphones, wearable devices

[0326] Software to use: Applications with notification functionality

[0327] Specific example:

[0328] The server encrypts the generated advice and sends it to the user's device. The device receives and decrypts it, and displays it using the smartphone's notification function. The user enters feedback such as "I took a deep breath. My heart rate is stable," and this is sent to the server.

[0329] 8. Processing and Learning from Feedback

[0330] Server: Receives user feedback and stores it in the database. Based on this, the accuracy of the analysis and generation methods is improved.

[0331] Software used: Machine learning algorithms for retraining

[0332] Specific example:

[0333] The server uses the feedback it receives to retrain its machine learning model, improving the accuracy of future advice.

[0334] Through the above process, users can understand their health and emotional state in real time and receive appropriate personalized advice. This makes health management and emotional care for users easy and effective.

[0335] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0336] Step 1: Data Collection

[0337] Subject: terminal

[0338] Description: The device collects the user's biometric information (heart rate, blood pressure, steps, sleep patterns, etc.), facial expression information, and voice data in real time.

[0339] Input: User's heart rate, blood pressure, steps, sleep patterns, facial expression images, audio files

[0340] Output: Collected biometric information, facial expression information, and voice data (before encryption)

[0341] Specific operation: The device measures heart rate every second using a heart rate sensor and takes a picture of the face every 10 seconds using a camera. When the microphone detects a conversation, it records the audio and saves it as data.

[0342] Step 2: Encrypt and transmit data

[0343] Subject: terminal

[0344] Description: The device encrypts all collected data and sends it to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[0345] Input: Collected biometric data, facial expression data, and voice data (before encryption).

[0346] Output: Encrypted biometric data, facial expression data, voice data

[0347] Specific operation: The device encrypts all collected data using AES (Advanced Encryption Standard) and sends it to the server via the HTTPS protocol using a Wi-Fi connection. If Bluetooth is used, the data is sent via the paired mobile device.

[0348] Step 3: Data reception and decoding

[0349] Subject: Server

[0350] Description: The server receives encrypted data sent from the terminal and decrypts it. The received data is stored in the database with a timestamp.

[0351] Input: Encrypted biometric information, facial expression information, voice data

[0352] Output: Decoded biometric information, facial expression information, and voice data

[0353] Specific operation: The server receives data via the HTTPS protocol, decrypts the encrypted data using a dedicated key management system, and then stores it in the database as time-series data.

[0354] Step 4: Data Analysis

[0355] Subject: Server

[0356] Description: The server analyzes biometric information using statistical methods to detect anomalies and patterns. Simultaneously, it analyzes the user's emotions using facial recognition and voice analysis algorithms.

[0357] Input: Decoded biometric information, facial expression information, voice data

[0358] Output: Analysis results (health status, emotional status)

[0359] Specific operation: The server executes a Python script to analyze time-series biometric data. If an anomaly is detected, the information is recorded. Furthermore, it analyzes facial expression data using the OpenCV library and analyzes audio data using Scikit-learn.

[0360] Step 5: Emotion Recognition

[0361] Subject: Server

[0362] Description: The server's emotion engine analyzes facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[0363] Input: Analyzed facial expression information, audio data

[0364] Output: Estimated emotional state

[0365] Specific operation: The server uses TensorFlow and applies a convolutional neural network (CNN) model to analyze facial expression data. A natural language processing (NLP) model is applied to audio data to estimate emotional states.

[0366] Step 6: Generate personalized advice

[0367] Subject: Server

[0368] Description: The server generates personalized advice using generative artificial intelligence (e.g., GPT-4) based on the analysis results and emotion recognition results.

[0369] Input: Analysis results (health status, emotional status)

[0370] Output: Personalized advice

[0371] Specific operation: The server inputs prompt text into GPT-4 and generates advice tailored to the user's current health status and emotions.

[0372] Example of a prompt:

[0373] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[0374] Step 7: Send advice and collect feedback

[0375] Subject: Server, terminal, user

[0376] Description: Generated advice is sent to the user's device, and the user acts based on that advice. Later, the user sends feedback from their device to the server.

[0377] Input: Personalized advice

[0378] Output: User feedback

[0379] Specific operation: The server encrypts the generated advice and sends it to the user's device. The device decrypts it and displays it to the user using its notification function. The user enters feedback into the device, which is then sent to the server.

[0380] Step 8: Feedback Processing and Learning

[0381] Subject: Server

[0382] Description: User feedback is received and stored in a database. This is used to improve the accuracy of analysis and generation methods.

[0383] Input: User feedback

[0384] Output: Retrained model, improved analysis and advice accuracy.

[0385] Specific operation: The server uses the received feedback to retrain the machine learning model, improving the accuracy of subsequent advice.

[0386] Through these steps, users can understand their health and emotional state in real time and receive personalized advice. By utilizing the feedback obtained during this process, the system can continuously improve its accuracy.

[0387] (Application Example 2)

[0388] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0389] In modern brick-and-mortar stores, it is difficult to accurately understand what customers want and under what circumstances they are shopping. This makes it challenging to provide personalized services and conduct appropriate promotions. Furthermore, the inability to provide services that take into account customers' health and emotional states makes improving customer satisfaction a challenge.

[0390] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information and facial expression information collected by the terminal to the server, analysis means that analyzes the biometric information and facial expression information and evaluates the health state and emotional state, generation means that generates advice that recommends specific actions to the user based on the evaluation of the health state and emotional state, transmission means that transmits the generated advice to the user's terminal, learning means that receives user feedback based on the advice and improves the accuracy of the analysis means, and notification means that also notifies store staff of the advice. This makes it possible to grasp the customer's health state and emotional state in real time and provide appropriate services and promotions.

[0391] A "device that collects user biometric information in real time" is a device that, when worn by a user, measures and collects biometric information such as heart rate, blood pressure, and steps taken in real time.

[0392] "Communication methods" refers to the protocols and equipment used to encrypt biometric and facial expression information collected by a terminal and securely transmit it to a server.

[0393] "Analysis means for analyzing biological information and facial expression information" refers to devices or programs that analyze collected biological information and facial expression information using statistical methods and machine learning algorithms.

[0394] "Assessing health and emotional state" means determining the user's current health and emotional state based on the analysis results of biometric and facial information.

[0395] "Generation means" refers to programs or algorithms that generate advice recommending specific actions based on analysis results.

[0396] "Transmission means" refers to the communication protocols and devices used to send the generated advice to the user's terminal.

[0397] "Learning tools" refer to programs and algorithms that collect user feedback and use that data to improve the accuracy of analysis and generation tools.

[0398] "Notification methods" refers to a general term for protocols and devices used to notify store staff of generated advice and information in real time.

[0399] This invention is a system that collects a user's biometric information and emotional state in real time and provides specific behavioral advice based on the analysis results. This system consists of the following components.

[0400] 1. Data collection:

[0401] The device, when worn by the user, collects biometric information such as heart rate, blood pressure, and steps taken, as well as facial expression data, in real time. The device is equipped with a camera and microphone, which enables facial recognition and the collection of voice data.

[0402] Hardware used: Smartphones, smart glasses, and other wearable devices

[0403] Software used: Biometric data collection app, facial recognition app

[0404] 2. Data transmission:

[0405] The device encrypts the collected data and securely transmits it to the server. Wi-Fi and Bluetooth are used as communication methods.

[0406] Software used: encryption protocol, communication application

[0407] 3. Data reception and storage:

[0408] The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database.

[0409] Hardware used: Server, database management system

[0410] Software to use: Database management software (e.g., MySQL, PostgreSQL)

[0411] 4. Data Analysis:

[0412] The server analyzes biometric and facial expression information using statistical methods and machine learning algorithms to evaluate health and emotional states.

[0413] Software used: Sentiment analysis engine (CNN, RNN, etc.), biometric information analysis algorithms

[0414] 5. Emotion recognition and advice generation:

[0415] The server uses generative AI (e.g., GPT-4) to generate specific action advice based on the analysis results.

[0416] Software used: Generative AI model (GPT-4)

[0417] 6. Send advice:

[0418] The server sends the generated advice to the user's terminal and simultaneously notifies the store staff.

[0419] Software used: Notification management software, notification protocol

[0420] 7. Gathering and learning from feedback:

[0421] Users submit feedback on the effectiveness of the advice and promotions provided. This data is stored on the server and used for future analysis and advice generation.

[0422] Software used: Feedback collection app, feedback analysis engine

[0423] For example, if a customer using smart glasses in a store shows interest in a particular product but experiences slight stress, the following advice will be provided:

[0424] Advice to the customer: "You seem interested in this product. Please try it. We'll give you a special coupon."

[0425] Notice to store staff: "Customer A has shown interest in a particular product, but seems a little hesitant. Please provide appropriate support."

[0426] Examples of prompt messages are as follows:

[0427] "Based on the customer's heart rate and facial expression data, it indicates their level of interest and slight stress. Please generate personalized advice based on these conditions."

[0428] This makes it possible to understand customers' health and emotional states in real time and provide appropriate services and promotions.

[0429] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0430] Step 1: The device collects the user's heart rate, blood pressure, steps, and facial expression information in real time. The input is the user's biometric and facial expression data, and the output is the result of this data collection. Specifically, sensors equipped on the device periodically read the data, and the built-in camera and microphone collect facial recognition and voice data.

[0431] Step 2: The device encrypts the collected biometric and facial expression information and securely transmits it to the server. The input is the data collected in Step 1, and the output is the encrypted data. Specifically, an encryption algorithm operates within the device and transmits the data to the server via Wi-Fi or Bluetooth.

[0432] Step 3: The server receives and decrypts the encrypted data sent from the terminal. The input is encrypted data, and the output is decrypted data. Specifically, the decryption algorithm runs on the server to decrypt the received data and add a timestamp.

[0433] Step 4: The server analyzes biometric and facial expression information. The input is decoded data, and the output is the analysis result. Specifically, biometric analysis algorithms and emotion analysis engines (e.g., CNN or RNN) operate to perform data analysis to evaluate health and emotional states.

[0434] Step 5: The server uses a generative AI (e.g., GPT-4) based on the analysis results to generate specific action advice. The input is the analysis results, and the output is personalized advice. Specifically, the generative AI operates and generates advice using the analysis results and specific prompt sentences.

[0435] Step 6: The server sends the generated advice to the user's terminal and simultaneously notifies the store staff. The input is the generated advice, and the output is the result of the advice being delivered to the terminal and the store staff. Specifically, notification management software operates to send notifications to the user's terminal and the store staff.

[0436] Step 7: Users submit feedback on the effectiveness of the advice and promotions provided. The input is the user's feedback, and the output is feedback data. Specifically, a feedback collection app operates, and users input their thoughts and suggestions for improvement through the app.

[0437] Step 8: The server receives feedback from the user, analyzes the data to improve the accuracy of the analysis and generation methods, and uses it for future analyses and advice generation. The input is feedback data, and the output is the learning result for accuracy improvement. Specifically, the feedback analysis engine runs and retrains the machine learning algorithm.

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0441] [Second Embodiment]

[0442] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0443] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0448] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0449] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0450] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0453] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0454] This invention is implemented using a terminal that collects a user's biometric information in real time and transmits that information to a server, through the following six steps:

[0455] 1. Data Collection

[0456] Device: A wearable device worn by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. For example, a heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes.

[0457] 2. Data transmission

[0458] Terminal: Encrypts collected biometric information and transmits it to the server at regular intervals. Wireless communication methods such as Bluetooth and Wi-Fi are used for communication.

[0459] 3. Data reception and storage

[0460] Server: Receives biometric information transmitted from terminals and stores it in the database. This data is necessary for analysis, so it is registered with a timestamp when saved.

[0461] 4. Data Analysis

[0462] Server: To analyze biometric information stored in the database, it uses statistical methods to detect anomalies and evaluate the user's health status. This analysis employs techniques that combine historical and current data.

[0463] 5. Generating personalized advice

[0464] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT) to generate customized health advice. The generated advice is based on the user's past behavior and specific biometric information for that day.

[0465] 6. Sending advice and collecting feedback

[0466] Server: Sends generated advice to the user's terminal. The user receives the advice and acts accordingly. They also use their terminal to input feedback on the advice and send it to the server.

[0467] User: Follow the advice provided and provide feedback on the results and your impressions via your device.

[0468] Specific examples

[0469] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[0470] Sensors: Equipped with heart rate sensors, blood pressure sensors, etc., to measure biometric information in real time.

[0471] Communication module: Communicates with the server via Wi-Fi or Bluetooth.

[0472] At 9:00 AM, users begin wearing the device. The device measures heart rate every second and blood pressure every 30 minutes. The collected data is encrypted and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is analyzed in real time using Python's pandas library.

[0473] At 9:10 AM, the server detected that the user's heart rate was higher than normal. Based on this, the generative artificial intelligence generated the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sent this advice to the device, and the user received a notification on their device.

[0474] The user follows the advice and tries deep breathing. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and performed deep breathing, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[0475] This entire process allows users to constantly monitor their health status and receive real-time guidance on appropriate actions. Furthermore, the system can utilize user feedback to continuously improve the accuracy of its advice.

[0476] The following describes the processing flow.

[0477] Step 1:

[0478] Device: The user wears the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. An accelerometer detects the user's movements and records steps and activity levels.

[0479] Step 2:

[0480] Terminal: Collected biometric information is batch-processed at regular intervals (e.g., every 10 minutes), encrypted, and sent to the server. Bluetooth or Wi-Fi is used as the communication method, and SSL / TLS is applied to maintain data confidentiality.

[0481] Step 3:

[0482] Server: Receives encrypted data sent from terminals and decrypts it. The received data is stored in a database (e.g., MariaDB) with a timestamp. The data format is JSON or CSV.

[0483] Step 4:

[0484] Server: The stored data is analyzed using the Python pandas library. In particular, statistical methods (e.g., values ​​exceeding the mean ± 2 standard deviations) are used to detect abnormal heart rate and blood pressure. The analysis results are recorded in the database as an assessment of each user's health status.

[0485] Step 5:

[0486] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT-4) to generate specific advice for the user. The advice is personalized according to the user's past behavioral history and current health status.

[0487] Step 6:

[0488] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol. After transmission, a log indicating success is recorded in the database.

[0489] Step 7:

[0490] Terminal: Decodes advice received from the server and notifies the user. Notification methods include push notifications on smartphones and displays on the terminal's screen. For example, it might display a message such as, "Your heart rate is high, please take a break."

[0491] Step 8:

[0492] User: Receives advice notifications from their device and acts accordingly. Later, they input their impressions and the effectiveness of the advice into a feedback form and send it from their device to the server.

[0493] Step 9:

[0494] Server: Receives feedback submitted by users and stores it in the database. The feedback is used to improve future analysis and advice generation.

[0495] Step 10:

[0496] Server: Analyzes received feedback and improves the algorithms of the analysis and generation methods. For example, it trains the model with successful advice and the conditions under which it provides it, thereby improving the accuracy of future advice.

[0497] (Example 1)

[0498] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0499] Conventional health management systems struggle to collect users' biometric information in real time and provide accurate advice based on that information. Furthermore, they lack mechanisms to improve the accuracy of advice by incorporating user feedback into the system. Therefore, there is a challenge in continuously and appropriately managing users' health status.

[0500] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0501] In this invention, the server includes means for receiving and storing the user's biometric information in a database, means for analyzing the biometric information stored in the database and evaluating the user's health status, means for generating advice that recommends specific actions to the user based on the evaluation of the health status, means for transmitting the generated advice to the user's terminal, and learning means for receiving user feedback based on the advice and improving the accuracy of the analysis and generation means. This makes it possible to provide appropriate and personalized advice based on the real-time collection and analysis of the user's biometric information. Furthermore, by utilizing user feedback, the accuracy of the system can be improved, enabling continuous and accurate health management.

[0502] "User biometric information" refers to physiological data that indicates the user's health and activity level, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0503] A "terminal" refers to a device, such as a wearable device or smartphone worn by a user, that collects biometric information and transmits it to a server.

[0504] "Communication methods" refer to protocols and technologies for transmitting biometric information from a terminal to a server, specifically referring to wireless communication technologies such as Bluetooth and Wi-Fi.

[0505] "Receiving and storage means" refers to a system in which a server receives biometric information transmitted from a terminal and stores that data in a database.

[0506] A "database" is a data storage system that accumulates biometric information and retrieves and analyzes it as needed.

[0507] "Analysis means" refers to a system that analyzes biometric information stored in a database using statistical methods and other analytical methods to evaluate the user's health status.

[0508] A "generation method" is a system for generating advice that recommends appropriate actions to the user based on the analysis results, and it uses artificial intelligence (AI) models, etc.

[0509] "Transmission means" refers to the protocols and technologies used to send the generated advice to the user's terminal.

[0510] A "learning tool" is a system that receives feedback from users and improves the accuracy of the analysis and generation tools.

[0511] "Feedback" refers to information that users report to the server via their device, including actions taken in response to the advice provided, the results, and their impressions.

[0512] This invention is implemented using a terminal that collects the user's biometric information in real time and transmits that information to a server. This makes it possible to appropriately evaluate the user's health status and provide personalized advice.

[0513] Hardware and software to be used

[0514] hardware

[0515] Device: A wearable device worn by the user (e.g., fitness band or smartwatch). This device is equipped with a heart rate sensor and blood pressure sensor to collect biometric information in real time. It is equipped with either Bluetooth or Wi-Fi as a communication module.

[0516] Server: A computer system used for receiving, storing, and analyzing data. This server includes a database (e.g., MariaDB) and analysis software (e.g., Python, pandas library).

[0517] software

[0518] Database: Biometric information will be stored using a relational database such as MariaDB.

[0519] Analysis method: Biological information is analyzed using statistical methods with Python and the pandas library.

[0520] Generation method: Generative artificial intelligence (e.g., GPT model) is used to generate personalized advice for the user based on the analysis results.

[0521] Specific Examples of the System

[0522] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[0523] Sensors: Equipped with heart rate and blood pressure sensors, it measures biometric information in real time.

[0524] Communication module: Communicates with the server using Wi-Fi or Bluetooth.

[0525] At 9:00 AM, when the user puts on the device, it begins measuring heart rate every second and blood pressure every 30 minutes. The collected data is encrypted by the device and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is then analyzed in real time using Python's pandas library.

[0526] At 9:10 AM, the server detects that the user's heart rate is higher than normal. Based on this, the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sends this advice to the device, and the user receives a notification from the device.

[0527] The user follows the advice and attempts to take deep breaths. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and took deep breaths, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[0528] Example of a prompt

[0529] "Please describe a program that uses a wearable device to measure heart rate and blood pressure daily."

[0530] "Please provide a specific example of a system in which a user collects biometric information from a wearable device and transmits it to a server."

[0531] Please describe a system that has real-time data analysis and health advice generation capabilities, and provide specific examples.

[0532] Thus, the present invention realizes a system that collects a user's biometric information in real time and generates and provides personalized advice based on analysis. This system allows users to continuously monitor their health status and take appropriate actions. Furthermore, by incorporating feedback into the system, the accuracy of the generated advice can be improved.

[0533] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0534] Step 1:

[0535] Data collection

[0536] Device: The user wears a wearable device. The device uses built-in heart rate and blood pressure sensors to collect biometric information such as heart rate and blood pressure in real time. Specifically, the heart rate sensor measures heart rate every second, and the blood pressure sensor measures blood pressure every 30 minutes.

[0537] Input: User's real-time biometric information

[0538] Output: Collected biometric data (heart rate, blood pressure, etc.)

[0539] Step 2:

[0540] Data transmission

[0541] Device: Encrypts collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used for communication.

[0542] Input: Collected biometric data

[0543] Data processing: Encryption of biometric information

[0544] Output: Encrypted biometric data

[0545] Step 3:

[0546] Data reception and storage

[0547] Server: Receives encrypted data sent from the terminal and decrypts it. Stores the decrypted data, along with a timestamp, in a database such as MariaDB.

[0548] Input: Encrypted biometric data

[0549] Data processing: Decrypting data and adding timestamps.

[0550] Output: Biometric data stored in the database

[0551] Step 4:

[0552] Data Analysis

[0553] Server: Analyzes biometric information stored in a database in real time using Python's pandas library. Statistical methods are used to detect anomalies and assess the user's health status. For example, it issues a warning if the heart rate exceeds the normal range.

[0554] Input: Biometric data stored in the database

[0555] Data processing: Anomaly detection using statistical analysis

[0556] Output: Analysis results (evaluation of health status)

[0557] Step 5:

[0558] Personalized advice generation

[0559] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT model) to generate personalized health advice for the user. For example, it might create specific advice such as, "Your heart rate is high, so take some deep breaths and relax."

[0560] Input: Analysis results (health status assessment)

[0561] Data processing: AI-generated advice

[0562] Output: Generated health advice

[0563] Step 6:

[0564] Sending advice and collecting feedback

[0565] Server: Sends generated advice to the user's terminal. The user acts according to the advice and sends the results back to the server as feedback from the terminal. The server receives this feedback and uses it to improve the accuracy of the analysis and generation methods.

[0566] Input: Generated health advice, user feedback

[0567] Data processing: Advice distribution and feedback analysis

[0568] Output: Delivered advice, feedback data

[0569] As a specific example, if a user puts on the device at 9:00 AM, the following actions will occur:

[0570] 9:00: The device starts measuring heart rate every second and blood pressure every 30 minutes.

[0571] 9:10: The device collects biometric information for 10 minutes and sends encrypted data to the server.

[0572] 9:11: The server receives the data and saves it to MariaDB. Then, it is analyzed using Python's pandas library.

[0573] 9:12: The server detects an abnormal heart rate, and the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking some deep breaths or a short walk."

[0574] 9:13: The server sends the generated advice to the terminal.

[0575] 9:14: The user follows the advice and takes deep breaths. As a result, their heart rate stabilizes.

[0576] 9:15: The user sends feedback to the server via their device. The message reads, "I followed the advice and took deep breaths, and my heart rate stabilized."

[0577] 9:16: The server receives feedback and incorporates it into future advice generation.

[0578] This allows users to receive appropriate health advice in real time and improve the system's accuracy based on their feedback.

[0579] (Application Example 1)

[0580] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0581] In today's brick-and-mortar stores, there is a growing demand to enhance the customer experience by providing personalized services to each individual customer. However, the technology to quickly and accurately assess customers' health conditions and stress levels and provide appropriate advice based on that information is still insufficient. Furthermore, there is a lack of mechanisms to improve the accuracy of services by incorporating this feedback.

[0582] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0583] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information collected by the terminal to the server, and analysis means that analyzes the biometric information and evaluates the health status. This makes it possible to grasp the customer's health status in real time and provide personalized services and product recommendations based on that information.

[0584] "User biometric information" refers to data that indicates the user's health status, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0585] A "terminal" refers to a device worn by the user, such as a wearable device or a smartphone, through which biometric information is collected.

[0586] "Communication methods" refer to wireless communication technologies used to transmit collected biometric information to a server. Specifically, this includes technologies such as Bluetooth and Wi-Fi.

[0587] A "server" is a computer system that receives and stores collected biometric information, and is a device that performs data analysis and generates advice.

[0588] "Analysis means" refers to algorithms and software used by the server to analyze biometric information and evaluate the user's health status. This includes statistical methods and anomaly detection techniques.

[0589] The "generation method" refers to a technology that generates advice recommending specific actions to the user based on the analysis results, and artificial intelligence is used for this purpose.

[0590] "Transmission method" refers to the technology used to send the generated advice to the user's terminal. This primarily involves communication via the internet.

[0591] "Learning methods" refer to techniques for receiving user feedback and improving the accuracy of analysis and generation methods. Machine learning is applied to these methods.

[0592] "Service delivery means" refers to technologies that provide personalized services and product recommendations within physical stores based on the user's biometric information.

[0593] This invention is a system that collects users' biometric information in real time and provides personalized services within stores. Specific embodiments for implementing this invention are described below.

[0594] Hardware and software to be used

[0595] Device: Wearable devices and smartphones worn by the user.

[0596] Communication methods: Bluetooth and Wi-Fi

[0597] Server: A computer system that receives, stores, analyzes, and generates advice from data.

[0598] Database: A system for storing biometric information (e.g., MariaDB)

[0599] Analysis software: Python libraries used for data analysis (e.g., pandas, scikit-learn)

[0600] Generative AI models: Artificial intelligence models that generate advice (e.g., GPT)

[0601] Communication protocol: HTTPS

[0602] Process Overview

[0603] The server receives biometric information transmitted from the terminal and stores it in a database. This data is registered with a timestamp, making it possible to analyze past and present data together. Statistical methods are used in the analysis to detect outliers. Based on the analysis results, the user's health status is evaluated, and a generative AI model generates appropriate advice based on this evaluation. The generated advice is sent to the user's terminal for the user to receive. User feedback is also received and used to improve the accuracy of the analysis and generation methods.

[0604] Specific example

[0605] For example, suppose a customer comes into a physical store wearing a wearable device. This device is equipped with heart rate sensors, blood pressure sensors, etc., and measures biometric information in real time. The measured data is transmitted to a smartphone via Bluetooth, and then from the smartphone to a server via Wi-Fi.

[0606] The server receives data in real time and stores it in a database. The stored data is analyzed in real time using the Python pandas library. For example, if a user's heart rate is higher than normal, the server detects this information. Using a generative AI model (e.g., GPT), it generates advice such as, "Your heart rate is high, try deep breathing or a short walk to relax." This improves the customer experience.

[0607] The generated advice is sent to the user's smartphone, and the user receives a notification on their device. The user then provides feedback, such as, "I followed the advice and took deep breaths, and my heart rate stabilized." This feedback is sent back to the server and used as data to improve the system's analysis and generation accuracy.

[0608] Example of a prompt

[0609] Here are some examples of prompt statements to input into the generated AI model.

[0610] The user's current heart rate is 120 bpm at 1:45 PM. This heart rate is higher than normal, suggesting the user is likely experiencing stress. What advice would you offer in this situation?

[0611] With the above configuration, this invention makes it possible to provide personalized services in physical stores by utilizing the user's biometric information, thereby improving the customer experience.

[0612] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0613] Step 1:

[0614] The device collects the user's biometric information. Specifically, it uses heart rate and blood pressure sensors to measure heart rate every second and blood pressure at regular intervals (e.g., every 30 minutes). The input is the user's real-time heart rate and blood pressure data, and the output is this biometric data. This data is temporarily stored inside the device.

[0615] Step 2:

[0616] The device encrypts the collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method. The input is the biometric data collected in step 1, and the output is the encrypted biometric data sent to the server. Specifically, the data is protected using an encryption algorithm and transmitted via a wireless communication protocol.

[0617] Step 3:

[0618] The server receives biometric information transmitted from the terminal and stores it in a database. The software used in this step includes a database management system (e.g., MariaDB). The input is encrypted biometric data, and the output is the data stored in the database. The server registers the data with a timestamp.

[0619] Step 4:

[0620] The server uses Python libraries (e.g., pandas, scikit-learn) to analyze stored biometric data. The input is biometric data from a database, and the output is the analysis results. Specifically, it performs data preprocessing (e.g., normalization and cleansing) and uses statistical methods to detect outliers.

[0621] Step 5:

[0622] The server uses a generative AI model (e.g., GPT) to generate personalized advice based on the analysis results. The input is the analysis results and prompt text obtained in step 4, and the output is the generated advice. Specifically, the prompt text is input to the generative AI model based on the analysis results, and the generated text is retrieved.

[0623] Example of a prompt:

[0624] "The user's current heart rate is 120 at 1:45 PM. This heart rate is higher than normal, so we've determined that they are likely experiencing stress. What advice would you offer in this situation?"

[0625] Step 6:

[0626] The server sends the generated advice to the user's terminal. The input is the generated advice, and the output is the advice displayed on the user's terminal. Specifically, the server sends the advice using a communication method (e.g., HTTPS).

[0627] Step 7:

[0628] The user acts based on the advice received and inputs the results and their impressions as feedback into the device. The input is the user's feedback data, and the output is the feedback data sent from the device to the server.

[0629] Step 8:

[0630] The server receives feedback from users and uses it to improve the accuracy of the analysis and generation methods. The input is feedback data sent by the user, and the output is the updated analysis algorithm and generation model. Specifically, the model is updated using a machine learning algorithm.

[0631] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0632] This invention is a system that collects a user's biometric information in real time, uses an emotion engine to recognize the user's emotions based on that information, and provides specific behavioral advice based on the user's health and emotional state. The components of this invention are as follows:

[0633] 1. Data Collection

[0634] Device: The user wears a wearable device that collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. At the same time, the device is equipped with a camera and microphone to collect the user's facial expressions and voice data.

[0635] Example: Heart rate is measured every second, and facial expression information is saved as a photograph at regular intervals. Audio data is collected during active conversation.

[0636] 2. Data transmission

[0637] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[0638] 3. Data reception and storage

[0639] Server: Receives and decrypts encrypted data sent from the terminal. Received data is stored in the database with a timestamp. Biometric information is stored in JSON format, and media data (facial expressions, voice) is stored in an appropriate format.

[0640] 4. Data Analysis

[0641] Server: Uses statistical methods to analyze biometric information and detect anomalies and patterns. In addition, it operates an emotion engine that estimates the user's emotions using facial recognition algorithms and voice analysis algorithms.

[0642] Example: Stress levels are assessed by analyzing heart rate, and facial recognition software is used to classify the user's emotional state from their face into categories such as "joy," "anger," and "sadness."

[0643] 5. Emotion recognition

[0644] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used to estimate the emotional state.

[0645] Example: The system recognizes a user's smiling face as "joy" and detects "stress" from their tone of voice during conversation.

[0646] 6. Generating personalized advice

[0647] Server: Based on analysis results and emotion recognition results, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. The generated advice is customized according to the user's current health status and emotions.

[0648] Example: Generates advice such as, "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[0649] 7. Sending advice and collecting feedback

[0650] Server: Sends the generated advice to the user's terminal, encrypts it, and transmits it using a communication protocol. After transmission, it maintains a success log.

[0651] Terminal: Decodes advice received from the server and notifies the user. The terminal displays the advice to the user using the smartphone's notification function and display.

[0652] User: Receives advice and acts accordingly. Later, sends feedback on the effectiveness of the advice from the device to the server.

[0653] Example: Enter feedback such as, "I took a deep breath. My heart rate has stabilized."

[0654] 8. Processing and Learning from Feedback

[0655] Server: Receives user feedback and stores it in the database. Analyzes the feedback content and uses it to improve the accuracy of future analyses and advice generation.

[0656] Through the above process, users are constantly aware of their health and emotional state and receive real-time recommendations for specific actions accordingly. Furthermore, the system learns from user feedback, continuously improving the accuracy of subsequent advice.

[0657] The following describes the processing flow.

[0658] Step 1:

[0659] Device: The user puts on the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. Simultaneously, a camera module collects facial expression information, and a microphone collects audio data.

[0660] Step 2:

[0661] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method.

[0662] Step 3:

[0663] Server: Receives and decrypts encrypted data sent from the terminal. The received data is stored in the database with a timestamp. Biometric information is stored in JSON format, while facial expression and voice data are stored in appropriate formats (e.g., JPEG, WAV).

[0664] Step 4:

[0665] Server: To analyze biometric information, statistical analysis is performed using Python's pandas library and other libraries. Anomalies and specific health risks are detected and flagged if applicable. In addition, facial recognition algorithms are run using OpenCV or TensorFlow to estimate emotional states. For audio data, an audio analysis algorithm (e.g., LibROSA) is used.

[0666] Step 5:

[0667] Server: The emotion engine integrates facial expression information and voice data to estimate the user's emotional state. For example, if a smiling facial expression and a calm tone of voice are detected, the emotional state is determined to be "joyful."

[0668] Step 6:

[0669] Server: Based on emotional and health status, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. If health is good and emotions are "joyful," advice recommending continued care is generated. Conversely, if stress or a high heart rate is detected, advice recommending relaxation or rest is generated.

[0670] Step 7:

[0671] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol (e.g., HTTPS). After transmission, a log indicating success is recorded in the database.

[0672] Step 8:

[0673] Terminal: Decodes advice received from the server and notifies the user. Specific action instructions are displayed to the user using the smartphone's notification function or the terminal's display.

[0674] For example, it might display: "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[0675] Step 9:

[0676] User: Follow the advice provided. For example, take deep breaths or go for a short walk. Enter the results and your thoughts into the feedback form and send it from your device to the server.

[0677] Step 10:

[0678] Server: Receives feedback submitted by users and stores it in a database. Analyzes the content of the feedback and improves the algorithms of the analysis and generation methods. For example, successful advice and the conditions under which it was given are incorporated into the model as training data to improve the accuracy of future advice.

[0679] (Example 2)

[0680] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0681] In recent years, there has been a growing demand for technologies that can understand users' health and emotional states in real time and recommend specific actions based on that information. However, existing systems merely collect biometric information and have low accuracy in using that information to provide optimal advice to users. Furthermore, the accuracy of data analysis and emotional state estimation is insufficient, often resulting in ineffective advice. Moreover, the lack of functionality to incorporate user feedback makes continuous system improvement difficult. Therefore, the challenge is to develop a system that can accurately analyze users' biometric information and emotional states and provide personalized advice based on that analysis.

[0682] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0683] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information, facial expression information, and voice data collected by the terminal to the server, analysis means that analyzes the biometric information, facial expression information, and voice data and evaluates the user's health and emotional state, generation means that generates advice recommending specific actions to the user based on the evaluation of the health and emotional state, transmission means that transmits the generated advice to the user's terminal, and learning means that receives user feedback based on the advice and improves the accuracy of the analysis means. This enables accurate analysis of the user's biometric information and emotional state and provides personalized advice, thereby enabling health management and emotional care for the user.

[0684] A "terminal" is a device that a user can carry with them and that collects biometric information, facial expression information, and voice data in real time.

[0685] "Communication methods" refer to technologies that use wireless communication to transmit data collected by a terminal to a server, and include Wi-Fi and Bluetooth.

[0686] "Biometric information" refers to data collected to assess a user's health status, and includes heart rate, blood pressure, steps taken, sleep patterns, and more.

[0687] "Facial expression information" refers to data collected to analyze a user's facial expressions, and includes image data recorded by a camera.

[0688] "Audio data" refers to data collected to analyze user speech and conversations, and includes audio files recorded by a microphone.

[0689] A "server" is a central management device that receives, stores, and analyzes data transmitted from terminals.

[0690] "Analysis methods" refer to techniques for analyzing collected data and evaluating the user's health and emotional state. These include statistical methods and machine learning algorithms.

[0691] "Generation method" refers to a technology that generates specific actionable advice for the user based on analysis results. This includes generation AI models.

[0692] "Transmission means" refers to the technology used to send the generated advice to the user's terminal.

[0693] "Learning methods" refer to technologies for receiving user feedback and improving the accuracy of analysis methods. This includes retraining processes, such as machine learning.

[0694] "Emotional state" is an indicator that shows the user's psychological state, and is estimated by analyzing facial expression information and voice data.

[0695] "Personalized advice" refers to specific action recommendations tailored to each user's current health and emotional state.

[0696] Modes for carrying out the invention

[0697] This invention is a system that collects a user's biometric information, facial expression information, and voice data in real time, analyzes this data to evaluate their health and emotional state, and provides personalized advice to the user. This system includes the following components:

[0698] 1. Data Collection

[0699] Device: The wearable device carried by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. In addition, it simultaneously collects facial expression information and voice data using the camera and microphone built into the device.

[0700] Hardware to use: Common wearable devices (e.g., smartwatches, fitness trackers)

[0701] Software to use: OS (e.g., iOS, Android)

[0702] Specific example:

[0703] The user wears a smartwatch, and a heart rate sensor measures their heart rate every second. The camera takes a picture of their face every 10 seconds, and the microphone collects voice data during conversations.

[0704] 2. Data transmission

[0705] Device: The wearable device encrypts the collected data and sends it to the server at regular intervals. It uses Wi-Fi or Bluetooth.

[0706] Hardware used: Wireless communication module

[0707] Software used: Communication protocol (e.g., HTTPS, Bluetooth protocol)

[0708] Specific example:

[0709] The device checks for a Wi-Fi connection, and once the connection is confirmed, it sends encrypted data to the server using a secure protocol.

[0710] 3. Data reception and storage

[0711] Server: The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database with a timestamp.

[0712] Hardware used: High-performance servers, database systems

[0713] Software to use: Database management system (e.g., MySQL, PostgreSQL)

[0714] Specific example:

[0715] The server receives the data and decrypts it using a dedicated encryption key. Then, it saves biometric information such as heart rate and steps in JSON format, and facial expressions and voice data in the appropriate media format.

[0716] 4. Data Analysis

[0717] Server: The server analyzes biometric information using statistical methods to detect anomalies and patterns. It also estimates the user's emotional state using facial recognition algorithms and voice analysis algorithms.

[0718] Software used: Statistical analysis tools (e.g., Python, Scikit-learn), facial recognition software (e.g., OpenCV)

[0719] Specific example:

[0720] The server performs time-series analysis of heart rate data and generates an alert if an abnormal heart rate is detected. Furthermore, it uses OpenCV to classify emotions from facial images and Scikit-learn to analyze audio data.

[0721] 5. Emotion recognition

[0722] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[0723] Software to be used: Machine learning platform (e.g., TensorFlow, Keras)

[0724] Specific example:

[0725] The server uses TensorFlow to apply a convolutional neural network (CNN) model to analyze facial images and classify emotions. It also applies a natural language processing (NLP) model to audio data to assess stress levels.

[0726] 6. Generating personalized advice

[0727] Server: Based on the analysis results and emotion recognition results, it generates personalized advice using generative artificial intelligence (e.g., generative AI model).

[0728] Software used: Generative AI model (e.g., GPT-4)

[0729] Specific example:

[0730] The server inputs prompt messages into GPT-4 and generates advice tailored to the user's current health status and emotions.

[0731] Example of a prompt:

[0732] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[0733] 7. Sending advice and collecting feedback

[0734] Server, terminal, and user: Generated advice is sent to the user's terminal, and the user acts based on that advice. Later, the user sends feedback from the terminal to the server.

[0735] Hardware used: Smartphones, wearable devices

[0736] Software to use: Applications with notification functionality

[0737] Specific example:

[0738] The server encrypts the generated advice and sends it to the user's device. The device receives and decrypts it, and displays it using the smartphone's notification function. The user enters feedback such as "I took a deep breath. My heart rate is stable," and this is sent to the server.

[0739] 8. Processing and Learning from Feedback

[0740] Server: Receives user feedback and stores it in the database. Based on this, the accuracy of the analysis and generation methods is improved.

[0741] Software used: Machine learning algorithms for retraining

[0742] Specific example:

[0743] The server uses the feedback it receives to retrain its machine learning model, improving the accuracy of future advice.

[0744] Through the above process, users can understand their health and emotional state in real time and receive appropriate personalized advice. This makes health management and emotional care for users easy and effective.

[0745] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0746] Step 1: Data Collection

[0747] Subject: terminal

[0748] Description: The device collects the user's biometric information (heart rate, blood pressure, steps, sleep patterns, etc.), facial expression information, and voice data in real time.

[0749] Input: User's heart rate, blood pressure, steps, sleep patterns, facial expression images, audio files

[0750] Output: Collected biometric information, facial expression information, and voice data (before encryption)

[0751] Specific operation: The device measures heart rate every second using a heart rate sensor and takes a picture of the face every 10 seconds using a camera. When the microphone detects a conversation, it records the audio and saves it as data.

[0752] Step 2: Encrypt and transmit data

[0753] Subject: terminal

[0754] Description: The device encrypts all collected data and sends it to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[0755] Input: Collected biometric data, facial expression data, and voice data (before encryption).

[0756] Output: Encrypted biometric data, facial expression data, voice data

[0757] Specific operation: The device encrypts all collected data using AES (Advanced Encryption Standard) and sends it to the server via the HTTPS protocol using a Wi-Fi connection. If Bluetooth is used, the data is sent via the paired mobile device.

[0758] Step 3: Data reception and decoding

[0759] Subject: Server

[0760] Description: The server receives encrypted data sent from the terminal and decrypts it. The received data is stored in the database with a timestamp.

[0761] Input: Encrypted biometric information, facial expression information, voice data

[0762] Output: Decoded biometric information, facial expression information, and voice data

[0763] Specific operation: The server receives data via the HTTPS protocol, decrypts the encrypted data using a dedicated key management system, and then stores it in the database as time-series data.

[0764] Step 4: Data Analysis

[0765] Subject: Server

[0766] Description: The server analyzes biometric information using statistical methods to detect anomalies and patterns. Simultaneously, it analyzes the user's emotions using facial recognition and voice analysis algorithms.

[0767] Input: Decoded biometric information, facial expression information, voice data

[0768] Output: Analysis results (health status, emotional status)

[0769] Specific operation: The server executes a Python script to analyze time-series biometric data. If an anomaly is detected, the information is recorded. Furthermore, it analyzes facial expression data using the OpenCV library and analyzes audio data using Scikit-learn.

[0770] Step 5: Emotion Recognition

[0771] Subject: Server

[0772] Description: The server's emotion engine analyzes facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[0773] Input: Analyzed facial expression information, audio data

[0774] Output: Estimated emotional state

[0775] Specific operation: The server uses TensorFlow and applies a convolutional neural network (CNN) model to analyze facial expression data. A natural language processing (NLP) model is applied to audio data to estimate emotional states.

[0776] Step 6: Generate personalized advice

[0777] Subject: Server

[0778] Description: The server generates personalized advice using generative artificial intelligence (e.g., GPT-4) based on the analysis results and emotion recognition results.

[0779] Input: Analysis results (health status, emotional status)

[0780] Output: Personalized advice

[0781] Specific operation: The server inputs prompt text into GPT-4 and generates advice tailored to the user's current health status and emotions.

[0782] Example of a prompt:

[0783] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[0784] Step 7: Send advice and collect feedback

[0785] Subject: Server, terminal, user

[0786] Description: Generated advice is sent to the user's device, and the user acts based on that advice. Later, the user sends feedback from their device to the server.

[0787] Input: Personalized advice

[0788] Output: User feedback

[0789] Specific operation: The server encrypts the generated advice and sends it to the user's device. The device decrypts it and displays it to the user using its notification function. The user enters feedback into the device, which is then sent to the server.

[0790] Step 8: Feedback Processing and Learning

[0791] Subject: Server

[0792] Description: User feedback is received and stored in a database. This is used to improve the accuracy of analysis and generation methods.

[0793] Input: User feedback

[0794] Output: Retrained model, improved analysis and advice accuracy.

[0795] Specific operation: The server uses the received feedback to retrain the machine learning model, improving the accuracy of subsequent advice.

[0796] Through these steps, users can understand their health and emotional state in real time and receive personalized advice. By utilizing the feedback obtained during this process, the system can continuously improve its accuracy.

[0797] (Application Example 2)

[0798] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0799] In modern brick-and-mortar stores, it is difficult to accurately understand what customers want and under what circumstances they are shopping. This makes it challenging to provide personalized services and conduct appropriate promotions. Furthermore, the inability to provide services that take into account customers' health and emotional states makes improving customer satisfaction a challenge.

[0800] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information and facial expression information collected by the terminal to the server, analysis means that analyzes the biometric information and facial expression information and evaluates the health state and emotional state, generation means that generates advice that recommends specific actions to the user based on the evaluation of the health state and emotional state, transmission means that transmits the generated advice to the user's terminal, learning means that receives user feedback based on the advice and improves the accuracy of the analysis means, and notification means that also notifies store staff of the advice. This makes it possible to grasp the customer's health state and emotional state in real time and provide appropriate services and promotions.

[0801] A "device that collects user biometric information in real time" is a device that, when worn by a user, measures and collects biometric information such as heart rate, blood pressure, and steps taken in real time.

[0802] "Communication methods" refers to the protocols and equipment used to encrypt biometric and facial expression information collected by a terminal and securely transmit it to a server.

[0803] "Analysis means for analyzing biological information and facial expression information" refers to devices or programs that analyze collected biological information and facial expression information using statistical methods and machine learning algorithms.

[0804] "Assessing health and emotional state" means determining the user's current health and emotional state based on the analysis results of biometric and facial information.

[0805] "Generation means" refers to programs or algorithms that generate advice recommending specific actions based on analysis results.

[0806] "Transmission means" refers to the communication protocols and devices used to send the generated advice to the user's terminal.

[0807] "Learning tools" refer to programs and algorithms that collect user feedback and use that data to improve the accuracy of analysis and generation tools.

[0808] "Notification methods" refers to a general term for protocols and devices used to notify store staff of generated advice and information in real time.

[0809] This invention is a system that collects a user's biometric information and emotional state in real time and provides specific behavioral advice based on the analysis results. This system consists of the following components.

[0810] 1. Data collection:

[0811] The device, when worn by the user, collects biometric information such as heart rate, blood pressure, and steps taken, as well as facial expression data, in real time. The device is equipped with a camera and microphone, which enables facial recognition and the collection of voice data.

[0812] Hardware used: Smartphones, smart glasses, and other wearable devices

[0813] Software used: Biometric data collection app, facial recognition app

[0814] 2. Data transmission:

[0815] The device encrypts the collected data and securely transmits it to the server. Wi-Fi and Bluetooth are used as communication methods.

[0816] Software used: encryption protocol, communication application

[0817] 3. Data reception and storage:

[0818] The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database.

[0819] Hardware used: Server, database management system

[0820] Software to use: Database management software (e.g., MySQL, PostgreSQL)

[0821] 4. Data Analysis:

[0822] The server analyzes biometric and facial expression information using statistical methods and machine learning algorithms to evaluate health and emotional states.

[0823] Software used: Sentiment analysis engine (CNN, RNN, etc.), biometric information analysis algorithms

[0824] 5. Emotion recognition and advice generation:

[0825] The server uses generative AI (e.g., GPT-4) to generate specific action advice based on the analysis results.

[0826] Software used: Generative AI model (GPT-4)

[0827] 6. Send advice:

[0828] The server sends the generated advice to the user's terminal and simultaneously notifies the store staff.

[0829] Software used: Notification management software, notification protocol

[0830] 7. Gathering and learning from feedback:

[0831] Users submit feedback on the effectiveness of the advice and promotions provided. This data is stored on the server and used for future analysis and advice generation.

[0832] Software used: Feedback collection app, feedback analysis engine

[0833] For example, if a customer using smart glasses in a store shows interest in a particular product but experiences slight stress, the following advice will be provided:

[0834] Advice to the customer: "You seem interested in this product. Please try it. We'll give you a special coupon."

[0835] Notice to store staff: "Customer A has shown interest in a particular product, but seems a little hesitant. Please provide appropriate support."

[0836] Examples of prompt messages are as follows:

[0837] "Based on the customer's heart rate and facial expression data, it indicates their level of interest and slight stress. Please generate personalized advice based on these conditions."

[0838] This makes it possible to understand customers' health and emotional states in real time and provide appropriate services and promotions.

[0839] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0840] Step 1: The device collects the user's heart rate, blood pressure, steps, and facial expression information in real time. The input is the user's biometric and facial expression data, and the output is the result of this data collection. Specifically, sensors equipped on the device periodically read the data, and the built-in camera and microphone collect facial recognition and voice data.

[0841] Step 2: The device encrypts the collected biometric and facial expression information and securely transmits it to the server. The input is the data collected in Step 1, and the output is the encrypted data. Specifically, an encryption algorithm operates within the device and transmits the data to the server via Wi-Fi or Bluetooth.

[0842] Step 3: The server receives and decrypts the encrypted data sent from the terminal. The input is encrypted data, and the output is decrypted data. Specifically, the decryption algorithm runs on the server to decrypt the received data and add a timestamp.

[0843] Step 4: The server analyzes biometric and facial expression information. The input is decoded data, and the output is the analysis result. Specifically, biometric analysis algorithms and emotion analysis engines (e.g., CNN or RNN) operate to perform data analysis to evaluate health and emotional states.

[0844] Step 5: The server uses a generative AI (e.g., GPT-4) based on the analysis results to generate specific action advice. The input is the analysis results, and the output is personalized advice. Specifically, the generative AI operates and generates advice using the analysis results and specific prompt sentences.

[0845] Step 6: The server sends the generated advice to the user's terminal and simultaneously notifies the store staff. The input is the generated advice, and the output is the result of the advice being delivered to the terminal and the store staff. Specifically, notification management software operates to send notifications to the user's terminal and the store staff.

[0846] Step 7: Users submit feedback on the effectiveness of the advice and promotions provided. The input is the user's feedback, and the output is feedback data. Specifically, a feedback collection app operates, and users input their thoughts and suggestions for improvement through the app.

[0847] Step 8: The server receives feedback from the user, analyzes the data to improve the accuracy of the analysis and generation methods, and uses it for future analyses and advice generation. The input is feedback data, and the output is the learning result for accuracy improvement. Specifically, the feedback analysis engine runs and retrains the machine learning algorithm.

[0848] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0849] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0850] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0851] [Third Embodiment]

[0852] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0853] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0854] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0855] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0856] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0857] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0858] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0859] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0860] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0861] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0862] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0863] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0864] This invention is implemented using a terminal that collects a user's biometric information in real time and transmits that information to a server, through the following six steps:

[0865] 1. Data Collection

[0866] Device: A wearable device worn by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. For example, a heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes.

[0867] 2. Data transmission

[0868] Terminal: Encrypts collected biometric information and transmits it to the server at regular intervals. Wireless communication methods such as Bluetooth and Wi-Fi are used for communication.

[0869] 3. Data reception and storage

[0870] Server: Receives biometric information transmitted from terminals and stores it in the database. This data is necessary for analysis, so it is registered with a timestamp when saved.

[0871] 4. Data Analysis

[0872] Server: To analyze biometric information stored in the database, it uses statistical methods to detect anomalies and evaluate the user's health status. This analysis employs techniques that combine historical and current data.

[0873] 5. Generating personalized advice

[0874] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT) to generate customized health advice. The generated advice is based on the user's past behavior and specific biometric information for that day.

[0875] 6. Sending advice and collecting feedback

[0876] Server: Sends generated advice to the user's terminal. The user receives the advice and acts accordingly. They also use their terminal to input feedback on the advice and send it to the server.

[0877] User: Follow the advice provided and provide feedback on the results and your impressions via your device.

[0878] Specific examples

[0879] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[0880] Sensors: Equipped with heart rate sensors, blood pressure sensors, etc., to measure biometric information in real time.

[0881] Communication module: Communicates with the server via Wi-Fi or Bluetooth.

[0882] At 9:00 AM, users begin wearing the device. The device measures heart rate every second and blood pressure every 30 minutes. The collected data is encrypted and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is analyzed in real time using Python's pandas library.

[0883] At 9:10 AM, the server detected that the user's heart rate was higher than normal. Based on this, the generative artificial intelligence generated the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sent this advice to the device, and the user received a notification on their device.

[0884] The user follows the advice and tries deep breathing. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and performed deep breathing, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[0885] This entire process allows users to constantly monitor their health status and receive real-time guidance on appropriate actions. Furthermore, the system can utilize user feedback to continuously improve the accuracy of its advice.

[0886] The following describes the processing flow.

[0887] Step 1:

[0888] Device: The user wears the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. An accelerometer detects the user's movements and records steps and activity levels.

[0889] Step 2:

[0890] Terminal: Collected biometric information is batch-processed at regular intervals (e.g., every 10 minutes), encrypted, and sent to the server. Bluetooth or Wi-Fi is used as the communication method, and SSL / TLS is applied to maintain data confidentiality.

[0891] Step 3:

[0892] Server: Receives encrypted data sent from terminals and decrypts it. The received data is stored in a database (e.g., MariaDB) with a timestamp. The data format is JSON or CSV.

[0893] Step 4:

[0894] Server: The stored data is analyzed using the Python pandas library. In particular, statistical methods (e.g., values ​​exceeding the mean ± 2 standard deviations) are used to detect abnormal heart rate and blood pressure. The analysis results are recorded in the database as an assessment of each user's health status.

[0895] Step 5:

[0896] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT-4) to generate specific advice for the user. The advice is personalized according to the user's past behavioral history and current health status.

[0897] Step 6:

[0898] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol. After transmission, a log indicating success is recorded in the database.

[0899] Step 7:

[0900] Terminal: Decodes advice received from the server and notifies the user. Notification methods include push notifications on smartphones and displays on the terminal's screen. For example, it might display a message such as, "Your heart rate is high, please take a break."

[0901] Step 8:

[0902] User: Receives advice notifications from their device and acts accordingly. Later, they input their impressions and the effectiveness of the advice into a feedback form and send it from their device to the server.

[0903] Step 9:

[0904] Server: Receives feedback submitted by users and stores it in the database. The feedback is used to improve future analysis and advice generation.

[0905] Step 10:

[0906] Server: Analyzes received feedback and improves the algorithms of the analysis and generation methods. For example, it trains the model with successful advice and the conditions under which it provides it, thereby improving the accuracy of future advice.

[0907] (Example 1)

[0908] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0909] Conventional health management systems struggle to collect users' biometric information in real time and provide accurate advice based on that information. Furthermore, they lack mechanisms to improve the accuracy of advice by incorporating user feedback into the system. Therefore, there is a challenge in continuously and appropriately managing users' health status.

[0910] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0911] In this invention, the server includes means for receiving and storing the user's biometric information in a database, means for analyzing the biometric information stored in the database and evaluating the user's health status, means for generating advice that recommends specific actions to the user based on the evaluation of the health status, means for transmitting the generated advice to the user's terminal, and learning means for receiving user feedback based on the advice and improving the accuracy of the analysis and generation means. This makes it possible to provide appropriate and personalized advice based on the real-time collection and analysis of the user's biometric information. Furthermore, by utilizing user feedback, the accuracy of the system can be improved, enabling continuous and accurate health management.

[0912] "User biometric information" refers to physiological data that indicates the user's health and activity level, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0913] A "terminal" refers to a device, such as a wearable device or smartphone worn by a user, that collects biometric information and transmits it to a server.

[0914] "Communication methods" refer to protocols and technologies for transmitting biometric information from a terminal to a server, specifically referring to wireless communication technologies such as Bluetooth and Wi-Fi.

[0915] "Receiving and storage means" refers to a system in which a server receives biometric information transmitted from a terminal and stores that data in a database.

[0916] A "database" is a data storage system that accumulates biometric information and retrieves and analyzes it as needed.

[0917] "Analysis means" refers to a system that analyzes biometric information stored in a database using statistical methods and other analytical methods to evaluate the user's health status.

[0918] A "generation method" is a system for generating advice that recommends appropriate actions to the user based on the analysis results, and it uses artificial intelligence (AI) models, etc.

[0919] "Transmission means" refers to the protocols and technologies used to send the generated advice to the user's terminal.

[0920] A "learning tool" is a system that receives feedback from users and improves the accuracy of the analysis and generation tools.

[0921] "Feedback" refers to information that users report to the server via their device, including actions taken in response to the advice provided, the results, and their impressions.

[0922] This invention is implemented using a terminal that collects the user's biometric information in real time and transmits that information to a server. This makes it possible to appropriately evaluate the user's health status and provide personalized advice.

[0923] Hardware and software to be used

[0924] hardware

[0925] Device: A wearable device worn by the user (e.g., fitness band or smartwatch). This device is equipped with a heart rate sensor and blood pressure sensor to collect biometric information in real time. It is equipped with either Bluetooth or Wi-Fi as a communication module.

[0926] Server: A computer system used for receiving, storing, and analyzing data. This server includes a database (e.g., MariaDB) and analysis software (e.g., Python, pandas library).

[0927] software

[0928] Database: Biometric information will be stored using a relational database such as MariaDB.

[0929] Analysis method: Biological information is analyzed using statistical methods with Python and the pandas library.

[0930] Generation method: Generative artificial intelligence (e.g., GPT model) is used to generate personalized advice for the user based on the analysis results.

[0931] Specific Examples of the System

[0932] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[0933] Sensors: Equipped with heart rate and blood pressure sensors, it measures biometric information in real time.

[0934] Communication module: Communicates with the server using Wi-Fi or Bluetooth.

[0935] At 9:00 AM, when the user puts on the device, it begins measuring heart rate every second and blood pressure every 30 minutes. The collected data is encrypted by the device and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is then analyzed in real time using Python's pandas library.

[0936] At 9:10 AM, the server detects that the user's heart rate is higher than normal. Based on this, the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sends this advice to the device, and the user receives a notification from the device.

[0937] The user follows the advice and attempts to take deep breaths. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and took deep breaths, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[0938] Example of a prompt

[0939] "Please describe a program that uses a wearable device to measure heart rate and blood pressure daily."

[0940] "Please provide a specific example of a system in which a user collects biometric information from a wearable device and transmits it to a server."

[0941] Please describe a system that has real-time data analysis and health advice generation capabilities, and provide specific examples.

[0942] Thus, the present invention realizes a system that collects a user's biometric information in real time and generates and provides personalized advice based on analysis. This system allows users to continuously monitor their health status and take appropriate actions. Furthermore, by incorporating feedback into the system, the accuracy of the generated advice can be improved.

[0943] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0944] Step 1:

[0945] Data collection

[0946] Device: The user wears a wearable device. The device uses built-in heart rate and blood pressure sensors to collect biometric information such as heart rate and blood pressure in real time. Specifically, the heart rate sensor measures heart rate every second, and the blood pressure sensor measures blood pressure every 30 minutes.

[0947] Input: User's real-time biometric information

[0948] Output: Collected biometric data (heart rate, blood pressure, etc.)

[0949] Step 2:

[0950] Data transmission

[0951] Device: Encrypts collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used for communication.

[0952] Input: Collected biometric data

[0953] Data processing: Encryption of biometric information

[0954] Output: Encrypted biometric data

[0955] Step 3:

[0956] Data reception and storage

[0957] Server: Receives encrypted data sent from the terminal and decrypts it. Stores the decrypted data, along with a timestamp, in a database such as MariaDB.

[0958] Input: Encrypted biometric data

[0959] Data processing: Decrypting data and adding timestamps.

[0960] Output: Biometric data stored in the database

[0961] Step 4:

[0962] Data Analysis

[0963] Server: Analyzes biometric information stored in a database in real time using Python's pandas library. Statistical methods are used to detect anomalies and assess the user's health status. For example, it issues a warning if the heart rate exceeds the normal range.

[0964] Input: Biometric data stored in the database

[0965] Data processing: Anomaly detection using statistical analysis

[0966] Output: Analysis results (evaluation of health status)

[0967] Step 5:

[0968] Personalized advice generation

[0969] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT model) to generate personalized health advice for the user. For example, it might create specific advice such as, "Your heart rate is high, so take some deep breaths and relax."

[0970] Input: Analysis results (health status assessment)

[0971] Data processing: AI-generated advice

[0972] Output: Generated health advice

[0973] Step 6:

[0974] Sending advice and collecting feedback

[0975] Server: Sends generated advice to the user's terminal. The user acts according to the advice and sends the results back to the server as feedback from the terminal. The server receives this feedback and uses it to improve the accuracy of the analysis and generation methods.

[0976] Input: Generated health advice, user feedback

[0977] Data processing: Advice distribution and feedback analysis

[0978] Output: Delivered advice, feedback data

[0979] As a specific example, if a user puts on the device at 9:00 AM, the following actions will occur:

[0980] 9:00: The device starts measuring heart rate every second and blood pressure every 30 minutes.

[0981] 9:10: The device collects biometric information for 10 minutes and sends encrypted data to the server.

[0982] 9:11: The server receives the data and saves it to MariaDB. Then, it is analyzed using Python's pandas library.

[0983] 9:12: The server detects an abnormal heart rate, and the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking some deep breaths or a short walk."

[0984] 9:13: The server sends the generated advice to the terminal.

[0985] 9:14: The user follows the advice and takes deep breaths. As a result, their heart rate stabilizes.

[0986] 9:15: The user sends feedback to the server via their device. The message reads, "I followed the advice and took deep breaths, and my heart rate stabilized."

[0987] 9:16: The server receives feedback and incorporates it into future advice generation.

[0988] This allows users to receive appropriate health advice in real time and improve the system's accuracy based on their feedback.

[0989] (Application Example 1)

[0990] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0991] In today's brick-and-mortar stores, there is a growing demand to enhance the customer experience by providing personalized services to each individual customer. However, the technology to quickly and accurately assess customers' health conditions and stress levels and provide appropriate advice based on that information is still insufficient. Furthermore, there is a lack of mechanisms to improve the accuracy of services by incorporating this feedback.

[0992] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0993] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information collected by the terminal to the server, and analysis means that analyzes the biometric information and evaluates the health status. This makes it possible to grasp the customer's health status in real time and provide personalized services and product recommendations based on that information.

[0994] "User biometric information" refers to data that indicates the user's health status, such as heart rate, blood pressure, steps taken, and sleep patterns.

[0995] A "terminal" refers to a device worn by the user, such as a wearable device or a smartphone, through which biometric information is collected.

[0996] "Communication methods" refer to wireless communication technologies used to transmit collected biometric information to a server. Specifically, this includes technologies such as Bluetooth and Wi-Fi.

[0997] A "server" is a computer system that receives and stores collected biometric information, and is a device that performs data analysis and generates advice.

[0998] "Analysis means" refers to algorithms and software used by the server to analyze biometric information and evaluate the user's health status. This includes statistical methods and anomaly detection techniques.

[0999] The "generation method" refers to a technology that generates advice recommending specific actions to the user based on the analysis results, and artificial intelligence is used for this purpose.

[1000] "Transmission method" refers to the technology used to send the generated advice to the user's terminal. This primarily involves communication via the internet.

[1001] "Learning methods" refer to techniques for receiving user feedback and improving the accuracy of analysis and generation methods. Machine learning is applied to these methods.

[1002] "Service delivery means" refers to technologies that provide personalized services and product recommendations within physical stores based on the user's biometric information.

[1003] This invention is a system that collects users' biometric information in real time and provides personalized services within stores. Specific embodiments for implementing this invention are described below.

[1004] Hardware and software to be used

[1005] Device: Wearable devices and smartphones worn by the user.

[1006] Communication methods: Bluetooth and Wi-Fi

[1007] Server: A computer system that receives, stores, analyzes, and generates advice from data.

[1008] Database: A system for storing biometric information (e.g., MariaDB)

[1009] Analysis software: Python libraries used for data analysis (e.g., pandas, scikit-learn)

[1010] Generative AI models: Artificial intelligence models that generate advice (e.g., GPT)

[1011] Communication protocol: HTTPS

[1012] Process Overview

[1013] The server receives biometric information transmitted from the terminal and stores it in a database. This data is registered with a timestamp, making it possible to analyze past and present data together. Statistical methods are used in the analysis to detect outliers. Based on the analysis results, the user's health status is evaluated, and a generative AI model generates appropriate advice based on this evaluation. The generated advice is sent to the user's terminal for the user to receive. User feedback is also received and used to improve the accuracy of the analysis and generation methods.

[1014] Specific example

[1015] For example, suppose a customer comes into a physical store wearing a wearable device. This device is equipped with heart rate sensors, blood pressure sensors, etc., and measures biometric information in real time. The measured data is transmitted to a smartphone via Bluetooth, and then from the smartphone to a server via Wi-Fi.

[1016] The server receives data in real time and stores it in a database. The stored data is analyzed in real time using the Python pandas library. For example, if a user's heart rate is higher than normal, the server detects this information. Using a generative AI model (e.g., GPT), it generates advice such as, "Your heart rate is high, try deep breathing or a short walk to relax." This improves the customer experience.

[1017] The generated advice is sent to the user's smartphone, and the user receives a notification on their device. The user then provides feedback, such as, "I followed the advice and took deep breaths, and my heart rate stabilized." This feedback is sent back to the server and used as data to improve the system's analysis and generation accuracy.

[1018] Example of a prompt

[1019] Here are some examples of prompt statements to input into the generated AI model.

[1020] The user's current heart rate is 120 bpm at 1:45 PM. This heart rate is higher than normal, suggesting the user is likely experiencing stress. What advice would you offer in this situation?

[1021] With the above configuration, this invention makes it possible to provide personalized services in physical stores by utilizing the user's biometric information, thereby improving the customer experience.

[1022] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1023] Step 1:

[1024] The device collects the user's biometric information. Specifically, it uses heart rate and blood pressure sensors to measure heart rate every second and blood pressure at regular intervals (e.g., every 30 minutes). The input is the user's real-time heart rate and blood pressure data, and the output is this biometric data. This data is temporarily stored inside the device.

[1025] Step 2:

[1026] The device encrypts the collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method. The input is the biometric data collected in step 1, and the output is the encrypted biometric data sent to the server. Specifically, the data is protected using an encryption algorithm and transmitted via a wireless communication protocol.

[1027] Step 3:

[1028] The server receives biometric information transmitted from the terminal and stores it in a database. The software used in this step includes a database management system (e.g., MariaDB). The input is encrypted biometric data, and the output is the data stored in the database. The server registers the data with a timestamp.

[1029] Step 4:

[1030] The server uses Python libraries (e.g., pandas, scikit-learn) to analyze stored biometric data. The input is biometric data from a database, and the output is the analysis results. Specifically, it performs data preprocessing (e.g., normalization and cleansing) and uses statistical methods to detect outliers.

[1031] Step 5:

[1032] The server uses a generative AI model (e.g., GPT) to generate personalized advice based on the analysis results. The input is the analysis results and prompt text obtained in step 4, and the output is the generated advice. Specifically, the prompt text is input to the generative AI model based on the analysis results, and the generated text is retrieved.

[1033] Example of a prompt:

[1034] "The user's current heart rate is 120 at 1:45 PM. This heart rate is higher than normal, so we've determined that they are likely experiencing stress. What advice would you offer in this situation?"

[1035] Step 6:

[1036] The server sends the generated advice to the user's terminal. The input is the generated advice, and the output is the advice displayed on the user's terminal. Specifically, the server sends the advice using a communication method (e.g., HTTPS).

[1037] Step 7:

[1038] The user acts based on the advice received and inputs the results and their impressions as feedback into the device. The input is the user's feedback data, and the output is the feedback data sent from the device to the server.

[1039] Step 8:

[1040] The server receives feedback from users and uses it to improve the accuracy of the analysis and generation methods. The input is feedback data sent by the user, and the output is the updated analysis algorithm and generation model. Specifically, the model is updated using a machine learning algorithm.

[1041] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1042] This invention is a system that collects a user's biometric information in real time, uses an emotion engine to recognize the user's emotions based on that information, and provides specific behavioral advice based on the user's health and emotional state. The components of this invention are as follows:

[1043] 1. Data Collection

[1044] Device: The user wears a wearable device that collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. At the same time, the device is equipped with a camera and microphone to collect the user's facial expressions and voice data.

[1045] Example: Heart rate is measured every second, and facial expression information is saved as a photograph at regular intervals. Audio data is collected during active conversation.

[1046] 2. Data transmission

[1047] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[1048] 3. Data reception and storage

[1049] Server: Receives and decrypts encrypted data sent from the terminal. Received data is stored in the database with a timestamp. Biometric information is stored in JSON format, and media data (facial expressions, voice) is stored in an appropriate format.

[1050] 4. Data Analysis

[1051] Server: Uses statistical methods to analyze biometric information and detect anomalies and patterns. In addition, it operates an emotion engine that estimates the user's emotions using facial recognition algorithms and voice analysis algorithms.

[1052] Example: Stress levels are assessed by analyzing heart rate, and facial recognition software is used to classify the user's emotional state from their face into categories such as "joy," "anger," and "sadness."

[1053] 5. Emotion recognition

[1054] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used to estimate the emotional state.

[1055] Example: The system recognizes a user's smiling face as "joy" and detects "stress" from their tone of voice during conversation.

[1056] 6. Generating personalized advice

[1057] Server: Based on analysis results and emotion recognition results, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. The generated advice is customized according to the user's current health status and emotions.

[1058] Example: Generates advice such as, "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[1059] 7. Sending advice and collecting feedback

[1060] Server: Sends the generated advice to the user's terminal, encrypts it, and transmits it using a communication protocol. After transmission, it maintains a success log.

[1061] Terminal: Decodes advice received from the server and notifies the user. The terminal displays the advice to the user using the smartphone's notification function and display.

[1062] User: Receives advice and acts accordingly. Later, sends feedback on the effectiveness of the advice from the device to the server.

[1063] Example: Enter feedback such as, "I took a deep breath. My heart rate has stabilized."

[1064] 8. Processing and Learning from Feedback

[1065] Server: Receives user feedback and stores it in the database. Analyzes the feedback content and uses it to improve the accuracy of future analyses and advice generation.

[1066] Through the above process, users are constantly aware of their health and emotional state and receive real-time recommendations for specific actions accordingly. Furthermore, the system learns from user feedback, continuously improving the accuracy of subsequent advice.

[1067] The following describes the processing flow.

[1068] Step 1:

[1069] Device: The user puts on the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. Simultaneously, a camera module collects facial expression information, and a microphone collects audio data.

[1070] Step 2:

[1071] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method.

[1072] Step 3:

[1073] Server: Receives and decrypts encrypted data sent from the terminal. The received data is stored in the database with a timestamp. Biometric information is stored in JSON format, while facial expression and voice data are stored in appropriate formats (e.g., JPEG, WAV).

[1074] Step 4:

[1075] Server: To analyze biometric information, statistical analysis is performed using Python's pandas library and other libraries. Anomalies and specific health risks are detected and flagged if applicable. In addition, facial recognition algorithms are run using OpenCV or TensorFlow to estimate emotional states. For audio data, an audio analysis algorithm (e.g., LibROSA) is used.

[1076] Step 5:

[1077] Server: The emotion engine integrates facial expression information and voice data to estimate the user's emotional state. For example, if a smiling facial expression and a calm tone of voice are detected, the emotional state is determined to be "joyful."

[1078] Step 6:

[1079] Server: Based on emotional and health status, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. If health is good and emotions are "joyful," advice recommending continued care is generated. Conversely, if stress or a high heart rate is detected, advice recommending relaxation or rest is generated.

[1080] Step 7:

[1081] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol (e.g., HTTPS). After transmission, a log indicating success is recorded in the database.

[1082] Step 8:

[1083] Terminal: Decodes advice received from the server and notifies the user. Specific action instructions are displayed to the user using the smartphone's notification function or the terminal's display.

[1084] For example, it might display: "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[1085] Step 9:

[1086] User: Follow the advice provided. For example, take deep breaths or go for a short walk. Enter the results and your thoughts into the feedback form and send it from your device to the server.

[1087] Step 10:

[1088] Server: Receives feedback submitted by users and stores it in a database. Analyzes the content of the feedback and improves the algorithms of the analysis and generation methods. For example, successful advice and the conditions under which it was given are incorporated into the model as training data to improve the accuracy of future advice.

[1089] (Example 2)

[1090] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1091] In recent years, there has been a growing demand for technologies that can understand users' health and emotional states in real time and recommend specific actions based on that information. However, existing systems merely collect biometric information and have low accuracy in using that information to provide optimal advice to users. Furthermore, the accuracy of data analysis and emotional state estimation is insufficient, often resulting in ineffective advice. Moreover, the lack of functionality to incorporate user feedback makes continuous system improvement difficult. Therefore, the challenge is to develop a system that can accurately analyze users' biometric information and emotional states and provide personalized advice based on that analysis.

[1092] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1093] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information, facial expression information, and voice data collected by the terminal to the server, analysis means that analyzes the biometric information, facial expression information, and voice data and evaluates the user's health and emotional state, generation means that generates advice recommending specific actions to the user based on the evaluation of the health and emotional state, transmission means that transmits the generated advice to the user's terminal, and learning means that receives user feedback based on the advice and improves the accuracy of the analysis means. This enables accurate analysis of the user's biometric information and emotional state and provides personalized advice, thereby enabling health management and emotional care for the user.

[1094] A "terminal" is a device that a user can carry with them and that collects biometric information, facial expression information, and voice data in real time.

[1095] "Communication methods" refer to technologies that use wireless communication to transmit data collected by a terminal to a server, and include Wi-Fi and Bluetooth.

[1096] "Biometric information" refers to data collected to assess a user's health status, and includes heart rate, blood pressure, steps taken, sleep patterns, and more.

[1097] "Facial expression information" refers to data collected to analyze a user's facial expressions, and includes image data recorded by a camera.

[1098] "Audio data" refers to data collected to analyze user speech and conversations, and includes audio files recorded by a microphone.

[1099] A "server" is a central management device that receives, stores, and analyzes data transmitted from terminals.

[1100] "Analysis methods" refer to techniques for analyzing collected data and evaluating the user's health and emotional state. These include statistical methods and machine learning algorithms.

[1101] "Generation method" refers to a technology that generates specific actionable advice for the user based on analysis results. This includes generation AI models.

[1102] "Transmission means" refers to the technology used to send the generated advice to the user's terminal.

[1103] "Learning methods" refer to technologies for receiving user feedback and improving the accuracy of analysis methods. This includes retraining processes, such as machine learning.

[1104] "Emotional state" is an indicator that shows the user's psychological state, and is estimated by analyzing facial expression information and voice data.

[1105] "Personalized advice" refers to specific action recommendations tailored to each user's current health and emotional state.

[1106] Modes for carrying out the invention

[1107] This invention is a system that collects a user's biometric information, facial expression information, and voice data in real time, analyzes this data to evaluate their health and emotional state, and provides personalized advice to the user. This system includes the following components:

[1108] 1. Data Collection

[1109] Device: The wearable device carried by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. In addition, it simultaneously collects facial expression information and voice data using the camera and microphone built into the device.

[1110] Hardware to use: Common wearable devices (e.g., smartwatches, fitness trackers)

[1111] Software to use: OS (e.g., iOS, Android)

[1112] Specific example:

[1113] The user wears a smartwatch, and a heart rate sensor measures their heart rate every second. The camera takes a picture of their face every 10 seconds, and the microphone collects voice data during conversations.

[1114] 2. Data transmission

[1115] Device: The wearable device encrypts the collected data and sends it to the server at regular intervals. It uses Wi-Fi or Bluetooth.

[1116] Hardware used: Wireless communication module

[1117] Software used: Communication protocol (e.g., HTTPS, Bluetooth protocol)

[1118] Specific example:

[1119] The device checks for a Wi-Fi connection, and once the connection is confirmed, it sends encrypted data to the server using a secure protocol.

[1120] 3. Data reception and storage

[1121] Server: The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database with a timestamp.

[1122] Hardware used: High-performance servers, database systems

[1123] Software to use: Database management system (e.g., MySQL, PostgreSQL)

[1124] Specific example:

[1125] The server receives the data and decrypts it using a dedicated encryption key. Then, it saves biometric information such as heart rate and steps in JSON format, and facial expressions and voice data in the appropriate media format.

[1126] 4. Data Analysis

[1127] Server: The server analyzes biometric information using statistical methods to detect anomalies and patterns. It also estimates the user's emotional state using facial recognition algorithms and voice analysis algorithms.

[1128] Software used: Statistical analysis tools (e.g., Python, Scikit-learn), facial recognition software (e.g., OpenCV)

[1129] Specific example:

[1130] The server performs time-series analysis of heart rate data and generates an alert if an abnormal heart rate is detected. Furthermore, it uses OpenCV to classify emotions from facial images and Scikit-learn to analyze audio data.

[1131] 5. Emotion recognition

[1132] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[1133] Software to be used: Machine learning platform (e.g., TensorFlow, Keras)

[1134] Specific example:

[1135] The server uses TensorFlow to apply a convolutional neural network (CNN) model to analyze facial images and classify emotions. It also applies a natural language processing (NLP) model to audio data to assess stress levels.

[1136] 6. Generating personalized advice

[1137] Server: Based on the analysis results and emotion recognition results, it generates personalized advice using generative artificial intelligence (e.g., generative AI model).

[1138] Software used: Generative AI model (e.g., GPT-4)

[1139] Specific example:

[1140] The server inputs prompt messages into GPT-4 and generates advice tailored to the user's current health status and emotions.

[1141] Example of a prompt:

[1142] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[1143] 7. Sending advice and collecting feedback

[1144] Server, terminal, and user: Generated advice is sent to the user's terminal, and the user acts based on that advice. Later, the user sends feedback from the terminal to the server.

[1145] Hardware used: Smartphones, wearable devices

[1146] Software to use: Applications with notification functionality

[1147] Specific example:

[1148] The server encrypts the generated advice and sends it to the user's device. The device receives and decrypts it, and displays it using the smartphone's notification function. The user enters feedback such as "I took a deep breath. My heart rate is stable," and this is sent to the server.

[1149] 8. Processing and Learning from Feedback

[1150] Server: Receives user feedback and stores it in the database. Based on this, the accuracy of the analysis and generation methods is improved.

[1151] Software used: Machine learning algorithms for retraining

[1152] Specific example:

[1153] The server uses the feedback it receives to retrain its machine learning model, improving the accuracy of future advice.

[1154] Through the above process, users can understand their health and emotional state in real time and receive appropriate personalized advice. This makes health management and emotional care for users easy and effective.

[1155] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1156] Step 1: Data Collection

[1157] Subject: terminal

[1158] Description: The device collects the user's biometric information (heart rate, blood pressure, steps, sleep patterns, etc.), facial expression information, and voice data in real time.

[1159] Input: User's heart rate, blood pressure, steps, sleep patterns, facial expression images, audio files

[1160] Output: Collected biometric information, facial expression information, and voice data (before encryption)

[1161] Specific operation: The device measures heart rate every second using a heart rate sensor and takes a picture of the face every 10 seconds using a camera. When the microphone detects a conversation, it records the audio and saves it as data.

[1162] Step 2: Encrypt and transmit data

[1163] Subject: terminal

[1164] Description: The device encrypts all collected data and sends it to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[1165] Input: Collected biometric data, facial expression data, and voice data (before encryption).

[1166] Output: Encrypted biometric data, facial expression data, voice data

[1167] Specific operation: The device encrypts all collected data using AES (Advanced Encryption Standard) and sends it to the server via the HTTPS protocol using a Wi-Fi connection. If Bluetooth is used, the data is sent via the paired mobile device.

[1168] Step 3: Data reception and decoding

[1169] Subject: Server

[1170] Description: The server receives encrypted data sent from the terminal and decrypts it. The received data is stored in the database with a timestamp.

[1171] Input: Encrypted biometric information, facial expression information, voice data

[1172] Output: Decoded biometric information, facial expression information, and voice data

[1173] Specific operation: The server receives data via the HTTPS protocol, decrypts the encrypted data using a dedicated key management system, and then stores it in the database as time-series data.

[1174] Step 4: Data Analysis

[1175] Subject: Server

[1176] Description: The server analyzes biometric information using statistical methods to detect anomalies and patterns. Simultaneously, it analyzes the user's emotions using facial recognition and voice analysis algorithms.

[1177] Input: Decoded biometric information, facial expression information, voice data

[1178] Output: Analysis results (health status, emotional status)

[1179] Specific operation: The server executes a Python script to analyze time-series biometric data. If an anomaly is detected, the information is recorded. Furthermore, it analyzes facial expression data using the OpenCV library and analyzes audio data using Scikit-learn.

[1180] Step 5: Emotion Recognition

[1181] Subject: Server

[1182] Description: The server's emotion engine analyzes facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[1183] Input: Analyzed facial expression information, audio data

[1184] Output: Estimated emotional state

[1185] Specific operation: The server uses TensorFlow and applies a convolutional neural network (CNN) model to analyze facial expression data. A natural language processing (NLP) model is applied to audio data to estimate emotional states.

[1186] Step 6: Generate personalized advice

[1187] Subject: Server

[1188] Description: The server generates personalized advice using generative artificial intelligence (e.g., GPT-4) based on the analysis results and emotion recognition results.

[1189] Input: Analysis results (health status, emotional status)

[1190] Output: Personalized advice

[1191] Specific operation: The server inputs prompt text into GPT-4 and generates advice tailored to the user's current health status and emotions.

[1192] Example of a prompt:

[1193] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[1194] Step 7: Send advice and collect feedback

[1195] Subject: Server, terminal, user

[1196] Description: Generated advice is sent to the user's device, and the user acts based on that advice. Later, the user sends feedback from their device to the server.

[1197] Input: Personalized advice

[1198] Output: User feedback

[1199] Specific operation: The server encrypts the generated advice and sends it to the user's device. The device decrypts it and displays it to the user using its notification function. The user enters feedback into the device, which is then sent to the server.

[1200] Step 8: Feedback Processing and Learning

[1201] Subject: Server

[1202] Description: User feedback is received and stored in a database. This is used to improve the accuracy of analysis and generation methods.

[1203] Input: User feedback

[1204] Output: Retrained model, improved analysis and advice accuracy.

[1205] Specific operation: The server uses the received feedback to retrain the machine learning model, improving the accuracy of subsequent advice.

[1206] Through these steps, users can understand their health and emotional state in real time and receive personalized advice. By utilizing the feedback obtained during this process, the system can continuously improve its accuracy.

[1207] (Application Example 2)

[1208] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1209] In modern brick-and-mortar stores, it is difficult to accurately understand what customers want and under what circumstances they are shopping. This makes it challenging to provide personalized services and conduct appropriate promotions. Furthermore, the inability to provide services that take into account customers' health and emotional states makes improving customer satisfaction a challenge.

[1210] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information and facial expression information collected by the terminal to the server, analysis means that analyzes the biometric information and facial expression information and evaluates the health state and emotional state, generation means that generates advice that recommends specific actions to the user based on the evaluation of the health state and emotional state, transmission means that transmits the generated advice to the user's terminal, learning means that receives user feedback based on the advice and improves the accuracy of the analysis means, and notification means that also notifies store staff of the advice. This makes it possible to grasp the customer's health state and emotional state in real time and provide appropriate services and promotions.

[1211] A "device that collects user biometric information in real time" is a device that, when worn by a user, measures and collects biometric information such as heart rate, blood pressure, and steps taken in real time.

[1212] "Communication methods" refers to the protocols and equipment used to encrypt biometric and facial expression information collected by a terminal and securely transmit it to a server.

[1213] "Analysis means for analyzing biological information and facial expression information" refers to devices or programs that analyze collected biological information and facial expression information using statistical methods and machine learning algorithms.

[1214] "Assessing health and emotional state" means determining the user's current health and emotional state based on the analysis results of biometric and facial information.

[1215] "Generation means" refers to programs or algorithms that generate advice recommending specific actions based on analysis results.

[1216] "Transmission means" refers to the communication protocols and devices used to send the generated advice to the user's terminal.

[1217] "Learning tools" refer to programs and algorithms that collect user feedback and use that data to improve the accuracy of analysis and generation tools.

[1218] "Notification methods" refers to a general term for protocols and devices used to notify store staff of generated advice and information in real time.

[1219] This invention is a system that collects a user's biometric information and emotional state in real time and provides specific behavioral advice based on the analysis results. This system consists of the following components.

[1220] 1. Data collection:

[1221] The device, when worn by the user, collects biometric information such as heart rate, blood pressure, and steps taken, as well as facial expression data, in real time. The device is equipped with a camera and microphone, which enables facial recognition and the collection of voice data.

[1222] Hardware used: Smartphones, smart glasses, and other wearable devices

[1223] Software used: Biometric data collection app, facial recognition app

[1224] 2. Data transmission:

[1225] The device encrypts the collected data and securely transmits it to the server. Wi-Fi and Bluetooth are used as communication methods.

[1226] Software used: encryption protocol, communication application

[1227] 3. Data reception and storage:

[1228] The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database.

[1229] Hardware used: Server, database management system

[1230] Software to use: Database management software (e.g., MySQL, PostgreSQL)

[1231] 4. Data Analysis:

[1232] The server analyzes biometric and facial expression information using statistical methods and machine learning algorithms to evaluate health and emotional states.

[1233] Software used: Sentiment analysis engine (CNN, RNN, etc.), biometric information analysis algorithms

[1234] 5. Emotion recognition and advice generation:

[1235] The server uses generative AI (e.g., GPT-4) to generate specific action advice based on the analysis results.

[1236] Software used: Generative AI model (GPT-4)

[1237] 6. Send advice:

[1238] The server sends the generated advice to the user's terminal and simultaneously notifies the store staff.

[1239] Software used: Notification management software, notification protocol

[1240] 7. Gathering and learning from feedback:

[1241] Users submit feedback on the effectiveness of the advice and promotions provided. This data is stored on the server and used for future analysis and advice generation.

[1242] Software used: Feedback collection app, feedback analysis engine

[1243] For example, if a customer using smart glasses in a store shows interest in a particular product but experiences slight stress, the following advice will be provided:

[1244] Advice to the customer: "You seem interested in this product. Please try it. We'll give you a special coupon."

[1245] Notice to store staff: "Customer A has shown interest in a particular product, but seems a little hesitant. Please provide appropriate support."

[1246] Examples of prompt messages are as follows:

[1247] "Based on the customer's heart rate and facial expression data, it indicates their level of interest and slight stress. Please generate personalized advice based on these conditions."

[1248] This makes it possible to understand customers' health and emotional states in real time and provide appropriate services and promotions.

[1249] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1250] Step 1: The device collects the user's heart rate, blood pressure, steps, and facial expression information in real time. The input is the user's biometric and facial expression data, and the output is the result of this data collection. Specifically, sensors equipped on the device periodically read the data, and the built-in camera and microphone collect facial recognition and voice data.

[1251] Step 2: The device encrypts the collected biometric and facial expression information and securely transmits it to the server. The input is the data collected in Step 1, and the output is the encrypted data. Specifically, an encryption algorithm operates within the device and transmits the data to the server via Wi-Fi or Bluetooth.

[1252] Step 3: The server receives and decrypts the encrypted data sent from the terminal. The input is encrypted data, and the output is decrypted data. Specifically, the decryption algorithm runs on the server to decrypt the received data and add a timestamp.

[1253] Step 4: The server analyzes biometric and facial expression information. The input is decoded data, and the output is the analysis result. Specifically, biometric analysis algorithms and emotion analysis engines (e.g., CNN or RNN) operate to perform data analysis to evaluate health and emotional states.

[1254] Step 5: The server uses a generative AI (e.g., GPT-4) based on the analysis results to generate specific action advice. The input is the analysis results, and the output is personalized advice. Specifically, the generative AI operates and generates advice using the analysis results and specific prompt sentences.

[1255] Step 6: The server sends the generated advice to the user's terminal and simultaneously notifies the store staff. The input is the generated advice, and the output is the result of the advice being delivered to the terminal and the store staff. Specifically, notification management software operates to send notifications to the user's terminal and the store staff.

[1256] Step 7: Users submit feedback on the effectiveness of the advice and promotions provided. The input is the user's feedback, and the output is feedback data. Specifically, a feedback collection app operates, and users input their thoughts and suggestions for improvement through the app.

[1257] Step 8: The server receives feedback from the user, analyzes the data to improve the accuracy of the analysis and generation methods, and uses it for future analyses and advice generation. The input is feedback data, and the output is the learning result for accuracy improvement. Specifically, the feedback analysis engine runs and retrains the machine learning algorithm.

[1258] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1259] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1260] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1261] [Fourth Embodiment]

[1262] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1263] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1264] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1265] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1266] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1267] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1268] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1269] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1270] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1271] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1272] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1273] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1274] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1275] This invention is implemented using a terminal that collects a user's biometric information in real time and transmits that information to a server, through the following six steps:

[1276] 1. Data Collection

[1277] Device: A wearable device worn by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. For example, a heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes.

[1278] 2. Data transmission

[1279] Terminal: Encrypts collected biometric information and transmits it to the server at regular intervals. Wireless communication methods such as Bluetooth and Wi-Fi are used for communication.

[1280] 3. Data reception and storage

[1281] Server: Receives biometric information transmitted from terminals and stores it in the database. This data is necessary for analysis, so it is registered with a timestamp when saved.

[1282] 4. Data Analysis

[1283] Server: To analyze biometric information stored in the database, it uses statistical methods to detect anomalies and evaluate the user's health status. This analysis employs techniques that combine historical and current data.

[1284] 5. Generating personalized advice

[1285] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT) to generate customized health advice. The generated advice is based on the user's past behavior and specific biometric information for that day.

[1286] 6. Sending advice and collecting feedback

[1287] Server: Sends generated advice to the user's terminal. The user receives the advice and acts accordingly. They also use their terminal to input feedback on the advice and send it to the server.

[1288] User: Follow the advice provided and provide feedback on the results and your impressions via your device.

[1289] Specific examples

[1290] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[1291] Sensors: Equipped with heart rate sensors, blood pressure sensors, etc., to measure biometric information in real time.

[1292] Communication module: Communicates with the server via Wi-Fi or Bluetooth.

[1293] At 9:00 AM, users begin wearing the device. The device measures heart rate every second and blood pressure every 30 minutes. The collected data is encrypted and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is analyzed in real time using Python's pandas library.

[1294] At 9:10 AM, the server detected that the user's heart rate was higher than normal. Based on this, the generative artificial intelligence generated the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sent this advice to the device, and the user received a notification on their device.

[1295] The user follows the advice and tries deep breathing. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and performed deep breathing, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[1296] This entire process allows users to constantly monitor their health status and receive real-time guidance on appropriate actions. Furthermore, the system can utilize user feedback to continuously improve the accuracy of its advice.

[1297] The following describes the processing flow.

[1298] Step 1:

[1299] Device: The user wears the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. An accelerometer detects the user's movements and records steps and activity levels.

[1300] Step 2:

[1301] Terminal: Collected biometric information is batch-processed at regular intervals (e.g., every 10 minutes), encrypted, and sent to the server. Bluetooth or Wi-Fi is used as the communication method, and SSL / TLS is applied to maintain data confidentiality.

[1302] Step 3:

[1303] Server: Receives encrypted data sent from terminals and decrypts it. The received data is stored in a database (e.g., MariaDB) with a timestamp. The data format is JSON or CSV.

[1304] Step 4:

[1305] Server: The stored data is analyzed using the Python pandas library. In particular, statistical methods (e.g., values ​​exceeding the mean ± 2 standard deviations) are used to detect abnormal heart rate and blood pressure. The analysis results are recorded in the database as an assessment of each user's health status.

[1306] Step 5:

[1307] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT-4) to generate specific advice for the user. The advice is personalized according to the user's past behavioral history and current health status.

[1308] Step 6:

[1309] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol. After transmission, a log indicating success is recorded in the database.

[1310] Step 7:

[1311] Terminal: Decodes advice received from the server and notifies the user. Notification methods include push notifications on smartphones and displays on the terminal's screen. For example, it might display a message such as, "Your heart rate is high, please take a break."

[1312] Step 8:

[1313] User: Receives advice notifications from their device and acts accordingly. Later, they input their impressions and the effectiveness of the advice into a feedback form and send it from their device to the server.

[1314] Step 9:

[1315] Server: Receives feedback submitted by users and stores it in the database. The feedback is used to improve future analysis and advice generation.

[1316] Step 10:

[1317] Server: Analyzes received feedback and improves the algorithms of the analysis and generation methods. For example, it trains the model with successful advice and the conditions under which it provides it, thereby improving the accuracy of future advice.

[1318] (Example 1)

[1319] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1320] Conventional health management systems struggle to collect users' biometric information in real time and provide accurate advice based on that information. Furthermore, they lack mechanisms to improve the accuracy of advice by incorporating user feedback into the system. Therefore, there is a challenge in continuously and appropriately managing users' health status.

[1321] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1322] In this invention, the server includes means for receiving and storing the user's biometric information in a database, means for analyzing the biometric information stored in the database and evaluating the user's health status, means for generating advice that recommends specific actions to the user based on the evaluation of the health status, means for transmitting the generated advice to the user's terminal, and learning means for receiving user feedback based on the advice and improving the accuracy of the analysis and generation means. This makes it possible to provide appropriate and personalized advice based on the real-time collection and analysis of the user's biometric information. Furthermore, by utilizing user feedback, the accuracy of the system can be improved, enabling continuous and accurate health management.

[1323] "User biometric information" refers to physiological data that indicates the user's health and activity level, such as heart rate, blood pressure, steps taken, and sleep patterns.

[1324] A "terminal" refers to a device, such as a wearable device or smartphone worn by a user, that collects biometric information and transmits it to a server.

[1325] "Communication methods" refer to protocols and technologies for transmitting biometric information from a terminal to a server, specifically referring to wireless communication technologies such as Bluetooth and Wi-Fi.

[1326] "Receiving and storage means" refers to a system in which a server receives biometric information transmitted from a terminal and stores that data in a database.

[1327] A "database" is a data storage system that accumulates biometric information and retrieves and analyzes it as needed.

[1328] "Analysis means" refers to a system that analyzes biometric information stored in a database using statistical methods and other analytical methods to evaluate the user's health status.

[1329] A "generation method" is a system for generating advice that recommends appropriate actions to the user based on the analysis results, and it uses artificial intelligence (AI) models, etc.

[1330] "Transmission means" refers to the protocols and technologies used to send the generated advice to the user's terminal.

[1331] A "learning tool" is a system that receives feedback from users and improves the accuracy of the analysis and generation tools.

[1332] "Feedback" refers to information that users report to the server via their device, including actions taken in response to the advice provided, the results, and their impressions.

[1333] This invention is implemented using a terminal that collects the user's biometric information in real time and transmits that information to a server. This makes it possible to appropriately evaluate the user's health status and provide personalized advice.

[1334] Hardware and software to be used

[1335] hardware

[1336] Device: A wearable device worn by the user (e.g., fitness band or smartwatch). This device is equipped with a heart rate sensor and blood pressure sensor to collect biometric information in real time. It is equipped with either Bluetooth or Wi-Fi as a communication module.

[1337] Server: A computer system used for receiving, storing, and analyzing data. This server includes a database (e.g., MariaDB) and analysis software (e.g., Python, pandas library).

[1338] software

[1339] Database: Biometric information will be stored using a relational database such as MariaDB.

[1340] Analysis method: Biological information is analyzed using statistical methods with Python and the pandas library.

[1341] Generation method: Generative artificial intelligence (e.g., GPT model) is used to generate personalized advice for the user based on the analysis results.

[1342] Specific Examples of the System

[1343] For example, suppose a user wears a wearable device on a daily basis. This device has the following functions:

[1344] Sensors: Equipped with heart rate and blood pressure sensors, it measures biometric information in real time.

[1345] Communication module: Communicates with the server using Wi-Fi or Bluetooth.

[1346] At 9:00 AM, when the user puts on the device, it begins measuring heart rate every second and blood pressure every 30 minutes. The collected data is encrypted by the device and sent to the server every 10 minutes. The server receives the data and stores it in a database such as MariaDB. The stored data is then analyzed in real time using Python's pandas library.

[1347] At 9:10 AM, the server detects that the user's heart rate is higher than normal. Based on this, the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking deep breaths or a short walk." The server sends this advice to the device, and the user receives a notification from the device.

[1348] The user follows the advice and attempts to take deep breaths. If their heart rate stabilizes as a result, the user sends feedback to the server via their device stating, "I followed the advice and took deep breaths, and my heart rate stabilized." The server receives this feedback data and uses it to improve the accuracy of the analysis and generation methods.

[1349] Example of a prompt

[1350] "Please describe a program that uses a wearable device to measure heart rate and blood pressure daily."

[1351] "Please provide a specific example of a system in which a user collects biometric information from a wearable device and transmits it to a server."

[1352] Please describe a system that has real-time data analysis and health advice generation capabilities, and provide specific examples.

[1353] Thus, the present invention realizes a system that collects a user's biometric information in real time and generates and provides personalized advice based on analysis. This system allows users to continuously monitor their health status and take appropriate actions. Furthermore, by incorporating feedback into the system, the accuracy of the generated advice can be improved.

[1354] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1355] Step 1:

[1356] Data collection

[1357] Device: The user wears a wearable device. The device uses built-in heart rate and blood pressure sensors to collect biometric information such as heart rate and blood pressure in real time. Specifically, the heart rate sensor measures heart rate every second, and the blood pressure sensor measures blood pressure every 30 minutes.

[1358] Input: User's real-time biometric information

[1359] Output: Collected biometric data (heart rate, blood pressure, etc.)

[1360] Step 2:

[1361] Data transmission

[1362] Device: Encrypts collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used for communication.

[1363] Input: Collected biometric data

[1364] Data processing: Encryption of biometric information

[1365] Output: Encrypted biometric data

[1366] Step 3:

[1367] Data reception and storage

[1368] Server: Receives encrypted data sent from the terminal and decrypts it. Stores the decrypted data, along with a timestamp, in a database such as MariaDB.

[1369] Input: Encrypted biometric data

[1370] Data processing: Decrypting data and adding timestamps.

[1371] Output: Biometric data stored in the database

[1372] Step 4:

[1373] Data Analysis

[1374] Server: Analyzes biometric information stored in a database in real time using Python's pandas library. Statistical methods are used to detect anomalies and assess the user's health status. For example, it issues a warning if the heart rate exceeds the normal range.

[1375] Input: Biometric data stored in the database

[1376] Data processing: Anomaly detection using statistical analysis

[1377] Output: Analysis results (evaluation of health status)

[1378] Step 5:

[1379] Personalized advice generation

[1380] Server: Based on the analysis results, it uses generative artificial intelligence (e.g., GPT model) to generate personalized health advice for the user. For example, it might create specific advice such as, "Your heart rate is high, so take some deep breaths and relax."

[1381] Input: Analysis results (health status assessment)

[1382] Data processing: AI-generated advice

[1383] Output: Generated health advice

[1384] Step 6:

[1385] Sending advice and collecting feedback

[1386] Server: Sends generated advice to the user's terminal. The user acts according to the advice and sends the results back to the server as feedback from the terminal. The server receives this feedback and uses it to improve the accuracy of the analysis and generation methods.

[1387] Input: Generated health advice, user feedback

[1388] Data processing: Advice distribution and feedback analysis

[1389] Output: Delivered advice, feedback data

[1390] As a specific example, if a user puts on the device at 9:00 AM, the following actions will occur:

[1391] 9:00: The device starts measuring heart rate every second and blood pressure every 30 minutes.

[1392] 9:10: The device collects biometric information for 10 minutes and sends encrypted data to the server.

[1393] 9:11: The server receives the data and saves it to MariaDB. Then, it is analyzed using Python's pandas library.

[1394] 9:12: The server detects an abnormal heart rate, and the generative artificial intelligence generates the advice, "You appear to be stressed. Try taking some deep breaths or a short walk."

[1395] 9:13: The server sends the generated advice to the terminal.

[1396] 9:14: The user follows the advice and takes deep breaths. As a result, their heart rate stabilizes.

[1397] 9:15: The user sends feedback to the server via their device. The message reads, "I followed the advice and took deep breaths, and my heart rate stabilized."

[1398] 9:16: The server receives feedback and incorporates it into future advice generation.

[1399] This allows users to receive appropriate health advice in real time and improve the system's accuracy based on their feedback.

[1400] (Application Example 1)

[1401] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1402] In today's brick-and-mortar stores, there is a growing demand to enhance the customer experience by providing personalized services to each individual customer. However, the technology to quickly and accurately assess customers' health conditions and stress levels and provide appropriate advice based on that information is still insufficient. Furthermore, there is a lack of mechanisms to improve the accuracy of services by incorporating this feedback.

[1403] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1404] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information collected by the terminal to the server, and analysis means that analyzes the biometric information and evaluates the health status. This makes it possible to grasp the customer's health status in real time and provide personalized services and product recommendations based on that information.

[1405] "User biometric information" refers to data that indicates the user's health status, such as heart rate, blood pressure, steps taken, and sleep patterns.

[1406] A "terminal" refers to a device worn by the user, such as a wearable device or a smartphone, through which biometric information is collected.

[1407] "Communication methods" refer to wireless communication technologies used to transmit collected biometric information to a server. Specifically, this includes technologies such as Bluetooth and Wi-Fi.

[1408] A "server" is a computer system that receives and stores collected biometric information, and is a device that performs data analysis and generates advice.

[1409] "Analysis means" refers to algorithms and software used by the server to analyze biometric information and evaluate the user's health status. This includes statistical methods and anomaly detection techniques.

[1410] The "generation method" refers to a technology that generates advice recommending specific actions to the user based on the analysis results, and artificial intelligence is used for this purpose.

[1411] "Transmission method" refers to the technology used to send the generated advice to the user's terminal. This primarily involves communication via the internet.

[1412] "Learning methods" refer to techniques for receiving user feedback and improving the accuracy of analysis and generation methods. Machine learning is applied to these methods.

[1413] "Service delivery means" refers to technologies that provide personalized services and product recommendations within physical stores based on the user's biometric information.

[1414] This invention is a system that collects users' biometric information in real time and provides personalized services within stores. Specific embodiments for implementing this invention are described below.

[1415] Hardware and software to be used

[1416] Device: Wearable devices and smartphones worn by the user.

[1417] Communication methods: Bluetooth and Wi-Fi

[1418] Server: A computer system that receives, stores, analyzes, and generates advice from data.

[1419] Database: A system for storing biometric information (e.g., MariaDB)

[1420] Analysis software: Python libraries used for data analysis (e.g., pandas, scikit-learn)

[1421] Generative AI models: Artificial intelligence models that generate advice (e.g., GPT)

[1422] Communication protocol: HTTPS

[1423] Process Overview

[1424] The server receives biometric information transmitted from the terminal and stores it in a database. This data is registered with a timestamp, making it possible to analyze past and present data together. Statistical methods are used in the analysis to detect outliers. Based on the analysis results, the user's health status is evaluated, and a generative AI model generates appropriate advice based on this evaluation. The generated advice is sent to the user's terminal for the user to receive. User feedback is also received and used to improve the accuracy of the analysis and generation methods.

[1425] Specific example

[1426] For example, suppose a customer comes into a physical store wearing a wearable device. This device is equipped with heart rate sensors, blood pressure sensors, etc., and measures biometric information in real time. The measured data is transmitted to a smartphone via Bluetooth, and then from the smartphone to a server via Wi-Fi.

[1427] The server receives data in real time and stores it in a database. The stored data is analyzed in real time using the Python pandas library. For example, if a user's heart rate is higher than normal, the server detects this information. Using a generative AI model (e.g., GPT), it generates advice such as, "Your heart rate is high, try deep breathing or a short walk to relax." This improves the customer experience.

[1428] The generated advice is sent to the user's smartphone, and the user receives a notification on their device. The user then provides feedback, such as, "I followed the advice and took deep breaths, and my heart rate stabilized." This feedback is sent back to the server and used as data to improve the system's analysis and generation accuracy.

[1429] Example of a prompt

[1430] Here are some examples of prompt statements to input into the generated AI model.

[1431] The user's current heart rate is 120 bpm at 1:45 PM. This heart rate is higher than normal, suggesting the user is likely experiencing stress. What advice would you offer in this situation?

[1432] With the above configuration, this invention makes it possible to provide personalized services in physical stores by utilizing the user's biometric information, thereby improving the customer experience.

[1433] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1434] Step 1:

[1435] The device collects the user's biometric information. Specifically, it uses heart rate and blood pressure sensors to measure heart rate every second and blood pressure at regular intervals (e.g., every 30 minutes). The input is the user's real-time heart rate and blood pressure data, and the output is this biometric data. This data is temporarily stored inside the device.

[1436] Step 2:

[1437] The device encrypts the collected biometric information and sends it to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method. The input is the biometric data collected in step 1, and the output is the encrypted biometric data sent to the server. Specifically, the data is protected using an encryption algorithm and transmitted via a wireless communication protocol.

[1438] Step 3:

[1439] The server receives biometric information transmitted from the terminal and stores it in a database. The software used in this step includes a database management system (e.g., MariaDB). The input is encrypted biometric data, and the output is the data stored in the database. The server registers the data with a timestamp.

[1440] Step 4:

[1441] The server uses Python libraries (e.g., pandas, scikit-learn) to analyze stored biometric data. The input is biometric data from a database, and the output is the analysis results. Specifically, it performs data preprocessing (e.g., normalization and cleansing) and uses statistical methods to detect outliers.

[1442] Step 5:

[1443] The server uses a generative AI model (e.g., GPT) to generate personalized advice based on the analysis results. The input is the analysis results and prompt text obtained in step 4, and the output is the generated advice. Specifically, the prompt text is input to the generative AI model based on the analysis results, and the generated text is retrieved.

[1444] Example of a prompt:

[1445] "The user's current heart rate is 120 at 1:45 PM. This heart rate is higher than normal, so we've determined that they are likely experiencing stress. What advice would you offer in this situation?"

[1446] Step 6:

[1447] The server sends the generated advice to the user's terminal. The input is the generated advice, and the output is the advice displayed on the user's terminal. Specifically, the server sends the advice using a communication method (e.g., HTTPS).

[1448] Step 7:

[1449] The user acts based on the advice received and inputs the results and their impressions as feedback into the device. The input is the user's feedback data, and the output is the feedback data sent from the device to the server.

[1450] Step 8:

[1451] The server receives feedback from users and uses it to improve the accuracy of the analysis and generation methods. The input is feedback data sent by the user, and the output is the updated analysis algorithm and generation model. Specifically, the model is updated using a machine learning algorithm.

[1452] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1453] This invention is a system that collects a user's biometric information in real time, uses an emotion engine to recognize the user's emotions based on that information, and provides specific behavioral advice based on the user's health and emotional state. The components of this invention are as follows:

[1454] 1. Data Collection

[1455] Device: The user wears a wearable device that collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. At the same time, the device is equipped with a camera and microphone to collect the user's facial expressions and voice data.

[1456] Example: Heart rate is measured every second, and facial expression information is saved as a photograph at regular intervals. Audio data is collected during active conversation.

[1457] 2. Data transmission

[1458] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[1459] 3. Data reception and storage

[1460] Server: Receives and decrypts encrypted data sent from the terminal. Received data is stored in the database with a timestamp. Biometric information is stored in JSON format, and media data (facial expressions, voice) is stored in an appropriate format.

[1461] 4. Data Analysis

[1462] Server: Uses statistical methods to analyze biometric information and detect anomalies and patterns. In addition, it operates an emotion engine that estimates the user's emotions using facial recognition algorithms and voice analysis algorithms.

[1463] Example: Stress levels are assessed by analyzing heart rate, and facial recognition software is used to classify the user's emotional state from their face into categories such as "joy," "anger," and "sadness."

[1464] 5. Emotion recognition

[1465] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used to estimate the emotional state.

[1466] Example: The system recognizes a user's smiling face as "joy" and detects "stress" from their tone of voice during conversation.

[1467] 6. Generating personalized advice

[1468] Server: Based on analysis results and emotion recognition results, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. The generated advice is customized according to the user's current health status and emotions.

[1469] Example: Generates advice such as, "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[1470] 7. Sending advice and collecting feedback

[1471] Server: Sends the generated advice to the user's terminal, encrypts it, and transmits it using a communication protocol. After transmission, it maintains a success log.

[1472] Terminal: Decodes advice received from the server and notifies the user. The terminal displays the advice to the user using the smartphone's notification function and display.

[1473] User: Receives advice and acts accordingly. Later, sends feedback on the effectiveness of the advice from the device to the server.

[1474] Example: Enter feedback such as, "I took a deep breath. My heart rate has stabilized."

[1475] 8. Processing and Learning from Feedback

[1476] Server: Receives user feedback and stores it in the database. Analyzes the feedback content and uses it to improve the accuracy of future analyses and advice generation.

[1477] Through the above process, users are constantly aware of their health and emotional state and receive real-time recommendations for specific actions accordingly. Furthermore, the system learns from user feedback, continuously improving the accuracy of subsequent advice.

[1478] The following describes the processing flow.

[1479] Step 1:

[1480] Device: The user puts on the wearable device, and data collection begins. A heart rate sensor measures heart rate every second, and a blood pressure sensor measures blood pressure every 30 minutes. Simultaneously, a camera module collects facial expression information, and a microphone collects audio data.

[1481] Step 2:

[1482] Terminal: Encrypts collected biometric information, facial expression information, and voice data, and transmits them to the server at regular intervals (e.g., every 10 minutes). Bluetooth or Wi-Fi is used as the communication method.

[1483] Step 3:

[1484] Server: Receives and decrypts encrypted data sent from the terminal. The received data is stored in the database with a timestamp. Biometric information is stored in JSON format, while facial expression and voice data are stored in appropriate formats (e.g., JPEG, WAV).

[1485] Step 4:

[1486] Server: To analyze biometric information, statistical analysis is performed using Python's pandas library and other libraries. Anomalies and specific health risks are detected and flagged if applicable. In addition, facial recognition algorithms are run using OpenCV or TensorFlow to estimate emotional states. For audio data, an audio analysis algorithm (e.g., LibROSA) is used.

[1487] Step 5:

[1488] Server: The emotion engine integrates facial expression information and voice data to estimate the user's emotional state. For example, if a smiling facial expression and a calm tone of voice are detected, the emotional state is determined to be "joyful."

[1489] Step 6:

[1490] Server: Based on emotional and health status, it uses generative artificial intelligence (e.g., GPT-4) to generate personalized advice. If health is good and emotions are "joyful," advice recommending continued care is generated. Conversely, if stress or a high heart rate is detected, advice recommending relaxation or rest is generated.

[1491] Step 7:

[1492] Server: Sends the generated advice to the user's terminal. The data is re-encrypted and sent using a secure communication protocol (e.g., HTTPS). After transmission, a log indicating success is recorded in the database.

[1493] Step 8:

[1494] Terminal: Decodes advice received from the server and notifies the user. Specific action instructions are displayed to the user using the smartphone's notification function or the terminal's display.

[1495] For example, it might display: "Your heart rate is high, and you appear to be stressed. Try taking 5 minutes of deep breathing exercises."

[1496] Step 9:

[1497] User: Follow the advice provided. For example, take deep breaths or go for a short walk. Enter the results and your thoughts into the feedback form and send it from your device to the server.

[1498] Step 10:

[1499] Server: Receives feedback submitted by users and stores it in a database. Analyzes the content of the feedback and improves the algorithms of the analysis and generation methods. For example, successful advice and the conditions under which it was given are incorporated into the model as training data to improve the accuracy of future advice.

[1500] (Example 2)

[1501] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1502] In recent years, there has been a growing demand for technologies that can understand users' health and emotional states in real time and recommend specific actions based on that information. However, existing systems merely collect biometric information and have low accuracy in using that information to provide optimal advice to users. Furthermore, the accuracy of data analysis and emotional state estimation is insufficient, often resulting in ineffective advice. Moreover, the lack of functionality to incorporate user feedback makes continuous system improvement difficult. Therefore, the challenge is to develop a system that can accurately analyze users' biometric information and emotional states and provide personalized advice based on that analysis.

[1503] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1504] In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information, facial expression information, and voice data collected by the terminal to the server, analysis means that analyzes the biometric information, facial expression information, and voice data and evaluates the user's health and emotional state, generation means that generates advice recommending specific actions to the user based on the evaluation of the health and emotional state, transmission means that transmits the generated advice to the user's terminal, and learning means that receives user feedback based on the advice and improves the accuracy of the analysis means. This enables accurate analysis of the user's biometric information and emotional state and provides personalized advice, thereby enabling health management and emotional care for the user.

[1505] A "terminal" is a device that a user can carry with them and that collects biometric information, facial expression information, and voice data in real time.

[1506] "Communication methods" refer to technologies that use wireless communication to transmit data collected by a terminal to a server, and include Wi-Fi and Bluetooth.

[1507] "Biometric information" refers to data collected to assess a user's health status, and includes heart rate, blood pressure, steps taken, sleep patterns, and more.

[1508] "Facial expression information" refers to data collected to analyze a user's facial expressions, and includes image data recorded by a camera.

[1509] "Audio data" refers to data collected to analyze user speech and conversations, and includes audio files recorded by a microphone.

[1510] A "server" is a central management device that receives, stores, and analyzes data transmitted from terminals.

[1511] "Analysis methods" refer to techniques for analyzing collected data and evaluating the user's health and emotional state. These include statistical methods and machine learning algorithms.

[1512] "Generation method" refers to a technology that generates specific actionable advice for the user based on analysis results. This includes generation AI models.

[1513] "Transmission means" refers to the technology used to send the generated advice to the user's terminal.

[1514] "Learning methods" refer to technologies for receiving user feedback and improving the accuracy of analysis methods. This includes retraining processes, such as machine learning.

[1515] "Emotional state" is an indicator that shows the user's psychological state, and is estimated by analyzing facial expression information and voice data.

[1516] "Personalized advice" refers to specific action recommendations tailored to each user's current health and emotional state.

[1517] Modes for carrying out the invention

[1518] This invention is a system that collects a user's biometric information, facial expression information, and voice data in real time, analyzes this data to evaluate their health and emotional state, and provides personalized advice to the user. This system includes the following components:

[1519] 1. Data Collection

[1520] Device: The wearable device carried by the user collects biometric information such as heart rate, blood pressure, steps taken, and sleep patterns in real time. In addition, it simultaneously collects facial expression information and voice data using the camera and microphone built into the device.

[1521] Hardware to use: Common wearable devices (e.g., smartwatches, fitness trackers)

[1522] Software to use: OS (e.g., iOS, Android)

[1523] Specific example:

[1524] The user wears a smartwatch, and a heart rate sensor measures their heart rate every second. The camera takes a picture of their face every 10 seconds, and the microphone collects voice data during conversations.

[1525] 2. Data transmission

[1526] Device: The wearable device encrypts the collected data and sends it to the server at regular intervals. It uses Wi-Fi or Bluetooth.

[1527] Hardware used: Wireless communication module

[1528] Software used: Communication protocol (e.g., HTTPS, Bluetooth protocol)

[1529] Specific example:

[1530] The device checks for a Wi-Fi connection, and once the connection is confirmed, it sends encrypted data to the server using a secure protocol.

[1531] 3. Data reception and storage

[1532] Server: The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database with a timestamp.

[1533] Hardware used: High-performance servers, database systems

[1534] Software to use: Database management system (e.g., MySQL, PostgreSQL)

[1535] Specific example:

[1536] The server receives the data and decrypts it using a dedicated encryption key. Then, it saves biometric information such as heart rate and steps in JSON format, and facial expressions and voice data in the appropriate media format.

[1537] 4. Data Analysis

[1538] Server: The server analyzes biometric information using statistical methods to detect anomalies and patterns. It also estimates the user's emotional state using facial recognition algorithms and voice analysis algorithms.

[1539] Software used: Statistical analysis tools (e.g., Python, Scikit-learn), facial recognition software (e.g., OpenCV)

[1540] Specific example:

[1541] The server performs time-series analysis of heart rate data and generates an alert if an abnormal heart rate is detected. Furthermore, it uses OpenCV to classify emotions from facial images and Scikit-learn to analyze audio data.

[1542] 5. Emotion recognition

[1543] Server: The emotion engine analyzes collected facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[1544] Software to be used: Machine learning platform (e.g., TensorFlow, Keras)

[1545] Specific example:

[1546] The server uses TensorFlow to apply a convolutional neural network (CNN) model to analyze facial images and classify emotions. It also applies a natural language processing (NLP) model to audio data to assess stress levels.

[1547] 6. Generating personalized advice

[1548] Server: Based on the analysis results and emotion recognition results, it generates personalized advice using generative artificial intelligence (e.g., generative AI model).

[1549] Software used: Generative AI model (e.g., GPT-4)

[1550] Specific example:

[1551] The server inputs prompt messages into GPT-4 and generates advice tailored to the user's current health status and emotions.

[1552] Example of a prompt:

[1553] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[1554] 7. Sending advice and collecting feedback

[1555] Server, terminal, and user: Generated advice is sent to the user's terminal, and the user acts based on that advice. Later, the user sends feedback from the terminal to the server.

[1556] Hardware used: Smartphones, wearable devices

[1557] Software to use: Applications with notification functionality

[1558] Specific example:

[1559] The server encrypts the generated advice and sends it to the user's device. The device receives and decrypts it, and displays it using the smartphone's notification function. The user enters feedback such as "I took a deep breath. My heart rate is stable," and this is sent to the server.

[1560] 8. Processing and Learning from Feedback

[1561] Server: Receives user feedback and stores it in the database. Based on this, the accuracy of the analysis and generation methods is improved.

[1562] Software used: Machine learning algorithms for retraining

[1563] Specific example:

[1564] The server uses the feedback it receives to retrain its machine learning model, improving the accuracy of future advice.

[1565] Through the above process, users can understand their health and emotional state in real time and receive appropriate personalized advice. This makes health management and emotional care for users easy and effective.

[1566] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1567] Step 1: Data Collection

[1568] Subject: terminal

[1569] Description: The device collects the user's biometric information (heart rate, blood pressure, steps, sleep patterns, etc.), facial expression information, and voice data in real time.

[1570] Input: User's heart rate, blood pressure, steps, sleep patterns, facial expression images, audio files

[1571] Output: Collected biometric information, facial expression information, and voice data (before encryption)

[1572] Specific operation: The device measures heart rate every second using a heart rate sensor and takes a picture of the face every 10 seconds using a camera. When the microphone detects a conversation, it records the audio and saves it as data.

[1573] Step 2: Encrypt and transmit data

[1574] Subject: terminal

[1575] Description: The device encrypts all collected data and sends it to the server at regular intervals. Wi-Fi and Bluetooth are used as communication methods.

[1576] Input: Collected biometric data, facial expression data, and voice data (before encryption).

[1577] Output: Encrypted biometric data, facial expression data, voice data

[1578] Specific operation: The device encrypts all collected data using AES (Advanced Encryption Standard) and sends it to the server via the HTTPS protocol using a Wi-Fi connection. If Bluetooth is used, the data is sent via the paired mobile device.

[1579] Step 3: Data reception and decoding

[1580] Subject: Server

[1581] Description: The server receives encrypted data sent from the terminal and decrypts it. The received data is stored in the database with a timestamp.

[1582] Input: Encrypted biometric information, facial expression information, voice data

[1583] Output: Decoded biometric information, facial expression information, and voice data

[1584] Specific operation: The server receives data via the HTTPS protocol, decrypts the encrypted data using a dedicated key management system, and then stores it in the database as time-series data.

[1585] Step 4: Data Analysis

[1586] Subject: Server

[1587] Description: The server analyzes biometric information using statistical methods to detect anomalies and patterns. Simultaneously, it analyzes the user's emotions using facial recognition and voice analysis algorithms.

[1588] Input: Decoded biometric information, facial expression information, voice data

[1589] Output: Analysis results (health status, emotional status)

[1590] Specific operation: The server executes a Python script to analyze time-series biometric data. If an anomaly is detected, the information is recorded. Furthermore, it analyzes facial expression data using the OpenCV library and analyzes audio data using Scikit-learn.

[1591] Step 5: Emotion Recognition

[1592] Subject: Server

[1593] Description: The server's emotion engine analyzes facial expression and audio data to estimate the user's emotional state. Machine learning models (e.g., convolutional neural networks) are used for estimation.

[1594] Input: Analyzed facial expression information, audio data

[1595] Output: Estimated emotional state

[1596] Specific operation: The server uses TensorFlow and applies a convolutional neural network (CNN) model to analyze facial expression data. A natural language processing (NLP) model is applied to audio data to estimate emotional states.

[1597] Step 6: Generate personalized advice

[1598] Subject: Server

[1599] Description: The server generates personalized advice using generative artificial intelligence (e.g., GPT-4) based on the analysis results and emotion recognition results.

[1600] Input: Analysis results (health status, emotional status)

[1601] Output: Personalized advice

[1602] Specific operation: The server inputs prompt text into GPT-4 and generates advice tailored to the user's current health status and emotions.

[1603] Example of a prompt:

[1604] "The user's heart rate is high, indicating they are experiencing stress. Provide actionable advice to the user. Generate text recommending 5 minutes of deep breathing."

[1605] Step 7: Send advice and collect feedback

[1606] Subject: Server, terminal, user

[1607] Description: Generated advice is sent to the user's device, and the user acts based on that advice. Later, the user sends feedback from their device to the server.

[1608] Input: Personalized advice

[1609] Output: User feedback

[1610] Specific operation: The server encrypts the generated advice and sends it to the user's device. The device decrypts it and displays it to the user using its notification function. The user enters feedback into the device, which is then sent to the server.

[1611] Step 8: Feedback Processing and Learning

[1612] Subject: Server

[1613] Description: User feedback is received and stored in a database. This is used to improve the accuracy of analysis and generation methods.

[1614] Input: User feedback

[1615] Output: Retrained model, improved analysis and advice accuracy.

[1616] Specific operation: The server uses the received feedback to retrain the machine learning model, improving the accuracy of subsequent advice.

[1617] Through these steps, users can understand their health and emotional state in real time and receive personalized advice. By utilizing the feedback obtained during this process, the system can continuously improve its accuracy.

[1618] (Application Example 2)

[1619] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1620] In modern brick-and-mortar stores, it is difficult to accurately understand what customers want and under what circumstances they are shopping. This makes it challenging to provide personalized services and conduct appropriate promotions. Furthermore, the inability to provide services that take into account customers' health and emotional states makes improving customer satisfaction a challenge.

[1621] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes a terminal that collects the user's biometric information in real time, communication means that transmits the biometric information and facial expression information collected by the terminal to the server, analysis means that analyzes the biometric information and facial expression information and evaluates the health state and emotional state, generation means that generates advice that recommends specific actions to the user based on the evaluation of the health state and emotional state, transmission means that transmits the generated advice to the user's terminal, learning means that receives user feedback based on the advice and improves the accuracy of the analysis means, and notification means that also notifies store staff of the advice. This makes it possible to grasp the customer's health state and emotional state in real time and provide appropriate services and promotions.

[1622] A "device that collects user biometric information in real time" is a device that, when worn by a user, measures and collects biometric information such as heart rate, blood pressure, and steps taken in real time.

[1623] "Communication methods" refers to the protocols and equipment used to encrypt biometric and facial expression information collected by a terminal and securely transmit it to a server.

[1624] "Analysis means for analyzing biological information and facial expression information" refers to devices or programs that analyze collected biological information and facial expression information using statistical methods and machine learning algorithms.

[1625] "Assessing health and emotional state" means determining the user's current health and emotional state based on the analysis results of biometric and facial information.

[1626] "Generation means" refers to programs or algorithms that generate advice recommending specific actions based on analysis results.

[1627] "Transmission means" refers to the communication protocols and devices used to send the generated advice to the user's terminal.

[1628] "Learning tools" refer to programs and algorithms that collect user feedback and use that data to improve the accuracy of analysis and generation tools.

[1629] "Notification methods" refers to a general term for protocols and devices used to notify store staff of generated advice and information in real time.

[1630] This invention is a system that collects a user's biometric information and emotional state in real time and provides specific behavioral advice based on the analysis results. This system consists of the following components.

[1631] 1. Data collection:

[1632] The device, when worn by the user, collects biometric information such as heart rate, blood pressure, and steps taken, as well as facial expression data, in real time. The device is equipped with a camera and microphone, which enables facial recognition and the collection of voice data.

[1633] Hardware used: Smartphones, smart glasses, and other wearable devices

[1634] Software used: Biometric data collection app, facial recognition app

[1635] 2. Data transmission:

[1636] The device encrypts the collected data and securely transmits it to the server. Wi-Fi and Bluetooth are used as communication methods.

[1637] Software used: encryption protocol, communication application

[1638] 3. Data reception and storage:

[1639] The server receives encrypted data sent from the terminal, decrypts it, and stores it in the database.

[1640] Hardware used: Server, database management system

[1641] Software to use: Database management software (e.g., MySQL, PostgreSQL)

[1642] 4. Data Analysis:

[1643] The server analyzes biometric and facial expression information using statistical methods and machine learning algorithms to evaluate health and emotional states.

[1644] Software used: Sentiment analysis engine (CNN, RNN, etc.), biometric information analysis algorithms

[1645] 5. Emotion recognition and advice generation:

[1646] The server uses generative AI (e.g., GPT-4) to generate specific action advice based on the analysis results.

[1647] Software used: Generative AI model (GPT-4)

[1648] 6. Send advice:

[1649] The server sends the generated advice to the user's terminal and simultaneously notifies the store staff.

[1650] Software used: Notification management software, notification protocol

[1651] 7. Gathering and learning from feedback:

[1652] Users submit feedback on the effectiveness of the advice and promotions provided. This data is stored on the server and used for future analysis and advice generation.

[1653] Software used: Feedback collection app, feedback analysis engine

[1654] For example, if a customer using smart glasses in a store shows interest in a particular product but experiences slight stress, the following advice will be provided:

[1655] Advice to the customer: "You seem interested in this product. Please try it. We'll give you a special coupon."

[1656] Notice to store staff: "Customer A has shown interest in a particular product, but seems a little hesitant. Please provide appropriate support."

[1657] Examples of prompt messages are as follows:

[1658] "Based on the customer's heart rate and facial expression data, it indicates their level of interest and slight stress. Please generate personalized advice based on these conditions."

[1659] This makes it possible to understand customers' health and emotional states in real time and provide appropriate services and promotions.

[1660] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1661] Step 1: The device collects the user's heart rate, blood pressure, steps, and facial expression information in real time. The input is the user's biometric and facial expression data, and the output is the result of this data collection. Specifically, sensors equipped on the device periodically read the data, and the built-in camera and microphone collect facial recognition and voice data.

[1662] Step 2: The device encrypts the collected biometric and facial expression information and securely transmits it to the server. The input is the data collected in Step 1, and the output is the encrypted data. Specifically, an encryption algorithm operates within the device and transmits the data to the server via Wi-Fi or Bluetooth.

[1663] Step 3: The server receives and decrypts the encrypted data sent from the terminal. The input is encrypted data, and the output is decrypted data. Specifically, the decryption algorithm runs on the server to decrypt the received data and add a timestamp.

[1664] Step 4: The server analyzes biometric and facial expression information. The input is decoded data, and the output is the analysis result. Specifically, biometric analysis algorithms and emotion analysis engines (e.g., CNN or RNN) operate to perform data analysis to evaluate health and emotional states.

[1665] Step 5: The server uses a generative AI (e.g., GPT-4) based on the analysis results to generate specific action advice. The input is the analysis results, and the output is personalized advice. Specifically, the generative AI operates and generates advice using the analysis results and specific prompt sentences.

[1666] Step 6: The server sends the generated advice to the user's terminal and simultaneously notifies the store staff. The input is the generated advice, and the output is the result of the advice being delivered to the terminal and the store staff. Specifically, notification management software operates to send notifications to the user's terminal and the store staff.

[1667] Step 7: Users submit feedback on the effectiveness of the advice and promotions provided. The input is the user's feedback, and the output is feedback data. Specifically, a feedback collection app operates, and users input their thoughts and suggestions for improvement through the app.

[1668] Step 8: The server receives feedback from the user, analyzes the data to improve the accuracy of the analysis and generation methods, and uses it for future analyses and advice generation. The input is feedback data, and the output is the learning result for accuracy improvement. Specifically, the feedback analysis engine runs and retrains the machine learning algorithm.

[1669] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1670] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1671] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1672] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1673] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1674] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1675] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1676] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1677] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1678] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1679] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1680] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1681] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1682] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1683] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1684] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1685] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1686] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1687] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1688] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1689] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1690] The following is further disclosed regarding the embodiments described above.

[1691] (Claim 1)

[1692] A terminal that collects the user's biometric information in real time,

[1693] A communication means for transmitting biometric information collected by the aforementioned terminal to a server,

[1694] An analytical means for analyzing the aforementioned biological information and evaluating the health status,

[1695] A generation means for generating advice that recommends specific actions to the user based on the aforementioned health status assessment,

[1696] A transmission means for sending the generated advice to the user's terminal,

[1697] A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means,

[1698] A system that includes this.

[1699] (Claim 2)

[1700] The system according to claim 1, wherein the analysis means detects outliers using statistical methods.

[1701] (Claim 3)

[1702] The system according to claim 1, wherein the generation means generates personalized advice using artificial intelligence.

[1703] "Example 1"

[1704] (Claim 1)

[1705] A terminal that collects the user's biometric information in real time,

[1706] A communication means for transmitting biometric information collected by the aforementioned terminal to a server,

[1707] Receiving and storage means for receiving the aforementioned biological information and storing it in a database,

[1708] An analysis means for analyzing biological information stored in the aforementioned database and evaluating the health status,

[1709] A generation means for generating advice that recommends specific actions to the user based on the aforementioned health status assessment,

[1710] A transmission means for sending the generated advice to the user's terminal,

[1711] A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means and generation means,

[1712] A system that includes this.

[1713] (Claim 2)

[1714] The system according to claim 1, wherein the analysis means detects outliers using statistical methods.

[1715] (Claim 3)

[1716] The system according to claim 1, wherein the generation means generates personalized advice using artificial intelligence.

[1717] "Application Example 1"

[1718] (Claim 1)

[1719] A terminal that collects the user's biometric information in real time,

[1720] A communication means for transmitting biometric information collected by the aforementioned terminal to a server,

[1721] An analytical means for analyzing the aforementioned biological information and evaluating the health status,

[1722] A generation means for generating advice that recommends specific actions to the user based on the aforementioned health status assessment,

[1723] A transmission means for sending the generated advice to the user's terminal,

[1724] A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means,

[1725] A service provision means that provides in-store services and recommends products based on the user's biometric information,

[1726] A system that includes this.

[1727] (Claim 2)

[1728] The system according to claim 1, wherein the analysis means detects outliers using statistical methods.

[1729] (Claim 3)

[1730] The system according to claim 1, wherein the generation means generates personalized advice using artificial intelligence.

[1731] "Example 2 of combining an emotion engine"

[1732] (Claim 1)

[1733] A terminal that collects the user's biometric information in real time,

[1734] A communication means for transmitting biometric information, facial expression information, and voice data collected by the aforementioned terminal to a server,

[1735] An analysis means for analyzing the aforementioned biometric information, facial expression information, and voice data to evaluate health status and emotional state,

[1736] A generation means for generating advice that recommends specific actions to the user based on the aforementioned evaluation of health and emotional state,

[1737] A transmission means for sending the generated advice to the user's terminal,

[1738] A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means,

[1739] A system that includes this.

[1740] (Claim 2)

[1741] The system according to claim 1, wherein the analysis means detects abnormal values ​​in biological information using statistical methods and estimates emotional states from facial expression information and voice data using a machine learning algorithm.

[1742] (Claim 3)

[1743] The system according to claim 1, wherein the generation means generates personalized advice using a generation AI model.

[1744] "Application example 2 when combining with an emotional engine"

[1745] (Claim 1)

[1746] A terminal that collects the user's biometric information in real time,

[1747] A communication means for transmitting biometric information and facial expression information collected by the terminal to a server,

[1748] An analysis means for analyzing the aforementioned biological information and facial expression information to evaluate health status and emotional state,

[1749] A generation means for generating advice that recommends specific actions to the user based on the aforementioned assessment of health and emotional state,

[1750] A transmission means for sending the generated advice to the user's terminal,

[1751] A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means,

[1752] A notification method to inform store staff of the aforementioned advice,

[1753] A system that includes this.

[1754] (Claim 2)

[1755] The system according to claim 1, wherein the analysis means detects outliers using statistical methods and machine learning models.

[1756] (Claim 3)

[1757] The system according to claim 1, wherein the generation means uses artificial intelligence to generate personalized advice and appropriate services in a store. [Explanation of Symbols]

[1758] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A terminal that collects the user's biometric information in real time, A communication means for transmitting biometric information collected by the aforementioned terminal to a server, An analytical means for analyzing the aforementioned biological information and evaluating the health status, A generation means for generating advice that recommends specific actions to the user based on the aforementioned health status assessment, A transmission means for sending the generated advice to the user's terminal, A learning means that receives user feedback based on the aforementioned advice and improves the accuracy of the analysis means, A system that includes this.

2. The system according to claim 1, wherein the analysis means detects anomalies using statistical methods.

3. The system according to claim 1, wherein the generation means generates personalized advice using artificial intelligence.

Images