System

Abstract

A system is provided.SOLUTION: A system comprising: means for measuring biological data of a user in real time; means for analyzing a condition of the user such as a complexion and an expression; means for receiving the measured biological data and an analysis result; means for analyzing a physical condition and a condition of the user using the received data; means for referring to fortune data of the day; means for generating advice based on the analysis result and the fortune data; and means for providing the generated advice by voice and text.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people suffer from stress and poor health. This has created a demand for customized advice tailored to each individual's health condition. However, conventional systems lack the technology to analyze an individual's physical condition and condition in real time and provide appropriate lifestyle advice based on that analysis. Furthermore, there is no established method for generating advice that incorporates data on auspicious and unlucky days, meaning users are not adequately supported to take optimal actions based on their daily condition. This has created a challenge in improving users' quality of life. [Means for solving the problem]

[0005] The present invention is a system including means for measuring a user's biometric data in real time, means for analyzing the user's condition such as facial color and facial expression, means for receiving the measured biometric data and analysis results, means for analyzing the user's physical condition and condition using the received data, means for referencing the current day's fortune data, means for generating advice based on the analysis results and fortune data, and means for providing the generated advice in voice and text. Furthermore, the system also includes means for collecting feedback from the user and improving the accuracy of advice from the next time onwards, and means for converting the generated advice into voice using a voice synthesis engine and providing it via a speaker, thereby enabling the system to provide optimal advice tailored to individual physical condition and condition, thereby improving the quality of life of users.

[0006] "Biometric data" refers to data relating to the physical condition of a user, such as heart rate, body temperature, number of steps taken, and sleep patterns.

[0007] The "analysis means" is a system or device that has the function of determining the user's physical condition or condition from measured data and images.

[0008] "Good or bad fortune data" is calendar data for determining whether a particular day is auspicious or unlucky.

[0009] The "advice generation means" is a system or device that has the function of creating specific guidance for the user's daily activities based on the user's physical condition, condition, and good and bad fortune data.

[0010] A "speech synthesis engine" is a technology or device for converting text data into natural-sounding speech.

[0011] A "speaker" is a device that outputs generated audio data to the user as sound.

[0012] A "feedback collection means" is a system or device that has the ability to obtain responses or opinions from users.

[0013] "Real-time" is a term that refers to a time frame in which data is processed and transmitted immediately, with little or no delay.

[0014] The "analysis results" are judgment information and evaluation values ​​obtained from the measured and analyzed data.

[0015] A "generation means" is a system or device that has the function of generating new information or data based on input information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] A specific embodiment for carrying out the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that information. The program processing and specific examples will be described in detail below.

[0038] Overall system configuration

[0039] The system mainly consists of the following components:

[0040] Terminal (wearable device)

[0041] AI Smart Camera

[0042] server

[0043] speaker

[0044] 1. User data collection

[0045] Terminal

[0046] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[0047] The collected data is sent to the server in real time or in batches.

[0048] AI Smart Camera

[0049] It captures the user's face in real time and analyzes their facial expressions, complexion, and movements.

[0050] As an example, stress levels and fatigue levels can be determined from facial color.

[0051] The analysis results are sent as data to the server.

[0052] 2. Data Receipt and Analysis

[0053] server

[0054] It receives data sent from the device and the AI ​​smart camera and stores it in a database.

[0055] The user's physical condition and health are analyzed based on the saved data, using a machine learning model.

[0056] As an example, if the user's heart rate is higher than normal and the quality of sleep is poor, the user is determined to be in poor health.

[0057] 3. Reference to fortune data

[0058] server

[0059] Based on the user's profile, the system retrieves the day's fortune data from the database.

[0060] If the data on lucky and unlucky days has been updated, the latest information is acquired and referenced accordingly.

[0061] 4. Generating Advice

[0062] server

[0063] Based on the analysis results and good and bad fortune data, a generative AI model is used to generate specific advice.

[0064] As an example, if the user is in poor health and it is an unlucky day, the system generates advice such as "Try not to push yourself too hard today and try to relax." If the user is in good health and it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[0065] 5. Providing advice

[0066] server

[0067] The generated advice is sent to the speaker and the device.

[0068] Advice is provided in audio and text formats.

[0069] speaker

[0070] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[0071] As an example, advice may be given as "Try to relax and not push yourself too hard today."

[0072] Terminal

[0073] Display advice in text format on the screen.

[0074] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[0075] Specific examples

[0076] If the user is unwell

[0077] Terminal

[0078] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[0079] AI Smart Camera

[0080] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[0081] server

[0082] Based on the received data, it is determined that the user is in poor health.

[0083] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[0084] Taking into consideration the poor physical condition and the fact that it is an unlucky day, the advice generated is "Try not to push yourself today and try to relax."

[0085] The generated advice is sent to the speaker and the device.

[0086] speaker

[0087] The voice tells the user, "Try not to push yourself too hard today and try to relax."

[0088] Terminal

[0089] The display will say, "Try not to push yourself too hard today and try to relax."

[0090] If the user is in good health

[0091] Terminal

[0092] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[0093] AI Smart Camera

[0094] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[0095] server

[0096] Based on the received data, it is determined that the user's physical condition is good.

[0097] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[0098] Taking into account the fact that the person is in good health and that it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[0099] The generated advice is sent to the speaker and the device.

[0100] speaker

[0101] A voice tells the user, "Today is a great day to be active. Try a new hobby."

[0102] Terminal

[0103] The display will say, "Today is a great day to be active. Try a new hobby."

[0104] The above is a specific embodiment for carrying out the present invention.

[0105] The processing flow will be explained below.

[0106] Step 1:

[0107] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits the measured data to a server in real time.

[0108] Step 2:

[0109] The AI ​​smart camera captures the user's face and analyzes their facial color, facial expressions, and movements using image analysis algorithms. The analyzed data is then sent to a server as user condition information.

[0110] Step 3:

[0111] The server receives biometric and condition data from the device and AI smart camera, stores it in a database, checks the data for consistency, and requests the data again if there is any inconsistency.

[0112] Step 4:

[0113] The server retrieves the latest biometric and condition data from the database and uses a machine learning model to analyze the user's physical condition and state, combining factors such as a high heart rate or poor sleep quality to provide a comprehensive assessment of their physical condition.

[0114] Step 5:

[0115] The server retrieves the fortune data for that day from the database based on the user's profile information. If the fortune data has been updated, the server retrieves the new information and updates the database.

[0116] Step 6:

[0117] The server combines the results of the health analysis with the auspicious data and uses a generative AI model to generate optimal advice for the day. The advice includes encouraging rest when the user is feeling unwell and encouraging activity when the user is feeling well.

[0118] Step 7:

[0119] The server formats the generated advice into JSON or XML format and sends it to the speaker and device, allowing the advice to be provided appropriately in both audio and text formats.

[0120] Step 8:

[0121] The speaker receives advice from the server, converts it using a speech synthesis engine, and conveys it to the user in a natural voice. The speaker provides advice such as, "Try to relax and not push yourself too hard today," or "Today is a great day to be active. Why not try a new hobby?"

[0122] Step 9:

[0123] The device displays the advice received from the server in text format on the display. The displayed advice matches the audio advice from the speaker.

[0124] Step 10:

[0125] The user acts based on the advice provided, and then inputs feedback about the advice through the terminal, such as whether the advice was helpful or how their physical condition has changed.

[0126] Step 11:

[0127] The device sends user feedback to the server, which is used to improve the accuracy of advice.

[0128] Step 12:

[0129] The server stores the feedback in a database and uses it to generate future advice, continuously improving the accuracy of the entire system.

[0130] Example 1

[0131] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] Conventional health management systems only measure a user's biometric data and condition, but lack the ability to provide individually optimized advice. Furthermore, they are unable to generate advice that takes into account the user's daily auspicious and unlucky data, and do not take into account the user's actual situation or day. As a result, the advice provided is general, making it difficult to provide specific help for the user's health condition or activities. Furthermore, there is a lack of utilization of user feedback to improve the accuracy of the advice.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0134] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition, such as facial color and facial expression, and a means for receiving the measured biometric data and the analysis results. This makes it possible to grasp the user's health state and condition in real time and provide individually optimized advice. The system also includes a means for automatically generating advice using a generative AI model and a means for generating prompt sentences based on the analysis results, making it possible to quickly provide highly personalized advice. Furthermore, the generated advice is converted into speech using a speech synthesis engine and provided through a speaker, allowing for quick and effective notification to the user. These means enable the provision of personalized health advice desired by the user and improved accuracy.

[0135] "Biometric data" refers to data that indicates the user's health condition, including heart rate, body temperature, number of steps taken, sleep patterns, etc.

[0136] "Condition" is information that indicates the user's physical and mental state, and is data that is analyzed based on facial color, facial expressions, movements, etc.

[0137] "Means for measuring in real time" refers to a device or method for continuously measuring a user's biometric data and instantly obtaining the results.

[0138] The term "analyzing means" refers to a device or method for evaluating the user's physical condition or health status based on measured biometric data and user condition data.

[0139] "Means for receiving" refers to a device or method for receiving data transmitted from a measurement device or an analytical device.

[0140] "Good and bad fortune data" is information that indicates the fortune and lucky data for that day, and serves as a reference when the user decides what to do.

[0141] "Means for generating advice" refers to a device or method that generates specific advice or recommended actions to be provided to a user based on the analysis results and good or bad luck data.

[0142] "Generative AI model" refers to a machine learning model that uses artificial intelligence technology to generate appropriate advice or responses from data.

[0143] A "prompt sentence" is an instruction sentence input into a generative AI model that prompts the model to generate specific advice.

[0144] A "speech synthesis engine" refers to software or hardware technology for converting text data into natural-sounding speech.

[0145] MODE FOR CARRYING OUT THE INVENTION

[0146] Overall system configuration

[0147] This system monitors the user's physical condition and condition in real time and provides optimal advice. Specifically, it consists of the following components:

[0148] Terminal (wearable device): Continuously measures the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and transmits it to the server in real time or in batches.

[0149] AI Smart Camera: Captures the user's face in real time, analyzes their facial expressions, complexion, and movements, and sends the results to a server.

[0150] Server: Stores and analyzes the received data and generates advice using generative AI models.

[0151] Speaker: Provides the generated advice aloud using a speech synthesis engine.

[0152] 1. User data collection

[0153] Terminal

[0154] The terminal constantly measures the user's biometric data. Examples of wearable devices include Apple Watch and Fitbit.

[0155] Heart rate is measured every minute and body temperature every hour, and the data is sent to the server via HTTP POST requests.

[0156] AI Smart Camera

[0157] The camera captures the user's face in real time, and uses hue and saturation analysis for facial color analysis, and a face recognition library (e.g., OpenCV) for facial expression analysis.

[0158] The analysis results are sent to the server in real time.

[0159] 2. Data Receipt and Analysis

[0160] server

[0161] The server receives the data sent from the device and the AI ​​smart camera and stores it in a database (e.g., MySQL, PostgreSQL).

[0162] Based on the stored data, the user's physical condition and health are analyzed using a machine learning model (e.g., TensorFlow, PyTorch).

[0163] 3. Reference to fortune data

[0164] server

[0165] The server retrieves the day's fortune data from the database based on the user's profile data.

[0166] The latest fortune data is updated daily from public APIs, etc.

[0167] 4. Generating Advice

[0168] server

[0169] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate specific advice based on the analysis results and the fortune data.

[0170] An example of a specific prompt sentence is "Generate advice when the user's heart rate is higher than normal and the quality of sleep is poor. Also, when the fortune data indicates an unlucky day."

[0171] 5. Providing advice

[0172] server

[0173] The generated advice is sent to the speaker and device via a REST API in JSON format.

[0174] speaker

[0175] The speaker uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert text data into natural-sounding speech and convey it to the user.

[0176] Advice such as "Try not to push yourself too hard today and try to relax" is provided via voice.

[0177] Terminal

[0178] The device will display advice in text format on the display. For example, the smartwatch will say, "Today is a great day to be active. Try a new hobby."

[0179] Specific examples

[0180] As a specific example of how it works, if a user is feeling unwell, the device will send data such as a higher than normal heart rate and poor quality sleep to the server, and the AI ​​smart camera will analyze that the user looks pale and tired, and send this to the server. After receiving this data, the server determines that the user is feeling unwell and checks the fortune data to see if that day is an unlucky day. As a result, it generates advice such as "Try not to push yourself today and try to relax," and sends this to the speaker and device.

[0181] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0182] Step 1: Measure and send data

[0183] Terminal

[0184] Input: Real-time biometric data such as the user's heart rate, body temperature, steps taken, and sleep patterns.

[0185] What it does: A wearable device (e.g., a smartwatch) measures your heart rate every minute and your body temperature every hour.

[0186] Data processing: These biometric data are collected and sent to the server via an HTTP POST request.

[0187] Output: Formatted biometric data (JSON format).

[0188] Step 2: Analyzing the user's facial data

[0189] AI Smart Camera

[0190] Input: User's facial image and video data.

[0191] How it works: The camera captures the user's face in real time and analyzes their facial color and facial expressions. Hue and saturation analysis is used for facial color analysis, and a face recognition library (e.g., OpenCV) is used for facial expression analysis.

[0192] Data calculation: Facial color analysis and facial expression recognition algorithms are applied to quantify stress levels and fatigue levels.

[0193] Output: Analysis results (e.g., data such as pale complexion or tired facial expression).

[0194] Step 3: Receiving and storing data

[0195] server

[0196] Input: Biometric and facial data sent from the device and AI smart camera.

[0197] Specific operation: The server receives the HTTP POST request and saves it in a database (e.g. MySQL, PostgreSQL).

[0198] Data processing: The received data is properly formatted and a timestamp is added to each data.

[0199] Output: Data stored in a database.

[0200] Step 4: Analyzing the user's physical condition and condition

[0201] server

[0202] Input: Stored biometric and facial data.

[0203] Specific operation: Uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the user's physical condition and condition.

[0204] Data calculations: Combining multiple data points to assess your physical condition based on specific conditions (e.g., higher than normal heart rate and poor sleep quality).

[0205] Output: Analysis results (e.g., poor health).

[0206] Step 5: Check the luck data

[0207] server

[0208] Input: User profile data.

[0209] Specific operation: Obtains the day's fortune data from the database and updates the latest information from the public API as needed.

[0210] Data calculation: Calculates fortune based on the user's date of birth and other auspicious and inauspicious factors.

[0211] Output: Fortune data for that day (e.g. lucky day, unlucky day).

[0212] Step 6: Generating Advice

[0213] server

[0214] Input: Analysis results and fortune data.

[0215] Specific operation: Uses a generative AI model (e.g., OpenAI GPT-4) to generate advice based on the prompt.

[0216] Data processing: The prompt text "Generate advice when the user's heart rate is higher than normal and the quality of their sleep is poor. Also, what is the auspicious / unlucky date?" is input into the generative AI model, and natural language processing is performed.

[0217] Output: The generated advice (e.g., "Try to relax and take it easy today.").

[0218] Step 7: Providing advice

[0219] server

[0220] Input: The generated advice.

[0221] Specific operation: Advice is sent to the speaker and device in JSON format.

[0222] Output: Sending advice data.

[0223] speaker

[0224] Input: Advice data in JSON format.

[0225] Specific operation: Uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert into natural-sounding speech.

[0226] Output: Spoken advice (e.g., "Try to relax and take it easy today.").

[0227] Terminal

[0228] Input: Advice data in JSON format.

[0229] Specific operation: Display advice in text format on the device display.

[0230] Output: Advice in text form (e.g., "Today is a great day to be active. Try a new hobby.").

[0231] (Application example 1)

[0232] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] In today's world, there is a demand for a system that analyzes a user's physical condition and mood in real time and provides optimal advice when the user is selecting a product in a virtual store. Conventional virtual stores do not provide product suggestions that take into account the user's instantaneous physical condition and mood, and as a result, the user's satisfaction and purchasing motivation are not fully stimulated. The object of the present invention is to solve this problem by providing a system that provides optimal advice and product suggestions in real time based on the user's physical condition and mood.

[0234] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0235] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition such as complexion and facial expression, and a means for receiving the measured biometric data and the analysis results, which makes it possible to analyze the user's physical condition and provide optimal advice and product suggestions in real time within the virtual store.

[0236] "User's biometric data" refers to data that indicates the user's physical condition, such as the user's heart rate, body temperature, number of steps taken, and sleep patterns.

[0237] "Analysis of facial color and facial expression" is the process of analyzing the user's facial color and facial expression and determining the user's condition from that information.

[0238] "Means for receiving measured biometric data and analysis results" refers to a device or system that receives data sent from a user's wearable device or AI smart camera.

[0239] "Analysis of the user's physical condition and condition" is the process of comprehensively analyzing the user's health and mood based on biometric data, facial color, and facial expression data.

[0240] "Good or bad fortune data" is fortune information such as whether it is an auspicious day or an unlucky day based on the current date.

[0241] The "means for generating advice" is a system or algorithm that uses the analysis results and the fortune data to generate specific advice tailored to the user's condition.

[0242] The "means for providing the generated advice by voice and text" refers to means for providing the generated advice by voice and on a display, and is a mechanism for conveying the advice to the user.

[0243] "A means for suggesting optimal products based on the user's physical condition and condition within a virtual store" is a system that suggests optimal products in a virtual reality environment, taking into account the user's health condition and mood.

[0244] The system embodying the present invention monitors the user's physical condition in real time and provides optimal advice and product suggestions. This system is composed of the following main components:

[0245] Overall system configuration

[0246] The system mainly consists of the following components:

[0247] Wearable devices: Continuously measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and send it to a server in real time or in batches.

[0248] AI Smart Camera: Captures the user's face in real time and analyzes their facial expressions, complexion, and movements, and sends the analysis results to the server.

[0249] Server: Receives data sent from the wearable device and AI smart camera and stores it in a database. Analyzes the user's physical condition and health based on the stored data. A machine learning model is used for the analysis.

[0250] Speaker and display: Generated advice and product suggestions are delivered to the user via voice and text.

[0251] Data collection and analysis

[0252] The server receives biometric data, facial color, and facial expression data sent from the wearable device and AI smart camera. This data is used to analyze the user's physical condition and condition using a machine learning model. Based on the analysis results, the server determines the user's condition and refers to the fortune data for the day.

[0253] Generating advice and product recommendations

[0254] The server uses a generative AI model based on the analysis results and auspicious / lucky data to generate specific advice and product suggestions. For example, if you are feeling unwell and it's an unlucky day, the server generates advice such as, "Try not to push yourself too hard today and relax." If you are feeling well and it's an auspicious day, the server generates a product suggestion such as, "Today is a great day to be active. Try a new product."

[0255] Providing advice and product suggestions

[0256] The generated advice and product suggestions are provided to the user through a speaker and a display. The speaker uses a speech synthesis engine to convert text data into natural-sounding speech, and the display displays the information in text format.

[0257] The specific hardware and software used

[0258] Hardware:

[0259] Wearable devices: Apple Watch, Fitbit, etc.

[0260] AI smart camera: OpenCV, Azure Face API, etc.

[0261] Speakers and Displays

[0262] software:

[0263] Data analysis: TensorFlow, AWS SageMaker, etc.

[0264] Speech synthesis engine: Google Text-to-Speech API, IBM Watson TTS, etc.

[0265] Generative AI models: GPT-3, etc.

[0266] Specific examples

[0267] simulation:

[0268] A user logs into the virtual shop.

[0269] The wearable device measures the user's heart rate, body temperature, steps taken, and sleep patterns and sends the data to a server.

[0270] The AI ​​smart camera analyzes the user's complexion and facial expressions and sends the results to the server.

[0271] The server analyzes this data and determines the user's physical condition in real time.

[0272] If you are in poor health and it is an unlucky day, the advice is, "Don't push yourself today, take your time choosing your products."

[0273] If you are in good health and it's an auspicious day, the advice is, "Be active, have fun, and try a new product today!"

[0274] Example prompt sentence:

[0275] Prompt for a generative AI model (e.g. GPT-3):

[0276] "Generate advice for when a user is in poor health and having an unlucky day. Specific advice: 'Try to relax and not push yourself too hard today.'"

[0277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0278] Step 1:

[0279] The user wears a wearable device, which measures the user's biometric data, such as heart rate, body temperature, number of steps, and sleep patterns, in real time and transmits it to a server. The input is the measured biometric data, and the output is the data sent to the server.

[0280] Step 2:

[0281] The server receives the biometric data sent from the wearable device and stores the data in a database. The input is the biometric data from the wearable device, and the output is the data stored in the database. A storage system is used to store the data.

[0282] Step 3:

[0283] The user stands in front of the AI ​​smart camera. The camera captures the user's face and analyzes their facial expressions, complexion, and movements. The input is the captured image data, and the output is the analysis results. Image analysis software (e.g., OpenCV, Azure Face API) is used for the analysis.

[0284] Step 4:

[0285] The AI ​​smart camera sends the analysis results to the server. The input is the image analysis result, and the output is the data sent to the server. The server also stores this data in a database.

[0286] Step 5:

[0287] The server analyzes the user's physical condition and health using the received biometric data and image analysis results. The input is the biometric data and image analysis data stored in the database, and the output is the analysis results. Machine learning models (e.g., TensorFlow, AWS SageMaker) are used for the analysis.

[0288] Step 6:

[0289] Based on the analysis results, the server retrieves the current day's fortune data from the database. The input is the analysis results, and the output is the fortune data. The latest fortune data is retrieved by a database query.

[0290] Step 7:

[0291] The server uses a generative AI model (e.g., GPT-3) based on the analysis results and the fortune data to generate specific advice and product suggestions. The input is the analysis results and fortune data, and the output is the generated advice and product suggestions. A prompt sentence is input into the generative AI model, which generates the optimal advice.

[0292] Step 8:

[0293] The generated advice and product suggestions are sent from the server to the speaker and display. The input is the generated advice and product suggestions, and the output is text and audio that is displayed on the display and played back from the speaker.

[0294] Step 9:

[0295] The user receives advice and product suggestions through a speaker and a display. The input is the text displayed on the display and the audio played from the speaker, and the output is the user's behavior and purchasing decisions.

[0296] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0297] A specific embodiment for implementing the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized advice can be achieved.

[0298] Overall system configuration

[0299] The system mainly consists of the following components:

[0300] Terminal (wearable device)

[0301] AI Smart Camera

[0302] Emotion Engine

[0303] server

[0304] speaker

[0305] 1. User data collection

[0306] Terminal

[0307] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[0308] The collected data is sent to the server in real time or in batches.

[0309] AI Smart Camera

[0310] It captures the user's face in real time and analyzes their facial expression, facial expression, and movements.

[0311] As an example, stress levels and fatigue levels can be determined from facial color.

[0312] The analysis results are sent as data to the server.

[0313] 2. Emotion recognition

[0314] Emotion Engine

[0315] It analyzes the user's facial images and voice data obtained from the AI ​​smart camera and recognizes the user's emotions (joy, sadness, anger, etc.).

[0316] The recognized emotion data is sent to the server.

[0317] 3. Data Receipt and Analysis

[0318] server

[0319] It receives biometric data, condition data, and emotion data from the device, AI smart camera, and emotion engine, and stores them in a database.

[0320] Check the consistency of the data, and if there is any inconsistency, request the data again.

[0321] 4. Analysis of user's physical condition and emotions

[0322] server

[0323] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to analyze the user's physical condition and emotions. Here, factors such as a high heart rate, poor sleep quality, and the frequency of happy facial expressions are combined to form a comprehensive evaluation.

[0324] 5. Reference to fortune data

[0325] server

[0326] Based on the user's profile information, the system retrieves the day's fortune data from the database.

[0327] If the data on lucky and unlucky days has been updated, the latest information is acquired accordingly and the database is updated.

[0328] 6. Generating Advice

[0329] server

[0330] The analysis results, emotional data, and good and bad luck data are combined and the optimal advice for the day is generated using a generative AI model.

[0331] The advice includes encouraging rest when you're feeling unwell and activity when you're feeling well, and it adjusts the advice based on the perceived emotions.

[0332] 7. Providing advice

[0333] server

[0334] The generated advice is formatted in JSON or XML and sent to the speaker and device, allowing the advice to be provided appropriately in both voice and text.

[0335] speaker

[0336] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[0337] As an example, advice can be given as "Try to relax and take it easy today" or "Today is a great day to be active. Try a new hobby."

[0338] Terminal

[0339] Display advice in text format on the screen.

[0340] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[0341] Specific examples

[0342] When the user is unwell and emotionally sad

[0343] Terminal

[0344] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[0345] AI Smart Camera

[0346] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[0347] Emotion Engine

[0348] The sadness emotion is recognized from the user's face and transmitted to the server.

[0349] server

[0350] Based on the received data, it is determined that the user is in poor health.

[0351] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[0352] Taking into consideration that the person is feeling unwell, that it is an unlucky day, and that the emotion is sadness, the advice generated is, "Try not to push yourself today and try to relax. Read your favorite book to change your mood."

[0353] The generated advice is sent to the speaker and the device.

[0354] speaker

[0355] The voice tells the user, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[0356] Terminal

[0357] The display will say, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[0358] If the user is in good health and the emotion is joy

[0359] Terminal

[0360] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[0361] AI Smart Camera

[0362] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[0363] Emotion Engine

[0364] The joyful emotion is recognized from the user's face and transmitted to the server.

[0365] server

[0366] Based on the received data, it is determined that the user's physical condition is good.

[0367] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[0368] Taking into consideration that the emotion is joy in addition to the person being in good health and that it is an auspicious day, the advice generated is "Today is a great day to be active. Have a good time with your friends."

[0369] The generated advice is sent to the speaker and the device.

[0370] speaker

[0371] A voice tells the user, "Today is a great day to be active and have fun with friends."

[0372] Terminal

[0373] The display will say, "Today is a great day to be active and have fun with friends."

[0374] The above is a specific embodiment for carrying out the present invention.

[0375] The processing flow will be explained below.

[0376] Step 1:

[0377] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits this data to a server in real time.

[0378] Step 2:

[0379] The AI ​​smart camera captures the user's face in real time and analyzes their facial expressions, complexion, and movements. The analyzed data is sent to a server as information about the user's condition.

[0380] Step 3:

[0381] The emotion engine analyzes the user's facial images and voice data acquired from the AI ​​smart camera and recognizes the user's emotions (happiness, sadness, anger, etc.). The recognized emotion data is sent to the server.

[0382] Step 4:

[0383] The server receives biometric data, condition data, and emotion data sent from the device, AI smart camera, and emotion engine, and stores it in a database. It checks the consistency of the received data and requests the data again if there is any inconsistency.

[0384] Step 5:

[0385] The server retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, the overall analysis results may show a high heart rate, poor sleep quality, and sadness.

[0386] Step 6:

[0387] The server refers to the user's profile information and retrieves the lucky / unlucky data for that day from the database. If the lucky / unlucky data has been updated, the new information is updated in the database.

[0388] Step 7:

[0389] The server combines the results of the physical condition analysis, emotional data, and auspicious / unlucky data, and uses a generative AI model to generate the optimal advice for that day. The advice is based on the individual's physical condition and emotions. For example, if you are feeling unwell and sad, the advice might be, "Try to relax and not push yourself today. I recommend reading your favorite book to change your mood." If you are feeling good and happy, the advice might be, "Today is a great day. Have some fun outdoors with friends."

[0390] Step 8:

[0391] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The transmitted data is output as voice and text.

[0392] Step 9:

[0393] The speaker receives advice from the server and converts it into natural-sounding speech using a speech synthesis engine, which then relays it to the user. For example, the advice could be, "Try not to push yourself too hard today, but relax. Read your favorite book to change your mood."

[0394] Step 10:

[0395] The device will display the received advice in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[0396] Step 11:

[0397] The user acts on the advice provided. The effectiveness of the advice and the user's opinion are input into the device as feedback. For example, the user can input whether the advice was helpful or whether their physical condition improved.

[0398] Step 12:

[0399] The device sends user feedback to the server, which is used to make future advice more effective.

[0400] Step 13:

[0401] The server stores the feedback in a database and uses it to generate advice in future. This feedback information continuously improves the accuracy of the entire system.

[0402] Example 2

[0403] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] Conventional healthcare systems tend to collect only a user's biometric data and provide advice based solely on their physical condition. However, providing personalized advice that also takes into account the user's emotions and fortunes is important for comprehensive health management. Furthermore, the lack of a means to improve the accuracy of advice based on feedback makes continuous improvement difficult. This has led to the issue of not being able to fully meet user needs.

[0405] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric information of the user in real time, means for analyzing the user's condition such as complexion, facial expression, and movement, means for receiving the measured biometric data and analysis results, means for generating advice using a generative AI model, means for providing the generated advice in audio and text, and means for collecting feedback from the user and improving the accuracy of advice from the next time onwards. This makes it possible to provide personalized advice that comprehensively takes into account the user's physical condition, emotions, and fortune, and to continuously improve the accuracy of that advice.

[0406] "User's biological information" is data that indicates the user's physical condition, such as heart rate, body temperature, number of steps, and sleep pattern.

[0407] "Real-time acquisition means" refers to the ability of sensors or devices to collect data immediately, without time delay, and transmit that data to the system.

[0408] "User's condition such as complexion, facial expression, and movement" is subjective information that indicates the user's emotions, stress level, activity level, and so on.

[0409] "Means of analysis" refers to the algorithms and software functions used to analyze data and detect specific patterns or anomalies.

[0410] "Measured biometric data and analysis results" refers to biometric information obtained from sensors or devices and the results of analyzing that data.

[0411] "Means for receiving" refers to the interface or protocol for importing and storing data from the outside.

[0412] A "generative AI model" is an artificial intelligence algorithm or system that automatically generates an output based on specific input data.

[0413] The "means for generating advice" is a function that automatically creates recommended actions and points of caution for the user based on the collected data and analysis results.

[0414] The "means of providing by voice and text" is a function of reading out the generated advice as voice and displaying it as text.

[0415] "Means for collecting feedback" refers to interfaces and protocols for capturing opinions and reactions from users and reflecting them in the system.

[0416] A specific embodiment of this invention will be described. This system monitors the user's physical condition and emotions in real time and provides optimal advice based on the results. The entire system mainly consists of the following components: a terminal, an AI smart camera, an emotion engine, a server, and a speaker.

[0417] User data collection

[0418] Terminal (wearable device):

[0419] The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The wearable device is equipped with sensors such as a heart rate monitor, a body temperature sensor, and an accelerometer. For example, the heart rate monitor measures values ​​such as "90 BPM" and the body temperature sensor measures "36.5°C."

[0420] These biometric data are transmitted to a server periodically or in real time via Bluetooth or Wi-Fi.

[0421] AI Smart Camera:

[0422] The camera captures the user's facial color, facial expressions, and movements, and analyzes this data in real time. The camera has an image analysis algorithm built in to analyze facial color and facial expressions. For example, if the user's face is pale, it will be determined that the stress level is high.

[0423] The analyzed data is immediately sent to the server.

[0424] Emotion recognition and data reception

[0425] Emotion Engine:

[0426] Using image and audio data obtained from an AI smart camera, the system recognizes the user's emotions (happiness, sadness, anger, etc.). For example, it uses facial expression analysis technology to determine that the user is expressing happiness.

[0427] The recognized emotion data is transmitted to a server.

[0428] server:

[0429] It receives all data acquired from devices, AI smart cameras, and emotion engines, stores it in a database, checks the data consistency, and requests the data again if there is any inconsistency.

[0430] Analyzing physical condition and emotions and generating advice

[0431] server:

[0432] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to comprehensively analyze the user's physical condition and emotions. For example, if the user has a high heart rate, poor sleep quality, and sadness, the system will determine that the user is in poor health.

[0433] Based on the user's profile information, the system retrieves the day's fortune data from the database. If the fortune data is not up to date, the system retrieves the latest data from an external data source and updates the database.

[0434] The analysis results are combined with fortune data and a generative AI model (e.g., GPT-3) is used to generate prompts. Examples of prompts include:

[0435] Example prompt sentence:

[0436] User health data:

[0437] Heart rate: 90 BPM

[0438] Body temperature: 36.5℃

[0439] Steps: 10,000

[0440] Sleep Pattern: Good

[0441] User sentiment data:

[0442] Emotion: Joy

[0443] Expression: Cheerful

[0444] Activity: Active

[0445] Fortune Data:

[0446] Today's Fortune: Auspicious Day

[0447] Use this information to generate the best advice for your users.

[0448] A generative AI model receives the prompts and generates the best advice for the day based on them.

[0449] Providing advice

[0450] server:

[0451] The generated advice is formatted into JSON or XML format and sent to the speaker and device.

[0452] speaker:

[0453] A speech synthesis engine is used to convert the generated advice into natural-sounding speech and convey it to the user.

[0454] Device:

[0455] Advice is displayed in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[0456] This system will provide personalized advice that takes into account the user's physical condition, emotions, and fortune, and will be able to continuously improve its accuracy.

[0457] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0458] Program processing flow

[0459] Step 1:

[0460] Data Acquisition

[0461] Device: The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The device periodically reads data from the heart rate monitor and body temperature sensor, converts the data into JSON format, and stores it in a transmission queue.

[0462] Input: Biometric data from sensors.

[0463] Output: Biometric data in JSON format.

[0464] Step 2:

[0465] Sending data

[0466] Terminal: The terminal sends the collected biometric data to the server via Bluetooth or Wi-Fi. Specifically, the terminal sends the collected JSON data to the server's receiving API as an HTTP POST request.

[0467] Input: Biometric data obtained in step 1 in JSON format.

[0468] Output: HTTP request to the server.

[0469] Step 3:

[0470] Camera-based facial color and facial expression analysis

[0471] AI Smart Camera: Captures the user's facial expressions, facial expressions, and movements, and analyzes them in real time using image analysis algorithms. For example, facial expression analysis software analyzes the user's facial photo to determine emotions such as "sadness" or "happiness."

[0472] Input: A face image captured by the camera.

[0473] Output: Analyzed facial color and expression data.

[0474] Step 4:

[0475] Sending camera data

[0476] AI Smart Camera: Converts the analysis results into JSON format and sends them to the server. Sends the analysis data using an HTTP POST request.

[0477] Input: Facial color and expression data analyzed in step 3.

[0478] Output: JSON formatted data to the server.

[0479] Step 5:

[0480] Processing received data

[0481] Server: Receives biometric data and facial image analysis data sent from the device and AI smart camera. The receiving API receives this data and stores it in a database.

[0482] Input: HTTP requests from the device and the AI ​​smart camera.

[0483] Output: Biometric data and facial image analysis data stored in a database.

[0484] Step 6:

[0485] emotion recognition

[0486] Emotion Engine: Analyzes facial image data retrieved from a database to recognize the user's emotions. It uses an AI model to identify emotions such as "happiness" or "sadness."

[0487] Input: Facial image data stored in a database.

[0488] Output: Parsed emotion data.

[0489] Step 7:

[0490] Comprehensive analysis of data

[0491] Server: Retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, if the heart rate is high, sleep quality is poor, and the user is expressing sadness, it determines that the user is in poor health.

[0492] Input: Biometric data, facial expression data, and emotion data in the database.

[0493] Output: Comprehensively analyzed user physical and emotional data.

[0494] Step 8:

[0495] Fortune data reference

[0496] Server: Based on the user's profile information, retrieves the day's fortune data from the database. If the fortune data is not up to date, retrieves the latest data from an external interface and updates the database.

[0497] Input: User profile information.

[0498] Output: Fortune data for the day.

[0499] Step 9:

[0500] Prompt creation for advice generation

[0501] Server: Combines the user's biometric, emotional, and fortune data to generate prompts to input into the generative AI model. Examples of prompts include:

[0502] Example prompt sentence:

[0503] User health data:

[0504] Heart rate: 90 BPM

[0505] Body temperature: 36.5℃

[0506] Steps: 10,000

[0507] Sleep Pattern: Good

[0508] User sentiment data:

[0509] Emotion: Joy

[0510] Expression: Cheerful

[0511] Activity: Active

[0512] Fortune Data:

[0513] Today's Fortune: Auspicious Day

[0514] Use this information to generate the best advice for your users.

[0515] Input: Biometric data, emotional data, fortune data.

[0516] Output: Prompts to the generative AI model.

[0517] Step 10:

[0518] Generating Advice

[0519] Generative AI model: Generates optimal advice for the user based on the prompt. For example, it generates advice such as, "Try not to push yourself too hard today and try to relax."

[0520] Input: Prompt statement.

[0521] Output: The generated advice.

[0522] Step 11:

[0523] Sending Advice

[0524] Server: Formats the generated advice into JSON or XML format and sends it to the speaker and device, providing advice in voice and text.

[0525] Input: The generated advice.

[0526] Output: JSON or XML data to speakers and devices.

[0527] Step 12:

[0528] Voice and text advice provided

[0529] Speaker: Uses a speech synthesis engine to convert the JSON or XML data into speech and tells the user, "Try to relax and take it easy today."

[0530] Terminal: Display the text "Try not to push yourself too hard today and try to relax" on the display.

[0531] Input: Advice sent by the server.

[0532] Output: Providing audio and text advice to the user.

[0533] These are the specific processing steps of this system, which allows users to receive optimal advice based on their physical condition, emotions, and fortune.

[0534] (Application example 2)

[0535] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0536] Conventional self-driving vehicles lack real-time feedback based on the physical condition and emotions of the driver and passengers, making it difficult to provide optimal driving modes and entertainment. They also lack the ability to provide personalized advice based on the day's auspicious and unlucky data or the individual user's condition. These circumstances have led to insufficient safety and comfort within the vehicle.

[0537] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for measuring the user's biological data in real time, means for analyzing the user's condition such as facial color and facial expression, and means for receiving the measured biological data and the analysis results. This makes it possible to monitor the user's physical condition and emotions in real time in an autonomous vehicle and optimize the driving mode and entertainment based on the monitoring results.

[0538] "User's biometric data" refers to data that indicates a person's physical condition, such as heart rate, body temperature, number of steps taken, and sleep patterns.

[0539] "User condition such as facial color and facial expression" is information indicating the physical and emotional state of the user, including the color and facial expression of the user's face, and movements.

[0540] "Analysis means" refers to a device or software that evaluates and judges the user's physical condition and emotions based on measured data and images.

[0541] "Today's fortune data" is traditional or astrological information that indicates whether the day is auspicious or inauspicious for the user.

[0542] The "means for generating advice" is a device or software that suggests appropriate actions or measures to the user based on the collected data and analysis results.

[0543] "Means for providing generated advice in audio and text" refers to a device or software that displays and plays generated advice in audio or text format in order to convey the advice to the user in an easy-to-understand manner.

[0544] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[0545] "Driving modes" are settings that adjust the style and conditions under which an autonomous vehicle is driven.

[0546] "Entertainment providing means" refers to devices or software for playing entertainment content such as music and video for users.

[0547] "Means for recognizing emotions" refers to a device or software that analyzes a user's facial expressions, actions, and voice data to determine their emotional state.

[0548] A "generative AI model" is an algorithm or program that is trained by artificial intelligence to generate appropriate advice or suggestions from specific data.

[0549] A "prompt" is a specific input sentence used to generate an appropriate answer or output for a generative AI model.

[0550] A specific embodiment of the present invention will be described. This invention is a system that monitors the physical condition and emotions of occupants in an autonomous vehicle in real time and provides optimal driving modes and entertainment based on the monitoring results. The entire system is composed of the following main components.

[0551] System configuration

[0552] The system mainly consists of the following components:

[0553] Wearable devices

[0554] AI Camera

[0555] Emotion Engine

[0556] server

[0557] speaker

[0558] 1. Data Collection

[0559] Wearable devices

[0560] Wearable devices (e.g., smartwatches) constantly measure biometric data such as heart rate, body temperature, steps taken, and sleep patterns.

[0561] The collected data is sent to a server in real time at irregular intervals.

[0562] AI Camera

[0563] An AI camera installed inside the vehicle captures and analyzes the facial expressions, facial expressions, and movements of the occupants in real time.

[0564] The analysis results are sent to a server as data to determine each passenger's stress level and fatigue level.

[0565] 2. Emotion recognition

[0566] Emotion Engine

[0567] The emotion engine analyzes facial images and audio data obtained from the AI ​​camera to recognize the emotions of the occupants (joy, sadness, anger, etc.).

[0568] The recognized emotion data is transmitted to a server.

[0569] 3. Data Receipt and Analysis

[0570] server

[0571] It receives biometric data, condition data, and emotion data from the wearable device, AI camera, and emotion engine, and stores them in a database.

[0572] Check the consistency of the data and re-request the data if there is any inconsistency.

[0573] 4. Data analysis and advice generation

[0574] server

[0575] The latest biometric, condition, and emotional data is collected and the occupants' physical condition and emotions are analyzed using machine learning models.

[0576] The data on the day's fortunes is referenced and retrieved from the database.

[0577] The analysis results, emotional data, and good and bad fortune data are combined and a generative AI model is used to generate the best advice for the day.

[0578] The advice includes specific suggestions for action, such as when drivers should take a break and relax, or when they should play music to help them relax.

[0579] 5. Providing advice

[0580] speaker

[0581] The advice is given to the passengers using a speech synthesis engine, which converts text data into natural-sounding speech.

[0582] Terminal

[0583] Advice is displayed in text format on the in-car display.

[0584] Specific examples

[0585] Example 1: When the user is unwell and sad

[0586] The wearable device measures high heart rate and poor sleep quality and sends the results to a server.

[0587] The AI ​​camera analyzes whether the person's complexion is pale and they look tired, and sends the results to the server.

[0588] The emotion engine recognizes the sadness emotion from the user's face and sends it to the server.

[0589] The server determines that the user is feeling unwell and that it is an unlucky day, generates advice such as "Try not to push yourself today and try to relax. Read your favorite book to change your mood," and sends this advice to the device and speaker.

[0590] Example 2: When the user is in good health and the emotion is joy

[0591] The wearable device measures the heart rate, body temperature, and number of steps taken, and sends the results to the server.

[0592] The AI ​​camera analyzes whether the person has a good complexion and a bright expression, and sends the results to the server.

[0593] The emotion engine recognizes the emotion of joy from the user's face and transmits it to the server.

[0594] The server determines that the person is in good health and that it is an auspicious day, generates advice such as "Today is a great day to be active. Have a good time with friends," and sends it to the device and speaker.

[0595] Prompt example

[0596] "If the user has a high heart rate but is showing signs of joy, provide appropriate advice."

[0597] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0598] Step 1:

[0599] The wearable device measures the user's biometric data in real time, specifically, heart rate, body temperature, number of steps taken, sleep patterns, etc. This acquired data is input, and the wearable device sends it to a server.

[0600] Step 2:

[0601] The server stores the received biometric data in a database and checks its consistency. Specifically, it checks whether the data is missing or consistent. If there is an inconsistency, it requests the data again. The input to this process is the biometric data sent from the wearable device, and the output is the biometric data whose consistency has been confirmed.

[0602] Step 3:

[0603] The AI ​​camera captures the facial color and facial expressions of the user in the car in real time. Specific operations include analyzing facial color and facial expressions. These analysis results are input, and the AI ​​camera sends the analysis results to the server.

[0604] Step 4:

[0605] The server receives the analysis results sent from the AI ​​camera and stores them in a database. The input is the analysis result data, and the output is the saved analysis result data. The server also checks the consistency of this data.

[0606] Step 5:

[0607] The emotion engine analyzes emotions based on facial image data obtained from the AI ​​camera. Specifically, it uses facial expression data to recognize emotional states (happiness, sadness, anger, etc.). This recognition result is input, and the emotion engine sends it to the server.

[0608] Step 6:

[0609] The server receives the emotion data sent from the emotion engine and stores it in a database. The input is emotion data, and the output is the stored emotion data. The server also checks the integrity of the emotion data.

[0610] Step 7:

[0611] The server integrates all data received from the wearable device, AI camera, and emotion engine to obtain the latest biometric, condition, and emotion data. These data are the input, and the output is the integrated dataset.

[0612] Step 8:

[0613] The server uses the integrated data set to analyze the user's physical condition and emotions using a machine learning model. Specifically, it evaluates heart rate and facial expression changes to determine the user's overall physical condition and emotions. The results of this analysis are the input, and the output is the analysis result data.

[0614] Step 9:

[0615] The server retrieves the fortune data for that day from the database. The input is the date information, and the output is the fortune data for that day. The server combines this with the analysis results.

[0616] Step 10:

[0617] The server uses a generative AI model to generate optimal advice based on the analysis results and the good and bad fortune data. Specifically, a prompt sentence is used as input to the model to generate appropriate advice. An example of this prompt sentence is, "If the user's heart rate is high but they are showing signs of joy, please provide appropriate advice." The input is the integrated data and the prompt sentence, and the output is the generated advice.

[0618] Step 11:

[0619] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The input is the generated advice, and the output is the formatted advice data.

[0620] Step 12:

[0621] The terminal displays the advice in text format on the in-car display. Specifically, it converts the transmitted format data into text and displays it on the display. The input is the formatted advice data, and the output is the display.

[0622] Step 13:

[0623] The speaker uses a speech synthesis engine to convert the advice into natural-sounding speech and convey it to the passengers. Specifically, it converts text data into speech data and plays it back. The input is formatted advice data, and the output is advice conveyed in speech.

[0624] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0625] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0627] [Second embodiment]

[0628] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0629] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0630] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0632] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0634] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0635] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0636] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0637] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0638] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0639] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0640] A specific embodiment for carrying out the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that information. The program processing and specific examples will be described in detail below.

[0641] Overall system configuration

[0642] The system mainly consists of the following components:

[0643] Terminal (wearable device)

[0644] AI Smart Camera

[0645] server

[0646] speaker

[0647] 1. User data collection

[0648] Terminal

[0649] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[0650] The collected data is sent to the server in real time or in batches.

[0651] AI Smart Camera

[0652] It captures the user's face in real time and analyzes their facial expressions, complexion, and movements.

[0653] As an example, stress levels and fatigue levels can be determined from facial color.

[0654] The analysis results are sent as data to the server.

[0655] 2. Data Receipt and Analysis

[0656] server

[0657] It receives data sent from the device and the AI ​​smart camera and stores it in a database.

[0658] The user's physical condition and health are analyzed based on the saved data, using a machine learning model.

[0659] As an example, if the user's heart rate is higher than normal and the quality of sleep is poor, the user is determined to be in poor health.

[0660] 3. Reference to fortune data

[0661] server

[0662] Based on the user's profile, the system retrieves the day's fortune data from the database.

[0663] If the data on lucky and unlucky days has been updated, the latest information is acquired and referenced accordingly.

[0664] 4. Generating Advice

[0665] server

[0666] Based on the analysis results and good and bad fortune data, a generative AI model is used to generate specific advice.

[0667] As an example, if the user is in poor health and it is an unlucky day, the system generates advice such as "Try not to push yourself too hard today and try to relax." If the user is in good health and it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[0668] 5. Providing advice

[0669] server

[0670] The generated advice is sent to the speaker and the device.

[0671] Advice is provided in audio and text formats.

[0672] speaker

[0673] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[0674] As an example, advice may be given as "Try to relax and not push yourself too hard today."

[0675] Terminal

[0676] Display advice in text format on the screen.

[0677] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[0678] Specific examples

[0679] If the user is unwell

[0680] Terminal

[0681] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[0682] AI Smart Camera

[0683] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[0684] server

[0685] Based on the received data, it is determined that the user is in poor health.

[0686] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[0687] Taking into consideration the poor physical condition and the fact that it is an unlucky day, the advice generated is "Try not to push yourself today and try to relax."

[0688] The generated advice is sent to the speaker and the device.

[0689] speaker

[0690] The voice tells the user, "Try not to push yourself too hard today and try to relax."

[0691] Terminal

[0692] The display will say, "Try not to push yourself too hard today and try to relax."

[0693] If the user is in good health

[0694] Terminal

[0695] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[0696] AI Smart Camera

[0697] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[0698] server

[0699] Based on the received data, it is determined that the user's physical condition is good.

[0700] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[0701] Taking into account the fact that the person is in good health and that it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[0702] The generated advice is sent to the speaker and the device.

[0703] speaker

[0704] A voice tells the user, "Today is a great day to be active. Try a new hobby."

[0705] Terminal

[0706] The display will say, "Today is a great day to be active. Try a new hobby."

[0707] The above is a specific embodiment for carrying out the present invention.

[0708] The processing flow will be explained below.

[0709] Step 1:

[0710] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits the measured data to a server in real time.

[0711] Step 2:

[0712] The AI ​​smart camera captures the user's face and analyzes their facial color, facial expressions, and movements using image analysis algorithms. The analyzed data is then sent to a server as user condition information.

[0713] Step 3:

[0714] The server receives biometric and condition data from the device and AI smart camera, stores it in a database, checks the data for consistency, and requests the data again if there is any inconsistency.

[0715] Step 4:

[0716] The server retrieves the latest biometric and condition data from the database and uses a machine learning model to analyze the user's physical condition and state, combining factors such as a high heart rate or poor sleep quality to provide a comprehensive assessment of their physical condition.

[0717] Step 5:

[0718] The server retrieves the fortune data for that day from the database based on the user's profile information. If the fortune data has been updated, the server retrieves the new information and updates the database.

[0719] Step 6:

[0720] The server combines the results of the health analysis with the auspicious data and uses a generative AI model to generate optimal advice for the day. The advice includes encouraging rest when the user is feeling unwell and encouraging activity when the user is feeling well.

[0721] Step 7:

[0722] The server formats the generated advice into JSON or XML format and sends it to the speaker and device, allowing the advice to be provided appropriately in both audio and text formats.

[0723] Step 8:

[0724] The speaker receives advice from the server, converts it using a speech synthesis engine, and conveys it to the user in a natural voice. The speaker provides advice such as, "Try to relax and not push yourself too hard today," or "Today is a great day to be active. Why not try a new hobby?"

[0725] Step 9:

[0726] The device displays the advice received from the server in text format on the display. The displayed advice matches the audio advice from the speaker.

[0727] Step 10:

[0728] The user acts based on the advice provided, and then inputs feedback about the advice through the terminal, such as whether the advice was helpful or how their physical condition has changed.

[0729] Step 11:

[0730] The device sends user feedback to the server, which is used to improve the accuracy of advice.

[0731] Step 12:

[0732] The server stores the feedback in a database and uses it to generate future advice, continuously improving the accuracy of the entire system.

[0733] Example 1

[0734] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0735] Conventional health management systems only measure a user's biometric data and condition, but lack the ability to provide individually optimized advice. Furthermore, they are unable to generate advice that takes into account the user's daily auspicious and unlucky data, and do not take into account the user's actual situation or day. As a result, the advice provided is general, making it difficult to provide specific help for the user's health condition or activities. Furthermore, there is a lack of utilization of user feedback to improve the accuracy of the advice.

[0736] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0737] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition, such as facial color and facial expression, and a means for receiving the measured biometric data and the analysis results. This makes it possible to grasp the user's health state and condition in real time and provide individually optimized advice. The system also includes a means for automatically generating advice using a generative AI model and a means for generating prompt sentences based on the analysis results, making it possible to quickly provide highly personalized advice. Furthermore, the generated advice is converted into speech using a speech synthesis engine and provided through a speaker, allowing for quick and effective notification to the user. These means enable the provision of personalized health advice desired by the user and improved accuracy.

[0738] "Biometric data" refers to data that indicates the user's health condition, including heart rate, body temperature, number of steps taken, sleep patterns, etc.

[0739] "Condition" is information that indicates the user's physical and mental state, and is data that is analyzed based on facial color, facial expressions, movements, etc.

[0740] "Means for measuring in real time" refers to a device or method for continuously measuring a user's biometric data and instantly obtaining the results.

[0741] The term "analyzing means" refers to a device or method for evaluating the user's physical condition or health status based on measured biometric data and user condition data.

[0742] "Means for receiving" refers to a device or method for receiving data transmitted from a measurement device or an analytical device.

[0743] "Good and bad fortune data" is information that indicates the fortune and lucky data for that day, and serves as a reference when the user decides what to do.

[0744] "Means for generating advice" refers to a device or method that generates specific advice or recommended actions to be provided to a user based on the analysis results and good or bad luck data.

[0745] "Generative AI model" refers to a machine learning model that uses artificial intelligence technology to generate appropriate advice or responses from data.

[0746] A "prompt sentence" is an instruction sentence input into a generative AI model that prompts the model to generate specific advice.

[0747] A "speech synthesis engine" refers to software or hardware technology for converting text data into natural-sounding speech.

[0748] MODE FOR CARRYING OUT THE INVENTION

[0749] Overall system configuration

[0750] This system monitors the user's physical condition and condition in real time and provides optimal advice. Specifically, it consists of the following components:

[0751] Terminal (wearable device): Continuously measures the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and transmits it to the server in real time or in batches.

[0752] AI Smart Camera: Captures the user's face in real time, analyzes their facial expressions, complexion, and movements, and sends the results to a server.

[0753] Server: Stores and analyzes the received data and generates advice using generative AI models.

[0754] Speaker: Provides the generated advice aloud using a speech synthesis engine.

[0755] 1. User data collection

[0756] Terminal

[0757] The terminal constantly measures the user's biometric data. Examples of wearable devices include Apple Watch and Fitbit.

[0758] Heart rate is measured every minute and body temperature every hour, and the data is sent to the server via HTTP POST requests.

[0759] AI Smart Camera

[0760] The camera captures the user's face in real time, and uses hue and saturation analysis for facial color analysis, and a face recognition library (e.g., OpenCV) for facial expression analysis.

[0761] The analysis results are sent to the server in real time.

[0762] 2. Data Receipt and Analysis

[0763] server

[0764] The server receives the data sent from the device and the AI ​​smart camera and stores it in a database (e.g., MySQL, PostgreSQL).

[0765] Based on the stored data, the user's physical condition and health are analyzed using a machine learning model (e.g., TensorFlow, PyTorch).

[0766] 3. Reference to fortune data

[0767] server

[0768] The server retrieves the day's fortune data from the database based on the user's profile data.

[0769] The latest fortune data is updated daily from public APIs, etc.

[0770] 4. Generating Advice

[0771] server

[0772] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate specific advice based on the analysis results and the fortune data.

[0773] An example of a specific prompt sentence is "Generate advice when the user's heart rate is higher than normal and the quality of sleep is poor. Also, when the fortune data indicates an unlucky day."

[0774] 5. Providing advice

[0775] server

[0776] The generated advice is sent to the speaker and device via a REST API in JSON format.

[0777] speaker

[0778] The speaker uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert text data into natural-sounding speech and convey it to the user.

[0779] Advice such as "Try not to push yourself too hard today and try to relax" is provided via voice.

[0780] Terminal

[0781] The device will display advice in text format on the display. For example, the smartwatch will say, "Today is a great day to be active. Try a new hobby."

[0782] Specific examples

[0783] As a specific example of how it works, if a user is feeling unwell, the device will send data such as a higher than normal heart rate and poor quality sleep to the server, and the AI ​​smart camera will analyze that the user looks pale and tired, and send this to the server. After receiving this data, the server determines that the user is feeling unwell and checks the fortune data to see if that day is an unlucky day. As a result, it generates advice such as "Try not to push yourself today and try to relax," and sends this to the speaker and device.

[0784] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0785] Step 1: Measure and send data

[0786] Terminal

[0787] Input: Real-time biometric data such as the user's heart rate, body temperature, steps taken, and sleep patterns.

[0788] What it does: A wearable device (e.g., a smartwatch) measures your heart rate every minute and your body temperature every hour.

[0789] Data processing: These biometric data are collected and sent to the server via an HTTP POST request.

[0790] Output: Formatted biometric data (JSON format).

[0791] Step 2: Analyzing the user's facial data

[0792] AI Smart Camera

[0793] Input: User's facial image and video data.

[0794] How it works: The camera captures the user's face in real time and analyzes their facial color and facial expressions. Hue and saturation analysis is used for facial color analysis, and a face recognition library (e.g., OpenCV) is used for facial expression analysis.

[0795] Data calculation: Facial color analysis and facial expression recognition algorithms are applied to quantify stress levels and fatigue levels.

[0796] Output: Analysis results (e.g., data such as pale complexion or tired facial expression).

[0797] Step 3: Receiving and storing data

[0798] server

[0799] Input: Biometric and facial data sent from the device and AI smart camera.

[0800] Specific operation: The server receives the HTTP POST request and saves it in a database (e.g. MySQL, PostgreSQL).

[0801] Data processing: The received data is properly formatted and a timestamp is added to each data.

[0802] Output: Data stored in a database.

[0803] Step 4: Analyzing the user's physical condition and condition

[0804] server

[0805] Input: Stored biometric and facial data.

[0806] Specific operation: Uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the user's physical condition and condition.

[0807] Data calculations: Combining multiple data points to assess your physical condition based on specific conditions (e.g., higher than normal heart rate and poor sleep quality).

[0808] Output: Analysis results (e.g., poor health).

[0809] Step 5: Check the luck data

[0810] server

[0811] Input: User profile data.

[0812] Specific operation: Obtains the day's fortune data from the database and updates the latest information from the public API as needed.

[0813] Data calculation: Calculates fortune based on the user's date of birth and other auspicious and inauspicious factors.

[0814] Output: Fortune data for that day (e.g. lucky day, unlucky day).

[0815] Step 6: Generating Advice

[0816] server

[0817] Input: Analysis results and fortune data.

[0818] Specific operation: Uses a generative AI model (e.g., OpenAI GPT-4) to generate advice based on the prompt.

[0819] Data processing: The prompt text "Generate advice when the user's heart rate is higher than normal and the quality of their sleep is poor. Also, what is the auspicious / unlucky date?" is input into the generative AI model, and natural language processing is performed.

[0820] Output: The generated advice (e.g., "Try to relax and take it easy today.").

[0821] Step 7: Providing advice

[0822] server

[0823] Input: The generated advice.

[0824] Specific operation: Advice is sent to the speaker and device in JSON format.

[0825] Output: Sending advice data.

[0826] speaker

[0827] Input: Advice data in JSON format.

[0828] Specific operation: Uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert into natural-sounding speech.

[0829] Output: Spoken advice (e.g., "Try to relax and take it easy today.").

[0830] Terminal

[0831] Input: Advice data in JSON format.

[0832] Specific operation: Display advice in text format on the device display.

[0833] Output: Advice in text form (e.g., "Today is a great day to be active. Try a new hobby.").

[0834] (Application example 1)

[0835] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0836] In today's world, there is a demand for a system that analyzes a user's physical condition and mood in real time and provides optimal advice when the user is selecting a product in a virtual store. Conventional virtual stores do not provide product suggestions that take into account the user's instantaneous physical condition and mood, and as a result, the user's satisfaction and purchasing motivation are not fully stimulated. The object of the present invention is to solve this problem by providing a system that provides optimal advice and product suggestions in real time based on the user's physical condition and mood.

[0837] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0838] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition such as complexion and facial expression, and a means for receiving the measured biometric data and the analysis results, which makes it possible to analyze the user's physical condition and provide optimal advice and product suggestions in real time within the virtual store.

[0839] "User's biometric data" refers to data that indicates the user's physical condition, such as the user's heart rate, body temperature, number of steps taken, and sleep patterns.

[0840] "Analysis of facial color and facial expression" is the process of analyzing the user's facial color and facial expression and determining the user's condition from that information.

[0841] "Means for receiving measured biometric data and analysis results" refers to a device or system that receives data sent from a user's wearable device or AI smart camera.

[0842] "Analysis of the user's physical condition and condition" is the process of comprehensively analyzing the user's health and mood based on biometric data, facial color, and facial expression data.

[0843] "Good or bad fortune data" is fortune information such as whether it is an auspicious day or an unlucky day based on the current date.

[0844] The "means for generating advice" is a system or algorithm that uses the analysis results and the fortune data to generate specific advice tailored to the user's condition.

[0845] The "means for providing the generated advice by voice and text" refers to means for providing the generated advice by voice and on a display, and is a mechanism for conveying the advice to the user.

[0846] "A means for suggesting optimal products based on the user's physical condition and condition within a virtual store" is a system that suggests optimal products in a virtual reality environment, taking into account the user's health condition and mood.

[0847] The system embodying the present invention monitors the user's physical condition in real time and provides optimal advice and product suggestions. This system is composed of the following main components:

[0848] Overall system configuration

[0849] The system mainly consists of the following components:

[0850] Wearable devices: Continuously measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and send it to a server in real time or in batches.

[0851] AI Smart Camera: Captures the user's face in real time and analyzes their facial expressions, complexion, and movements, and sends the analysis results to the server.

[0852] Server: Receives data sent from the wearable device and AI smart camera and stores it in a database. Analyzes the user's physical condition and health based on the stored data. A machine learning model is used for the analysis.

[0853] Speaker and display: Generated advice and product suggestions are delivered to the user via voice and text.

[0854] Data collection and analysis

[0855] The server receives biometric data, facial color, and facial expression data sent from the wearable device and AI smart camera. This data is used to analyze the user's physical condition and condition using a machine learning model. Based on the analysis results, the server determines the user's condition and refers to the fortune data for the day.

[0856] Generating advice and product recommendations

[0857] The server uses a generative AI model based on the analysis results and auspicious / lucky data to generate specific advice and product suggestions. For example, if you are feeling unwell and it's an unlucky day, the server generates advice such as, "Try not to push yourself too hard today and relax." If you are feeling well and it's an auspicious day, the server generates a product suggestion such as, "Today is a great day to be active. Try a new product."

[0858] Providing advice and product suggestions

[0859] The generated advice and product suggestions are provided to the user through a speaker and a display. The speaker uses a speech synthesis engine to convert text data into natural-sounding speech, and the display displays the information in text format.

[0860] The specific hardware and software used

[0861] Hardware:

[0862] Wearable devices: Apple Watch, Fitbit, etc.

[0863] AI smart camera: OpenCV, Azure Face API, etc.

[0864] Speakers and Displays

[0865] software:

[0866] Data analysis: TensorFlow, AWS SageMaker, etc.

[0867] Speech synthesis engine: Google Text-to-Speech API, IBM Watson TTS, etc.

[0868] Generative AI models: GPT-3, etc.

[0869] Specific examples

[0870] simulation:

[0871] A user logs into the virtual shop.

[0872] The wearable device measures the user's heart rate, body temperature, steps taken, and sleep patterns and sends the data to a server.

[0873] The AI ​​smart camera analyzes the user's complexion and facial expressions and sends the results to the server.

[0874] The server analyzes this data and determines the user's physical condition in real time.

[0875] If you are in poor health and it is an unlucky day, the advice is, "Don't push yourself today, take your time choosing your products."

[0876] If you are in good health and it's an auspicious day, the advice is, "Be active, have fun, and try a new product today!"

[0877] Example prompt sentence:

[0878] Prompt for a generative AI model (e.g. GPT-3):

[0879] "Generate advice for when a user is in poor health and having an unlucky day. Specific advice: 'Try to relax and not push yourself too hard today.'"

[0880] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0881] Step 1:

[0882] The user wears a wearable device, which measures the user's biometric data, such as heart rate, body temperature, number of steps, and sleep patterns, in real time and transmits it to a server. The input is the measured biometric data, and the output is the data sent to the server.

[0883] Step 2:

[0884] The server receives the biometric data sent from the wearable device and stores the data in a database. The input is the biometric data from the wearable device, and the output is the data stored in the database. A storage system is used to store the data.

[0885] Step 3:

[0886] The user stands in front of the AI ​​smart camera. The camera captures the user's face and analyzes their facial expressions, complexion, and movements. The input is the captured image data, and the output is the analysis results. Image analysis software (e.g., OpenCV, Azure Face API) is used for the analysis.

[0887] Step 4:

[0888] The AI ​​smart camera sends the analysis results to the server. The input is the image analysis result, and the output is the data sent to the server. The server also stores this data in a database.

[0889] Step 5:

[0890] The server analyzes the user's physical condition and health using the received biometric data and image analysis results. The input is the biometric data and image analysis data stored in the database, and the output is the analysis results. Machine learning models (e.g., TensorFlow, AWS SageMaker) are used for the analysis.

[0891] Step 6:

[0892] Based on the analysis results, the server retrieves the current day's fortune data from the database. The input is the analysis results, and the output is the fortune data. The latest fortune data is retrieved by a database query.

[0893] Step 7:

[0894] The server uses a generative AI model (e.g., GPT-3) based on the analysis results and the fortune data to generate specific advice and product suggestions. The input is the analysis results and fortune data, and the output is the generated advice and product suggestions. A prompt sentence is input into the generative AI model, which generates the optimal advice.

[0895] Step 8:

[0896] The generated advice and product suggestions are sent from the server to the speaker and display. The input is the generated advice and product suggestions, and the output is text and audio that is displayed on the display and played back from the speaker.

[0897] Step 9:

[0898] The user receives advice and product suggestions through a speaker and a display. The input is the text displayed on the display and the audio played from the speaker, and the output is the user's behavior and purchasing decisions.

[0899] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0900] A specific embodiment for implementing the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized advice can be achieved.

[0901] Overall system configuration

[0902] The system mainly consists of the following components:

[0903] Terminal (wearable device)

[0904] AI Smart Camera

[0905] Emotion Engine

[0906] server

[0907] speaker

[0908] 1. User data collection

[0909] Terminal

[0910] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[0911] The collected data is sent to the server in real time or in batches.

[0912] AI Smart Camera

[0913] It captures the user's face in real time and analyzes their facial expression, facial expression, and movements.

[0914] As an example, stress levels and fatigue levels can be determined from facial color.

[0915] The analysis results are sent as data to the server.

[0916] 2. Emotion recognition

[0917] Emotion Engine

[0918] It analyzes the user's facial images and voice data obtained from the AI ​​smart camera and recognizes the user's emotions (joy, sadness, anger, etc.).

[0919] The recognized emotion data is sent to the server.

[0920] 3. Data Receipt and Analysis

[0921] server

[0922] It receives biometric data, condition data, and emotion data from the device, AI smart camera, and emotion engine, and stores them in a database.

[0923] Check the consistency of the data, and if there is any inconsistency, request the data again.

[0924] 4. Analysis of user's physical condition and emotions

[0925] server

[0926] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to analyze the user's physical condition and emotions. Here, factors such as a high heart rate, poor sleep quality, and the frequency of happy facial expressions are combined to form a comprehensive evaluation.

[0927] 5. Reference to fortune data

[0928] server

[0929] Based on the user's profile information, the system retrieves the day's fortune data from the database.

[0930] If the data on lucky and unlucky days has been updated, the latest information is acquired accordingly and the database is updated.

[0931] 6. Generating Advice

[0932] server

[0933] The analysis results, emotional data, and good and bad luck data are combined and the optimal advice for the day is generated using a generative AI model.

[0934] The advice includes encouraging rest when you're feeling unwell and activity when you're feeling well, and it adjusts the advice based on the perceived emotions.

[0935] 7. Providing advice

[0936] server

[0937] The generated advice is formatted in JSON or XML and sent to the speaker and device, allowing the advice to be provided appropriately in both voice and text.

[0938] speaker

[0939] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[0940] As an example, advice can be given as "Try to relax and take it easy today" or "Today is a great day to be active. Try a new hobby."

[0941] Terminal

[0942] Display advice in text format on the screen.

[0943] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[0944] Specific examples

[0945] When the user is unwell and emotionally sad

[0946] Terminal

[0947] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[0948] AI Smart Camera

[0949] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[0950] Emotion Engine

[0951] The sadness emotion is recognized from the user's face and transmitted to the server.

[0952] server

[0953] Based on the received data, it is determined that the user is in poor health.

[0954] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[0955] Taking into consideration that the person is feeling unwell, that it is an unlucky day, and that the emotion is sadness, the advice generated is, "Try not to push yourself today and try to relax. Read your favorite book to change your mood."

[0956] The generated advice is sent to the speaker and the device.

[0957] speaker

[0958] The voice tells the user, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[0959] Terminal

[0960] The display will say, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[0961] If the user is in good health and the emotion is joy

[0962] Terminal

[0963] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[0964] AI Smart Camera

[0965] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[0966] Emotion Engine

[0967] The joyful emotion is recognized from the user's face and transmitted to the server.

[0968] server

[0969] Based on the received data, it is determined that the user's physical condition is good.

[0970] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[0971] Taking into consideration that the emotion is joy in addition to the person being in good health and that it is an auspicious day, the advice generated is "Today is a great day to be active. Have a good time with your friends."

[0972] The generated advice is sent to the speaker and the device.

[0973] speaker

[0974] A voice tells the user, "Today is a great day to be active and have fun with friends."

[0975] Terminal

[0976] The display will say, "Today is a great day to be active and have fun with friends."

[0977] The above is a specific embodiment for carrying out the present invention.

[0978] The processing flow will be explained below.

[0979] Step 1:

[0980] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits this data to a server in real time.

[0981] Step 2:

[0982] The AI ​​smart camera captures the user's face in real time and analyzes their facial expressions, complexion, and movements. The analyzed data is sent to a server as information about the user's condition.

[0983] Step 3:

[0984] The emotion engine analyzes the user's facial images and voice data acquired from the AI ​​smart camera and recognizes the user's emotions (happiness, sadness, anger, etc.). The recognized emotion data is sent to the server.

[0985] Step 4:

[0986] The server receives biometric data, condition data, and emotion data sent from the device, AI smart camera, and emotion engine, and stores it in a database. It checks the consistency of the received data and requests the data again if there is any inconsistency.

[0987] Step 5:

[0988] The server retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, the overall analysis results may show a high heart rate, poor sleep quality, and sadness.

[0989] Step 6:

[0990] The server refers to the user's profile information and retrieves the lucky / unlucky data for that day from the database. If the lucky / unlucky data has been updated, the new information is updated in the database.

[0991] Step 7:

[0992] The server combines the results of the physical condition analysis, emotional data, and auspicious / unlucky data, and uses a generative AI model to generate the optimal advice for that day. The advice is based on the individual's physical condition and emotions. For example, if you are feeling unwell and sad, the advice might be, "Try to relax and not push yourself today. I recommend reading your favorite book to change your mood." If you are feeling good and happy, the advice might be, "Today is a great day. Have some fun outdoors with friends."

[0993] Step 8:

[0994] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The transmitted data is output as voice and text.

[0995] Step 9:

[0996] The speaker receives advice from the server and converts it into natural-sounding speech using a speech synthesis engine, which then relays it to the user. For example, the advice could be, "Try not to push yourself too hard today, but relax. Read your favorite book to change your mood."

[0997] Step 10:

[0998] The device will display the received advice in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[0999] Step 11:

[1000] The user acts on the advice provided. The effectiveness of the advice and the user's opinion are input into the device as feedback. For example, the user can input whether the advice was helpful or whether their physical condition improved.

[1001] Step 12:

[1002] The device sends user feedback to the server, which is used to make future advice more effective.

[1003] Step 13:

[1004] The server stores the feedback in a database and uses it to generate advice in future. This feedback information continuously improves the accuracy of the entire system.

[1005] Example 2

[1006] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1007] Conventional healthcare systems tend to collect only a user's biometric data and provide advice based solely on their physical condition. However, providing personalized advice that also takes into account the user's emotions and fortunes is important for comprehensive health management. Furthermore, the lack of a means to improve the accuracy of advice based on feedback makes continuous improvement difficult. This has led to the issue of not being able to fully meet user needs.

[1008] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric information of the user in real time, means for analyzing the user's condition such as complexion, facial expression, and movement, means for receiving the measured biometric data and analysis results, means for generating advice using a generative AI model, means for providing the generated advice in audio and text, and means for collecting feedback from the user and improving the accuracy of advice from the next time onwards. This makes it possible to provide personalized advice that comprehensively takes into account the user's physical condition, emotions, and fortune, and to continuously improve the accuracy of that advice.

[1009] "User's biological information" is data that indicates the user's physical condition, such as heart rate, body temperature, number of steps, and sleep pattern.

[1010] "Real-time acquisition means" refers to the ability of sensors or devices to collect data immediately, without time delay, and transmit that data to the system.

[1011] "User's condition such as complexion, facial expression, and movement" is subjective information that indicates the user's emotions, stress level, activity level, and so on.

[1012] "Means of analysis" refers to the algorithms and software functions used to analyze data and detect specific patterns or anomalies.

[1013] "Measured biometric data and analysis results" refers to biometric information obtained from sensors or devices and the results of analyzing that data.

[1014] "Means for receiving" refers to the interface or protocol for importing and storing data from the outside.

[1015] A "generative AI model" is an artificial intelligence algorithm or system that automatically generates an output based on specific input data.

[1016] The "means for generating advice" is a function that automatically creates recommended actions and points of caution for the user based on the collected data and analysis results.

[1017] The "means of providing by voice and text" is a function of reading out the generated advice as voice and displaying it as text.

[1018] "Means for collecting feedback" refers to interfaces and protocols for capturing opinions and reactions from users and reflecting them in the system.

[1019] A specific embodiment of this invention will be described. This system monitors the user's physical condition and emotions in real time and provides optimal advice based on the results. The entire system mainly consists of the following components: a terminal, an AI smart camera, an emotion engine, a server, and a speaker.

[1020] User data collection

[1021] Terminal (wearable device):

[1022] The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The wearable device is equipped with sensors such as a heart rate monitor, a body temperature sensor, and an accelerometer. For example, the heart rate monitor measures values ​​such as "90 BPM" and the body temperature sensor measures "36.5°C."

[1023] These biometric data are transmitted to a server periodically or in real time via Bluetooth or Wi-Fi.

[1024] AI Smart Camera:

[1025] The camera captures the user's facial color, facial expressions, and movements, and analyzes this data in real time. The camera has an image analysis algorithm built in to analyze facial color and facial expressions. For example, if the user's face is pale, it will be determined that the stress level is high.

[1026] The analyzed data is immediately sent to the server.

[1027] Emotion recognition and data reception

[1028] Emotion Engine:

[1029] Using image and audio data obtained from an AI smart camera, the system recognizes the user's emotions (happiness, sadness, anger, etc.). For example, it uses facial expression analysis technology to determine that the user is expressing happiness.

[1030] The recognized emotion data is transmitted to a server.

[1031] server:

[1032] It receives all data acquired from devices, AI smart cameras, and emotion engines, stores it in a database, checks the data consistency, and requests the data again if there is any inconsistency.

[1033] Analyzing physical condition and emotions and generating advice

[1034] server:

[1035] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to comprehensively analyze the user's physical condition and emotions. For example, if the user has a high heart rate, poor sleep quality, and sadness, the system will determine that the user is in poor health.

[1036] Based on the user's profile information, the system retrieves the day's fortune data from the database. If the fortune data is not up to date, the system retrieves the latest data from an external data source and updates the database.

[1037] The analysis results are combined with fortune data and a generative AI model (e.g., GPT-3) is used to generate prompts. Examples of prompts include:

[1038] Example prompt sentence:

[1039] User health data:

[1040] Heart rate: 90 BPM

[1041] Body temperature: 36.5℃

[1042] Steps: 10,000

[1043] Sleep Pattern: Good

[1044] User sentiment data:

[1045] Emotion: Joy

[1046] Expression: Cheerful

[1047] Activity: Active

[1048] Fortune Data:

[1049] Today's Fortune: Auspicious Day

[1050] Use this information to generate the best advice for your users.

[1051] A generative AI model receives the prompts and generates the best advice for the day based on them.

[1052] Providing advice

[1053] server:

[1054] The generated advice is formatted into JSON or XML format and sent to the speaker and device.

[1055] speaker:

[1056] A speech synthesis engine is used to convert the generated advice into natural-sounding speech and convey it to the user.

[1057] Device:

[1058] Advice is displayed in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[1059] This system will provide personalized advice that takes into account the user's physical condition, emotions, and fortune, and will be able to continuously improve its accuracy.

[1060] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1061] Program processing flow

[1062] Step 1:

[1063] Data Acquisition

[1064] Device: The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The device periodically reads data from the heart rate monitor and body temperature sensor, converts the data into JSON format, and stores it in a transmission queue.

[1065] Input: Biometric data from sensors.

[1066] Output: Biometric data in JSON format.

[1067] Step 2:

[1068] Sending data

[1069] Terminal: The terminal sends the collected biometric data to the server via Bluetooth or Wi-Fi. Specifically, the terminal sends the collected JSON data to the server's receiving API as an HTTP POST request.

[1070] Input: Biometric data obtained in step 1 in JSON format.

[1071] Output: HTTP request to the server.

[1072] Step 3:

[1073] Camera-based facial color and facial expression analysis

[1074] AI Smart Camera: Captures the user's facial expressions, facial expressions, and movements, and analyzes them in real time using image analysis algorithms. For example, facial expression analysis software analyzes the user's facial photo to determine emotions such as "sadness" or "happiness."

[1075] Input: A face image captured by the camera.

[1076] Output: Analyzed facial color and expression data.

[1077] Step 4:

[1078] Sending camera data

[1079] AI Smart Camera: Converts the analysis results into JSON format and sends them to the server. Sends the analysis data using an HTTP POST request.

[1080] Input: Facial color and expression data analyzed in step 3.

[1081] Output: JSON formatted data to the server.

[1082] Step 5:

[1083] Processing received data

[1084] Server: Receives biometric data and facial image analysis data sent from the device and AI smart camera. The receiving API receives this data and stores it in a database.

[1085] Input: HTTP requests from the device and the AI ​​smart camera.

[1086] Output: Biometric data and facial image analysis data stored in a database.

[1087] Step 6:

[1088] emotion recognition

[1089] Emotion Engine: Analyzes facial image data retrieved from a database to recognize the user's emotions. It uses an AI model to identify emotions such as "happiness" or "sadness."

[1090] Input: Facial image data stored in a database.

[1091] Output: Parsed emotion data.

[1092] Step 7:

[1093] Comprehensive analysis of data

[1094] Server: Retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, if the heart rate is high, sleep quality is poor, and the user is expressing sadness, it determines that the user is in poor health.

[1095] Input: Biometric data, facial expression data, and emotion data in the database.

[1096] Output: Comprehensively analyzed user physical and emotional data.

[1097] Step 8:

[1098] Fortune data reference

[1099] Server: Based on the user's profile information, retrieves the day's fortune data from the database. If the fortune data is not up to date, retrieves the latest data from an external interface and updates the database.

[1100] Input: User profile information.

[1101] Output: Fortune data for the day.

[1102] Step 9:

[1103] Prompt creation for advice generation

[1104] Server: Combines the user's biometric, emotional, and fortune data to generate prompts to input into the generative AI model. Examples of prompts include:

[1105] Example prompt sentence:

[1106] User health data:

[1107] Heart rate: 90 BPM

[1108] Body temperature: 36.5℃

[1109] Steps: 10,000

[1110] Sleep Pattern: Good

[1111] User sentiment data:

[1112] Emotion: Joy

[1113] Expression: Cheerful

[1114] Activity: Active

[1115] Fortune Data:

[1116] Today's Fortune: Auspicious Day

[1117] Use this information to generate the best advice for your users.

[1118] Input: Biometric data, emotional data, fortune data.

[1119] Output: Prompts to the generative AI model.

[1120] Step 10:

[1121] Generating Advice

[1122] Generative AI model: Generates optimal advice for the user based on the prompt. For example, it generates advice such as, "Try not to push yourself too hard today and try to relax."

[1123] Input: Prompt statement.

[1124] Output: The generated advice.

[1125] Step 11:

[1126] Sending Advice

[1127] Server: Formats the generated advice into JSON or XML format and sends it to the speaker and device, providing advice in voice and text.

[1128] Input: The generated advice.

[1129] Output: JSON or XML data to speakers and devices.

[1130] Step 12:

[1131] Voice and text advice provided

[1132] Speaker: Uses a speech synthesis engine to convert the JSON or XML data into speech and tells the user, "Try to relax and take it easy today."

[1133] Terminal: Display the text "Try not to push yourself too hard today and try to relax" on the display.

[1134] Input: Advice sent by the server.

[1135] Output: Providing audio and text advice to the user.

[1136] These are the specific processing steps of this system, which allows users to receive optimal advice based on their physical condition, emotions, and fortune.

[1137] (Application example 2)

[1138] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1139] Conventional self-driving vehicles lack real-time feedback based on the physical condition and emotions of the driver and passengers, making it difficult to provide optimal driving modes and entertainment. They also lack the ability to provide personalized advice based on the day's auspicious and unlucky data or the individual user's condition. These circumstances have led to insufficient safety and comfort within the vehicle.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for measuring the user's biological data in real time, means for analyzing the user's condition such as facial color and facial expression, and means for receiving the measured biological data and the analysis results. This makes it possible to monitor the user's physical condition and emotions in real time in an autonomous vehicle and optimize the driving mode and entertainment based on the monitoring results.

[1141] "User's biometric data" refers to data that indicates a person's physical condition, such as heart rate, body temperature, number of steps taken, and sleep patterns.

[1142] "User condition such as facial color and facial expression" is information indicating the physical and emotional state of the user, including the color and facial expression of the user's face, and movements.

[1143] "Analysis means" refers to a device or software that evaluates and judges the user's physical condition and emotions based on measured data and images.

[1144] "Today's fortune data" is traditional or astrological information that indicates whether the day is auspicious or inauspicious for the user.

[1145] The "means for generating advice" is a device or software that suggests appropriate actions or measures to the user based on the collected data and analysis results.

[1146] "Means for providing generated advice in audio and text" refers to a device or software that displays and plays generated advice in audio or text format in order to convey the advice to the user in an easy-to-understand manner.

[1147] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[1148] "Driving modes" are settings that adjust the style and conditions under which an autonomous vehicle is driven.

[1149] "Entertainment providing means" refers to devices or software for playing entertainment content such as music and video for users.

[1150] "Means for recognizing emotions" refers to a device or software that analyzes a user's facial expressions, actions, and voice data to determine their emotional state.

[1151] A "generative AI model" is an algorithm or program that is trained by artificial intelligence to generate appropriate advice or suggestions from specific data.

[1152] A "prompt" is a specific input sentence used to generate an appropriate answer or output for a generative AI model.

[1153] A specific embodiment of the present invention will be described. This invention is a system that monitors the physical condition and emotions of occupants in an autonomous vehicle in real time and provides optimal driving modes and entertainment based on the monitoring results. The entire system is composed of the following main components.

[1154] System configuration

[1155] The system mainly consists of the following components:

[1156] Wearable devices

[1157] AI Camera

[1158] Emotion Engine

[1159] server

[1160] speaker

[1161] 1. Data Collection

[1162] Wearable devices

[1163] Wearable devices (e.g., smartwatches) constantly measure biometric data such as heart rate, body temperature, steps taken, and sleep patterns.

[1164] The collected data is sent to a server in real time at irregular intervals.

[1165] AI Camera

[1166] An AI camera installed inside the vehicle captures and analyzes the facial expressions, facial expressions, and movements of the occupants in real time.

[1167] The analysis results are sent to a server as data to determine each passenger's stress level and fatigue level.

[1168] 2. Emotion recognition

[1169] Emotion Engine

[1170] The emotion engine analyzes facial images and audio data obtained from the AI ​​camera to recognize the emotions of the occupants (joy, sadness, anger, etc.).

[1171] The recognized emotion data is transmitted to a server.

[1172] 3. Data Receipt and Analysis

[1173] server

[1174] It receives biometric data, condition data, and emotion data from the wearable device, AI camera, and emotion engine, and stores them in a database.

[1175] Check the consistency of the data and re-request the data if there is any inconsistency.

[1176] 4. Data analysis and advice generation

[1177] server

[1178] The latest biometric, condition, and emotional data is collected and the occupants' physical condition and emotions are analyzed using machine learning models.

[1179] The data on the day's fortunes is referenced and retrieved from the database.

[1180] The analysis results, emotional data, and good and bad fortune data are combined and a generative AI model is used to generate the best advice for the day.

[1181] The advice includes specific suggestions for action, such as when drivers should take a break and relax, or when they should play music to help them relax.

[1182] 5. Providing advice

[1183] speaker

[1184] The advice is given to the passengers using a speech synthesis engine, which converts text data into natural-sounding speech.

[1185] Terminal

[1186] Advice is displayed in text format on the in-car display.

[1187] Specific examples

[1188] Example 1: When the user is unwell and sad

[1189] The wearable device measures high heart rate and poor sleep quality and sends the results to a server.

[1190] The AI ​​camera analyzes whether the person's complexion is pale and they look tired, and sends the results to the server.

[1191] The emotion engine recognizes the sadness emotion from the user's face and sends it to the server.

[1192] The server determines that the user is feeling unwell and that it is an unlucky day, generates advice such as "Try not to push yourself today and try to relax. Read your favorite book to change your mood," and sends this advice to the device and speaker.

[1193] Example 2: When the user is in good health and the emotion is joy

[1194] The wearable device measures the heart rate, body temperature, and number of steps taken, and sends the results to the server.

[1195] The AI ​​camera analyzes whether the person has a good complexion and a bright expression, and sends the results to the server.

[1196] The emotion engine recognizes the emotion of joy from the user's face and transmits it to the server.

[1197] The server determines that the person is in good health and that it is an auspicious day, generates advice such as "Today is a great day to be active. Have a good time with friends," and sends it to the device and speaker.

[1198] Prompt example

[1199] "If the user has a high heart rate but is showing signs of joy, provide appropriate advice."

[1200] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1201] Step 1:

[1202] The wearable device measures the user's biometric data in real time, specifically, heart rate, body temperature, number of steps taken, sleep patterns, etc. This acquired data is input, and the wearable device sends it to a server.

[1203] Step 2:

[1204] The server stores the received biometric data in a database and checks its consistency. Specifically, it checks whether the data is missing or consistent. If there is an inconsistency, it requests the data again. The input to this process is the biometric data sent from the wearable device, and the output is the biometric data whose consistency has been confirmed.

[1205] Step 3:

[1206] The AI ​​camera captures the facial color and facial expressions of the user in the car in real time. Specific operations include analyzing facial color and facial expressions. These analysis results are input, and the AI ​​camera sends the analysis results to the server.

[1207] Step 4:

[1208] The server receives the analysis results sent from the AI ​​camera and stores them in a database. The input is the analysis result data, and the output is the saved analysis result data. The server also checks the consistency of this data.

[1209] Step 5:

[1210] The emotion engine analyzes emotions based on facial image data obtained from the AI ​​camera. Specifically, it uses facial expression data to recognize emotional states (happiness, sadness, anger, etc.). This recognition result is input, and the emotion engine sends it to the server.

[1211] Step 6:

[1212] The server receives the emotion data sent from the emotion engine and stores it in a database. The input is emotion data, and the output is the stored emotion data. The server also checks the integrity of the emotion data.

[1213] Step 7:

[1214] The server integrates all data received from the wearable device, AI camera, and emotion engine to obtain the latest biometric, condition, and emotion data. These data are the input, and the output is the integrated dataset.

[1215] Step 8:

[1216] The server uses the integrated data set to analyze the user's physical condition and emotions using a machine learning model. Specifically, it evaluates heart rate and facial expression changes to determine the user's overall physical condition and emotions. The results of this analysis are the input, and the output is the analysis result data.

[1217] Step 9:

[1218] The server retrieves the fortune data for that day from the database. The input is the date information, and the output is the fortune data for that day. The server combines this with the analysis results.

[1219] Step 10:

[1220] The server uses a generative AI model to generate optimal advice based on the analysis results and the good and bad fortune data. Specifically, a prompt sentence is used as input to the model to generate appropriate advice. An example of this prompt sentence is, "If the user's heart rate is high but they are showing signs of joy, please provide appropriate advice." The input is the integrated data and the prompt sentence, and the output is the generated advice.

[1221] Step 11:

[1222] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The input is the generated advice, and the output is the formatted advice data.

[1223] Step 12:

[1224] The terminal displays the advice in text format on the in-car display. Specifically, it converts the transmitted format data into text and displays it on the display. The input is the formatted advice data, and the output is the display.

[1225] Step 13:

[1226] The speaker uses a speech synthesis engine to convert the advice into natural-sounding speech and convey it to the passengers. Specifically, it converts text data into speech data and plays it back. The input is formatted advice data, and the output is advice conveyed in speech.

[1227] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1228] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1229] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1230] [Third embodiment]

[1231] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1232] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1233] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1234] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1235] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1237] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1238] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1239] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1240] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1241] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1242] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1243] A specific embodiment for carrying out the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that information. The program processing and specific examples will be described in detail below.

[1244] Overall system configuration

[1245] The system mainly consists of the following components:

[1246] Terminal (wearable device)

[1247] AI Smart Camera

[1248] server

[1249] speaker

[1250] 1. User data collection

[1251] Terminal

[1252] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[1253] The collected data is sent to the server in real time or in batches.

[1254] AI Smart Camera

[1255] It captures the user's face in real time and analyzes their facial expressions, complexion, and movements.

[1256] As an example, stress levels and fatigue levels can be determined from facial color.

[1257] The analysis results are sent as data to the server.

[1258] 2. Data Receipt and Analysis

[1259] server

[1260] It receives data sent from the device and the AI ​​smart camera and stores it in a database.

[1261] The user's physical condition and health are analyzed based on the saved data, using a machine learning model.

[1262] As an example, if the user's heart rate is higher than normal and the quality of sleep is poor, the user is determined to be in poor health.

[1263] 3. Reference to fortune data

[1264] server

[1265] Based on the user's profile, the system retrieves the day's fortune data from the database.

[1266] If the data on lucky and unlucky days has been updated, the latest information is acquired and referenced accordingly.

[1267] 4. Generating Advice

[1268] server

[1269] Based on the analysis results and good and bad fortune data, a generative AI model is used to generate specific advice.

[1270] As an example, if the user is in poor health and it is an unlucky day, the system generates advice such as "Try not to push yourself too hard today and try to relax." If the user is in good health and it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[1271] 5. Providing advice

[1272] server

[1273] The generated advice is sent to the speaker and the device.

[1274] Advice is provided in audio and text formats.

[1275] speaker

[1276] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[1277] As an example, advice may be given as "Try to relax and not push yourself too hard today."

[1278] Terminal

[1279] Display advice in text format on the screen.

[1280] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[1281] Specific examples

[1282] If the user is unwell

[1283] Terminal

[1284] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[1285] AI Smart Camera

[1286] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[1287] server

[1288] Based on the received data, it is determined that the user is in poor health.

[1289] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[1290] Taking into consideration the poor physical condition and the fact that it is an unlucky day, the advice generated is "Try not to push yourself today and try to relax."

[1291] The generated advice is sent to the speaker and the device.

[1292] speaker

[1293] The voice tells the user, "Try not to push yourself too hard today and try to relax."

[1294] Terminal

[1295] The display will say, "Try not to push yourself too hard today and try to relax."

[1296] If the user is in good health

[1297] Terminal

[1298] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[1299] AI Smart Camera

[1300] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[1301] server

[1302] Based on the received data, it is determined that the user's physical condition is good.

[1303] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[1304] Taking into account the fact that the person is in good health and that it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[1305] The generated advice is sent to the speaker and the device.

[1306] speaker

[1307] A voice tells the user, "Today is a great day to be active. Try a new hobby."

[1308] Terminal

[1309] The display will say, "Today is a great day to be active. Try a new hobby."

[1310] The above is a specific embodiment for carrying out the present invention.

[1311] The processing flow will be explained below.

[1312] Step 1:

[1313] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits the measured data to a server in real time.

[1314] Step 2:

[1315] The AI ​​smart camera captures the user's face and analyzes their facial color, facial expressions, and movements using image analysis algorithms. The analyzed data is then sent to a server as user condition information.

[1316] Step 3:

[1317] The server receives biometric and condition data from the device and AI smart camera, stores it in a database, checks the data for consistency, and requests the data again if there is any inconsistency.

[1318] Step 4:

[1319] The server retrieves the latest biometric and condition data from the database and uses a machine learning model to analyze the user's physical condition and state, combining factors such as a high heart rate or poor sleep quality to provide a comprehensive assessment of their physical condition.

[1320] Step 5:

[1321] The server retrieves the fortune data for that day from the database based on the user's profile information. If the fortune data has been updated, the server retrieves the new information and updates the database.

[1322] Step 6:

[1323] The server combines the results of the health analysis with the auspicious data and uses a generative AI model to generate optimal advice for the day. The advice includes encouraging rest when the user is feeling unwell and encouraging activity when the user is feeling well.

[1324] Step 7:

[1325] The server formats the generated advice into JSON or XML format and sends it to the speaker and device, allowing the advice to be provided appropriately in both audio and text formats.

[1326] Step 8:

[1327] The speaker receives advice from the server, converts it using a speech synthesis engine, and conveys it to the user in a natural voice. The speaker provides advice such as, "Try to relax and not push yourself too hard today," or "Today is a great day to be active. Why not try a new hobby?"

[1328] Step 9:

[1329] The device displays the advice received from the server in text format on the display. The displayed advice matches the audio advice from the speaker.

[1330] Step 10:

[1331] The user acts based on the advice provided, and then inputs feedback about the advice through the terminal, such as whether the advice was helpful or how their physical condition has changed.

[1332] Step 11:

[1333] The device sends user feedback to the server, which is used to improve the accuracy of advice.

[1334] Step 12:

[1335] The server stores the feedback in a database and uses it to generate future advice, continuously improving the accuracy of the entire system.

[1336] Example 1

[1337] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1338] Conventional health management systems only measure a user's biometric data and condition, but lack the ability to provide individually optimized advice. Furthermore, they are unable to generate advice that takes into account the user's daily auspicious and unlucky data, and do not take into account the user's actual situation or day. As a result, the advice provided is general, making it difficult to provide specific help for the user's health condition or activities. Furthermore, there is a lack of utilization of user feedback to improve the accuracy of the advice.

[1339] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1340] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition, such as facial color and facial expression, and a means for receiving the measured biometric data and the analysis results. This makes it possible to grasp the user's health state and condition in real time and provide individually optimized advice. The system also includes a means for automatically generating advice using a generative AI model and a means for generating prompt sentences based on the analysis results, making it possible to quickly provide highly personalized advice. Furthermore, the generated advice is converted into speech using a speech synthesis engine and provided through a speaker, allowing for quick and effective notification to the user. These means enable the provision of personalized health advice desired by the user and improved accuracy.

[1341] "Biometric data" refers to data that indicates the user's health condition, including heart rate, body temperature, number of steps taken, sleep patterns, etc.

[1342] "Condition" is information that indicates the user's physical and mental state, and is data that is analyzed based on facial color, facial expressions, movements, etc.

[1343] "Means for measuring in real time" refers to a device or method for continuously measuring a user's biometric data and instantly obtaining the results.

[1344] The term "analyzing means" refers to a device or method for evaluating the user's physical condition or health status based on measured biometric data and user condition data.

[1345] "Means for receiving" refers to a device or method for receiving data transmitted from a measurement device or an analytical device.

[1346] "Good and bad fortune data" is information that indicates the fortune and lucky data for that day, and serves as a reference when the user decides what to do.

[1347] "Means for generating advice" refers to a device or method that generates specific advice or recommended actions to be provided to a user based on the analysis results and good or bad luck data.

[1348] "Generative AI model" refers to a machine learning model that uses artificial intelligence technology to generate appropriate advice or responses from data.

[1349] A "prompt sentence" is an instruction sentence input into a generative AI model that prompts the model to generate specific advice.

[1350] A "speech synthesis engine" refers to software or hardware technology for converting text data into natural-sounding speech.

[1351] MODE FOR CARRYING OUT THE INVENTION

[1352] Overall system configuration

[1353] This system monitors the user's physical condition and condition in real time and provides optimal advice. Specifically, it consists of the following components:

[1354] Terminal (wearable device): Continuously measures the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and transmits it to the server in real time or in batches.

[1355] AI Smart Camera: Captures the user's face in real time, analyzes their facial expressions, complexion, and movements, and sends the results to a server.

[1356] Server: Stores and analyzes the received data and generates advice using generative AI models.

[1357] Speaker: Provides the generated advice aloud using a speech synthesis engine.

[1358] 1. User data collection

[1359] Terminal

[1360] The terminal constantly measures the user's biometric data. Examples of wearable devices include Apple Watch and Fitbit.

[1361] Heart rate is measured every minute and body temperature every hour, and the data is sent to the server via HTTP POST requests.

[1362] AI Smart Camera

[1363] The camera captures the user's face in real time, and uses hue and saturation analysis for facial color analysis, and a face recognition library (e.g., OpenCV) for facial expression analysis.

[1364] The analysis results are sent to the server in real time.

[1365] 2. Data Receipt and Analysis

[1366] server

[1367] The server receives the data sent from the device and the AI ​​smart camera and stores it in a database (e.g., MySQL, PostgreSQL).

[1368] Based on the stored data, the user's physical condition and health are analyzed using a machine learning model (e.g., TensorFlow, PyTorch).

[1369] 3. Reference to fortune data

[1370] server

[1371] The server retrieves the day's fortune data from the database based on the user's profile data.

[1372] The latest fortune data is updated daily from public APIs, etc.

[1373] 4. Generating Advice

[1374] server

[1375] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate specific advice based on the analysis results and the fortune data.

[1376] An example of a specific prompt sentence is "Generate advice when the user's heart rate is higher than normal and the quality of sleep is poor. Also, when the fortune data indicates an unlucky day."

[1377] 5. Providing advice

[1378] server

[1379] The generated advice is sent to the speaker and device via a REST API in JSON format.

[1380] speaker

[1381] The speaker uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert text data into natural-sounding speech and convey it to the user.

[1382] Advice such as "Try not to push yourself too hard today and try to relax" is provided via voice.

[1383] Terminal

[1384] The device will display advice in text format on the display. For example, the smartwatch will say, "Today is a great day to be active. Try a new hobby."

[1385] Specific examples

[1386] As a specific example of how it works, if a user is feeling unwell, the device will send data such as a higher than normal heart rate and poor quality sleep to the server, and the AI ​​smart camera will analyze that the user looks pale and tired, and send this to the server. After receiving this data, the server determines that the user is feeling unwell and checks the fortune data to see if that day is an unlucky day. As a result, it generates advice such as "Try not to push yourself today and try to relax," and sends this to the speaker and device.

[1387] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1388] Step 1: Measure and send data

[1389] Terminal

[1390] Input: Real-time biometric data such as the user's heart rate, body temperature, steps taken, and sleep patterns.

[1391] What it does: A wearable device (e.g., a smartwatch) measures your heart rate every minute and your body temperature every hour.

[1392] Data processing: These biometric data are collected and sent to the server via an HTTP POST request.

[1393] Output: Formatted biometric data (JSON format).

[1394] Step 2: Analyzing the user's facial data

[1395] AI Smart Camera

[1396] Input: User's facial image and video data.

[1397] How it works: The camera captures the user's face in real time and analyzes their facial color and facial expressions. Hue and saturation analysis is used for facial color analysis, and a face recognition library (e.g., OpenCV) is used for facial expression analysis.

[1398] Data calculation: Facial color analysis and facial expression recognition algorithms are applied to quantify stress levels and fatigue levels.

[1399] Output: Analysis results (e.g., data such as pale complexion or tired facial expression).

[1400] Step 3: Receiving and storing data

[1401] server

[1402] Input: Biometric and facial data sent from the device and AI smart camera.

[1403] Specific operation: The server receives the HTTP POST request and saves it in a database (e.g. MySQL, PostgreSQL).

[1404] Data processing: The received data is properly formatted and a timestamp is added to each data.

[1405] Output: Data stored in a database.

[1406] Step 4: Analyzing the user's physical condition and condition

[1407] server

[1408] Input: Stored biometric and facial data.

[1409] Specific operation: Uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the user's physical condition and condition.

[1410] Data calculations: Combining multiple data points to assess your physical condition based on specific conditions (e.g., higher than normal heart rate and poor sleep quality).

[1411] Output: Analysis results (e.g., poor health).

[1412] Step 5: Check the luck data

[1413] server

[1414] Input: User profile data.

[1415] Specific operation: Obtains the day's fortune data from the database and updates the latest information from the public API as needed.

[1416] Data calculation: Calculates fortune based on the user's date of birth and other auspicious and inauspicious factors.

[1417] Output: Fortune data for that day (e.g. lucky day, unlucky day).

[1418] Step 6: Generating Advice

[1419] server

[1420] Input: Analysis results and fortune data.

[1421] Specific operation: Uses a generative AI model (e.g., OpenAI GPT-4) to generate advice based on the prompt.

[1422] Data processing: The prompt text "Generate advice when the user's heart rate is higher than normal and the quality of their sleep is poor. Also, what is the auspicious / unlucky date?" is input into the generative AI model, and natural language processing is performed.

[1423] Output: The generated advice (e.g., "Try to relax and take it easy today.").

[1424] Step 7: Providing advice

[1425] server

[1426] Input: The generated advice.

[1427] Specific operation: Advice is sent to the speaker and device in JSON format.

[1428] Output: Sending advice data.

[1429] speaker

[1430] Input: Advice data in JSON format.

[1431] Specific operation: Uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert into natural-sounding speech.

[1432] Output: Spoken advice (e.g., "Try to relax and take it easy today.").

[1433] Terminal

[1434] Input: Advice data in JSON format.

[1435] Specific operation: Display advice in text format on the device display.

[1436] Output: Advice in text form (e.g., "Today is a great day to be active. Try a new hobby.").

[1437] (Application example 1)

[1438] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1439] In today's world, there is a demand for a system that analyzes a user's physical condition and mood in real time and provides optimal advice when the user is selecting a product in a virtual store. Conventional virtual stores do not provide product suggestions that take into account the user's instantaneous physical condition and mood, and as a result, the user's satisfaction and purchasing motivation are not fully stimulated. The object of the present invention is to solve this problem by providing a system that provides optimal advice and product suggestions in real time based on the user's physical condition and mood.

[1440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1441] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition such as complexion and facial expression, and a means for receiving the measured biometric data and the analysis results, which makes it possible to analyze the user's physical condition and provide optimal advice and product suggestions in real time within the virtual store.

[1442] "User's biometric data" refers to data that indicates the user's physical condition, such as the user's heart rate, body temperature, number of steps taken, and sleep patterns.

[1443] "Analysis of facial color and facial expression" is the process of analyzing the user's facial color and facial expression and determining the user's condition from that information.

[1444] "Means for receiving measured biometric data and analysis results" refers to a device or system that receives data sent from a user's wearable device or AI smart camera.

[1445] "Analysis of the user's physical condition and condition" is the process of comprehensively analyzing the user's health and mood based on biometric data, facial color, and facial expression data.

[1446] "Good or bad fortune data" is fortune information such as whether it is an auspicious day or an unlucky day based on the current date.

[1447] The "means for generating advice" is a system or algorithm that uses the analysis results and the fortune data to generate specific advice tailored to the user's condition.

[1448] The "means for providing the generated advice by voice and text" refers to means for providing the generated advice by voice and on a display, and is a mechanism for conveying the advice to the user.

[1449] "A means for suggesting optimal products based on the user's physical condition and condition within a virtual store" is a system that suggests optimal products in a virtual reality environment, taking into account the user's health condition and mood.

[1450] The system embodying the present invention monitors the user's physical condition in real time and provides optimal advice and product suggestions. This system is composed of the following main components:

[1451] Overall system configuration

[1452] The system mainly consists of the following components:

[1453] Wearable devices: Continuously measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and send it to a server in real time or in batches.

[1454] AI Smart Camera: Captures the user's face in real time and analyzes their facial expressions, complexion, and movements, and sends the analysis results to the server.

[1455] Server: Receives data sent from the wearable device and AI smart camera and stores it in a database. Analyzes the user's physical condition and health based on the stored data. A machine learning model is used for the analysis.

[1456] Speaker and display: Generated advice and product suggestions are delivered to the user via voice and text.

[1457] Data collection and analysis

[1458] The server receives biometric data, facial color, and facial expression data sent from the wearable device and AI smart camera. This data is used to analyze the user's physical condition and condition using a machine learning model. Based on the analysis results, the server determines the user's condition and refers to the fortune data for the day.

[1459] Generating advice and product recommendations

[1460] The server uses a generative AI model based on the analysis results and auspicious / lucky data to generate specific advice and product suggestions. For example, if you are feeling unwell and it's an unlucky day, the server generates advice such as, "Try not to push yourself too hard today and relax." If you are feeling well and it's an auspicious day, the server generates a product suggestion such as, "Today is a great day to be active. Try a new product."

[1461] Providing advice and product suggestions

[1462] The generated advice and product suggestions are provided to the user through a speaker and a display. The speaker uses a speech synthesis engine to convert text data into natural-sounding speech, and the display displays the information in text format.

[1463] The specific hardware and software used

[1464] Hardware:

[1465] Wearable devices: Apple Watch, Fitbit, etc.

[1466] AI smart camera: OpenCV, Azure Face API, etc.

[1467] Speakers and Displays

[1468] software:

[1469] Data analysis: TensorFlow, AWS SageMaker, etc.

[1470] Speech synthesis engine: Google Text-to-Speech API, IBM Watson TTS, etc.

[1471] Generative AI models: GPT-3, etc.

[1472] Specific examples

[1473] simulation:

[1474] A user logs into the virtual shop.

[1475] The wearable device measures the user's heart rate, body temperature, steps taken, and sleep patterns and sends the data to a server.

[1476] The AI ​​smart camera analyzes the user's complexion and facial expressions and sends the results to the server.

[1477] The server analyzes this data and determines the user's physical condition in real time.

[1478] If you are in poor health and it is an unlucky day, the advice is, "Don't push yourself today, take your time choosing your products."

[1479] If you are in good health and it's an auspicious day, the advice is, "Be active, have fun, and try a new product today!"

[1480] Example prompt sentence:

[1481] Prompt for a generative AI model (e.g. GPT-3):

[1482] "Generate advice for when a user is in poor health and having an unlucky day. Specific advice: 'Try to relax and not push yourself too hard today.'"

[1483] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1484] Step 1:

[1485] The user wears a wearable device, which measures the user's biometric data, such as heart rate, body temperature, number of steps, and sleep patterns, in real time and transmits it to a server. The input is the measured biometric data, and the output is the data sent to the server.

[1486] Step 2:

[1487] The server receives the biometric data sent from the wearable device and stores the data in a database. The input is the biometric data from the wearable device, and the output is the data stored in the database. A storage system is used to store the data.

[1488] Step 3:

[1489] The user stands in front of the AI ​​smart camera. The camera captures the user's face and analyzes their facial expressions, complexion, and movements. The input is the captured image data, and the output is the analysis results. Image analysis software (e.g., OpenCV, Azure Face API) is used for the analysis.

[1490] Step 4:

[1491] The AI ​​smart camera sends the analysis results to the server. The input is the image analysis result, and the output is the data sent to the server. The server also stores this data in a database.

[1492] Step 5:

[1493] The server analyzes the user's physical condition and health using the received biometric data and image analysis results. The input is the biometric data and image analysis data stored in the database, and the output is the analysis results. Machine learning models (e.g., TensorFlow, AWS SageMaker) are used for the analysis.

[1494] Step 6:

[1495] Based on the analysis results, the server retrieves the current day's fortune data from the database. The input is the analysis results, and the output is the fortune data. The latest fortune data is retrieved by a database query.

[1496] Step 7:

[1497] The server uses a generative AI model (e.g., GPT-3) based on the analysis results and the fortune data to generate specific advice and product suggestions. The input is the analysis results and fortune data, and the output is the generated advice and product suggestions. A prompt sentence is input into the generative AI model, which generates the optimal advice.

[1498] Step 8:

[1499] The generated advice and product suggestions are sent from the server to the speaker and display. The input is the generated advice and product suggestions, and the output is text and audio that is displayed on the display and played back from the speaker.

[1500] Step 9:

[1501] The user receives advice and product suggestions through a speaker and a display. The input is the text displayed on the display and the audio played from the speaker, and the output is the user's behavior and purchasing decisions.

[1502] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1503] A specific embodiment for implementing the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized advice can be achieved.

[1504] Overall system configuration

[1505] The system mainly consists of the following components:

[1506] Terminal (wearable device)

[1507] AI Smart Camera

[1508] Emotion Engine

[1509] server

[1510] speaker

[1511] 1. User data collection

[1512] Terminal

[1513] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[1514] The collected data is sent to the server in real time or in batches.

[1515] AI Smart Camera

[1516] It captures the user's face in real time and analyzes their facial expression, facial expression, and movements.

[1517] As an example, stress levels and fatigue levels can be determined from facial color.

[1518] The analysis results are sent as data to the server.

[1519] 2. Emotion recognition

[1520] Emotion Engine

[1521] It analyzes the user's facial images and voice data obtained from the AI ​​smart camera and recognizes the user's emotions (joy, sadness, anger, etc.).

[1522] The recognized emotion data is sent to the server.

[1523] 3. Data Receipt and Analysis

[1524] server

[1525] It receives biometric data, condition data, and emotion data from the device, AI smart camera, and emotion engine, and stores them in a database.

[1526] Check the consistency of the data, and if there is any inconsistency, request the data again.

[1527] 4. Analysis of user's physical condition and emotions

[1528] server

[1529] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to analyze the user's physical condition and emotions. Here, factors such as a high heart rate, poor sleep quality, and the frequency of happy facial expressions are combined to form a comprehensive evaluation.

[1530] 5. Reference to fortune data

[1531] server

[1532] Based on the user's profile information, the system retrieves the day's fortune data from the database.

[1533] If the data on lucky and unlucky days has been updated, the latest information is acquired accordingly and the database is updated.

[1534] 6. Generating Advice

[1535] server

[1536] The analysis results, emotional data, and good and bad luck data are combined and the optimal advice for the day is generated using a generative AI model.

[1537] The advice includes encouraging rest when you're feeling unwell and activity when you're feeling well, and it adjusts the advice based on the perceived emotions.

[1538] 7. Providing advice

[1539] server

[1540] The generated advice is formatted in JSON or XML and sent to the speaker and device, allowing the advice to be provided appropriately in both voice and text.

[1541] speaker

[1542] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[1543] As an example, advice can be given as "Try to relax and take it easy today" or "Today is a great day to be active. Try a new hobby."

[1544] Terminal

[1545] Display advice in text format on the screen.

[1546] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[1547] Specific examples

[1548] When the user is unwell and emotionally sad

[1549] Terminal

[1550] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[1551] AI Smart Camera

[1552] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[1553] Emotion Engine

[1554] The sadness emotion is recognized from the user's face and transmitted to the server.

[1555] server

[1556] Based on the received data, it is determined that the user is in poor health.

[1557] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[1558] Taking into consideration that the person is feeling unwell, that it is an unlucky day, and that the emotion is sadness, the advice generated is, "Try not to push yourself today and try to relax. Read your favorite book to change your mood."

[1559] The generated advice is sent to the speaker and the device.

[1560] speaker

[1561] The voice tells the user, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[1562] Terminal

[1563] The display will say, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[1564] If the user is in good health and the emotion is joy

[1565] Terminal

[1566] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[1567] AI Smart Camera

[1568] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[1569] Emotion Engine

[1570] The joyful emotion is recognized from the user's face and transmitted to the server.

[1571] server

[1572] Based on the received data, it is determined that the user's physical condition is good.

[1573] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[1574] Taking into consideration that the emotion is joy in addition to the person being in good health and that it is an auspicious day, the advice generated is "Today is a great day to be active. Have a good time with your friends."

[1575] The generated advice is sent to the speaker and the device.

[1576] speaker

[1577] A voice tells the user, "Today is a great day to be active and have fun with friends."

[1578] Terminal

[1579] The display will say, "Today is a great day to be active and have fun with friends."

[1580] The above is a specific embodiment for carrying out the present invention.

[1581] The processing flow will be explained below.

[1582] Step 1:

[1583] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits this data to a server in real time.

[1584] Step 2:

[1585] The AI ​​smart camera captures the user's face in real time and analyzes their facial expressions, complexion, and movements. The analyzed data is sent to a server as information about the user's condition.

[1586] Step 3:

[1587] The emotion engine analyzes the user's facial images and voice data acquired from the AI ​​smart camera and recognizes the user's emotions (happiness, sadness, anger, etc.). The recognized emotion data is sent to the server.

[1588] Step 4:

[1589] The server receives biometric data, condition data, and emotion data sent from the device, AI smart camera, and emotion engine, and stores it in a database. It checks the consistency of the received data and requests the data again if there is any inconsistency.

[1590] Step 5:

[1591] The server retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, the overall analysis results may show a high heart rate, poor sleep quality, and sadness.

[1592] Step 6:

[1593] The server refers to the user's profile information and retrieves the lucky / unlucky data for that day from the database. If the lucky / unlucky data has been updated, the new information is updated in the database.

[1594] Step 7:

[1595] The server combines the results of the physical condition analysis, emotional data, and auspicious / unlucky data, and uses a generative AI model to generate the optimal advice for that day. The advice is based on the individual's physical condition and emotions. For example, if you are feeling unwell and sad, the advice might be, "Try to relax and not push yourself today. I recommend reading your favorite book to change your mood." If you are feeling good and happy, the advice might be, "Today is a great day. Have some fun outdoors with friends."

[1596] Step 8:

[1597] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The transmitted data is output as voice and text.

[1598] Step 9:

[1599] The speaker receives advice from the server and converts it into natural-sounding speech using a speech synthesis engine, which then relays it to the user. For example, the advice could be, "Try not to push yourself too hard today, but relax. Read your favorite book to change your mood."

[1600] Step 10:

[1601] The device will display the received advice in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[1602] Step 11:

[1603] The user acts on the advice provided. The effectiveness of the advice and the user's opinion are input into the device as feedback. For example, the user can input whether the advice was helpful or whether their physical condition improved.

[1604] Step 12:

[1605] The device sends user feedback to the server, which is used to make future advice more effective.

[1606] Step 13:

[1607] The server stores the feedback in a database and uses it to generate advice in future. This feedback information continuously improves the accuracy of the entire system.

[1608] Example 2

[1609] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1610] Conventional healthcare systems tend to collect only a user's biometric data and provide advice based solely on their physical condition. However, providing personalized advice that also takes into account the user's emotions and fortunes is important for comprehensive health management. Furthermore, the lack of a means to improve the accuracy of advice based on feedback makes continuous improvement difficult. This has led to the issue of not being able to fully meet user needs.

[1611] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric information of the user in real time, means for analyzing the user's condition such as complexion, facial expression, and movement, means for receiving the measured biometric data and analysis results, means for generating advice using a generative AI model, means for providing the generated advice in audio and text, and means for collecting feedback from the user and improving the accuracy of advice from the next time onwards. This makes it possible to provide personalized advice that comprehensively takes into account the user's physical condition, emotions, and fortune, and to continuously improve the accuracy of that advice.

[1612] "User's biological information" is data that indicates the user's physical condition, such as heart rate, body temperature, number of steps, and sleep pattern.

[1613] "Real-time acquisition means" refers to the ability of sensors or devices to collect data immediately, without time delay, and transmit that data to the system.

[1614] "User's condition such as complexion, facial expression, and movement" is subjective information that indicates the user's emotions, stress level, activity level, and so on.

[1615] "Means of analysis" refers to the algorithms and software functions used to analyze data and detect specific patterns or anomalies.

[1616] "Measured biometric data and analysis results" refers to biometric information obtained from sensors or devices and the results of analyzing that data.

[1617] "Means for receiving" refers to the interface or protocol for importing and storing data from the outside.

[1618] A "generative AI model" is an artificial intelligence algorithm or system that automatically generates an output based on specific input data.

[1619] The "means for generating advice" is a function that automatically creates recommended actions and points of caution for the user based on the collected data and analysis results.

[1620] The "means of providing by voice and text" is a function of reading out the generated advice as voice and displaying it as text.

[1621] "Means for collecting feedback" refers to interfaces and protocols for capturing opinions and reactions from users and reflecting them in the system.

[1622] A specific embodiment of this invention will be described. This system monitors the user's physical condition and emotions in real time and provides optimal advice based on the results. The entire system mainly consists of the following components: a terminal, an AI smart camera, an emotion engine, a server, and a speaker.

[1623] User data collection

[1624] Terminal (wearable device):

[1625] The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The wearable device is equipped with sensors such as a heart rate monitor, a body temperature sensor, and an accelerometer. For example, the heart rate monitor measures values ​​such as "90 BPM" and the body temperature sensor measures "36.5°C."

[1626] These biometric data are transmitted to a server periodically or in real time via Bluetooth or Wi-Fi.

[1627] AI Smart Camera:

[1628] The camera captures the user's facial color, facial expressions, and movements, and analyzes this data in real time. The camera has an image analysis algorithm built in to analyze facial color and facial expressions. For example, if the user's face is pale, it will be determined that the stress level is high.

[1629] The analyzed data is immediately sent to the server.

[1630] Emotion recognition and data reception

[1631] Emotion Engine:

[1632] Using image and audio data obtained from an AI smart camera, the system recognizes the user's emotions (happiness, sadness, anger, etc.). For example, it uses facial expression analysis technology to determine that the user is expressing happiness.

[1633] The recognized emotion data is transmitted to a server.

[1634] server:

[1635] It receives all data acquired from devices, AI smart cameras, and emotion engines, stores it in a database, checks the data consistency, and requests the data again if there is any inconsistency.

[1636] Analyzing physical condition and emotions and generating advice

[1637] server:

[1638] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to comprehensively analyze the user's physical condition and emotions. For example, if the user has a high heart rate, poor sleep quality, and sadness, the system will determine that the user is in poor health.

[1639] Based on the user's profile information, the system retrieves the day's fortune data from the database. If the fortune data is not up to date, the system retrieves the latest data from an external data source and updates the database.

[1640] The analysis results are combined with fortune data and a generative AI model (e.g., GPT-3) is used to generate prompts. Examples of prompts include:

[1641] Example prompt sentence:

[1642] User health data:

[1643] Heart rate: 90 BPM

[1644] Body temperature: 36.5℃

[1645] Steps: 10,000

[1646] Sleep Pattern: Good

[1647] User sentiment data:

[1648] Emotion: Joy

[1649] Expression: Cheerful

[1650] Activity: Active

[1651] Fortune Data:

[1652] Today's Fortune: Auspicious Day

[1653] Use this information to generate the best advice for your users.

[1654] A generative AI model receives the prompts and generates the best advice for the day based on them.

[1655] Providing advice

[1656] server:

[1657] The generated advice is formatted into JSON or XML format and sent to the speaker and device.

[1658] speaker:

[1659] A speech synthesis engine is used to convert the generated advice into natural-sounding speech and convey it to the user.

[1660] Device:

[1661] Advice is displayed in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[1662] This system will provide personalized advice that takes into account the user's physical condition, emotions, and fortune, and will be able to continuously improve its accuracy.

[1663] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1664] Program processing flow

[1665] Step 1:

[1666] Data Acquisition

[1667] Device: The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The device periodically reads data from the heart rate monitor and body temperature sensor, converts the data into JSON format, and stores it in a transmission queue.

[1668] Input: Biometric data from sensors.

[1669] Output: Biometric data in JSON format.

[1670] Step 2:

[1671] Sending data

[1672] Terminal: The terminal sends the collected biometric data to the server via Bluetooth or Wi-Fi. Specifically, the terminal sends the collected JSON data to the server's receiving API as an HTTP POST request.

[1673] Input: Biometric data obtained in step 1 in JSON format.

[1674] Output: HTTP request to the server.

[1675] Step 3:

[1676] Camera-based facial color and facial expression analysis

[1677] AI Smart Camera: Captures the user's facial expressions, facial expressions, and movements, and analyzes them in real time using image analysis algorithms. For example, facial expression analysis software analyzes the user's facial photo to determine emotions such as "sadness" or "happiness."

[1678] Input: A face image captured by the camera.

[1679] Output: Analyzed facial color and expression data.

[1680] Step 4:

[1681] Sending camera data

[1682] AI Smart Camera: Converts the analysis results into JSON format and sends them to the server. Sends the analysis data using an HTTP POST request.

[1683] Input: Facial color and expression data analyzed in step 3.

[1684] Output: JSON formatted data to the server.

[1685] Step 5:

[1686] Processing received data

[1687] Server: Receives biometric data and facial image analysis data sent from the device and AI smart camera. The receiving API receives this data and stores it in a database.

[1688] Input: HTTP requests from the device and the AI ​​smart camera.

[1689] Output: Biometric data and facial image analysis data stored in a database.

[1690] Step 6:

[1691] emotion recognition

[1692] Emotion Engine: Analyzes facial image data retrieved from a database to recognize the user's emotions. It uses an AI model to identify emotions such as "happiness" or "sadness."

[1693] Input: Facial image data stored in a database.

[1694] Output: Parsed emotion data.

[1695] Step 7:

[1696] Comprehensive analysis of data

[1697] Server: Retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, if the heart rate is high, sleep quality is poor, and the user is expressing sadness, it determines that the user is in poor health.

[1698] Input: Biometric data, facial expression data, and emotion data in the database.

[1699] Output: Comprehensively analyzed user physical and emotional data.

[1700] Step 8:

[1701] Fortune data reference

[1702] Server: Based on the user's profile information, retrieves the day's fortune data from the database. If the fortune data is not up to date, retrieves the latest data from an external interface and updates the database.

[1703] Input: User profile information.

[1704] Output: Fortune data for the day.

[1705] Step 9:

[1706] Prompt creation for advice generation

[1707] Server: Combines the user's biometric, emotional, and fortune data to generate prompts to input into the generative AI model. Examples of prompts include:

[1708] Example prompt sentence:

[1709] User health data:

[1710] Heart rate: 90 BPM

[1711] Body temperature: 36.5℃

[1712] Steps: 10,000

[1713] Sleep Pattern: Good

[1714] User sentiment data:

[1715] Emotion: Joy

[1716] Expression: Cheerful

[1717] Activity: Active

[1718] Fortune Data:

[1719] Today's Fortune: Auspicious Day

[1720] Use this information to generate the best advice for your users.

[1721] Input: Biometric data, emotional data, fortune data.

[1722] Output: Prompts to the generative AI model.

[1723] Step 10:

[1724] Generating Advice

[1725] Generative AI model: Generates optimal advice for the user based on the prompt. For example, it generates advice such as, "Try not to push yourself too hard today and try to relax."

[1726] Input: Prompt statement.

[1727] Output: The generated advice.

[1728] Step 11:

[1729] Sending Advice

[1730] Server: Formats the generated advice into JSON or XML format and sends it to the speaker and device, providing advice in voice and text.

[1731] Input: The generated advice.

[1732] Output: JSON or XML data to speakers and devices.

[1733] Step 12:

[1734] Voice and text advice provided

[1735] Speaker: Uses a speech synthesis engine to convert the JSON or XML data into speech and tells the user, "Try to relax and take it easy today."

[1736] Terminal: Display the text "Try not to push yourself too hard today and try to relax" on the display.

[1737] Input: Advice sent by the server.

[1738] Output: Providing audio and text advice to the user.

[1739] These are the specific processing steps of this system, which allows users to receive optimal advice based on their physical condition, emotions, and fortune.

[1740] (Application example 2)

[1741] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1742] Conventional self-driving vehicles lack real-time feedback based on the physical condition and emotions of the driver and passengers, making it difficult to provide optimal driving modes and entertainment. They also lack the ability to provide personalized advice based on the day's auspicious and unlucky data or the individual user's condition. These circumstances have led to insufficient safety and comfort within the vehicle.

[1743] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for measuring the user's biological data in real time, means for analyzing the user's condition such as facial color and facial expression, and means for receiving the measured biological data and the analysis results. This makes it possible to monitor the user's physical condition and emotions in real time in an autonomous vehicle and optimize the driving mode and entertainment based on the monitoring results.

[1744] "User's biometric data" refers to data that indicates a person's physical condition, such as heart rate, body temperature, number of steps taken, and sleep patterns.

[1745] "User condition such as facial color and facial expression" is information indicating the physical and emotional state of the user, including the color and facial expression of the user's face, and movements.

[1746] "Analysis means" refers to a device or software that evaluates and judges the user's physical condition and emotions based on measured data and images.

[1747] "Today's fortune data" is traditional or astrological information that indicates whether the day is auspicious or inauspicious for the user.

[1748] The "means for generating advice" is a device or software that suggests appropriate actions or measures to the user based on the collected data and analysis results.

[1749] "Means for providing generated advice in audio and text" refers to a device or software that displays and plays generated advice in audio or text format in order to convey the advice to the user in an easy-to-understand manner.

[1750] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[1751] "Driving modes" are settings that adjust the style and conditions under which an autonomous vehicle is driven.

[1752] "Entertainment providing means" refers to devices or software for playing entertainment content such as music and video for users.

[1753] "Means for recognizing emotions" refers to a device or software that analyzes a user's facial expressions, actions, and voice data to determine their emotional state.

[1754] A "generative AI model" is an algorithm or program that is trained by artificial intelligence to generate appropriate advice or suggestions from specific data.

[1755] A "prompt" is a specific input sentence used to generate an appropriate answer or output for a generative AI model.

[1756] A specific embodiment of the present invention will be described. This invention is a system that monitors the physical condition and emotions of occupants in an autonomous vehicle in real time and provides optimal driving modes and entertainment based on the monitoring results. The entire system is composed of the following main components.

[1757] System configuration

[1758] The system mainly consists of the following components:

[1759] Wearable devices

[1760] AI Camera

[1761] Emotion Engine

[1762] server

[1763] speaker

[1764] 1. Data Collection

[1765] Wearable devices

[1766] Wearable devices (e.g., smartwatches) constantly measure biometric data such as heart rate, body temperature, steps taken, and sleep patterns.

[1767] The collected data is sent to a server in real time at irregular intervals.

[1768] AI Camera

[1769] An AI camera installed inside the vehicle captures and analyzes the facial expressions, facial expressions, and movements of the occupants in real time.

[1770] The analysis results are sent to a server as data to determine each passenger's stress level and fatigue level.

[1771] 2. Emotion recognition

[1772] Emotion Engine

[1773] The emotion engine analyzes facial images and audio data obtained from the AI ​​camera to recognize the emotions of the occupants (joy, sadness, anger, etc.).

[1774] The recognized emotion data is transmitted to a server.

[1775] 3. Data Receipt and Analysis

[1776] server

[1777] It receives biometric data, condition data, and emotion data from the wearable device, AI camera, and emotion engine, and stores them in a database.

[1778] Check the consistency of the data and re-request the data if there is any inconsistency.

[1779] 4. Data analysis and advice generation

[1780] server

[1781] The latest biometric, condition, and emotional data is collected and the occupants' physical condition and emotions are analyzed using machine learning models.

[1782] The data on the day's fortunes is referenced and retrieved from the database.

[1783] The analysis results, emotional data, and good and bad fortune data are combined and a generative AI model is used to generate the best advice for the day.

[1784] The advice includes specific suggestions for action, such as when drivers should take a break and relax, or when they should play music to help them relax.

[1785] 5. Providing advice

[1786] speaker

[1787] The advice is given to the passengers using a speech synthesis engine, which converts text data into natural-sounding speech.

[1788] Terminal

[1789] Advice is displayed in text format on the in-car display.

[1790] Specific examples

[1791] Example 1: When the user is unwell and sad

[1792] The wearable device measures high heart rate and poor sleep quality and sends the results to a server.

[1793] The AI ​​camera analyzes whether the person's complexion is pale and they look tired, and sends the results to the server.

[1794] The emotion engine recognizes the sadness emotion from the user's face and sends it to the server.

[1795] The server determines that the user is feeling unwell and that it is an unlucky day, generates advice such as "Try not to push yourself today and try to relax. Read your favorite book to change your mood," and sends this advice to the device and speaker.

[1796] Example 2: When the user is in good health and the emotion is joy

[1797] The wearable device measures the heart rate, body temperature, and number of steps taken, and sends the results to the server.

[1798] The AI ​​camera analyzes whether the person has a good complexion and a bright expression, and sends the results to the server.

[1799] The emotion engine recognizes the emotion of joy from the user's face and transmits it to the server.

[1800] The server determines that the person is in good health and that it is an auspicious day, generates advice such as "Today is a great day to be active. Have a good time with friends," and sends it to the device and speaker.

[1801] Prompt example

[1802] "If the user has a high heart rate but is showing signs of joy, provide appropriate advice."

[1803] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1804] Step 1:

[1805] The wearable device measures the user's biometric data in real time, specifically, heart rate, body temperature, number of steps taken, sleep patterns, etc. This acquired data is input, and the wearable device sends it to a server.

[1806] Step 2:

[1807] The server stores the received biometric data in a database and checks its consistency. Specifically, it checks whether the data is missing or consistent. If there is an inconsistency, it requests the data again. The input to this process is the biometric data sent from the wearable device, and the output is the biometric data whose consistency has been confirmed.

[1808] Step 3:

[1809] The AI ​​camera captures the facial color and facial expressions of the user in the car in real time. Specific operations include analyzing facial color and facial expressions. These analysis results are input, and the AI ​​camera sends the analysis results to the server.

[1810] Step 4:

[1811] The server receives the analysis results sent from the AI ​​camera and stores them in a database. The input is the analysis result data, and the output is the saved analysis result data. The server also checks the consistency of this data.

[1812] Step 5:

[1813] The emotion engine analyzes emotions based on facial image data obtained from the AI ​​camera. Specifically, it uses facial expression data to recognize emotional states (happiness, sadness, anger, etc.). This recognition result is input, and the emotion engine sends it to the server.

[1814] Step 6:

[1815] The server receives the emotion data sent from the emotion engine and stores it in a database. The input is emotion data, and the output is the stored emotion data. The server also checks the integrity of the emotion data.

[1816] Step 7:

[1817] The server integrates all data received from the wearable device, AI camera, and emotion engine to obtain the latest biometric, condition, and emotion data. These data are the input, and the output is the integrated dataset.

[1818] Step 8:

[1819] The server uses the integrated data set to analyze the user's physical condition and emotions using a machine learning model. Specifically, it evaluates heart rate and facial expression changes to determine the user's overall physical condition and emotions. The results of this analysis are the input, and the output is the analysis result data.

[1820] Step 9:

[1821] The server retrieves the fortune data for that day from the database. The input is the date information, and the output is the fortune data for that day. The server combines this with the analysis results.

[1822] Step 10:

[1823] The server uses a generative AI model to generate optimal advice based on the analysis results and the good and bad fortune data. Specifically, a prompt sentence is used as input to the model to generate appropriate advice. An example of this prompt sentence is, "If the user's heart rate is high but they are showing signs of joy, please provide appropriate advice." The input is the integrated data and the prompt sentence, and the output is the generated advice.

[1824] Step 11:

[1825] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The input is the generated advice, and the output is the formatted advice data.

[1826] Step 12:

[1827] The terminal displays the advice in text format on the in-car display. Specifically, it converts the transmitted format data into text and displays it on the display. The input is the formatted advice data, and the output is the display.

[1828] Step 13:

[1829] The speaker uses a speech synthesis engine to convert the advice into natural-sounding speech and convey it to the passengers. Specifically, it converts text data into speech data and plays it back. The input is formatted advice data, and the output is advice conveyed in speech.

[1830] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1831] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1832] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1833] [Fourth embodiment]

[1834] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1835] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1836] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1837] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1838] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1839] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1840] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1841] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1842] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1843] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1844] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1845] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1846] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1847] A specific embodiment for carrying out the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that information. The program processing and specific examples will be described in detail below.

[1848] Overall system configuration

[1849] The system mainly consists of the following components:

[1850] Terminal (wearable device)

[1851] AI Smart Camera

[1852] server

[1853] speaker

[1854] 1. User data collection

[1855] Terminal

[1856] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[1857] The collected data is sent to the server in real time or in batches.

[1858] AI Smart Camera

[1859] It captures the user's face in real time and analyzes their facial expressions, complexion, and movements.

[1860] As an example, stress levels and fatigue levels can be determined from facial color.

[1861] The analysis results are sent as data to the server.

[1862] 2. Data Receipt and Analysis

[1863] server

[1864] It receives data sent from the device and the AI ​​smart camera and stores it in a database.

[1865] The user's physical condition and health are analyzed based on the saved data, using a machine learning model.

[1866] As an example, if the user's heart rate is higher than normal and the quality of sleep is poor, the user is determined to be in poor health.

[1867] 3. Reference to fortune data

[1868] server

[1869] Based on the user's profile, the system retrieves the day's fortune data from the database.

[1870] If the data on lucky and unlucky days has been updated, the latest information is acquired and referenced accordingly.

[1871] 4. Generating Advice

[1872] server

[1873] Based on the analysis results and good and bad fortune data, a generative AI model is used to generate specific advice.

[1874] As an example, if the user is in poor health and it is an unlucky day, the system generates advice such as "Try not to push yourself too hard today and try to relax." If the user is in good health and it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[1875] 5. Providing advice

[1876] server

[1877] The generated advice is sent to the speaker and the device.

[1878] Advice is provided in audio and text formats.

[1879] speaker

[1880] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[1881] As an example, advice may be given as "Try to relax and not push yourself too hard today."

[1882] Terminal

[1883] Display advice in text format on the screen.

[1884] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[1885] Specific examples

[1886] If the user is unwell

[1887] Terminal

[1888] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[1889] AI Smart Camera

[1890] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[1891] server

[1892] Based on the received data, it is determined that the user is in poor health.

[1893] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[1894] Taking into consideration the poor physical condition and the fact that it is an unlucky day, the advice generated is "Try not to push yourself today and try to relax."

[1895] The generated advice is sent to the speaker and the device.

[1896] speaker

[1897] The voice tells the user, "Try not to push yourself too hard today and try to relax."

[1898] Terminal

[1899] The display will say, "Try not to push yourself too hard today and try to relax."

[1900] If the user is in good health

[1901] Terminal

[1902] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[1903] AI Smart Camera

[1904] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[1905] server

[1906] Based on the received data, it is determined that the user's physical condition is good.

[1907] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[1908] Taking into account the fact that the person is in good health and that it is an auspicious day, the system generates advice such as "Today is a great day to be active. Why not try a new hobby?"

[1909] The generated advice is sent to the speaker and the device.

[1910] speaker

[1911] A voice tells the user, "Today is a great day to be active. Try a new hobby."

[1912] Terminal

[1913] The display will say, "Today is a great day to be active. Try a new hobby."

[1914] The above is a specific embodiment for carrying out the present invention.

[1915] The processing flow will be explained below.

[1916] Step 1:

[1917] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits the measured data to a server in real time.

[1918] Step 2:

[1919] The AI ​​smart camera captures the user's face and analyzes their facial color, facial expressions, and movements using image analysis algorithms. The analyzed data is then sent to a server as user condition information.

[1920] Step 3:

[1921] The server receives biometric and condition data from the device and AI smart camera, stores it in a database, checks the data for consistency, and requests the data again if there is any inconsistency.

[1922] Step 4:

[1923] The server retrieves the latest biometric and condition data from the database and uses a machine learning model to analyze the user's physical condition and state, combining factors such as a high heart rate or poor sleep quality to provide a comprehensive assessment of their physical condition.

[1924] Step 5:

[1925] The server retrieves the fortune data for that day from the database based on the user's profile information. If the fortune data has been updated, the server retrieves the new information and updates the database.

[1926] Step 6:

[1927] The server combines the results of the health analysis with the auspicious data and uses a generative AI model to generate optimal advice for the day. The advice includes encouraging rest when the user is feeling unwell and encouraging activity when the user is feeling well.

[1928] Step 7:

[1929] The server formats the generated advice into JSON or XML format and sends it to the speaker and device, allowing the advice to be provided appropriately in both audio and text formats.

[1930] Step 8:

[1931] The speaker receives advice from the server, converts it using a speech synthesis engine, and conveys it to the user in a natural voice. The speaker provides advice such as, "Try to relax and not push yourself too hard today," or "Today is a great day to be active. Why not try a new hobby?"

[1932] Step 9:

[1933] The device displays the advice received from the server in text format on the display. The displayed advice matches the audio advice from the speaker.

[1934] Step 10:

[1935] The user acts based on the advice provided, and then inputs feedback about the advice through the terminal, such as whether the advice was helpful or how their physical condition has changed.

[1936] Step 11:

[1937] The device sends user feedback to the server, which is used to improve the accuracy of advice.

[1938] Step 12:

[1939] The server stores the feedback in a database and uses it to generate future advice, continuously improving the accuracy of the entire system.

[1940] Example 1

[1941] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1942] Conventional health management systems only measure a user's biometric data and condition, but lack the ability to provide individually optimized advice. Furthermore, they are unable to generate advice that takes into account the user's daily auspicious and unlucky data, and do not take into account the user's actual situation or day. As a result, the advice provided is general, making it difficult to provide specific help for the user's health condition or activities. Furthermore, there is a lack of utilization of user feedback to improve the accuracy of the advice.

[1943] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1944] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition, such as facial color and facial expression, and a means for receiving the measured biometric data and the analysis results. This makes it possible to grasp the user's health state and condition in real time and provide individually optimized advice. The system also includes a means for automatically generating advice using a generative AI model and a means for generating prompt sentences based on the analysis results, making it possible to quickly provide highly personalized advice. Furthermore, the generated advice is converted into speech using a speech synthesis engine and provided through a speaker, allowing for quick and effective notification to the user. These means enable the provision of personalized health advice desired by the user and improved accuracy.

[1945] "Biometric data" refers to data that indicates the user's health condition, including heart rate, body temperature, number of steps taken, sleep patterns, etc.

[1946] "Condition" is information that indicates the user's physical and mental state, and is data that is analyzed based on facial color, facial expressions, movements, etc.

[1947] "Means for measuring in real time" refers to a device or method for continuously measuring a user's biometric data and instantly obtaining the results.

[1948] The term "analyzing means" refers to a device or method for evaluating the user's physical condition or health status based on measured biometric data and user condition data.

[1949] "Means for receiving" refers to a device or method for receiving data transmitted from a measurement device or an analytical device.

[1950] "Good and bad fortune data" is information that indicates the fortune and lucky data for that day, and serves as a reference when the user decides what to do.

[1951] "Means for generating advice" refers to a device or method that generates specific advice or recommended actions to be provided to a user based on the analysis results and good or bad luck data.

[1952] "Generative AI model" refers to a machine learning model that uses artificial intelligence technology to generate appropriate advice or responses from data.

[1953] A "prompt sentence" is an instruction sentence input into a generative AI model that prompts the model to generate specific advice.

[1954] A "speech synthesis engine" refers to software or hardware technology for converting text data into natural-sounding speech.

[1955] MODE FOR CARRYING OUT THE INVENTION

[1956] Overall system configuration

[1957] This system monitors the user's physical condition and condition in real time and provides optimal advice. Specifically, it consists of the following components:

[1958] Terminal (wearable device): Continuously measures the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and transmits it to the server in real time or in batches.

[1959] AI Smart Camera: Captures the user's face in real time, analyzes their facial expressions, complexion, and movements, and sends the results to a server.

[1960] Server: Stores and analyzes the received data and generates advice using generative AI models.

[1961] Speaker: Provides the generated advice aloud using a speech synthesis engine.

[1962] 1. User data collection

[1963] Terminal

[1964] The terminal constantly measures the user's biometric data. Examples of wearable devices include Apple Watch and Fitbit.

[1965] Heart rate is measured every minute and body temperature every hour, and the data is sent to the server via HTTP POST requests.

[1966] AI Smart Camera

[1967] The camera captures the user's face in real time, and uses hue and saturation analysis for facial color analysis, and a face recognition library (e.g., OpenCV) for facial expression analysis.

[1968] The analysis results are sent to the server in real time.

[1969] 2. Data Receipt and Analysis

[1970] server

[1971] The server receives the data sent from the device and the AI ​​smart camera and stores it in a database (e.g., MySQL, PostgreSQL).

[1972] Based on the stored data, the user's physical condition and health are analyzed using a machine learning model (e.g., TensorFlow, PyTorch).

[1973] 3. Reference to fortune data

[1974] server

[1975] The server retrieves the day's fortune data from the database based on the user's profile data.

[1976] The latest fortune data is updated daily from public APIs, etc.

[1977] 4. Generating Advice

[1978] server

[1979] The server uses a generative AI model (e.g., OpenAI GPT-4) to generate specific advice based on the analysis results and the fortune data.

[1980] An example of a specific prompt sentence is "Generate advice when the user's heart rate is higher than normal and the quality of sleep is poor. Also, when the fortune data indicates an unlucky day."

[1981] 5. Providing advice

[1982] server

[1983] The generated advice is sent to the speaker and device via a REST API in JSON format.

[1984] speaker

[1985] The speaker uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert text data into natural-sounding speech and convey it to the user.

[1986] Advice such as "Try not to push yourself too hard today and try to relax" is provided via voice.

[1987] Terminal

[1988] The device will display advice in text format on the display. For example, the smartwatch will say, "Today is a great day to be active. Try a new hobby."

[1989] Specific examples

[1990] As a specific example of how it works, if a user is feeling unwell, the device will send data such as a higher than normal heart rate and poor quality sleep to the server, and the AI ​​smart camera will analyze that the user looks pale and tired, and send this to the server. After receiving this data, the server determines that the user is feeling unwell and checks the fortune data to see if that day is an unlucky day. As a result, it generates advice such as "Try not to push yourself today and try to relax," and sends this to the speaker and device.

[1991] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1992] Step 1: Measure and send data

[1993] Terminal

[1994] Input: Real-time biometric data such as the user's heart rate, body temperature, steps taken, and sleep patterns.

[1995] What it does: A wearable device (e.g., a smartwatch) measures your heart rate every minute and your body temperature every hour.

[1996] Data processing: These biometric data are collected and sent to the server via an HTTP POST request.

[1997] Output: Formatted biometric data (JSON format).

[1998] Step 2: Analyzing the user's facial data

[1999] AI Smart Camera

[2000] Input: User's facial image and video data.

[2001] How it works: The camera captures the user's face in real time and analyzes their facial color and facial expressions. Hue and saturation analysis is used for facial color analysis, and a face recognition library (e.g., OpenCV) is used for facial expression analysis.

[2002] Data calculation: Facial color analysis and facial expression recognition algorithms are applied to quantify stress levels and fatigue levels.

[2003] Output: Analysis results (e.g., data such as pale complexion or tired facial expression).

[2004] Step 3: Receiving and storing data

[2005] server

[2006] Input: Biometric and facial data sent from the device and AI smart camera.

[2007] Specific operation: The server receives the HTTP POST request and saves it in a database (e.g. MySQL, PostgreSQL).

[2008] Data processing: The received data is properly formatted and a timestamp is added to each data.

[2009] Output: Data stored in a database.

[2010] Step 4: Analyzing the user's physical condition and condition

[2011] server

[2012] Input: Stored biometric and facial data.

[2013] Specific operation: Uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the user's physical condition and condition.

[2014] Data calculations: Combining multiple data points to assess your physical condition based on specific conditions (e.g., higher than normal heart rate and poor sleep quality).

[2015] Output: Analysis results (e.g., poor health).

[2016] Step 5: Check the luck data

[2017] server

[2018] Input: User profile data.

[2019] Specific operation: Obtains the day's fortune data from the database and updates the latest information from the public API as needed.

[2020] Data calculation: Calculates fortune based on the user's date of birth and other auspicious and inauspicious factors.

[2021] Output: Fortune data for that day (e.g. lucky day, unlucky day).

[2022] Step 6: Generating Advice

[2023] server

[2024] Input: Analysis results and fortune data.

[2025] Specific operation: Uses a generative AI model (e.g., OpenAI GPT-4) to generate advice based on the prompt.

[2026] Data processing: The prompt text "Generate advice when the user's heart rate is higher than normal and the quality of their sleep is poor. Also, what is the auspicious / unlucky date?" is input into the generative AI model, and natural language processing is performed.

[2027] Output: The generated advice (e.g., "Try to relax and take it easy today.").

[2028] Step 7: Providing advice

[2029] server

[2030] Input: The generated advice.

[2031] Specific operation: Advice is sent to the speaker and device in JSON format.

[2032] Output: Sending advice data.

[2033] speaker

[2034] Input: Advice data in JSON format.

[2035] Specific operation: Uses a speech synthesis engine (e.g., Google Text-to-Speech) to convert into natural-sounding speech.

[2036] Output: Spoken advice (e.g., "Try to relax and take it easy today.").

[2037] Terminal

[2038] Input: Advice data in JSON format.

[2039] Specific operation: Display advice in text format on the device display.

[2040] Output: Advice in text form (e.g., "Today is a great day to be active. Try a new hobby.").

[2041] (Application example 1)

[2042] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2043] In today's world, there is a demand for a system that analyzes a user's physical condition and mood in real time and provides optimal advice when the user is selecting a product in a virtual store. Conventional virtual stores do not provide product suggestions that take into account the user's instantaneous physical condition and mood, and as a result, the user's satisfaction and purchasing motivation are not fully stimulated. The object of the present invention is to solve this problem by providing a system that provides optimal advice and product suggestions in real time based on the user's physical condition and mood.

[2044] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2045] In this invention, the server includes a means for measuring a user's biometric data in real time, a means for analyzing the user's condition such as complexion and facial expression, and a means for receiving the measured biometric data and the analysis results, which makes it possible to analyze the user's physical condition and provide optimal advice and product suggestions in real time within the virtual store.

[2046] "User's biometric data" refers to data that indicates the user's physical condition, such as the user's heart rate, body temperature, number of steps taken, and sleep patterns.

[2047] "Analysis of facial color and facial expression" is the process of analyzing the user's facial color and facial expression and determining the user's condition from that information.

[2048] "Means for receiving measured biometric data and analysis results" refers to a device or system that receives data sent from a user's wearable device or AI smart camera.

[2049] "Analysis of the user's physical condition and condition" is the process of comprehensively analyzing the user's health and mood based on biometric data, facial color, and facial expression data.

[2050] "Good or bad fortune data" is fortune information such as whether it is an auspicious day or an unlucky day based on the current date.

[2051] The "means for generating advice" is a system or algorithm that uses the analysis results and the fortune data to generate specific advice tailored to the user's condition.

[2052] The "means for providing the generated advice by voice and text" refers to means for providing the generated advice by voice and on a display, and is a mechanism for conveying the advice to the user.

[2053] "A means for suggesting optimal products based on the user's physical condition and condition within a virtual store" is a system that suggests optimal products in a virtual reality environment, taking into account the user's health condition and mood.

[2054] The system embodying the present invention monitors the user's physical condition in real time and provides optimal advice and product suggestions. This system is composed of the following main components:

[2055] Overall system configuration

[2056] The system mainly consists of the following components:

[2057] Wearable devices: Continuously measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.) and send it to a server in real time or in batches.

[2058] AI Smart Camera: Captures the user's face in real time and analyzes their facial expressions, complexion, and movements, and sends the analysis results to the server.

[2059] Server: Receives data sent from the wearable device and AI smart camera and stores it in a database. Analyzes the user's physical condition and health based on the stored data. A machine learning model is used for the analysis.

[2060] Speaker and display: Generated advice and product suggestions are delivered to the user via voice and text.

[2061] Data collection and analysis

[2062] The server receives biometric data, facial color, and facial expression data sent from the wearable device and AI smart camera. This data is used to analyze the user's physical condition and condition using a machine learning model. Based on the analysis results, the server determines the user's condition and refers to the fortune data for the day.

[2063] Generating advice and product recommendations

[2064] The server uses a generative AI model based on the analysis results and auspicious / lucky data to generate specific advice and product suggestions. For example, if you are feeling unwell and it's an unlucky day, the server generates advice such as, "Try not to push yourself too hard today and relax." If you are feeling well and it's an auspicious day, the server generates a product suggestion such as, "Today is a great day to be active. Try a new product."

[2065] Providing advice and product suggestions

[2066] The generated advice and product suggestions are provided to the user through a speaker and a display. The speaker uses a speech synthesis engine to convert text data into natural-sounding speech, and the display displays the information in text format.

[2067] The specific hardware and software used

[2068] Hardware:

[2069] Wearable devices: Apple Watch, Fitbit, etc.

[2070] AI smart camera: OpenCV, Azure Face API, etc.

[2071] Speakers and Displays

[2072] software:

[2073] Data analysis: TensorFlow, AWS SageMaker, etc.

[2074] Speech synthesis engine: Google Text-to-Speech API, IBM Watson TTS, etc.

[2075] Generative AI models: GPT-3, etc.

[2076] Specific examples

[2077] simulation:

[2078] A user logs into the virtual shop.

[2079] The wearable device measures the user's heart rate, body temperature, steps taken, and sleep patterns and sends the data to a server.

[2080] The AI ​​smart camera analyzes the user's complexion and facial expressions and sends the results to the server.

[2081] The server analyzes this data and determines the user's physical condition in real time.

[2082] If you are in poor health and it is an unlucky day, the advice is, "Don't push yourself today, take your time choosing your products."

[2083] If you are in good health and it's an auspicious day, the advice is, "Be active, have fun, and try a new product today!"

[2084] Example prompt sentence:

[2085] Prompt for a generative AI model (e.g. GPT-3):

[2086] "Generate advice for when a user is in poor health and having an unlucky day. Specific advice: 'Try to relax and not push yourself too hard today.'"

[2087] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2088] Step 1:

[2089] The user wears a wearable device, which measures the user's biometric data, such as heart rate, body temperature, number of steps, and sleep patterns, in real time and transmits it to a server. The input is the measured biometric data, and the output is the data sent to the server.

[2090] Step 2:

[2091] The server receives the biometric data sent from the wearable device and stores the data in a database. The input is the biometric data from the wearable device, and the output is the data stored in the database. A storage system is used to store the data.

[2092] Step 3:

[2093] The user stands in front of the AI ​​smart camera. The camera captures the user's face and analyzes their facial expressions, complexion, and movements. The input is the captured image data, and the output is the analysis results. Image analysis software (e.g., OpenCV, Azure Face API) is used for the analysis.

[2094] Step 4:

[2095] The AI ​​smart camera sends the analysis results to the server. The input is the image analysis result, and the output is the data sent to the server. The server also stores this data in a database.

[2096] Step 5:

[2097] The server analyzes the user's physical condition and health using the received biometric data and image analysis results. The input is the biometric data and image analysis data stored in the database, and the output is the analysis results. Machine learning models (e.g., TensorFlow, AWS SageMaker) are used for the analysis.

[2098] Step 6:

[2099] Based on the analysis results, the server retrieves the current day's fortune data from the database. The input is the analysis results, and the output is the fortune data. The latest fortune data is retrieved by a database query.

[2100] Step 7:

[2101] The server uses a generative AI model (e.g., GPT-3) based on the analysis results and the fortune data to generate specific advice and product suggestions. The input is the analysis results and fortune data, and the output is the generated advice and product suggestions. A prompt sentence is input into the generative AI model, which generates the optimal advice.

[2102] Step 8:

[2103] The generated advice and product suggestions are sent from the server to the speaker and display. The input is the generated advice and product suggestions, and the output is text and audio that is displayed on the display and played back from the speaker.

[2104] Step 9:

[2105] The user receives advice and product suggestions through a speaker and a display. The input is the text displayed on the display and the audio played from the speaker, and the output is the user's behavior and purchasing decisions.

[2106] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2107] A specific embodiment for implementing the present invention will now be described. This system monitors the user's physical condition in real time and provides optimal advice based on that. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more personalized advice can be achieved.

[2108] Overall system configuration

[2109] The system mainly consists of the following components:

[2110] Terminal (wearable device)

[2111] AI Smart Camera

[2112] Emotion Engine

[2113] server

[2114] speaker

[2115] 1. User data collection

[2116] Terminal

[2117] It has the ability to constantly measure the user's biometric data (heart rate, body temperature, number of steps, sleep patterns, etc.).

[2118] The collected data is sent to the server in real time or in batches.

[2119] AI Smart Camera

[2120] It captures the user's face in real time and analyzes their facial expression, facial expression, and movements.

[2121] As an example, stress levels and fatigue levels can be determined from facial color.

[2122] The analysis results are sent as data to the server.

[2123] 2. Emotion recognition

[2124] Emotion Engine

[2125] It analyzes the user's facial images and voice data obtained from the AI ​​smart camera and recognizes the user's emotions (joy, sadness, anger, etc.).

[2126] The recognized emotion data is sent to the server.

[2127] 3. Data Receipt and Analysis

[2128] server

[2129] It receives biometric data, condition data, and emotion data from the device, AI smart camera, and emotion engine, and stores them in a database.

[2130] Check the consistency of the data, and if there is any inconsistency, request the data again.

[2131] 4. Analysis of user's physical condition and emotions

[2132] server

[2133] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to analyze the user's physical condition and emotions. Here, factors such as a high heart rate, poor sleep quality, and the frequency of happy facial expressions are combined to form a comprehensive evaluation.

[2134] 5. Reference to fortune data

[2135] server

[2136] Based on the user's profile information, the system retrieves the day's fortune data from the database.

[2137] If the data on lucky and unlucky days has been updated, the latest information is acquired accordingly and the database is updated.

[2138] 6. Generating Advice

[2139] server

[2140] The analysis results, emotional data, and good and bad luck data are combined and the optimal advice for the day is generated using a generative AI model.

[2141] The advice includes encouraging rest when you're feeling unwell and activity when you're feeling well, and it adjusts the advice based on the perceived emotions.

[2142] 7. Providing advice

[2143] server

[2144] The generated advice is formatted in JSON or XML and sent to the speaker and device, allowing the advice to be provided appropriately in both voice and text.

[2145] speaker

[2146] A speech synthesis engine is used to convert text data into natural speech and communicate it to the user.

[2147] As an example, advice can be given as "Try to relax and take it easy today" or "Today is a great day to be active. Try a new hobby."

[2148] Terminal

[2149] Display advice in text format on the screen.

[2150] In one embodiment, the display may say, "Today is a great day to be active. Try a new hobby."

[2151] Specific examples

[2152] When the user is unwell and emotionally sad

[2153] Terminal

[2154] It measures data such as higher than normal heart rate and poor sleep quality and sends it to a server.

[2155] AI Smart Camera

[2156] The system analyzes that the user has a pale complexion and a tired expression, and sends the results to the server.

[2157] Emotion Engine

[2158] The sadness emotion is recognized from the user's face and transmitted to the server.

[2159] server

[2160] Based on the received data, it is determined that the user is in poor health.

[2161] Check the lucky / unlucky day data and confirm that the day is an unlucky day.

[2162] Taking into consideration that the person is feeling unwell, that it is an unlucky day, and that the emotion is sadness, the advice generated is, "Try not to push yourself today and try to relax. Read your favorite book to change your mood."

[2163] The generated advice is sent to the speaker and the device.

[2164] speaker

[2165] The voice tells the user, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[2166] Terminal

[2167] The display will say, "Try not to push yourself too hard today and try to relax. Read your favorite book to change your mood."

[2168] If the user is in good health and the emotion is joy

[2169] Terminal

[2170] The data is measured to show that the heart rate, body temperature, and number of steps are normal, and sent to the server.

[2171] AI Smart Camera

[2172] The system analyzes that the user has a bright expression and is active, and sends the results to the server.

[2173] Emotion Engine

[2174] The joyful emotion is recognized from the user's face and transmitted to the server.

[2175] server

[2176] Based on the received data, it is determined that the user's physical condition is good.

[2177] Check the auspicious / unlucky day data to confirm that the day is auspicious.

[2178] Taking into consideration that the emotion is joy in addition to the person being in good health and that it is an auspicious day, the advice generated is "Today is a great day to be active. Have a good time with your friends."

[2179] The generated advice is sent to the speaker and the device.

[2180] speaker

[2181] A voice tells the user, "Today is a great day to be active and have fun with friends."

[2182] Terminal

[2183] The display will say, "Today is a great day to be active and have fun with friends."

[2184] The above is a specific embodiment for carrying out the present invention.

[2185] The processing flow will be explained below.

[2186] Step 1:

[2187] The device uses sensors to measure the user's heart rate, body temperature, number of steps taken, and sleep patterns. This biometric data is collected through the wearable device, and the device transmits this data to a server in real time.

[2188] Step 2:

[2189] The AI ​​smart camera captures the user's face in real time and analyzes their facial expressions, complexion, and movements. The analyzed data is sent to a server as information about the user's condition.

[2190] Step 3:

[2191] The emotion engine analyzes the user's facial images and voice data acquired from the AI ​​smart camera and recognizes the user's emotions (happiness, sadness, anger, etc.). The recognized emotion data is sent to the server.

[2192] Step 4:

[2193] The server receives biometric data, condition data, and emotion data sent from the device, AI smart camera, and emotion engine, and stores it in a database. It checks the consistency of the received data and requests the data again if there is any inconsistency.

[2194] Step 5:

[2195] The server retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, the overall analysis results may show a high heart rate, poor sleep quality, and sadness.

[2196] Step 6:

[2197] The server refers to the user's profile information and retrieves the lucky / unlucky data for that day from the database. If the lucky / unlucky data has been updated, the new information is updated in the database.

[2198] Step 7:

[2199] The server combines the results of the physical condition analysis, emotional data, and auspicious / unlucky data, and uses a generative AI model to generate the optimal advice for that day. The advice is based on the individual's physical condition and emotions. For example, if you are feeling unwell and sad, the advice might be, "Try to relax and not push yourself today. I recommend reading your favorite book to change your mood." If you are feeling good and happy, the advice might be, "Today is a great day. Have some fun outdoors with friends."

[2200] Step 8:

[2201] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The transmitted data is output as voice and text.

[2202] Step 9:

[2203] The speaker receives advice from the server and converts it into natural-sounding speech using a speech synthesis engine, which then relays it to the user. For example, the advice could be, "Try not to push yourself too hard today, but relax. Read your favorite book to change your mood."

[2204] Step 10:

[2205] The device will display the received advice in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[2206] Step 11:

[2207] The user acts on the advice provided. The effectiveness of the advice and the user's opinion are input into the device as feedback. For example, the user can input whether the advice was helpful or whether their physical condition improved.

[2208] Step 12:

[2209] The device sends user feedback to the server, which is used to make future advice more effective.

[2210] Step 13:

[2211] The server stores the feedback in a database and uses it to generate advice in future. This feedback information continuously improves the accuracy of the entire system.

[2212] Example 2

[2213] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2214] Conventional healthcare systems tend to collect only a user's biometric data and provide advice based solely on their physical condition. However, providing personalized advice that also takes into account the user's emotions and fortunes is important for comprehensive health management. Furthermore, the lack of a means to improve the accuracy of advice based on feedback makes continuous improvement difficult. This has led to the issue of not being able to fully meet user needs.

[2215] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric information of the user in real time, means for analyzing the user's condition such as complexion, facial expression, and movement, means for receiving the measured biometric data and analysis results, means for generating advice using a generative AI model, means for providing the generated advice in audio and text, and means for collecting feedback from the user and improving the accuracy of advice from the next time onwards. This makes it possible to provide personalized advice that comprehensively takes into account the user's physical condition, emotions, and fortune, and to continuously improve the accuracy of that advice.

[2216] "User's biological information" is data that indicates the user's physical condition, such as heart rate, body temperature, number of steps, and sleep pattern.

[2217] "Real-time acquisition means" refers to the ability of sensors or devices to collect data immediately, without time delay, and transmit that data to the system.

[2218] "User's condition such as complexion, facial expression, and movement" is subjective information that indicates the user's emotions, stress level, activity level, and so on.

[2219] "Means of analysis" refers to the algorithms and software functions used to analyze data and detect specific patterns or anomalies.

[2220] "Measured biometric data and analysis results" refers to biometric information obtained from sensors or devices and the results of analyzing that data.

[2221] "Means for receiving" refers to the interface or protocol for importing and storing data from the outside.

[2222] A "generative AI model" is an artificial intelligence algorithm or system that automatically generates an output based on specific input data.

[2223] The "means for generating advice" is a function that automatically creates recommended actions and points of caution for the user based on the collected data and analysis results.

[2224] The "means of providing by voice and text" is a function of reading out the generated advice as voice and displaying it as text.

[2225] "Means for collecting feedback" refers to interfaces and protocols for capturing opinions and reactions from users and reflecting them in the system.

[2226] A specific embodiment of this invention will be described. This system monitors the user's physical condition and emotions in real time and provides optimal advice based on the results. The entire system mainly consists of the following components: a terminal, an AI smart camera, an emotion engine, a server, and a speaker.

[2227] User data collection

[2228] Terminal (wearable device):

[2229] The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The wearable device is equipped with sensors such as a heart rate monitor, a body temperature sensor, and an accelerometer. For example, the heart rate monitor measures values ​​such as "90 BPM" and the body temperature sensor measures "36.5°C."

[2230] These biometric data are transmitted to a server periodically or in real time via Bluetooth or Wi-Fi.

[2231] AI Smart Camera:

[2232] The camera captures the user's facial color, facial expressions, and movements, and analyzes this data in real time. The camera has an image analysis algorithm built in to analyze facial color and facial expressions. For example, if the user's face is pale, it will be determined that the stress level is high.

[2233] The analyzed data is immediately sent to the server.

[2234] Emotion recognition and data reception

[2235] Emotion Engine:

[2236] Using image and audio data obtained from an AI smart camera, the system recognizes the user's emotions (happiness, sadness, anger, etc.). For example, it uses facial expression analysis technology to determine that the user is expressing happiness.

[2237] The recognized emotion data is transmitted to a server.

[2238] server:

[2239] It receives all data acquired from devices, AI smart cameras, and emotion engines, stores it in a database, checks the data consistency, and requests the data again if there is any inconsistency.

[2240] Analyzing physical condition and emotions and generating advice

[2241] server:

[2242] The latest biometric, condition, and emotional data is retrieved from the database, and a machine learning model is used to comprehensively analyze the user's physical condition and emotions. For example, if the user has a high heart rate, poor sleep quality, and sadness, the system will determine that the user is in poor health.

[2243] Based on the user's profile information, the system retrieves the day's fortune data from the database. If the fortune data is not up to date, the system retrieves the latest data from an external data source and updates the database.

[2244] The analysis results are combined with fortune data and a generative AI model (e.g., GPT-3) is used to generate prompts. Examples of prompts include:

[2245] Example prompt sentence:

[2246] User health data:

[2247] Heart rate: 90 BPM

[2248] Body temperature: 36.5℃

[2249] Steps: 10,000

[2250] Sleep Pattern: Good

[2251] User sentiment data:

[2252] Emotion: Joy

[2253] Expression: Cheerful

[2254] Activity: Active

[2255] Fortune Data:

[2256] Today's Fortune: Auspicious Day

[2257] Use this information to generate the best advice for your users.

[2258] A generative AI model receives the prompts and generates the best advice for the day based on them.

[2259] Providing advice

[2260] server:

[2261] The generated advice is formatted into JSON or XML format and sent to the speaker and device.

[2262] speaker:

[2263] A speech synthesis engine is used to convert the generated advice into natural-sounding speech and convey it to the user.

[2264] Device:

[2265] Advice is displayed in text format on the display, such as "Try to relax and not push yourself too hard today. Read your favorite book to change your mood."

[2266] This system will provide personalized advice that takes into account the user's physical condition, emotions, and fortune, and will be able to continuously improve its accuracy.

[2267] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2268] Program processing flow

[2269] Step 1:

[2270] Data Acquisition

[2271] Device: The user wears a wearable device to collect real-time biometric information such as heart rate, body temperature, number of steps, and sleep patterns. The device periodically reads data from the heart rate monitor and body temperature sensor, converts the data into JSON format, and stores it in a transmission queue.

[2272] Input: Biometric data from sensors.

[2273] Output: Biometric data in JSON format.

[2274] Step 2:

[2275] Sending data

[2276] Terminal: The terminal sends the collected biometric data to the server via Bluetooth or Wi-Fi. Specifically, the terminal sends the collected JSON data to the server's receiving API as an HTTP POST request.

[2277] Input: Biometric data obtained in step 1 in JSON format.

[2278] Output: HTTP request to the server.

[2279] Step 3:

[2280] Camera-based facial color and facial expression analysis

[2281] AI Smart Camera: Captures the user's facial expressions, facial expressions, and movements, and analyzes them in real time using image analysis algorithms. For example, facial expression analysis software analyzes the user's facial photo to determine emotions such as "sadness" or "happiness."

[2282] Input: A face image captured by the camera.

[2283] Output: Analyzed facial color and expression data.

[2284] Step 4:

[2285] Sending camera data

[2286] AI Smart Camera: Converts the analysis results into JSON format and sends them to the server. Sends the analysis data using an HTTP POST request.

[2287] Input: Facial color and expression data analyzed in step 3.

[2288] Output: JSON formatted data to the server.

[2289] Step 5:

[2290] Processing received data

[2291] Server: Receives biometric data and facial image analysis data sent from the device and AI smart camera. The receiving API receives this data and stores it in a database.

[2292] Input: HTTP requests from the device and the AI ​​smart camera.

[2293] Output: Biometric data and facial image analysis data stored in a database.

[2294] Step 6:

[2295] emotion recognition

[2296] Emotion Engine: Analyzes facial image data retrieved from a database to recognize the user's emotions. It uses an AI model to identify emotions such as "happiness" or "sadness."

[2297] Input: Facial image data stored in a database.

[2298] Output: Parsed emotion data.

[2299] Step 7:

[2300] Comprehensive analysis of data

[2301] Server: Retrieves the latest biometric, condition, and emotional data from the database and uses a machine learning model to comprehensively analyze the user's physical condition and emotions. For example, if the heart rate is high, sleep quality is poor, and the user is expressing sadness, it determines that the user is in poor health.

[2302] Input: Biometric data, facial expression data, and emotion data in the database.

[2303] Output: Comprehensively analyzed user physical and emotional data.

[2304] Step 8:

[2305] Fortune data reference

[2306] Server: Based on the user's profile information, retrieves the day's fortune data from the database. If the fortune data is not up to date, retrieves the latest data from an external interface and updates the database.

[2307] Input: User profile information.

[2308] Output: Fortune data for the day.

[2309] Step 9:

[2310] Prompt creation for advice generation

[2311] Server: Combines the user's biometric, emotional, and fortune data to generate prompts to input into the generative AI model. Examples of prompts include:

[2312] Example prompt sentence:

[2313] User health data:

[2314] Heart rate: 90 BPM

[2315] Body temperature: 36.5℃

[2316] Steps: 10,000

[2317] Sleep Pattern: Good

[2318] User sentiment data:

[2319] Emotion: Joy

[2320] Expression: Cheerful

[2321] Activity: Active

[2322] Fortune Data:

[2323] Today's Fortune: Auspicious Day

[2324] Use this information to generate the best advice for your users.

[2325] Input: Biometric data, emotional data, fortune data.

[2326] Output: Prompts to the generative AI model.

[2327] Step 10:

[2328] Generating Advice

[2329] Generative AI model: Generates optimal advice for the user based on the prompt. For example, it generates advice such as, "Try not to push yourself too hard today and try to relax."

[2330] Input: Prompt statement.

[2331] Output: The generated advice.

[2332] Step 11:

[2333] Sending Advice

[2334] Server: Formats the generated advice into JSON or XML format and sends it to the speaker and device, providing advice in voice and text.

[2335] Input: The generated advice.

[2336] Output: JSON or XML data to speakers and devices.

[2337] Step 12:

[2338] Voice and text advice provided

[2339] Speaker: Uses a speech synthesis engine to convert the JSON or XML data into speech and tells the user, "Try to relax and take it easy today."

[2340] Terminal: Display the text "Try not to push yourself too hard today and try to relax" on the display.

[2341] Input: Advice sent by the server.

[2342] Output: Providing audio and text advice to the user.

[2343] These are the specific processing steps of this system, which allows users to receive optimal advice based on their physical condition, emotions, and fortune.

[2344] (Application example 2)

[2345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2346] Conventional self-driving vehicles lack real-time feedback based on the physical condition and emotions of the driver and passengers, making it difficult to provide optimal driving modes and entertainment. They also lack the ability to provide personalized advice based on the day's auspicious and unlucky data or the individual user's condition. These circumstances have led to insufficient safety and comfort within the vehicle.

[2347] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for measuring the user's biological data in real time, means for analyzing the user's condition such as facial color and facial expression, and means for receiving the measured biological data and the analysis results. This makes it possible to monitor the user's physical condition and emotions in real time in an autonomous vehicle and optimize the driving mode and entertainment based on the monitoring results.

[2348] "User's biometric data" refers to data that indicates a person's physical condition, such as heart rate, body temperature, number of steps taken, and sleep patterns.

[2349] "User condition such as facial color and facial expression" is information indicating the physical and emotional state of the user, including the color and facial expression of the user's face, and movements.

[2350] "Analysis means" refers to a device or software that evaluates and judges the user's physical condition and emotions based on measured data and images.

[2351] "Today's fortune data" is traditional or astrological information that indicates whether the day is auspicious or inauspicious for the user.

[2352] The "means for generating advice" is a device or software that suggests appropriate actions or measures to the user based on the collected data and analysis results.

[2353] "Means for providing generated advice in audio and text" refers to a device or software that displays and plays generated advice in audio or text format in order to convey the advice to the user in an easy-to-understand manner.

[2354] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[2355] "Driving modes" are settings that adjust the style and conditions under which an autonomous vehicle is driven.

[2356] "Entertainment providing means" refers to devices or software for playing entertainment content such as music and video for users.

[2357] "Means for recognizing emotions" refers to a device or software that analyzes a user's facial expressions, actions, and voice data to determine their emotional state.

[2358] A "generative AI model" is an algorithm or program that is trained by artificial intelligence to generate appropriate advice or suggestions from specific data.

[2359] A "prompt" is a specific input sentence used to generate an appropriate answer or output for a generative AI model.

[2360] A specific embodiment of the present invention will be described. This invention is a system that monitors the physical condition and emotions of occupants in an autonomous vehicle in real time and provides optimal driving modes and entertainment based on the monitoring results. The entire system is composed of the following main components.

[2361] System configuration

[2362] The system mainly consists of the following components:

[2363] Wearable devices

[2364] AI Camera

[2365] Emotion Engine

[2366] server

[2367] speaker

[2368] 1. Data Collection

[2369] Wearable devices

[2370] Wearable devices (e.g., smartwatches) constantly measure biometric data such as heart rate, body temperature, steps taken, and sleep patterns.

[2371] The collected data is sent to a server in real time at irregular intervals.

[2372] AI Camera

[2373] An AI camera installed inside the vehicle captures and analyzes the facial expressions, facial expressions, and movements of the occupants in real time.

[2374] The analysis results are sent to a server as data to determine each passenger's stress level and fatigue level.

[2375] 2. Emotion recognition

[2376] Emotion Engine

[2377] The emotion engine analyzes facial images and audio data obtained from the AI ​​camera to recognize the emotions of the occupants (joy, sadness, anger, etc.).

[2378] The recognized emotion data is transmitted to a server.

[2379] 3. Data Receipt and Analysis

[2380] server

[2381] It receives biometric data, condition data, and emotion data from the wearable device, AI camera, and emotion engine, and stores them in a database.

[2382] Check the consistency of the data and re-request the data if there is any inconsistency.

[2383] 4. Data analysis and advice generation

[2384] server

[2385] The latest biometric, condition, and emotional data is collected and the occupants' physical condition and emotions are analyzed using machine learning models.

[2386] The data on the day's fortunes is referenced and retrieved from the database.

[2387] The analysis results, emotional data, and good and bad fortune data are combined and a generative AI model is used to generate the best advice for the day.

[2388] The advice includes specific suggestions for action, such as when drivers should take a break and relax, or when they should play music to help them relax.

[2389] 5. Providing advice

[2390] speaker

[2391] The advice is given to the passengers using a speech synthesis engine, which converts text data into natural-sounding speech.

[2392] Terminal

[2393] Advice is displayed in text format on the in-car display.

[2394] Specific examples

[2395] Example 1: When the user is unwell and sad

[2396] The wearable device measures high heart rate and poor sleep quality and sends the results to a server.

[2397] The AI ​​camera analyzes whether the person's complexion is pale and they look tired, and sends the results to the server.

[2398] The emotion engine recognizes the sadness emotion from the user's face and sends it to the server.

[2399] The server determines that the user is feeling unwell and that it is an unlucky day, generates advice such as "Try not to push yourself today and try to relax. Read your favorite book to change your mood," and sends this advice to the device and speaker.

[2400] Example 2: When the user is in good health and the emotion is joy

[2401] The wearable device measures the heart rate, body temperature, and number of steps taken, and sends the results to the server.

[2402] The AI ​​camera analyzes whether the person has a good complexion and a bright expression, and sends the results to the server.

[2403] The emotion engine recognizes the emotion of joy from the user's face and transmits it to the server.

[2404] The server determines that the person is in good health and that it is an auspicious day, generates advice such as "Today is a great day to be active. Have a good time with friends," and sends it to the device and speaker.

[2405] Prompt example

[2406] "If the user has a high heart rate but is showing signs of joy, provide appropriate advice."

[2407] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2408] Step 1:

[2409] The wearable device measures the user's biometric data in real time, specifically, heart rate, body temperature, number of steps taken, sleep patterns, etc. This acquired data is input, and the wearable device sends it to a server.

[2410] Step 2:

[2411] The server stores the received biometric data in a database and checks its consistency. Specifically, it checks whether the data is missing or consistent. If there is an inconsistency, it requests the data again. The input to this process is the biometric data sent from the wearable device, and the output is the biometric data whose consistency has been confirmed.

[2412] Step 3:

[2413] The AI ​​camera captures the facial color and facial expressions of the user in the car in real time. Specific operations include analyzing facial color and facial expressions. These analysis results are input, and the AI ​​camera sends the analysis results to the server.

[2414] Step 4:

[2415] The server receives the analysis results sent from the AI ​​camera and stores them in a database. The input is the analysis result data, and the output is the saved analysis result data. The server also checks the consistency of this data.

[2416] Step 5:

[2417] The emotion engine analyzes emotions based on facial image data obtained from the AI ​​camera. Specifically, it uses facial expression data to recognize emotional states (happiness, sadness, anger, etc.). This recognition result is input, and the emotion engine sends it to the server.

[2418] Step 6:

[2419] The server receives the emotion data sent from the emotion engine and stores it in a database. The input is emotion data, and the output is the stored emotion data. The server also checks the integrity of the emotion data.

[2420] Step 7:

[2421] The server integrates all data received from the wearable device, AI camera, and emotion engine to obtain the latest biometric, condition, and emotion data. These data are the input, and the output is the integrated dataset.

[2422] Step 8:

[2423] The server uses the integrated data set to analyze the user's physical condition and emotions using a machine learning model. Specifically, it evaluates heart rate and facial expression changes to determine the user's overall physical condition and emotions. The results of this analysis are the input, and the output is the analysis result data.

[2424] Step 9:

[2425] The server retrieves the fortune data for that day from the database. The input is the date information, and the output is the fortune data for that day. The server combines this with the analysis results.

[2426] Step 10:

[2427] The server uses a generative AI model to generate optimal advice based on the analysis results and the good and bad fortune data. Specifically, a prompt sentence is used as input to the model to generate appropriate advice. An example of this prompt sentence is, "If the user's heart rate is high but they are showing signs of joy, please provide appropriate advice." The input is the integrated data and the prompt sentence, and the output is the generated advice.

[2428] Step 11:

[2429] The server formats the generated advice into JSON or XML format and sends it to the speaker and device. The input is the generated advice, and the output is the formatted advice data.

[2430] Step 12:

[2431] The terminal displays the advice in text format on the in-car display. Specifically, it converts the transmitted format data into text and displays it on the display. The input is the formatted advice data, and the output is the display.

[2432] Step 13:

[2433] The speaker uses a speech synthesis engine to convert the advice into natural-sounding speech and convey it to the passengers. Specifically, it converts text data into speech data and plays it back. The input is formatted advice data, and the output is advice conveyed in speech.

[2434] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2436] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2437] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2438] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2439] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2440] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2441] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2442] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2443] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2444] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2445] In the above embodiment, an example was given in which the specific proc...

Claims

1. a means for measuring biometric data of a user in real time; A means of analyzing the user's condition, such as facial color and facial expression, means for receiving the measured biometric data and analysis results; A means for analyzing the user's physical condition and condition using the received data; A way to check the fortune data for the day, A means for generating advice based on the analysis results and the fortune data; a means for providing the generated advice in audio and text; A system including:

2. The system of claim 1 further comprising means for collecting feedback from the user to improve the accuracy of subsequent advice.

3. The system according to claim 1 , further comprising means for converting the generated advice into speech using a speech synthesis engine and providing the advice through a speaker.

Images