System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an exercise record and a meal photograph from a user; means for transmitting the received data to a server; means for analyzing the exercise record and the meal photograph and generating individual advice based on an analysis result in the server; means for transmitting the generated advice to a terminal; means for notifying the user of the transmitted advice in the terminal; and means for adjusting next advice based on the received feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional healthcare applications provide advice through standard notifications, lacking specific, personalized advice. As a result, even if users initially use the app, they often end up leaving the notifications unread. Furthermore, because diet and exercise records are managed separately, it is difficult to grasp a comprehensive health status, leading to problems with effective health management. [Means for solving the problem]

[0005] The present invention includes a means for receiving exercise records and meal photos from a user, a means for transmitting the received data to a server, and a means for analyzing the exercise records and meal photos in the server and generating individualized advice based on the analysis results. It also includes a means for transmitting the generated advice to a terminal, a means for notifying the user of the transmitted advice in the terminal, and a means for receiving feedback from the user and transmitting it to the server. Furthermore, by providing a means for adjusting the next advice based on the received feedback in the server, it is possible to provide advice tailored to changes in the user's behavior. Furthermore, by providing a means for tracking whether the advice has been read and implemented, and a means for performing image analysis of meal photos and converting them into menu, nutrient, and calorie information, it is possible to provide comprehensive and specific health management support to the user.

[0006] A "user" is an individual who uses the healthcare application and provides data such as exercise records and meal photos.

[0007] "Exercise records" refers to data that includes information such as the type, distance, time, and intensity of the exercise performed by the user.

[0008] A "meal photo" is an image file of the meal contents taken by the user when he or she eats a meal.

[0009] A "terminal" refers to an electronic device, such as a smartphone or tablet, on which a user installs and uses a healthcare application.

[0010] A "server" is a computer system that stores data received from a terminal, analyzes the data, and transmits generated advice to the terminal.

[0011] "Analysis" refers to a series of processes that process data such as exercise records and meal photos, extract information, and give it meaning.

[0012] "Generated advice" refers to specific suggestions and guidance for improving health that are individually customized based on the analysis results and provided to the user.

[0013] "Feedback" refers to information provided by a user to report the results or status of actions taken based on advice.

[0014] The "read status" is information that tracks whether the user has opened the advice notification and checked its contents.

[0015] "Implementation" refers to information that tracks whether a user actually acts on the advice provided.

[0016] "Image analysis" refers to the technology used to process photographs and image data and extract features and information from them.

[0017] A "menu" refers to the specific items and names of dishes included in a meal.

[0018] "Nutrients" refer to the components of food such as proteins, lipids, carbohydrates, vitamins, and minerals.

[0019] "Calorie information" refers to information that indicates the amount of energy contained in a meal. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[0042] System Overview

[0043] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[0044] Specific processing of the program

[0045] 1. Data Collection Phase

[0046] User

[0047] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0048] Terminal

[0049] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[0050] 2. Data analysis phase

[0051] server

[0052] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0053] 3. Individual advice creation phase

[0054] server

[0055] The generative AI model on the server creates personalized healthcare advice based on the analyzed data. For example, it might generate specific advice such as, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[0056] 4. Advice sending phase

[0057] server

[0058] The server sends the generated advice to the device, along with metadata for managing the advice's read status and implementation status.

[0059] Terminal

[0060] The device notifies the user of the advice sent from the server. It displays the advice "walk for 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[0061] 5. Feedback Phase

[0062] User

[0063] The user acts based on the advice provided and inputs the results into the application, for example, reporting, "I walked for 30 minutes today."

[0064] Terminal

[0065] The terminal receives feedback from the user and transmits it to the server.

[0066] server

[0067] The server uses the received feedback to generate the next piece of advice, allowing the advice to be continuously optimized.

[0068] Example

[0069] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "eat less carbohydrates and more vegetables tomorrow. It is recommended that you jog for at least 30 minutes." The generated advice is sent to the user's device and notified to the user.

[0070] The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[0071] As described above, the present invention is a system that provides individualized advice based on a user's exercise records, health status, and dietary information, and effectively supports the user's health management.

[0072] The processing flow will be explained below.

[0073] Step 1:

[0074] User

[0075] Users use the application to enter photos of their daily exercise records and meals. Exercise records include walking distance, time, calorie consumption, etc. Meal photos are images that show the details of the meals they ate that day.

[0076] Step 2:

[0077] Terminal

[0078] The device receives user-entered exercise records and meal photos, organizes them according to format, and temporarily stores them in local storage. The device also syncs data from devices such as fitness bands and smartwatches.

[0079] Step 3:

[0080] Terminal

[0081] The device generates packets to send the organized data to the server, securely transmitting it using the SSL / TLS protocol, and displays a notification to the user if the data transmission is successful.

[0082] Step 4:

[0083] server

[0084] The server receives the exercise records and meal photos sent from the device, and the received data is stored in a database for use in the next analysis step.

[0085] Step 5:

[0086] server

[0087] The server analyzes exercise records and meal photos. For meal photos, image recognition technology is used to extract information on ingredients, menu items, nutrients, and calories. For exercise records, the server calculates the user's calorie expenditure and exercise volume from the data.

[0088] Step 6:

[0089] server

[0090] The server uses the generative AI model based on the analysis results to generate personalized healthcare advice for the user. For example, if the user's exercise volume is low, the server generates advice recommending additional exercise. If the user's diet is unbalanced, the server recommends a balanced diet.

[0091] Step 7:

[0092] server

[0093] The server converts the generated advice into packets and sends them to the device, along with metadata for tracking whether the advice has been read or implemented.

[0094] Step 8:

[0095] Terminal

[0096] The device receives the advice packet sent from the server and notifies the user of its contents. The user can check the advice by opening the notification.

[0097] Step 9:

[0098] User

[0099] The user acts according to the advice. For example, if a 30-minute jog is recommended, the user performs the exercise. Furthermore, the user also improves their diet based on the advice.

[0100] Step 10:

[0101] User

[0102] The user provides feedback to the application about the results of their actions, such as "I went jogging for 30 minutes" or "I ate a lot of vegetables at dinner."

[0103] Step 11:

[0104] Terminal

[0105] The terminal receives the feedback from the user and transmits it to the server, and notifies the user if the transmission is successful.

[0106] Step 12:

[0107] server

[0108] The server receives the feedback sent from the device, stores it in a database, and takes the feedback into account when generating the next advice, allowing for continuous optimization of personalized advice.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] As modern lifestyles become more diverse, it is becoming increasingly difficult to provide individually optimized healthcare advice. Existing systems often require cumbersome data collection and analysis from users, resulting in uniform and suboptimal advice. Furthermore, insufficient feedback collection, necessary to verify the effectiveness of advice, reduces the accuracy and effectiveness of the advice. To address this issue, it is necessary to effectively collect and analyze data from users, provide individually tailored advice, and further optimize the advice by incorporating user feedback.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes means for receiving exercise records and meal photos from a user, means for transmitting the received data to a network device, means for analyzing the exercise records and meal photos in the network device and generating personalized advice based on the analysis results, means for transmitting the generated advice to a terminal, means for notifying the user of the sent advice in the terminal, means for receiving feedback from the user and transmitting it to the network device, means for adjusting next advice based on the received feedback in the network device, and means for generating personalized advice using a generative AI model, thereby enabling the provision of healthcare advice optimized for each user.

[0114] "User" refers to an individual who uses the system to input exercise records, health status, and dietary information and receive healthcare advice.

[0115] "Exercise record" refers to information including the type of exercise performed by the user, the time, distance, calories burned, etc.

[0116] "Meal photos" refer to digital images that are photographs of the meals a user has eaten.

[0117] "Network device" refers to a computer system that has the capability to analyze data received from a user terminal and generate and send personalized advice.

[0118] "Generative AI Model" refers to the artificial intelligence algorithm used to generate personalized healthcare advice based on received and analyzed data.

[0119] "Feedback" refers to information including the results of a user's actions and evaluation of the advice provided.

[0120] "Advice" refers to health management instructions and recommendations created by a generative AI model based on a user's exercise records and dietary information.

[0121] "Analysis" refers to the act of performing calculations and processing on the received exercise record and meal photo data to extract useful information and patterns.

[0122] MODE FOR CARRYING OUT THE INVENTION

[0123] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[0124] System Overview

[0125] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[0126] Specific hardware and software

[0127] User device:

[0128] The user uses a smartphone, tablet, or other device. An application is installed on the device to input and save exercise records and meal photos. This application also works in conjunction with a fitness band or smartwatch to collect data.

[0129] server:

[0130] A server responsible for analyzing the data and generating healthcare advice. The server has a database (e.g., MySQL) and an image analysis tool (e.g., Google Cloud Vision API) installed. It also requires computing resources to run the generative AI model (e.g., ChatGPT).

[0131] Specific processing of the program

[0132] Data collection phase:

[0133] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0134] Data transmission phase:

[0135] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[0136] Data analysis phase:

[0137] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology (e.g., Google Cloud Vision API) to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as, "Dinner is about 600 kcal, and is particularly high in carbohydrates."

[0138] Personalized advice creation phase:

[0139] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, reduce the carbohydrates in your dinner a little and eat more vegetables."

[0140] Advice sending phase:

[0141] The server sends the generated advice to the device. It also sends metadata to manage whether the advice has been read or implemented. The device notifies the user of the advice sent from the server. The device displays the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[0142] Feedback Phase:

[0143] The user acts based on the advice provided and inputs the results into the application. For example, the user reports, "I walked for 30 minutes today." The device receives the feedback from the user and sends it to the server. The server uses the received feedback to generate the next piece of advice. This allows the advice to be continuously optimized.

[0144] Examples of specific examples and prompts

[0145] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "Tomorrow, eat less carbohydrates and more vegetables. It is recommended that you jog for at least 30 minutes."

[0146] The generated advice is sent to the user's device and notified. The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[0147] Example prompt sentence:

[0148] User:

[0149] "I ran 5km today. I had salad, chicken, and bread for dinner."

[0150] server:

[0151] "The calorie analysis of your meal shows that it is approximately 600 kcal. Taking into consideration the amount of exercise required, we recommend that you walk for 30 minutes tomorrow. It would also be a good idea to reduce your carbohydrate intake at dinner and eat more vegetables."

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1: Data collection

[0154] Users enter their daily exercise records and meal photos through the application. Specifically, users send text and images such as "I ran 5km today" or "I had salad, chicken, and bread for dinner" to the app. Exercise record data and meal photo data are provided as input, and these data are temporarily saved on the device as output.

[0155] Step 2: Send data

[0156] The device sends the exercise record and meal photo data collected from the user to the server via HTTPS protocol. The temporarily stored data is used as input, and the output is securely transmitted to the server via the network.

[0157] Step 3: Receiving and storing data

[0158] The server receives the exercise record and meal photo data sent from the device and stores them in a database (e.g., MySQL). It receives the data sent from the device as input and stores this data in the database as output.

[0159] Step 4: Data analysis

[0160] The server analyzes the received data. Specifically, it uses Google Cloud Vision API to perform image analysis on food photos and converts them into menu, nutrient, and calorie information. It also uses natural language processing tools to perform text analysis on exercise records. It uses data obtained from the database as input and obtains the analysis results as output.

[0161] Step 5: Create personalized advice

[0162] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, it would be good to eat a little less carbohydrates and more vegetables at dinner." The analysis results above are used as input, and advice is generated as output.

[0163] Step 6: Send advice

[0164] The server sends the generated advice to the device. The advice also includes metadata for managing read status and implementation status. The generated advice is used as input and sent to the device as output.

[0165] Step 7: Advice Notification

[0166] The device receives advice sent from the server and displays advice such as "walk 30 minutes tomorrow" or "reduce carbohydrates" to the user via push notification. The device receives advice data from the server as input and notifies the user as output.

[0167] Step 8: Gather feedback

[0168] The user acts based on the advice provided and reports the results to the application. For example, they may provide feedback such as, "I walked for 30 minutes today." The user provides the action data they took as input, and this data is temporarily saved on the device as output.

[0169] Step 9: Submit your feedback

[0170] The terminal receives feedback data from the user and transmits it to the server, using the feedback data from the user as input and transmission to the server as output.

[0171] Step 10: Feedback analysis and next-time advice adjustment

[0172] The server reflects the received feedback in the next advice generation, which allows advice to be continuously optimized. Feedback data is received as input, and data that will be reflected in the next advice is generated as output.

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] Conventional health management systems only required users to input data and receive advice, which lacked real-timeness and convenience. Furthermore, there was no mechanism for dynamically providing individual advice when selecting food in a physical store. This made it difficult for users to receive health support in a physical store.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes a means for receiving exercise records and meal photos, a means for performing food recognition and nutritional analysis, and a means for generating personalized advice based on the analysis results, thereby enabling real-time health advice in a physical store.

[0178] A "user" is an individual who uses this system and provides exercise records and dietary information.

[0179] The "exercise record" is data relating to the exercise performed by the user on a daily basis, and includes information such as distance, time, and type of exercise.

[0180] "Meal photos" are image data that record meals consumed by the user and are used for subsequent analysis.

[0181] The "receiving means" is a function that allows the terminal to receive exercise records and meal photos from the user.

[0182] A "terminal" is a smartphone, smart glasses, or other electronic device used by a user, and is a device for inputting data and receiving advice.

[0183] A "server" is a computer system that aggregates received data and performs analysis and advice generation.

[0184] The "means of analysis" refers to algorithms and software that analyze exercise records and meal photos on a server and extract nutrient and calorie information.

[0185] "Personalized advice" refers to specific health management instructions or recommendations provided to the user based on the analyzed data.

[0186] The "notification means" is a function that allows the terminal to notify the user of the advice that has been generated.

[0187] "Feedback" is information about the actions taken by the user based on the advice and the results of those actions, and is used to optimize future advice.

[0188] The "adjustment means" is a function that allows the server to revise and optimize the advice content from the next time onwards based on feedback from the user.

[0189] "Food recognition" is a technology that uses smart devices in physical stores to take pictures of products and identify their contents.

[0190] "Nutritional analysis" is the process of calculating and analyzing the various nutrients and calories contained in recognized foods.

[0191] "Real-time health advice" is a system that provides on-the-spot health guidance and recommendations when users are choosing food in a physical store.

[0192] A specific embodiment for realizing this invention will be described. This system has the function of collecting and analyzing exercise records and dietary information to provide individualized advice to support the user's health management. The main components of the system are a terminal, a server, and a generative AI model.

[0193] System configuration

[0194] Hardware:

[0195] Devices: Smartphones and smart glasses used by users. These devices input exercise records, take photos of meals, and display and notify advice.

[0196] Server: A computer system located in the cloud that analyzes the received data and generates advice.

[0197] Software used

[0198] software:

[0199] Terminal application: An application that allows users to input exercise records and dietary information and receive advice, which is displayed on the screen in real time.

[0200] Server-side software: Software that analyzes data, runs generative AI models, and incorporates feedback.

[0201] Image analysis algorithm: An algorithm that analyzes food photos to extract menu items, nutrients, and calories.

[0202] System Operation

[0203] 1. Data Collection:

[0204] Users use their smartphones or smart glasses to record their exercise and take photos of their meals. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0205] 2. Data transmission:

[0206] The terminal transmits the entered exercise records and meal photos to the server.

[0207] 3. Data Analysis:

[0208] The server analyzes the received exercise records and meal photos. It uses an image analysis algorithm to analyze the meal photos and extract nutrient and calorie information. For example, the server uses image recognition technology to analyze a meal photo and obtain information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0209] 4. Advice Generation:

[0210] The generative AI model on the server generates personalized healthcare advice based on the analyzed data. For example, it might say, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[0211] 5. Advice Notice:

[0212] The generated advice is sent to the device and notified to the user. For example, the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" is displayed to the user.

[0213] 6. Feedback received:

[0214] The user acts based on the advice provided and inputs the results into the device, for example, reporting, "I walked for 30 minutes today."

[0215] 7. Data Transmission and Coordination:

[0216] The device sends feedback to the server, which then adjusts the next advice based on the received feedback, allowing for continuous optimization of the advice.

[0217] Use in physical stores

[0218] Examples:

[0219] When a user scans chicken in a physical store, the AI ​​model analyzes it and determines that "this chicken is high in protein but low in fat," and based on the user's exercise history and health status, displays advice such as "This chicken is high in protein, so it fits into your weekly exercise plan. Eat it with a salad."

[0220] Example prompt sentence:

[0221] "User data: Jogging 3 times a week, high protein, low carb diet, good health. Analysis results: Chicken, high protein, low fat. Based on this information, please provide weekly exercise and meal plan suggestions."

[0222] The above is an embodiment of the present invention. This system effectively supports users in managing their health and significantly improves convenience when selecting food items in physical stores.

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1:

[0225] Data collection and input

[0226] User: The user uses a smartphone or smart glasses to enter exercise records and take photos of meals. For example, "I ran 5km today" or "I had salad, chicken, and bread for dinner." This data is stored on the device and later sent to the server.

[0227] Step 2:

[0228] Data transmission

[0229] Device: The device sends the exercise records and meal photos entered by the user to the server. Here, HTTP requests are used as the protocol for sending input data. The input data includes exercise records and meal photos, and is sent to the server.

[0230] Step 3:

[0231] Data analysis

[0232] Server: The server analyzes the exercise records and meal photos it receives. It uses an image analysis algorithm on the server side to extract information about the menu, nutrients, and calories from the meal photos. The image analysis algorithm receives the photos as input and outputs the menu, nutrients, and calories. For example, it obtains information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0233] Step 4:

[0234] Advice Generation

[0235] Server: The generative AI model on the server generates personalized healthcare advice based on the analyzed data. The generative AI model inputs the analysis results and past user data as prompts and outputs specific advice. For example, the generated advice might be, "Since you haven't exercised much today, we recommend walking for 30 minutes tomorrow. Also, it would be good to reduce carbohydrates a little and eat more vegetables at dinner."

[0236] Step 5:

[0237] Advice Notice

[0238] Device: The advice generated by the server is sent to the device. The device notifies the user of this advice and displays it to them. Possible notification methods include push notifications and in-app notifications. For example, the user might be notified of advice such as "walk 30 minutes tomorrow" and "reduce carbohydrate intake."

[0239] Step 6:

[0240] Feedback Input

[0241] User: The user acts based on the advice provided and inputs the results into the device. The input is feedback information such as "I walked for 30 minutes today." This information is later sent to the server.

[0242] Step 7:

[0243] Feedback submission and processing

[0244] Terminal: Collects feedback information and sends it to the server. Again, a communication protocol such as an HTTP request is used. The input is the feedback data, which is then sent to the server.

[0245] Server: Adjusts and optimizes the next advice based on the received feedback. The server takes past feedback information as input and uses a generative AI model to continuously optimize the advice, making health management more effective for the user.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] This invention combines a system that collects a user's daily exercise record, health condition, and dietary information and uses a generative AI model to provide personalized healthcare advice, with an emotion engine. The system aims to provide more effective healthcare support by recognizing the user's emotions and adjusting advice based on those emotions.

[0248] System Overview

[0249] The system receives exercise records and meal photo data entered by the user on the device and sends it to the server. The server analyzes the received data and generates individualized advice based on the results. In doing so, it uses an emotion engine to analyze the user's emotional data and reflects it in the advice. The generated advice is sent to the device and notified to the user. In addition, user feedback is collected and used to generate the next advice, aiming for individualization and optimization.

[0250] Specific processing of the program

[0251] 1. Data Collection Phase

[0252] User

[0253] Users use the app to record their daily exercise and meal photos. The exercise records include walking distance, time, and calorie consumption, while the meal photos are images that show the specific meals they ate that day. In addition, data from fitness bands and smartwatches is also synced to the app.

[0254] Terminal

[0255] The device receives the exercise records and meal photos entered by the user, temporarily stores them in local storage, and prepares them to be sent to the server, where they are securely transmitted using the SSL / TLS protocol.

[0256] 2. Data analysis phase

[0257] server

[0258] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[0259] Emotion Engine

[0260] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. For example, if the user is feeling stressed, it detects their stress level.

[0261] 3. Individual advice creation phase

[0262] server

[0263] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[0264] 4. Advice sending phase

[0265] server

[0266] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0267] Terminal

[0268] The terminal receives the advice sent from the server, notifies the user, checks whether the advice has been read, and returns that information to the server.

[0269] 5. Feedback Phase

[0270] User

[0271] The user acts on the advice provided and provides feedback to the application, such as "I did 30 minutes of yoga" or "I ate more vegetables for dinner."

[0272] Terminal

[0273] The terminal receives feedback from the user and transmits it to the server.

[0274] server

[0275] The server receives feedback from the device and stores it in a database. Based on this feedback, the server adjusts the next advice it provides, allowing it to continually provide optimal advice tailored to the user's individual needs.

[0276] Example

[0277] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[0278] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[0279] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[0280] In this way, the present invention is a system that provides individual advice based on emotional data in addition to the user's exercise records, health status, and dietary information, thereby more effectively supporting the user's health management.

[0281] The processing flow will be explained below.

[0282] Step 1:

[0283] User

[0284] Users use the application to record their daily exercise and meal photos. The exercise records include distance traveled, time, and calorie consumption, and the meal photos are taken to show what they ate that day.

[0285] Step 2:

[0286] Terminal

[0287] The device receives the exercise records and meal photos entered by the user and temporarily stores them in local storage, preparing to send the stored data to the server.

[0288] Step 3:

[0289] Terminal

[0290] The device converts the organized data into packets and sends them securely to the server using the SSL / TLS protocol. If the data transmission was successful, the device notifies the user.

[0291] Step 4:

[0292] server

[0293] The server receives the exercise records and meal photos sent from the device, stores them in a database, and then processes the data for analysis.

[0294] Step 5:

[0295] server

[0296] The server analyzes the received data. It uses image recognition technology to analyze food photos and extract menu, nutrient, and calorie information. It also analyzes exercise records to calculate the user's calorie consumption and exercise performance.

[0297] Step 6:

[0298] Emotion Engine

[0299] The server uses an emotion engine to analyze the user's emotions, taking in the user's voice, text, and facial expression data to determine their emotional state, such as stress or happiness.

[0300] Step 7:

[0301] server

[0302] The generative AI model on the server generates personalized healthcare advice for each user based on the analyzed exercise records, dietary data, and emotional data. For example, it generates specific advice such as, "Due to lack of exercise, walk 30 minutes tomorrow. Due to high stress, we recommend doing yoga to relax."

[0303] Step 8:

[0304] server

[0305] The server sends the generated advice and associated metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0306] Step 9:

[0307] Terminal

[0308] The device notifies the user of the advice and metadata received from the server, and the user can check the content of the advice by opening the notification.

[0309] Step 10:

[0310] User

[0311] The user then takes action based on the advice provided, for example, following instructions such as "do 30 minutes of yoga and eat more vegetables for dinner."

[0312] Step 11:

[0313] User

[0314] The user provides feedback to the application about the results of their efforts, such as "I did 30 minutes of yoga today" or "I ate a lot of vegetables for dinner."

[0315] Step 12:

[0316] Terminal

[0317] The terminal receives the feedback from the user and transmits it to the server, and if the transmission is successful, notifies the user.

[0318] Step 13:

[0319] server

[0320] The server receives the feedback sent from the device and stores it in a database, which allows it to take the feedback into account the next time it generates advice and provide individually optimized advice.

[0321] Example

[0322] For example, a user eats "chicken, salad, and bread" for dinner, uploads a photo of it to the app, and records "3km jogging" as an exercise record. This information is sent from the device to the server, which analyzes it. From the photo of the meal, the server identifies "chicken, salad, and bread" and calculates the calories of each. The emotion engine also analyzes the user's voice data and determines that the user is "highly stressed."

[0323] Based on this data, the generative AI model generates specific advice such as "Tomorrow, we recommend 30 minutes of yoga to relax. Eat more vegetables at dinner." The advice is sent to the device and notified to the user. The next day, the user acts on the advice and enters the results into the app. Feedback is sent to the server and reflected in the next advice generation.

[0324] Thus, the present invention is a system that effectively supports the user's health management by providing individualized advice based on the user's exercise records, health status, dietary information, and emotional data.

[0325] Example 2

[0326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] In recent years, interest in lifestyle-related diseases and health management has increased, but there are still limited systems that provide appropriate advice based on individual health and emotional states. Current systems often only collect exercise and dietary data, and do not provide advice that takes into account the emotions of individual users. As a result, there is a challenge in that it is difficult for users to actually achieve effective and sustainable health management.

[0328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0329] In this invention, the server includes a means for analyzing exercise records and meal photos, a means for generating personalized advice based on the analyzed data and the user's emotional data, and a means for adjusting the next advice based on the received feedback, thereby enabling the provision of more personalized healthcare advice based on multifaceted data such as the user's exercise records, meal information, and even emotional state.

[0330] "Exercise records" are data containing information about the user's daily exercise, such as walking distance, exercise time, and calories burned.

[0331] "Meal photos" are image data of daily meals taken by a user, and include information indicating the type and amount of food and details of ingredients.

[0332] "Terminal" refers to an electronic device such as a smartphone or tablet used by a user, which inputs and sends / receives data through an application.

[0333] "Server" refers to a computer system on which a computing device and software for centrally managing data, performing analysis, and executing necessary processing operates.

[0334] "Analysis" refers to the process in which the server analyzes the exercise records and meal photos received and extracts specific information (e.g., calories, nutrients, etc.).

[0335] A "generative AI model" refers to artificial intelligence technology that automatically creates optimal advice for users based on analyzed data.

[0336] "Emotion data" refers to data that indicates the emotional state of a user, and refers to the analysis results of the emotional state obtained by analyzing information such as voice, text, and facial expressions.

[0337] "Feedback" refers to data that reports to the system the results and impressions of users who follow the advice provided.

[0338] "Personalized advice" refers to optimal health management advice for a user, created by a generative AI model based on the user's specific exercise records, dietary information, and emotional data.

[0339] MODE FOR CARRYING OUT THE INVENTION

[0340] This invention is a system that collects a user's daily exercise record, dietary information, health condition, and emotional data, and provides personalized healthcare advice based on these data. The system aims to realize more effective and personalized healthcare support by recognizing the user's emotions and adjusting the advice content based on those emotions.

[0341] Hardware and software used

[0342] Devices: smartphones, tablets, etc.

[0343] Server: A central system for database management and analysis.

[0344] Software: Applications, generative AI models, data analysis engines, emotion engines

[0345] Specific actions

[0346] User

[0347] Through the application, users can input their daily exercise records and photos of their meals. Exercise records include walking distance, exercise time, and calorie consumption, while meal photos are images that specifically show what they ate that day. Data from fitness bands and smartwatches is also synced to the application. For example, users can input information such as "I ran 3km today" or "I ate chicken, salad, and bread."

[0348] Terminal

[0349] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When the data send button is pressed, the data is securely sent to the server using SSL / TLS protocol. At this stage, the data is encrypted to minimize the risk of leakage.

[0350] server

[0351] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise records and meal photo data. It uses image recognition technology to extract specific menu items, nutrients, and calorie information from the meal photos. For example, it can determine that "a dinner of chicken, salad, and bread contains approximately 700 kcal."

[0352] Emotion Engine

[0353] The emotion engine analyzes the user's voice, text, and facial expression data to identify their emotional state. For example, it may determine that the user is likely feeling stressed.

[0354] Generative AI Models

[0355] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it provides specific advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend that they do 30 minutes of yoga to relax," or "We recommend that they increase the amount of vegetables they eat at dinner to maintain a balanced diet."

[0356] Terminal

[0357] The generated advice is sent from the server to the device and notified to the user via push notification or in-app notification. After the user confirms the advice, its read status is sent back to the server.

[0358] User

[0359] The user acts according to the advice provided and provides feedback to the application with the results. For example, the user might enter information such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner." This feedback is then sent to the server and used to generate advice for the next time.

[0360] Specific examples

[0361] A user uploads a photo of "chicken, salad, and bread" to the app after dinner and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calorie count is approximately 700kcal." Next, an emotion engine analyzes the user's emotions from voice and text messages and determines that "the stress level is high." Based on this information, the generative AI model generates advice such as "I recommend 30 minutes of yoga tomorrow to relax, and I also recommend increasing the amount of vegetables you eat at dinner."

[0362] This advice is sent to the device and notified to the user. The next day, the user acts according to the advice and enters the result into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[0363] Prompt Sentence Examples

[0364] "Please provide personalized advice based on the user's exercise record, dietary content, and emotional data. The user ran 3km today and had chicken, salad, and bread for dinner. Emotion analysis has determined that the user's stress level is high. Please create appropriate healthcare advice based on this."

[0365] In this way, the present invention provides individualized advice based on the user's exercise record, dietary information, health condition, and emotional data, thereby more effectively supporting the user's health management.

[0366] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0367] Step 1: Enter your data

[0368] User

[0369] Through the application, users can input photos of their daily exercise records and meals. For example, they can input information such as "I ran 3km today" or "I ate chicken, salad, and bread." Exercise data from fitness bands and smartwatches is also automatically synchronized. The input data is temporarily stored in the device's local storage.

[0370] Input: Exercise records (text), meal photos (images), wearable device data (calories, steps, etc.)

[0371] Output: Exercise and diet data stored in local storage

[0372] Step 2: Sending data

[0373] Terminal

[0374] The saved exercise records and meal photo data are sent to the server. When sending the data, the SSL / TLS protocol is used to encrypt the data and ensure secure communication. Once the data has been sent to the server, it is deleted from local storage.

[0375] Input: Locally stored exercise and food data

[0376] Output: Data sent to the server

[0377] Step 3: Receiving and storing data

[0378] server

[0379] The server receives exercise records and meal photo data sent from the device. The received data is stored in a database and organized using the user ID as a key. If necessary, filtering is performed to standardize the data format.

[0380] Input: Exercise and diet data sent from the device

[0381] Output: Data to be analyzed stored in a database

[0382] Step 4: Analyze the data

[0383] server

[0384] The server analyzes the received data. It uses image recognition technology to extract menu, nutrient, and calorie information from meal photos. It also calculates calories burned and exercise volume from exercise record data. For example, it might determine that a dinner of chicken, salad, and bread contains 700 kcal.

[0385] Input: Exercise and diet data stored in a database

[0386] Output: Parsed menu, nutritional and calorie information and exercise data

[0387] Step 5: Analyze the sentiment data

[0388] Emotion Engine

[0389] The emotion engine analyzes the user's voice, text, and facial expressions to identify the user's emotional state. For example, it may determine that the user is likely feeling stressed.

[0390] Input: User voice, text, and facial expression data

[0391] Output: Identified emotional state (e.g., high stress level)

[0392] Step 6: Generate personalized advice

[0393] server

[0394] The generative AI model generates personalized healthcare advice based on the analyzed exercise, diet, and emotional data, such as "recommended 30 minutes of yoga to relax" or "recommended to increase the amount of vegetables at dinner to maintain a balanced diet."

[0395] Input: Analyzed exercise data, diet data, emotion data

[0396] Output: personalized advice

[0397] Step 7: Submitting Advice

[0398] server

[0399] The server transmits the generated advice and its metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0400] Input: Generated advice and metadata

[0401] Output: Advice sent to terminal

[0402] Step 8: Notification of advice and confirmation of read receipt

[0403] Terminal

[0404] The device receives the advice and notifies the user via push notification or in-app notification. After the user checks the advice, the device sends back the read status to the server.

[0405] Input: Advice sent by the server

[0406] Output: Advice sent to the user, read status sent back to the server

[0407] Step 9: Enter and submit your feedback

[0408] User

[0409] The user acts on the advice provided and provides feedback to the application with the results, such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner."

[0410] Input: User feedback (results of actions)

[0411] Output: Feedback data

[0412] Terminal

[0413] The device receives the feedback data and transmits it securely to the server using the SSL / TLS protocol.

[0414] Input: User feedback

[0415] Output: Feedback sent to the server

[0416] Step 10: Analyze the feedback and incorporate it into your next advice

[0417] server

[0418] The server analyzes the feedback received from the user and stores it in a database. The AI ​​model adjusts the next advice based on the feedback. For example, it generates appropriate advice based on information such as, "I followed the previous advice and practiced yoga to relax, and as a result, my stress level decreased."

[0419] Input: Feedback received from the user

[0420] Output: Adjustment information to be reflected in the next advice

[0421] (Application example 2)

[0422] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0423] Conventional healthcare support systems primarily rely on data such as exercise records and meal photos, and do not take into account the user's emotional state. This poses a challenge: they are unable to provide appropriate advice for health problems that are influenced by the user's emotions. Furthermore, because the advice provided is uniform, it is difficult to provide optimal support tailored to the needs of individual users.

[0424] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0425] In this invention, the server includes means for collecting the user's emotional state and analyzing it using an emotion engine, means for adjusting advice based on the emotional data, and means for analyzing exercise records and meal photos and generating personalized advice based on the analysis results. This enables personalized advice that takes the user's emotions into consideration, making it possible to more effectively support the user's health management.

[0426] The "exercise record" is data on the exercise the user performs each day, including the number of steps taken, distance traveled, calorie consumption, and the like.

[0427] "Meal photos" are data in which the user records the meals they have eaten as photographs.

[0428] "Server" refers to a computer system that receives and analyzes data sent by users and generates personalized advice.

[0429] "Terminal" refers to a device used by a user, such as a smartphone or computer, that has the means to send exercise records and meal photos, and receive and display advice.

[0430] The "emotional state" indicates the mental state of the user and includes emotional categories such as positive, negative, and neutral.

[0431] An "emotion engine" refers to a software or hardware system that analyzes emotions from a user's voice, text, facial expressions, etc., and identifies their emotional state.

[0432] "Analysis" refers to the process of processing received data and extracting and calculating specific information.

[0433] "Advice" is information that serves as a guide for the user's health management and behavior, generated based on the analysis results and emotion data.

[0434] "Feedback" is data that a user inputs into an application, including the results and impressions of actions taken based on the advice provided.

[0435] "Personalization" refers to the process of customizing advice content to each user's specific needs and circumstances.

[0436] To implement this invention, a system must be built that uses a user's smartphone or other device, a server, an emotion engine, and a generative AI model. This system collects the user's daily exercise records and meal photos, and provides personalized health advice based on the analysis results. It can also recognize the user's emotional state and adjust advice accordingly.

[0437] Program Description

[0438] The program of this system is configured as follows:

[0439] 1. Data Collection

[0440] Users use a smartphone application to record their daily exercise and meal photos. The exercise records include the number of steps taken, distance traveled, and calories burned, while the meal photos are images that show the specific meals eaten that day. In addition, data from fitness bands and smartwatches is also synchronized with the application.

[0441] 2. Data Transmission

[0442] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When sending this data to the server, it is securely transmitted using the SSL / TLS protocol.

[0443] 3. Data Analysis

[0444] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[0445] 4. Emotion analysis

[0446] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. Specifically, it uses voice recognition technology to convert voice into text and analyzes the emotion in that text. For example, if the user is feeling stressed, it detects their stress level.

[0447] 5. Generating personalized advice

[0448] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[0449] 6. Advice Notice

[0450] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the device. The device receives the advice sent from the server and notifies the user. The device checks whether the advice has been read and sends that information back to the server.

[0451] 7. Feedback Collection

[0452] The user acts based on the advice provided and provides feedback to the application with the results. For example, the user may input feedback such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner." The device receives the feedback from the user and sends it to the server. The server receives the feedback sent from the device and stores it in a database. The next advice is adjusted based on this feedback.

[0453] Specific examples

[0454] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[0455] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[0456] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[0457] Examples of prompts include:

[0458] The user exercises less. Their stress level is high, so recommend 30 minutes of yoga to relax. Recommend adding more vegetables to dinner.

[0459] As a result, the present invention can provide individual advice based on the user's exercise record, health condition, dietary information, and emotional data, thereby more effectively supporting the user's health management.

[0460] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0461] Step 1:

[0462] Users launch a smartphone application and input their daily exercise records and meal photos. The input exercise records include the number of steps taken, distance traveled, and calorie consumption, and the meal photos are images showing the meals eaten that day. The input data is temporarily saved in local storage.

[0463] Input: Exercise record data, meal photos

[0464] Output: Exercise record data and meal photos saved in local storage

[0465] Step 2:

[0466] The device then sends the received exercise record data and meal photos to a server. Data transmission is secure using the SSL / TLS protocol.

[0467] Input: Exercise record data and meal photos stored in local storage

[0468] Output: Exercise record data and meal photos sent to the server

[0469] Step 3:

[0470] The server stores the exercise record data and meal photos received from the device in a database. It then uses image recognition technology to analyze the meal photos and extract information about the menu, nutrients, and calories. It also calculates calories burned and the amount of exercise from the exercise record data.

[0471] Input: Exercise record data and meal photos sent to the server

[0472] Output: Analysis results (menu, nutrients, calorie information, calories burned, exercise amount)

[0473] Step 4:

[0474] The server collects the user's emotional state from voice and text and performs analysis using an emotion engine. For example, it analyzes text messages and voice data entered by the user within the application and classifies them into emotional categories such as positive, negative, and neutral.

[0475] Input: Audio data, text data

[0476] Output: Parsed emotional state data

[0477] Step 5:

[0478] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it might generate advice like, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[0479] Input: Analysis results (exercise records, dietary data, emotional data)

[0480] Output: Generated healthcare advice

[0481] Step 6:

[0482] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0483] Input: Generated healthcare advice, metadata

[0484] Output: Advice and metadata sent to the device

[0485] Step 7:

[0486] The device receives the advice sent from the server and notifies the user. The user checks the notification and acts based on the advice. The device returns the advice read / implemented status to the server.

[0487] Input: Advice and metadata sent to the device

[0488] Output: Notification to user, read / implemented status returned to server

[0489] Step 8:

[0490] The user acts according to the advice provided and inputs the results into the application as feedback, such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner," and sends this information from the device to the server.

[0491] Input: User feedback data

[0492] Output: Feedback data sent to the server

[0493] Step 9:

[0494] The server receives the feedback sent from the device and stores it in a database, which then adjusts the next advice based on the feedback and provides the most appropriate advice to meet the user's individual needs.

[0495] Input: Feedback data sent to the server

[0496] Output: Adjusted next advice

[0497] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0498] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0499] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0500] [Second embodiment]

[0501] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0502] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0503] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0504] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0505] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0506] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0507] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0508] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0509] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0510] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0511] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0512] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0513] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[0514] System Overview

[0515] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[0516] Specific processing of the program

[0517] 1. Data Collection Phase

[0518] User

[0519] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0520] Terminal

[0521] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[0522] 2. Data analysis phase

[0523] server

[0524] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0525] 3. Individual advice creation phase

[0526] server

[0527] The generative AI model on the server creates personalized healthcare advice based on the analyzed data. For example, it might generate specific advice such as, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[0528] 4. Advice sending phase

[0529] server

[0530] The server sends the generated advice to the device, along with metadata for managing the advice's read status and implementation status.

[0531] Terminal

[0532] The device notifies the user of the advice sent from the server. It displays the advice "walk for 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[0533] 5. Feedback Phase

[0534] User

[0535] The user acts based on the advice provided and inputs the results into the application, for example, reporting, "I walked for 30 minutes today."

[0536] Terminal

[0537] The terminal receives feedback from the user and transmits it to the server.

[0538] server

[0539] The server uses the received feedback to generate the next piece of advice, allowing the advice to be continuously optimized.

[0540] Example

[0541] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "eat less carbohydrates and more vegetables tomorrow. It is recommended that you jog for at least 30 minutes." The generated advice is sent to the user's device and notified to the user.

[0542] The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[0543] As described above, the present invention is a system that provides individualized advice based on a user's exercise records, health status, and dietary information, and effectively supports the user's health management.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] User

[0547] Users use the application to enter photos of their daily exercise records and meals. Exercise records include walking distance, time, calorie consumption, etc. Meal photos are images that show the details of the meals they ate that day.

[0548] Step 2:

[0549] Terminal

[0550] The device receives user-entered exercise records and meal photos, organizes them according to format, and temporarily stores them in local storage. The device also syncs data from devices such as fitness bands and smartwatches.

[0551] Step 3:

[0552] Terminal

[0553] The device generates packets to send the organized data to the server, securely transmitting it using the SSL / TLS protocol, and displays a notification to the user if the data transmission is successful.

[0554] Step 4:

[0555] server

[0556] The server receives the exercise records and meal photos sent from the device, and the received data is stored in a database for use in the next analysis step.

[0557] Step 5:

[0558] server

[0559] The server analyzes exercise records and meal photos. For meal photos, image recognition technology is used to extract information on ingredients, menu items, nutrients, and calories. For exercise records, the server calculates the user's calorie expenditure and exercise volume from the data.

[0560] Step 6:

[0561] server

[0562] The server uses the generative AI model based on the analysis results to generate personalized healthcare advice for the user. For example, if the user's exercise volume is low, the server generates advice recommending additional exercise. If the user's diet is unbalanced, the server recommends a balanced diet.

[0563] Step 7:

[0564] server

[0565] The server converts the generated advice into packets and sends them to the device, along with metadata for tracking whether the advice has been read or implemented.

[0566] Step 8:

[0567] Terminal

[0568] The device receives the advice packet sent from the server and notifies the user of its contents. The user can check the advice by opening the notification.

[0569] Step 9:

[0570] User

[0571] The user acts according to the advice. For example, if a 30-minute jog is recommended, the user performs the exercise. Furthermore, the user also improves their diet based on the advice.

[0572] Step 10:

[0573] User

[0574] The user provides feedback to the application about the results of their actions, such as "I went jogging for 30 minutes" or "I ate a lot of vegetables at dinner."

[0575] Step 11:

[0576] Terminal

[0577] The terminal receives the feedback from the user and transmits it to the server, and notifies the user if the transmission is successful.

[0578] Step 12:

[0579] server

[0580] The server receives the feedback sent from the device, stores it in a database, and takes the feedback into account when generating the next advice, allowing for continuous optimization of personalized advice.

[0581] Example 1

[0582] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0583] As modern lifestyles become more diverse, it is becoming increasingly difficult to provide individually optimized healthcare advice. Existing systems often require cumbersome data collection and analysis from users, resulting in uniform and suboptimal advice. Furthermore, insufficient feedback collection, necessary to verify the effectiveness of advice, reduces the accuracy and effectiveness of the advice. To address this issue, it is necessary to effectively collect and analyze data from users, provide individually tailored advice, and further optimize the advice by incorporating user feedback.

[0584] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0585] In this invention, the server includes means for receiving exercise records and meal photos from a user, means for transmitting the received data to a network device, means for analyzing the exercise records and meal photos in the network device and generating personalized advice based on the analysis results, means for transmitting the generated advice to a terminal, means for notifying the user of the sent advice in the terminal, means for receiving feedback from the user and transmitting it to the network device, means for adjusting next advice based on the received feedback in the network device, and means for generating personalized advice using a generative AI model, thereby enabling the provision of healthcare advice optimized for each user.

[0586] "User" refers to an individual who uses the system to input exercise records, health status, and dietary information and receive healthcare advice.

[0587] "Exercise record" refers to information including the type of exercise performed by the user, the time, distance, calories burned, etc.

[0588] "Meal photos" refer to digital images that are photographs of the meals a user has eaten.

[0589] "Network device" refers to a computer system that has the capability to analyze data received from a user terminal and generate and send personalized advice.

[0590] "Generative AI Model" refers to the artificial intelligence algorithm used to generate personalized healthcare advice based on received and analyzed data.

[0591] "Feedback" refers to information including the results of a user's actions and evaluation of the advice provided.

[0592] "Advice" refers to health management instructions and recommendations created by a generative AI model based on a user's exercise records and dietary information.

[0593] "Analysis" refers to the act of performing calculations and processing on the received exercise record and meal photo data to extract useful information and patterns.

[0594] MODE FOR CARRYING OUT THE INVENTION

[0595] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[0596] System Overview

[0597] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[0598] Specific hardware and software

[0599] User device:

[0600] The user uses a smartphone, tablet, or other device. An application is installed on the device to input and save exercise records and meal photos. This application also works in conjunction with a fitness band or smartwatch to collect data.

[0601] server:

[0602] A server responsible for analyzing the data and generating healthcare advice. The server has a database (e.g., MySQL) and an image analysis tool (e.g., Google Cloud Vision API) installed. It also requires computing resources to run the generative AI model (e.g., ChatGPT).

[0603] Specific processing of the program

[0604] Data collection phase:

[0605] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0606] Data transmission phase:

[0607] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[0608] Data analysis phase:

[0609] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology (e.g., Google Cloud Vision API) to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as, "Dinner is about 600 kcal, and is particularly high in carbohydrates."

[0610] Personalized advice creation phase:

[0611] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, reduce the carbohydrates in your dinner a little and eat more vegetables."

[0612] Advice sending phase:

[0613] The server sends the generated advice to the device. It also sends metadata to manage whether the advice has been read or implemented. The device notifies the user of the advice sent from the server. The device displays the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[0614] Feedback Phase:

[0615] The user acts based on the advice provided and inputs the results into the application. For example, the user reports, "I walked for 30 minutes today." The device receives the feedback from the user and sends it to the server. The server uses the received feedback to generate the next piece of advice. This allows the advice to be continuously optimized.

[0616] Examples of specific examples and prompts

[0617] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "Tomorrow, eat less carbohydrates and more vegetables. It is recommended that you jog for at least 30 minutes."

[0618] The generated advice is sent to the user's device and notified. The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[0619] Example prompt sentence:

[0620] User:

[0621] "I ran 5km today. I had salad, chicken, and bread for dinner."

[0622] server:

[0623] "The calorie analysis of your meal shows that it is approximately 600 kcal. Taking into consideration the amount of exercise required, we recommend that you walk for 30 minutes tomorrow. It would also be a good idea to reduce your carbohydrate intake at dinner and eat more vegetables."

[0624] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0625] Step 1: Data collection

[0626] Users enter their daily exercise records and meal photos through the application. Specifically, users send text and images such as "I ran 5km today" or "I had salad, chicken, and bread for dinner" to the app. Exercise record data and meal photo data are provided as input, and these data are temporarily saved on the device as output.

[0627] Step 2: Send data

[0628] The device sends the exercise record and meal photo data collected from the user to the server via HTTPS protocol. The temporarily stored data is used as input, and the output is securely transmitted to the server via the network.

[0629] Step 3: Receiving and storing data

[0630] The server receives the exercise record and meal photo data sent from the device and stores them in a database (e.g., MySQL). It receives the data sent from the device as input and stores this data in the database as output.

[0631] Step 4: Data analysis

[0632] The server analyzes the received data. Specifically, it uses Google Cloud Vision API to perform image analysis on food photos and converts them into menu, nutrient, and calorie information. It also uses natural language processing tools to perform text analysis on exercise records. It uses data obtained from the database as input and obtains the analysis results as output.

[0633] Step 5: Create personalized advice

[0634] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, it would be good to eat a little less carbohydrates and more vegetables at dinner." The analysis results above are used as input, and advice is generated as output.

[0635] Step 6: Send advice

[0636] The server sends the generated advice to the device. The advice also includes metadata for managing read status and implementation status. The generated advice is used as input and sent to the device as output.

[0637] Step 7: Advice Notification

[0638] The device receives advice sent from the server and displays advice such as "walk 30 minutes tomorrow" or "reduce carbohydrates" to the user via push notification. The device receives advice data from the server as input and notifies the user as output.

[0639] Step 8: Gather feedback

[0640] The user acts based on the advice provided and reports the results to the application. For example, they may provide feedback such as, "I walked for 30 minutes today." The user provides the action data they took as input, and this data is temporarily saved on the device as output.

[0641] Step 9: Submit your feedback

[0642] The terminal receives feedback data from the user and transmits it to the server, using the feedback data from the user as input and transmission to the server as output.

[0643] Step 10: Feedback analysis and next-time advice adjustment

[0644] The server reflects the received feedback in the next advice generation, which allows advice to be continuously optimized. Feedback data is received as input, and data that will be reflected in the next advice is generated as output.

[0645] (Application example 1)

[0646] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] Conventional health management systems only required users to input data and receive advice, which lacked real-timeness and convenience. Furthermore, there was no mechanism for dynamically providing individual advice when selecting food in a physical store. This made it difficult for users to receive health support in a physical store.

[0648] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0649] In this invention, the server includes a means for receiving exercise records and meal photos, a means for performing food recognition and nutritional analysis, and a means for generating personalized advice based on the analysis results, thereby enabling real-time health advice in a physical store.

[0650] A "user" is an individual who uses this system and provides exercise records and dietary information.

[0651] The "exercise record" is data relating to the exercise performed by the user on a daily basis, and includes information such as distance, time, and type of exercise.

[0652] "Meal photos" are image data that record meals consumed by the user and are used for subsequent analysis.

[0653] The "receiving means" is a function that allows the terminal to receive exercise records and meal photos from the user.

[0654] A "terminal" is a smartphone, smart glasses, or other electronic device used by a user, and is a device for inputting data and receiving advice.

[0655] A "server" is a computer system that aggregates received data and performs analysis and advice generation.

[0656] The "means of analysis" refers to algorithms and software that analyze exercise records and meal photos on a server and extract nutrient and calorie information.

[0657] "Personalized advice" refers to specific health management instructions or recommendations provided to the user based on the analyzed data.

[0658] The "notification means" is a function that allows the terminal to notify the user of the advice that has been generated.

[0659] "Feedback" is information about the actions taken by the user based on the advice and the results of those actions, and is used to optimize future advice.

[0660] The "adjustment means" is a function that allows the server to revise and optimize the advice content from the next time onwards based on feedback from the user.

[0661] "Food recognition" is a technology that uses smart devices in physical stores to take pictures of products and identify their contents.

[0662] "Nutritional analysis" is the process of calculating and analyzing the various nutrients and calories contained in recognized foods.

[0663] "Real-time health advice" is a system that provides on-the-spot health guidance and recommendations when users are choosing food in a physical store.

[0664] A specific embodiment for realizing this invention will be described. This system has the function of collecting and analyzing exercise records and dietary information to provide individualized advice to support the user's health management. The main components of the system are a terminal, a server, and a generative AI model.

[0665] System configuration

[0666] Hardware:

[0667] Devices: Smartphones and smart glasses used by users. These devices input exercise records, take photos of meals, and display and notify advice.

[0668] Server: A computer system located in the cloud that analyzes the received data and generates advice.

[0669] Software used

[0670] software:

[0671] Terminal application: An application that allows users to input exercise records and dietary information and receive advice, which is displayed on the screen in real time.

[0672] Server-side software: Software that analyzes data, runs generative AI models, and incorporates feedback.

[0673] Image analysis algorithm: An algorithm that analyzes food photos to extract menu items, nutrients, and calories.

[0674] System Operation

[0675] 1. Data Collection:

[0676] Users use their smartphones or smart glasses to record their exercise and take photos of their meals. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0677] 2. Data transmission:

[0678] The terminal transmits the entered exercise records and meal photos to the server.

[0679] 3. Data Analysis:

[0680] The server analyzes the received exercise records and meal photos. It uses an image analysis algorithm to analyze the meal photos and extract nutrient and calorie information. For example, the server uses image recognition technology to analyze a meal photo and obtain information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0681] 4. Advice Generation:

[0682] The generative AI model on the server generates personalized healthcare advice based on the analyzed data. For example, it might say, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[0683] 5. Advice Notice:

[0684] The generated advice is sent to the device and notified to the user. For example, the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" is displayed to the user.

[0685] 6. Feedback received:

[0686] The user acts based on the advice provided and inputs the results into the device, for example, reporting, "I walked for 30 minutes today."

[0687] 7. Data Transmission and Coordination:

[0688] The device sends feedback to the server, which then adjusts the next advice based on the received feedback, allowing for continuous optimization of the advice.

[0689] Use in physical stores

[0690] Examples:

[0691] When a user scans chicken in a physical store, the AI ​​model analyzes it and determines that "this chicken is high in protein but low in fat," and based on the user's exercise history and health status, displays advice such as "This chicken is high in protein, so it fits into your weekly exercise plan. Eat it with a salad."

[0692] Example prompt sentence:

[0693] "User data: Jogging 3 times a week, high protein, low carb diet, good health. Analysis results: Chicken, high protein, low fat. Based on this information, please provide weekly exercise and meal plan suggestions."

[0694] The above is an embodiment of the present invention. This system effectively supports users in managing their health and significantly improves convenience when selecting food items in physical stores.

[0695] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0696] Step 1:

[0697] Data collection and input

[0698] User: The user uses a smartphone or smart glasses to enter exercise records and take photos of meals. For example, "I ran 5km today" or "I had salad, chicken, and bread for dinner." This data is stored on the device and later sent to the server.

[0699] Step 2:

[0700] Data transmission

[0701] Device: The device sends the exercise records and meal photos entered by the user to the server. Here, HTTP requests are used as the protocol for sending input data. The input data includes exercise records and meal photos, and is sent to the server.

[0702] Step 3:

[0703] Data analysis

[0704] Server: The server analyzes the exercise records and meal photos it receives. It uses an image analysis algorithm on the server side to extract information about the menu, nutrients, and calories from the meal photos. The image analysis algorithm receives the photos as input and outputs the menu, nutrients, and calories. For example, it obtains information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0705] Step 4:

[0706] Advice Generation

[0707] Server: The generative AI model on the server generates personalized healthcare advice based on the analyzed data. The generative AI model inputs the analysis results and past user data as prompts and outputs specific advice. For example, the generated advice might be, "Since you haven't exercised much today, we recommend walking for 30 minutes tomorrow. Also, it would be good to reduce carbohydrates a little and eat more vegetables at dinner."

[0708] Step 5:

[0709] Advice Notice

[0710] Device: The advice generated by the server is sent to the device. The device notifies the user of this advice and displays it to them. Possible notification methods include push notifications and in-app notifications. For example, the user might be notified of advice such as "walk 30 minutes tomorrow" and "reduce carbohydrate intake."

[0711] Step 6:

[0712] Feedback Input

[0713] User: The user acts based on the advice provided and inputs the results into the device. The input is feedback information such as "I walked for 30 minutes today." This information is later sent to the server.

[0714] Step 7:

[0715] Feedback submission and processing

[0716] Terminal: Collects feedback information and sends it to the server. Again, a communication protocol such as an HTTP request is used. The input is the feedback data, which is then sent to the server.

[0717] Server: Adjusts and optimizes the next advice based on the received feedback. The server takes past feedback information as input and uses a generative AI model to continuously optimize the advice, making health management more effective for the user.

[0718] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0719] This invention combines a system that collects a user's daily exercise record, health condition, and dietary information and uses a generative AI model to provide personalized healthcare advice, with an emotion engine. The system aims to provide more effective healthcare support by recognizing the user's emotions and adjusting advice based on those emotions.

[0720] System Overview

[0721] The system receives exercise records and meal photo data entered by the user on the device and sends it to the server. The server analyzes the received data and generates individualized advice based on the results. In doing so, it uses an emotion engine to analyze the user's emotional data and reflects it in the advice. The generated advice is sent to the device and notified to the user. In addition, user feedback is collected and used to generate the next advice, aiming for individualization and optimization.

[0722] Specific processing of the program

[0723] 1. Data Collection Phase

[0724] User

[0725] Users use the app to record their daily exercise and meal photos. The exercise records include walking distance, time, and calorie consumption, while the meal photos are images that show the specific meals they ate that day. In addition, data from fitness bands and smartwatches is also synced to the app.

[0726] Terminal

[0727] The device receives the exercise records and meal photos entered by the user, temporarily stores them in local storage, and prepares them to be sent to the server, where they are securely transmitted using the SSL / TLS protocol.

[0728] 2. Data analysis phase

[0729] server

[0730] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[0731] Emotion Engine

[0732] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. For example, if the user is feeling stressed, it detects their stress level.

[0733] 3. Individual advice creation phase

[0734] server

[0735] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[0736] 4. Advice sending phase

[0737] server

[0738] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0739] Terminal

[0740] The terminal receives the advice sent from the server, notifies the user, checks whether the advice has been read, and returns that information to the server.

[0741] 5. Feedback Phase

[0742] User

[0743] The user acts on the advice provided and provides feedback to the application, such as "I did 30 minutes of yoga" or "I ate more vegetables for dinner."

[0744] Terminal

[0745] The terminal receives feedback from the user and transmits it to the server.

[0746] server

[0747] The server receives feedback from the device and stores it in a database. Based on this feedback, the server adjusts the next advice it provides, allowing it to continually provide optimal advice tailored to the user's individual needs.

[0748] Example

[0749] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[0750] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[0751] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[0752] In this way, the present invention is a system that provides individual advice based on emotional data in addition to the user's exercise records, health status, and dietary information, thereby more effectively supporting the user's health management.

[0753] The processing flow will be explained below.

[0754] Step 1:

[0755] User

[0756] Users use the application to record their daily exercise and meal photos. The exercise records include distance traveled, time, and calorie consumption, and the meal photos are taken to show what they ate that day.

[0757] Step 2:

[0758] Terminal

[0759] The device receives the exercise records and meal photos entered by the user and temporarily stores them in local storage, preparing to send the stored data to the server.

[0760] Step 3:

[0761] Terminal

[0762] The device converts the organized data into packets and sends them securely to the server using the SSL / TLS protocol. If the data transmission was successful, the device notifies the user.

[0763] Step 4:

[0764] server

[0765] The server receives the exercise records and meal photos sent from the device, stores them in a database, and then processes the data for analysis.

[0766] Step 5:

[0767] server

[0768] The server analyzes the received data. It uses image recognition technology to analyze food photos and extract menu, nutrient, and calorie information. It also analyzes exercise records to calculate the user's calorie consumption and exercise performance.

[0769] Step 6:

[0770] Emotion Engine

[0771] The server uses an emotion engine to analyze the user's emotions, taking in the user's voice, text, and facial expression data to determine their emotional state, such as stress or happiness.

[0772] Step 7:

[0773] server

[0774] The generative AI model on the server generates personalized healthcare advice for each user based on the analyzed exercise records, dietary data, and emotional data. For example, it generates specific advice such as, "Due to lack of exercise, walk 30 minutes tomorrow. Due to high stress, we recommend doing yoga to relax."

[0775] Step 8:

[0776] server

[0777] The server sends the generated advice and associated metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0778] Step 9:

[0779] Terminal

[0780] The device notifies the user of the advice and metadata received from the server, and the user can check the content of the advice by opening the notification.

[0781] Step 10:

[0782] User

[0783] The user then takes action based on the advice provided, for example, following instructions such as "do 30 minutes of yoga and eat more vegetables for dinner."

[0784] Step 11:

[0785] User

[0786] The user provides feedback to the application about the results of their efforts, such as "I did 30 minutes of yoga today" or "I ate a lot of vegetables for dinner."

[0787] Step 12:

[0788] Terminal

[0789] The terminal receives the feedback from the user and transmits it to the server, and if the transmission is successful, notifies the user.

[0790] Step 13:

[0791] server

[0792] The server receives the feedback sent from the device and stores it in a database, which allows it to take the feedback into account the next time it generates advice and provide individually optimized advice.

[0793] Example

[0794] For example, a user eats "chicken, salad, and bread" for dinner, uploads a photo of it to the app, and records "3km jogging" as an exercise record. This information is sent from the device to the server, which analyzes it. From the photo of the meal, the server identifies "chicken, salad, and bread" and calculates the calories of each. The emotion engine also analyzes the user's voice data and determines that the user is "highly stressed."

[0795] Based on this data, the generative AI model generates specific advice such as "Tomorrow, we recommend 30 minutes of yoga to relax. Eat more vegetables at dinner." The advice is sent to the device and notified to the user. The next day, the user acts on the advice and enters the results into the app. Feedback is sent to the server and reflected in the next advice generation.

[0796] Thus, the present invention is a system that effectively supports the user's health management by providing individualized advice based on the user's exercise records, health status, dietary information, and emotional data.

[0797] Example 2

[0798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] In recent years, interest in lifestyle-related diseases and health management has increased, but there are still limited systems that provide appropriate advice based on individual health and emotional states. Current systems often only collect exercise and dietary data, and do not provide advice that takes into account the emotions of individual users. As a result, there is a challenge in that it is difficult for users to actually achieve effective and sustainable health management.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0801] In this invention, the server includes a means for analyzing exercise records and meal photos, a means for generating personalized advice based on the analyzed data and the user's emotional data, and a means for adjusting the next advice based on the received feedback, thereby enabling the provision of more personalized healthcare advice based on multifaceted data such as the user's exercise records, meal information, and even emotional state.

[0802] "Exercise records" are data containing information about the user's daily exercise, such as walking distance, exercise time, and calories burned.

[0803] "Meal photos" are image data of daily meals taken by a user, and include information indicating the type and amount of food and details of ingredients.

[0804] "Terminal" refers to an electronic device such as a smartphone or tablet used by a user, which inputs and sends / receives data through an application.

[0805] "Server" refers to a computer system on which a computing device and software for centrally managing data, performing analysis, and executing necessary processing operates.

[0806] "Analysis" refers to the process in which the server analyzes the exercise records and meal photos received and extracts specific information (e.g., calories, nutrients, etc.).

[0807] A "generative AI model" refers to artificial intelligence technology that automatically creates optimal advice for users based on analyzed data.

[0808] "Emotion data" refers to data that indicates the emotional state of a user, and refers to the analysis results of the emotional state obtained by analyzing information such as voice, text, and facial expressions.

[0809] "Feedback" refers to data that reports to the system the results and impressions of users who follow the advice provided.

[0810] "Personalized advice" refers to optimal health management advice for a user, created by a generative AI model based on the user's specific exercise records, dietary information, and emotional data.

[0811] MODE FOR CARRYING OUT THE INVENTION

[0812] This invention is a system that collects a user's daily exercise record, dietary information, health condition, and emotional data, and provides personalized healthcare advice based on these data. The system aims to realize more effective and personalized healthcare support by recognizing the user's emotions and adjusting the advice content based on those emotions.

[0813] Hardware and software used

[0814] Devices: smartphones, tablets, etc.

[0815] Server: A central system for database management and analysis.

[0816] Software: Applications, generative AI models, data analysis engines, emotion engines

[0817] Specific actions

[0818] User

[0819] Through the application, users can input their daily exercise records and photos of their meals. Exercise records include walking distance, exercise time, and calorie consumption, while meal photos are images that specifically show what they ate that day. Data from fitness bands and smartwatches is also synced to the application. For example, users can input information such as "I ran 3km today" or "I ate chicken, salad, and bread."

[0820] Terminal

[0821] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When the data send button is pressed, the data is securely sent to the server using SSL / TLS protocol. At this stage, the data is encrypted to minimize the risk of leakage.

[0822] server

[0823] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise records and meal photo data. It uses image recognition technology to extract specific menu items, nutrients, and calorie information from the meal photos. For example, it can determine that "a dinner of chicken, salad, and bread contains approximately 700 kcal."

[0824] Emotion Engine

[0825] The emotion engine analyzes the user's voice, text, and facial expression data to identify their emotional state. For example, it may determine that the user is likely feeling stressed.

[0826] Generative AI Models

[0827] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it provides specific advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend that they do 30 minutes of yoga to relax," or "We recommend that they increase the amount of vegetables they eat at dinner to maintain a balanced diet."

[0828] Terminal

[0829] The generated advice is sent from the server to the device and notified to the user via push notification or in-app notification. After the user confirms the advice, its read status is sent back to the server.

[0830] User

[0831] The user acts according to the advice provided and provides feedback to the application with the results. For example, the user might enter information such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner." This feedback is then sent to the server and used to generate advice for the next time.

[0832] Specific examples

[0833] A user uploads a photo of "chicken, salad, and bread" to the app after dinner and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calorie count is approximately 700kcal." Next, an emotion engine analyzes the user's emotions from voice and text messages and determines that "the stress level is high." Based on this information, the generative AI model generates advice such as "I recommend 30 minutes of yoga tomorrow to relax, and I also recommend increasing the amount of vegetables you eat at dinner."

[0834] This advice is sent to the device and notified to the user. The next day, the user acts according to the advice and enters the result into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[0835] Prompt Sentence Examples

[0836] "Please provide personalized advice based on the user's exercise record, dietary content, and emotional data. The user ran 3km today and had chicken, salad, and bread for dinner. Emotion analysis has determined that the user's stress level is high. Please create appropriate healthcare advice based on this."

[0837] In this way, the present invention provides individualized advice based on the user's exercise record, dietary information, health condition, and emotional data, thereby more effectively supporting the user's health management.

[0838] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0839] Step 1: Enter your data

[0840] User

[0841] Through the application, users can input photos of their daily exercise records and meals. For example, they can input information such as "I ran 3km today" or "I ate chicken, salad, and bread." Exercise data from fitness bands and smartwatches is also automatically synchronized. The input data is temporarily stored in the device's local storage.

[0842] Input: Exercise records (text), meal photos (images), wearable device data (calories, steps, etc.)

[0843] Output: Exercise and diet data stored in local storage

[0844] Step 2: Sending data

[0845] Terminal

[0846] The saved exercise records and meal photo data are sent to the server. When sending the data, the SSL / TLS protocol is used to encrypt the data and ensure secure communication. Once the data has been sent to the server, it is deleted from local storage.

[0847] Input: Locally stored exercise and food data

[0848] Output: Data sent to the server

[0849] Step 3: Receiving and storing data

[0850] server

[0851] The server receives exercise records and meal photo data sent from the device. The received data is stored in a database and organized using the user ID as a key. If necessary, filtering is performed to standardize the data format.

[0852] Input: Exercise and diet data sent from the device

[0853] Output: Data to be analyzed stored in a database

[0854] Step 4: Analyze the data

[0855] server

[0856] The server analyzes the received data. It uses image recognition technology to extract menu, nutrient, and calorie information from meal photos. It also calculates calories burned and exercise volume from exercise record data. For example, it might determine that a dinner of chicken, salad, and bread contains 700 kcal.

[0857] Input: Exercise and diet data stored in a database

[0858] Output: Parsed menu, nutritional and calorie information and exercise data

[0859] Step 5: Analyze the sentiment data

[0860] Emotion Engine

[0861] The emotion engine analyzes the user's voice, text, and facial expressions to identify the user's emotional state. For example, it may determine that the user is likely feeling stressed.

[0862] Input: User voice, text, and facial expression data

[0863] Output: Identified emotional state (e.g., high stress level)

[0864] Step 6: Generate personalized advice

[0865] server

[0866] The generative AI model generates personalized healthcare advice based on the analyzed exercise, diet, and emotional data, such as "recommended 30 minutes of yoga to relax" or "recommended to increase the amount of vegetables at dinner to maintain a balanced diet."

[0867] Input: Analyzed exercise data, diet data, emotion data

[0868] Output: personalized advice

[0869] Step 7: Submitting Advice

[0870] server

[0871] The server transmits the generated advice and its metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0872] Input: Generated advice and metadata

[0873] Output: Advice sent to terminal

[0874] Step 8: Notification of advice and confirmation of read receipt

[0875] Terminal

[0876] The device receives the advice and notifies the user via push notification or in-app notification. After the user checks the advice, the device sends back the read status to the server.

[0877] Input: Advice sent by the server

[0878] Output: Advice sent to the user, read status sent back to the server

[0879] Step 9: Enter and submit your feedback

[0880] User

[0881] The user acts on the advice provided and provides feedback to the application with the results, such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner."

[0882] Input: User feedback (results of actions)

[0883] Output: Feedback data

[0884] Terminal

[0885] The device receives the feedback data and transmits it securely to the server using the SSL / TLS protocol.

[0886] Input: User feedback

[0887] Output: Feedback sent to the server

[0888] Step 10: Analyze the feedback and incorporate it into your next advice

[0889] server

[0890] The server analyzes the feedback received from the user and stores it in a database. The AI ​​model adjusts the next advice based on the feedback. For example, it generates appropriate advice based on information such as, "I followed the previous advice and practiced yoga to relax, and as a result, my stress level decreased."

[0891] Input: Feedback received from the user

[0892] Output: Adjustment information to be reflected in the next advice

[0893] (Application example 2)

[0894] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0895] Conventional healthcare support systems primarily rely on data such as exercise records and meal photos, and do not take into account the user's emotional state. This poses a challenge: they are unable to provide appropriate advice for health problems that are influenced by the user's emotions. Furthermore, because the advice provided is uniform, it is difficult to provide optimal support tailored to the needs of individual users.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0897] In this invention, the server includes means for collecting the user's emotional state and analyzing it using an emotion engine, means for adjusting advice based on the emotional data, and means for analyzing exercise records and meal photos and generating personalized advice based on the analysis results. This enables personalized advice that takes the user's emotions into consideration, making it possible to more effectively support the user's health management.

[0898] The "exercise record" is data on the exercise the user performs each day, including the number of steps taken, distance traveled, calorie consumption, and the like.

[0899] "Meal photos" are data in which the user records the meals they have eaten as photographs.

[0900] "Server" refers to a computer system that receives and analyzes data sent by users and generates personalized advice.

[0901] "Terminal" refers to a device used by a user, such as a smartphone or computer, that has the means to send exercise records and meal photos, and receive and display advice.

[0902] The "emotional state" indicates the mental state of the user and includes emotional categories such as positive, negative, and neutral.

[0903] An "emotion engine" refers to a software or hardware system that analyzes emotions from a user's voice, text, facial expressions, etc., and identifies their emotional state.

[0904] "Analysis" refers to the process of processing received data and extracting and calculating specific information.

[0905] "Advice" is information that serves as a guide for the user's health management and behavior, generated based on the analysis results and emotion data.

[0906] "Feedback" is data that a user inputs into an application, including the results and impressions of actions taken based on the advice provided.

[0907] "Personalization" refers to the process of customizing advice content to each user's specific needs and circumstances.

[0908] To implement this invention, a system must be built that uses a user's smartphone or other device, a server, an emotion engine, and a generative AI model. This system collects the user's daily exercise records and meal photos, and provides personalized health advice based on the analysis results. It can also recognize the user's emotional state and adjust advice accordingly.

[0909] Program Description

[0910] The program of this system is configured as follows:

[0911] 1. Data Collection

[0912] Users use a smartphone application to record their daily exercise and meal photos. The exercise records include the number of steps taken, distance traveled, and calories burned, while the meal photos are images that show the specific meals eaten that day. In addition, data from fitness bands and smartwatches is also synchronized with the application.

[0913] 2. Data Transmission

[0914] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When sending this data to the server, it is securely transmitted using the SSL / TLS protocol.

[0915] 3. Data Analysis

[0916] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[0917] 4. Emotion analysis

[0918] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. Specifically, it uses voice recognition technology to convert voice into text and analyzes the emotion in that text. For example, if the user is feeling stressed, it detects their stress level.

[0919] 5. Generating personalized advice

[0920] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[0921] 6. Advice Notice

[0922] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the device. The device receives the advice sent from the server and notifies the user. The device checks whether the advice has been read and sends that information back to the server.

[0923] 7. Feedback Collection

[0924] The user acts based on the advice provided and provides feedback to the application with the results. For example, the user may input feedback such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner." The device receives the feedback from the user and sends it to the server. The server receives the feedback sent from the device and stores it in a database. The next advice is adjusted based on this feedback.

[0925] Specific examples

[0926] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[0927] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[0928] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[0929] Examples of prompts include:

[0930] The user exercises less. Their stress level is high, so recommend 30 minutes of yoga to relax. Recommend adding more vegetables to dinner.

[0931] As a result, the present invention can provide individual advice based on the user's exercise record, health condition, dietary information, and emotional data, thereby more effectively supporting the user's health management.

[0932] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0933] Step 1:

[0934] Users launch a smartphone application and input their daily exercise records and meal photos. The input exercise records include the number of steps taken, distance traveled, and calorie consumption, and the meal photos are images showing the meals eaten that day. The input data is temporarily saved in local storage.

[0935] Input: Exercise record data, meal photos

[0936] Output: Exercise record data and meal photos saved in local storage

[0937] Step 2:

[0938] The device then sends the received exercise record data and meal photos to a server. Data transmission is secure using the SSL / TLS protocol.

[0939] Input: Exercise record data and meal photos stored in local storage

[0940] Output: Exercise record data and meal photos sent to the server

[0941] Step 3:

[0942] The server stores the exercise record data and meal photos received from the device in a database. It then uses image recognition technology to analyze the meal photos and extract information about the menu, nutrients, and calories. It also calculates calories burned and the amount of exercise from the exercise record data.

[0943] Input: Exercise record data and meal photos sent to the server

[0944] Output: Analysis results (menu, nutrients, calorie information, calories burned, exercise amount)

[0945] Step 4:

[0946] The server collects the user's emotional state from voice and text and performs analysis using an emotion engine. For example, it analyzes text messages and voice data entered by the user within the application and classifies them into emotional categories such as positive, negative, and neutral.

[0947] Input: Audio data, text data

[0948] Output: Parsed emotional state data

[0949] Step 5:

[0950] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it might generate advice like, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[0951] Input: Analysis results (exercise records, dietary data, emotional data)

[0952] Output: Generated healthcare advice

[0953] Step 6:

[0954] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[0955] Input: Generated healthcare advice, metadata

[0956] Output: Advice and metadata sent to the device

[0957] Step 7:

[0958] The device receives the advice sent from the server and notifies the user. The user checks the notification and acts based on the advice. The device returns the advice read / implemented status to the server.

[0959] Input: Advice and metadata sent to the device

[0960] Output: Notification to user, read / implemented status returned to server

[0961] Step 8:

[0962] The user acts according to the advice provided and inputs the results into the application as feedback, such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner," and sends this information from the device to the server.

[0963] Input: User feedback data

[0964] Output: Feedback data sent to the server

[0965] Step 9:

[0966] The server receives the feedback sent from the device and stores it in a database, which then adjusts the next advice based on the feedback and provides the most appropriate advice to meet the user's individual needs.

[0967] Input: Feedback data sent to the server

[0968] Output: Adjusted next advice

[0969] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0970] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0971] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0972] [Third embodiment]

[0973] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0974] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0975] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0976] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0977] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0978] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0979] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0980] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0981] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0982] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0983] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0984] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0985] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[0986] System Overview

[0987] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[0988] Specific processing of the program

[0989] 1. Data Collection Phase

[0990] User

[0991] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[0992] Terminal

[0993] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[0994] 2. Data analysis phase

[0995] server

[0996] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[0997] 3. Individual advice creation phase

[0998] server

[0999] The generative AI model on the server creates personalized healthcare advice based on the analyzed data. For example, it might generate specific advice such as, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[1000] 4. Advice sending phase

[1001] server

[1002] The server sends the generated advice to the device, along with metadata for managing the advice's read status and implementation status.

[1003] Terminal

[1004] The device notifies the user of the advice sent from the server. It displays the advice "walk for 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[1005] 5. Feedback Phase

[1006] User

[1007] The user acts based on the advice provided and inputs the results into the application, for example, reporting, "I walked for 30 minutes today."

[1008] Terminal

[1009] The terminal receives feedback from the user and transmits it to the server.

[1010] server

[1011] The server uses the received feedback to generate the next piece of advice, allowing the advice to be continuously optimized.

[1012] Example

[1013] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "eat less carbohydrates and more vegetables tomorrow. It is recommended that you jog for at least 30 minutes." The generated advice is sent to the user's device and notified to the user.

[1014] The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[1015] As described above, the present invention is a system that provides individualized advice based on a user's exercise records, health status, and dietary information, and effectively supports the user's health management.

[1016] The processing flow will be explained below.

[1017] Step 1:

[1018] User

[1019] Users use the application to enter photos of their daily exercise records and meals. Exercise records include walking distance, time, calorie consumption, etc. Meal photos are images that show the details of the meals they ate that day.

[1020] Step 2:

[1021] Terminal

[1022] The device receives user-entered exercise records and meal photos, organizes them according to format, and temporarily stores them in local storage. The device also syncs data from devices such as fitness bands and smartwatches.

[1023] Step 3:

[1024] Terminal

[1025] The device generates packets to send the organized data to the server, securely transmitting it using the SSL / TLS protocol, and displays a notification to the user if the data transmission is successful.

[1026] Step 4:

[1027] server

[1028] The server receives the exercise records and meal photos sent from the device, and the received data is stored in a database for use in the next analysis step.

[1029] Step 5:

[1030] server

[1031] The server analyzes exercise records and meal photos. For meal photos, image recognition technology is used to extract information on ingredients, menu items, nutrients, and calories. For exercise records, the server calculates the user's calorie expenditure and exercise volume from the data.

[1032] Step 6:

[1033] server

[1034] The server uses the generative AI model based on the analysis results to generate personalized healthcare advice for the user. For example, if the user's exercise volume is low, the server generates advice recommending additional exercise. If the user's diet is unbalanced, the server recommends a balanced diet.

[1035] Step 7:

[1036] server

[1037] The server converts the generated advice into packets and sends them to the device, along with metadata for tracking whether the advice has been read or implemented.

[1038] Step 8:

[1039] Terminal

[1040] The device receives the advice packet sent from the server and notifies the user of its contents. The user can check the advice by opening the notification.

[1041] Step 9:

[1042] User

[1043] The user acts according to the advice. For example, if a 30-minute jog is recommended, the user performs the exercise. Furthermore, the user also improves their diet based on the advice.

[1044] Step 10:

[1045] User

[1046] The user provides feedback to the application about the results of their actions, such as "I went jogging for 30 minutes" or "I ate a lot of vegetables at dinner."

[1047] Step 11:

[1048] Terminal

[1049] The terminal receives the feedback from the user and transmits it to the server, and notifies the user if the transmission is successful.

[1050] Step 12:

[1051] server

[1052] The server receives the feedback sent from the device, stores it in a database, and takes the feedback into account when generating the next advice, allowing for continuous optimization of personalized advice.

[1053] Example 1

[1054] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1055] As modern lifestyles become more diverse, it is becoming increasingly difficult to provide individually optimized healthcare advice. Existing systems often require cumbersome data collection and analysis from users, resulting in uniform and suboptimal advice. Furthermore, insufficient feedback collection, necessary to verify the effectiveness of advice, reduces the accuracy and effectiveness of the advice. To address this issue, it is necessary to effectively collect and analyze data from users, provide individually tailored advice, and further optimize the advice by incorporating user feedback.

[1056] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1057] In this invention, the server includes means for receiving exercise records and meal photos from a user, means for transmitting the received data to a network device, means for analyzing the exercise records and meal photos in the network device and generating personalized advice based on the analysis results, means for transmitting the generated advice to a terminal, means for notifying the user of the sent advice in the terminal, means for receiving feedback from the user and transmitting it to the network device, means for adjusting next advice based on the received feedback in the network device, and means for generating personalized advice using a generative AI model, thereby enabling the provision of healthcare advice optimized for each user.

[1058] "User" refers to an individual who uses the system to input exercise records, health status, and dietary information and receive healthcare advice.

[1059] "Exercise record" refers to information including the type of exercise performed by the user, the time, distance, calories burned, etc.

[1060] "Meal photos" refer to digital images that are photographs of the meals a user has eaten.

[1061] "Network device" refers to a computer system that has the capability to analyze data received from a user terminal and generate and send personalized advice.

[1062] "Generative AI Model" refers to the artificial intelligence algorithm used to generate personalized healthcare advice based on received and analyzed data.

[1063] "Feedback" refers to information including the results of a user's actions and evaluation of the advice provided.

[1064] "Advice" refers to health management instructions and recommendations created by a generative AI model based on a user's exercise records and dietary information.

[1065] "Analysis" refers to the act of performing calculations and processing on the received exercise record and meal photo data to extract useful information and patterns.

[1066] MODE FOR CARRYING OUT THE INVENTION

[1067] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[1068] System Overview

[1069] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[1070] Specific hardware and software

[1071] User device:

[1072] The user uses a smartphone, tablet, or other device. An application is installed on the device to input and save exercise records and meal photos. This application also works in conjunction with a fitness band or smartwatch to collect data.

[1073] server:

[1074] A server responsible for analyzing the data and generating healthcare advice. The server has a database (e.g., MySQL) and an image analysis tool (e.g., Google Cloud Vision API) installed. It also requires computing resources to run the generative AI model (e.g., ChatGPT).

[1075] Specific processing of the program

[1076] Data collection phase:

[1077] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[1078] Data transmission phase:

[1079] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[1080] Data analysis phase:

[1081] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology (e.g., Google Cloud Vision API) to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as, "Dinner is about 600 kcal, and is particularly high in carbohydrates."

[1082] Personalized advice creation phase:

[1083] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, reduce the carbohydrates in your dinner a little and eat more vegetables."

[1084] Advice sending phase:

[1085] The server sends the generated advice to the device. It also sends metadata to manage whether the advice has been read or implemented. The device notifies the user of the advice sent from the server. The device displays the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[1086] Feedback Phase:

[1087] The user acts based on the advice provided and inputs the results into the application. For example, the user reports, "I walked for 30 minutes today." The device receives the feedback from the user and sends it to the server. The server uses the received feedback to generate the next piece of advice. This allows the advice to be continuously optimized.

[1088] Examples of specific examples and prompts

[1089] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "Tomorrow, eat less carbohydrates and more vegetables. It is recommended that you jog for at least 30 minutes."

[1090] The generated advice is sent to the user's device and notified. The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[1091] Example prompt sentence:

[1092] User:

[1093] "I ran 5km today. I had salad, chicken, and bread for dinner."

[1094] server:

[1095] "The calorie analysis of your meal shows that it is approximately 600 kcal. Taking into consideration the amount of exercise required, we recommend that you walk for 30 minutes tomorrow. It would also be a good idea to reduce your carbohydrate intake at dinner and eat more vegetables."

[1096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1097] Step 1: Data collection

[1098] Users enter their daily exercise records and meal photos through the application. Specifically, users send text and images such as "I ran 5km today" or "I had salad, chicken, and bread for dinner" to the app. Exercise record data and meal photo data are provided as input, and these data are temporarily saved on the device as output.

[1099] Step 2: Send data

[1100] The device sends the exercise record and meal photo data collected from the user to the server via HTTPS protocol. The temporarily stored data is used as input, and the output is securely transmitted to the server via the network.

[1101] Step 3: Receiving and storing data

[1102] The server receives the exercise record and meal photo data sent from the device and stores them in a database (e.g., MySQL). It receives the data sent from the device as input and stores this data in the database as output.

[1103] Step 4: Data analysis

[1104] The server analyzes the received data. Specifically, it uses Google Cloud Vision API to perform image analysis on food photos and converts them into menu, nutrient, and calorie information. It also uses natural language processing tools to perform text analysis on exercise records. It uses data obtained from the database as input and obtains the analysis results as output.

[1105] Step 5: Create personalized advice

[1106] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, it would be good to eat a little less carbohydrates and more vegetables at dinner." The analysis results above are used as input, and advice is generated as output.

[1107] Step 6: Send advice

[1108] The server sends the generated advice to the device. The advice also includes metadata for managing read status and implementation status. The generated advice is used as input and sent to the device as output.

[1109] Step 7: Advice Notification

[1110] The device receives advice sent from the server and displays advice such as "walk 30 minutes tomorrow" or "reduce carbohydrates" to the user via push notification. The device receives advice data from the server as input and notifies the user as output.

[1111] Step 8: Gather feedback

[1112] The user acts based on the advice provided and reports the results to the application. For example, they may provide feedback such as, "I walked for 30 minutes today." The user provides the action data they took as input, and this data is temporarily saved on the device as output.

[1113] Step 9: Submit your feedback

[1114] The terminal receives feedback data from the user and transmits it to the server, using the feedback data from the user as input and transmission to the server as output.

[1115] Step 10: Feedback analysis and next-time advice adjustment

[1116] The server reflects the received feedback in the next advice generation, which allows advice to be continuously optimized. Feedback data is received as input, and data that will be reflected in the next advice is generated as output.

[1117] (Application example 1)

[1118] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1119] Conventional health management systems only required users to input data and receive advice, which lacked real-timeness and convenience. Furthermore, there was no mechanism for dynamically providing individual advice when selecting food in a physical store. This made it difficult for users to receive health support in a physical store.

[1120] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1121] In this invention, the server includes a means for receiving exercise records and meal photos, a means for performing food recognition and nutritional analysis, and a means for generating personalized advice based on the analysis results, thereby enabling real-time health advice in a physical store.

[1122] A "user" is an individual who uses this system and provides exercise records and dietary information.

[1123] The "exercise record" is data relating to the exercise performed by the user on a daily basis, and includes information such as distance, time, and type of exercise.

[1124] "Meal photos" are image data that record meals consumed by the user and are used for subsequent analysis.

[1125] The "receiving means" is a function that allows the terminal to receive exercise records and meal photos from the user.

[1126] A "terminal" is a smartphone, smart glasses, or other electronic device used by a user, and is a device for inputting data and receiving advice.

[1127] A "server" is a computer system that aggregates received data and performs analysis and advice generation.

[1128] The "means of analysis" refers to algorithms and software that analyze exercise records and meal photos on a server and extract nutrient and calorie information.

[1129] "Personalized advice" refers to specific health management instructions or recommendations provided to the user based on the analyzed data.

[1130] The "notification means" is a function that allows the terminal to notify the user of the advice that has been generated.

[1131] "Feedback" is information about the actions taken by the user based on the advice and the results of those actions, and is used to optimize future advice.

[1132] The "adjustment means" is a function that allows the server to revise and optimize the advice content from the next time onwards based on feedback from the user.

[1133] "Food recognition" is a technology that uses smart devices in physical stores to take pictures of products and identify their contents.

[1134] "Nutritional analysis" is the process of calculating and analyzing the various nutrients and calories contained in recognized foods.

[1135] "Real-time health advice" is a system that provides on-the-spot health guidance and recommendations when users are choosing food in a physical store.

[1136] A specific embodiment for realizing this invention will be described. This system has the function of collecting and analyzing exercise records and dietary information to provide individualized advice to support the user's health management. The main components of the system are a terminal, a server, and a generative AI model.

[1137] System configuration

[1138] Hardware:

[1139] Devices: Smartphones and smart glasses used by users. These devices input exercise records, take photos of meals, and display and notify advice.

[1140] Server: A computer system located in the cloud that analyzes the received data and generates advice.

[1141] Software used

[1142] software:

[1143] Terminal application: An application that allows users to input exercise records and dietary information and receive advice, which is displayed on the screen in real time.

[1144] Server-side software: Software that analyzes data, runs generative AI models, and incorporates feedback.

[1145] Image analysis algorithm: An algorithm that analyzes food photos to extract menu items, nutrients, and calories.

[1146] System Operation

[1147] 1. Data Collection:

[1148] Users use their smartphones or smart glasses to record their exercise and take photos of their meals. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[1149] 2. Data transmission:

[1150] The terminal transmits the entered exercise records and meal photos to the server.

[1151] 3. Data Analysis:

[1152] The server analyzes the received exercise records and meal photos. It uses an image analysis algorithm to analyze the meal photos and extract nutrient and calorie information. For example, the server uses image recognition technology to analyze a meal photo and obtain information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[1153] 4. Advice Generation:

[1154] The generative AI model on the server generates personalized healthcare advice based on the analyzed data. For example, it might say, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[1155] 5. Advice Notice:

[1156] The generated advice is sent to the device and notified to the user. For example, the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" is displayed to the user.

[1157] 6. Feedback received:

[1158] The user acts based on the advice provided and inputs the results into the device, for example, reporting, "I walked for 30 minutes today."

[1159] 7. Data Transmission and Coordination:

[1160] The device sends feedback to the server, which then adjusts the next advice based on the received feedback, allowing for continuous optimization of the advice.

[1161] Use in physical stores

[1162] Examples:

[1163] When a user scans chicken in a physical store, the AI ​​model analyzes it and determines that "this chicken is high in protein but low in fat," and based on the user's exercise history and health status, displays advice such as "This chicken is high in protein, so it fits into your weekly exercise plan. Eat it with a salad."

[1164] Example prompt sentence:

[1165] "User data: Jogging 3 times a week, high protein, low carb diet, good health. Analysis results: Chicken, high protein, low fat. Based on this information, please provide weekly exercise and meal plan suggestions."

[1166] The above is an embodiment of the present invention. This system effectively supports users in managing their health and significantly improves convenience when selecting food items in physical stores.

[1167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1168] Step 1:

[1169] Data collection and input

[1170] User: The user uses a smartphone or smart glasses to enter exercise records and take photos of meals. For example, "I ran 5km today" or "I had salad, chicken, and bread for dinner." This data is stored on the device and later sent to the server.

[1171] Step 2:

[1172] Data transmission

[1173] Device: The device sends the exercise records and meal photos entered by the user to the server. Here, HTTP requests are used as the protocol for sending input data. The input data includes exercise records and meal photos, and is sent to the server.

[1174] Step 3:

[1175] Data analysis

[1176] Server: The server analyzes the exercise records and meal photos it receives. It uses an image analysis algorithm on the server side to extract information about the menu, nutrients, and calories from the meal photos. The image analysis algorithm receives the photos as input and outputs the menu, nutrients, and calories. For example, it obtains information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[1177] Step 4:

[1178] Advice Generation

[1179] Server: The generative AI model on the server generates personalized healthcare advice based on the analyzed data. The generative AI model inputs the analysis results and past user data as prompts and outputs specific advice. For example, the generated advice might be, "Since you haven't exercised much today, we recommend walking for 30 minutes tomorrow. Also, it would be good to reduce carbohydrates a little and eat more vegetables at dinner."

[1180] Step 5:

[1181] Advice Notice

[1182] Device: The advice generated by the server is sent to the device. The device notifies the user of this advice and displays it to them. Possible notification methods include push notifications and in-app notifications. For example, the user might be notified of advice such as "walk 30 minutes tomorrow" and "reduce carbohydrate intake."

[1183] Step 6:

[1184] Feedback Input

[1185] User: The user acts based on the advice provided and inputs the results into the device. The input is feedback information such as "I walked for 30 minutes today." This information is later sent to the server.

[1186] Step 7:

[1187] Feedback submission and processing

[1188] Terminal: Collects feedback information and sends it to the server. Again, a communication protocol such as an HTTP request is used. The input is the feedback data, which is then sent to the server.

[1189] Server: Adjusts and optimizes the next advice based on the received feedback. The server takes past feedback information as input and uses a generative AI model to continuously optimize the advice, making health management more effective for the user.

[1190] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1191] This invention combines a system that collects a user's daily exercise record, health condition, and dietary information and uses a generative AI model to provide personalized healthcare advice, with an emotion engine. The system aims to provide more effective healthcare support by recognizing the user's emotions and adjusting advice based on those emotions.

[1192] System Overview

[1193] The system receives exercise records and meal photo data entered by the user on the device and sends it to the server. The server analyzes the received data and generates individualized advice based on the results. In doing so, it uses an emotion engine to analyze the user's emotional data and reflects it in the advice. The generated advice is sent to the device and notified to the user. In addition, user feedback is collected and used to generate the next advice, aiming for individualization and optimization.

[1194] Specific processing of the program

[1195] 1. Data Collection Phase

[1196] User

[1197] Users use the app to record their daily exercise and meal photos. The exercise records include walking distance, time, and calorie consumption, while the meal photos are images that show the specific meals they ate that day. In addition, data from fitness bands and smartwatches is also synced to the app.

[1198] Terminal

[1199] The device receives the exercise records and meal photos entered by the user, temporarily stores them in local storage, and prepares them to be sent to the server, where they are securely transmitted using the SSL / TLS protocol.

[1200] 2. Data analysis phase

[1201] server

[1202] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[1203] Emotion Engine

[1204] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. For example, if the user is feeling stressed, it detects their stress level.

[1205] 3. Individual advice creation phase

[1206] server

[1207] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[1208] 4. Advice sending phase

[1209] server

[1210] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1211] Terminal

[1212] The terminal receives the advice sent from the server, notifies the user, checks whether the advice has been read, and returns that information to the server.

[1213] 5. Feedback Phase

[1214] User

[1215] The user acts on the advice provided and provides feedback to the application, such as "I did 30 minutes of yoga" or "I ate more vegetables for dinner."

[1216] Terminal

[1217] The terminal receives feedback from the user and transmits it to the server.

[1218] server

[1219] The server receives feedback from the device and stores it in a database. Based on this feedback, the server adjusts the next advice it provides, allowing it to continually provide optimal advice tailored to the user's individual needs.

[1220] Example

[1221] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[1222] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[1223] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[1224] In this way, the present invention is a system that provides individual advice based on emotional data in addition to the user's exercise records, health status, and dietary information, thereby more effectively supporting the user's health management.

[1225] The processing flow will be explained below.

[1226] Step 1:

[1227] User

[1228] Users use the application to record their daily exercise and meal photos. The exercise records include distance traveled, time, and calorie consumption, and the meal photos are taken to show what they ate that day.

[1229] Step 2:

[1230] Terminal

[1231] The device receives the exercise records and meal photos entered by the user and temporarily stores them in local storage, preparing to send the stored data to the server.

[1232] Step 3:

[1233] Terminal

[1234] The device converts the organized data into packets and sends them securely to the server using the SSL / TLS protocol. If the data transmission was successful, the device notifies the user.

[1235] Step 4:

[1236] server

[1237] The server receives the exercise records and meal photos sent from the device, stores them in a database, and then processes the data for analysis.

[1238] Step 5:

[1239] server

[1240] The server analyzes the received data. It uses image recognition technology to analyze food photos and extract menu, nutrient, and calorie information. It also analyzes exercise records to calculate the user's calorie consumption and exercise performance.

[1241] Step 6:

[1242] Emotion Engine

[1243] The server uses an emotion engine to analyze the user's emotions, taking in the user's voice, text, and facial expression data to determine their emotional state, such as stress or happiness.

[1244] Step 7:

[1245] server

[1246] The generative AI model on the server generates personalized healthcare advice for each user based on the analyzed exercise records, dietary data, and emotional data. For example, it generates specific advice such as, "Due to lack of exercise, walk 30 minutes tomorrow. Due to high stress, we recommend doing yoga to relax."

[1247] Step 8:

[1248] server

[1249] The server sends the generated advice and associated metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1250] Step 9:

[1251] Terminal

[1252] The device notifies the user of the advice and metadata received from the server, and the user can check the content of the advice by opening the notification.

[1253] Step 10:

[1254] User

[1255] The user then takes action based on the advice provided, for example, following instructions such as "do 30 minutes of yoga and eat more vegetables for dinner."

[1256] Step 11:

[1257] User

[1258] The user provides feedback to the application about the results of their efforts, such as "I did 30 minutes of yoga today" or "I ate a lot of vegetables for dinner."

[1259] Step 12:

[1260] Terminal

[1261] The terminal receives the feedback from the user and transmits it to the server, and if the transmission is successful, notifies the user.

[1262] Step 13:

[1263] server

[1264] The server receives the feedback sent from the device and stores it in a database, which allows it to take the feedback into account the next time it generates advice and provide individually optimized advice.

[1265] Example

[1266] For example, a user eats "chicken, salad, and bread" for dinner, uploads a photo of it to the app, and records "3km jogging" as an exercise record. This information is sent from the device to the server, which analyzes it. From the photo of the meal, the server identifies "chicken, salad, and bread" and calculates the calories of each. The emotion engine also analyzes the user's voice data and determines that the user is "highly stressed."

[1267] Based on this data, the generative AI model generates specific advice such as "Tomorrow, we recommend 30 minutes of yoga to relax. Eat more vegetables at dinner." The advice is sent to the device and notified to the user. The next day, the user acts on the advice and enters the results into the app. Feedback is sent to the server and reflected in the next advice generation.

[1268] Thus, the present invention is a system that effectively supports the user's health management by providing individualized advice based on the user's exercise records, health status, dietary information, and emotional data.

[1269] Example 2

[1270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1271] In recent years, interest in lifestyle-related diseases and health management has increased, but there are still limited systems that provide appropriate advice based on individual health and emotional states. Current systems often only collect exercise and dietary data, and do not provide advice that takes into account the emotions of individual users. As a result, there is a challenge in that it is difficult for users to actually achieve effective and sustainable health management.

[1272] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1273] In this invention, the server includes a means for analyzing exercise records and meal photos, a means for generating personalized advice based on the analyzed data and the user's emotional data, and a means for adjusting the next advice based on the received feedback, thereby enabling the provision of more personalized healthcare advice based on multifaceted data such as the user's exercise records, meal information, and even emotional state.

[1274] "Exercise records" are data containing information about the user's daily exercise, such as walking distance, exercise time, and calories burned.

[1275] "Meal photos" are image data of daily meals taken by a user, and include information indicating the type and amount of food and details of ingredients.

[1276] "Terminal" refers to an electronic device such as a smartphone or tablet used by a user, which inputs and sends / receives data through an application.

[1277] "Server" refers to a computer system on which a computing device and software for centrally managing data, performing analysis, and executing necessary processing operates.

[1278] "Analysis" refers to the process in which the server analyzes the exercise records and meal photos received and extracts specific information (e.g., calories, nutrients, etc.).

[1279] A "generative AI model" refers to artificial intelligence technology that automatically creates optimal advice for users based on analyzed data.

[1280] "Emotion data" refers to data that indicates the emotional state of a user, and refers to the analysis results of the emotional state obtained by analyzing information such as voice, text, and facial expressions.

[1281] "Feedback" refers to data that reports to the system the results and impressions of users who follow the advice provided.

[1282] "Personalized advice" refers to optimal health management advice for a user, created by a generative AI model based on the user's specific exercise records, dietary information, and emotional data.

[1283] MODE FOR CARRYING OUT THE INVENTION

[1284] This invention is a system that collects a user's daily exercise record, dietary information, health condition, and emotional data, and provides personalized healthcare advice based on these data. The system aims to realize more effective and personalized healthcare support by recognizing the user's emotions and adjusting the advice content based on those emotions.

[1285] Hardware and software used

[1286] Devices: smartphones, tablets, etc.

[1287] Server: A central system for database management and analysis.

[1288] Software: Applications, generative AI models, data analysis engines, emotion engines

[1289] Specific actions

[1290] User

[1291] Through the application, users can input their daily exercise records and photos of their meals. Exercise records include walking distance, exercise time, and calorie consumption, while meal photos are images that specifically show what they ate that day. Data from fitness bands and smartwatches is also synced to the application. For example, users can input information such as "I ran 3km today" or "I ate chicken, salad, and bread."

[1292] Terminal

[1293] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When the data send button is pressed, the data is securely sent to the server using SSL / TLS protocol. At this stage, the data is encrypted to minimize the risk of leakage.

[1294] server

[1295] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise records and meal photo data. It uses image recognition technology to extract specific menu items, nutrients, and calorie information from the meal photos. For example, it can determine that "a dinner of chicken, salad, and bread contains approximately 700 kcal."

[1296] Emotion Engine

[1297] The emotion engine analyzes the user's voice, text, and facial expression data to identify their emotional state. For example, it may determine that the user is likely feeling stressed.

[1298] Generative AI Models

[1299] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it provides specific advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend that they do 30 minutes of yoga to relax," or "We recommend that they increase the amount of vegetables they eat at dinner to maintain a balanced diet."

[1300] Terminal

[1301] The generated advice is sent from the server to the device and notified to the user via push notification or in-app notification. After the user confirms the advice, its read status is sent back to the server.

[1302] User

[1303] The user acts according to the advice provided and provides feedback to the application with the results. For example, the user might enter information such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner." This feedback is then sent to the server and used to generate advice for the next time.

[1304] Specific examples

[1305] A user uploads a photo of "chicken, salad, and bread" to the app after dinner and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calorie count is approximately 700kcal." Next, an emotion engine analyzes the user's emotions from voice and text messages and determines that "the stress level is high." Based on this information, the generative AI model generates advice such as "I recommend 30 minutes of yoga tomorrow to relax, and I also recommend increasing the amount of vegetables you eat at dinner."

[1306] This advice is sent to the device and notified to the user. The next day, the user acts according to the advice and enters the result into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[1307] Prompt Sentence Examples

[1308] "Please provide personalized advice based on the user's exercise record, dietary content, and emotional data. The user ran 3km today and had chicken, salad, and bread for dinner. Emotion analysis has determined that the user's stress level is high. Please create appropriate healthcare advice based on this."

[1309] In this way, the present invention provides individualized advice based on the user's exercise record, dietary information, health condition, and emotional data, thereby more effectively supporting the user's health management.

[1310] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1311] Step 1: Enter your data

[1312] User

[1313] Through the application, users can input photos of their daily exercise records and meals. For example, they can input information such as "I ran 3km today" or "I ate chicken, salad, and bread." Exercise data from fitness bands and smartwatches is also automatically synchronized. The input data is temporarily stored in the device's local storage.

[1314] Input: Exercise records (text), meal photos (images), wearable device data (calories, steps, etc.)

[1315] Output: Exercise and diet data stored in local storage

[1316] Step 2: Sending data

[1317] Terminal

[1318] The saved exercise records and meal photo data are sent to the server. When sending the data, the SSL / TLS protocol is used to encrypt the data and ensure secure communication. Once the data has been sent to the server, it is deleted from local storage.

[1319] Input: Locally stored exercise and food data

[1320] Output: Data sent to the server

[1321] Step 3: Receiving and storing data

[1322] server

[1323] The server receives exercise records and meal photo data sent from the device. The received data is stored in a database and organized using the user ID as a key. If necessary, filtering is performed to standardize the data format.

[1324] Input: Exercise and diet data sent from the device

[1325] Output: Data to be analyzed stored in a database

[1326] Step 4: Analyze the data

[1327] server

[1328] The server analyzes the received data. It uses image recognition technology to extract menu, nutrient, and calorie information from meal photos. It also calculates calories burned and exercise volume from exercise record data. For example, it might determine that a dinner of chicken, salad, and bread contains 700 kcal.

[1329] Input: Exercise and diet data stored in a database

[1330] Output: Parsed menu, nutritional and calorie information and exercise data

[1331] Step 5: Analyze the sentiment data

[1332] Emotion Engine

[1333] The emotion engine analyzes the user's voice, text, and facial expressions to identify the user's emotional state. For example, it may determine that the user is likely feeling stressed.

[1334] Input: User voice, text, and facial expression data

[1335] Output: Identified emotional state (e.g., high stress level)

[1336] Step 6: Generate personalized advice

[1337] server

[1338] The generative AI model generates personalized healthcare advice based on the analyzed exercise, diet, and emotional data, such as "recommended 30 minutes of yoga to relax" or "recommended to increase the amount of vegetables at dinner to maintain a balanced diet."

[1339] Input: Analyzed exercise data, diet data, emotion data

[1340] Output: personalized advice

[1341] Step 7: Submitting Advice

[1342] server

[1343] The server transmits the generated advice and its metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1344] Input: Generated advice and metadata

[1345] Output: Advice sent to terminal

[1346] Step 8: Notification of advice and confirmation of read receipt

[1347] Terminal

[1348] The device receives the advice and notifies the user via push notification or in-app notification. After the user checks the advice, the device sends back the read status to the server.

[1349] Input: Advice sent by the server

[1350] Output: Advice sent to the user, read status sent back to the server

[1351] Step 9: Enter and submit your feedback

[1352] User

[1353] The user acts on the advice provided and provides feedback to the application with the results, such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner."

[1354] Input: User feedback (results of actions)

[1355] Output: Feedback data

[1356] Terminal

[1357] The device receives the feedback data and transmits it securely to the server using the SSL / TLS protocol.

[1358] Input: User feedback

[1359] Output: Feedback sent to the server

[1360] Step 10: Analyze the feedback and incorporate it into your next advice

[1361] server

[1362] The server analyzes the feedback received from the user and stores it in a database. The AI ​​model adjusts the next advice based on the feedback. For example, it generates appropriate advice based on information such as, "I followed the previous advice and practiced yoga to relax, and as a result, my stress level decreased."

[1363] Input: Feedback received from the user

[1364] Output: Adjustment information to be reflected in the next advice

[1365] (Application example 2)

[1366] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1367] Conventional healthcare support systems primarily rely on data such as exercise records and meal photos, and do not take into account the user's emotional state. This poses a challenge: they are unable to provide appropriate advice for health problems that are influenced by the user's emotions. Furthermore, because the advice provided is uniform, it is difficult to provide optimal support tailored to the needs of individual users.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1369] In this invention, the server includes means for collecting the user's emotional state and analyzing it using an emotion engine, means for adjusting advice based on the emotional data, and means for analyzing exercise records and meal photos and generating personalized advice based on the analysis results. This enables personalized advice that takes the user's emotions into consideration, making it possible to more effectively support the user's health management.

[1370] The "exercise record" is data on the exercise the user performs each day, including the number of steps taken, distance traveled, calorie consumption, and the like.

[1371] "Meal photos" are data in which the user records the meals they have eaten as photographs.

[1372] "Server" refers to a computer system that receives and analyzes data sent by users and generates personalized advice.

[1373] "Terminal" refers to a device used by a user, such as a smartphone or computer, that has the means to send exercise records and meal photos, and receive and display advice.

[1374] The "emotional state" indicates the mental state of the user and includes emotional categories such as positive, negative, and neutral.

[1375] An "emotion engine" refers to a software or hardware system that analyzes emotions from a user's voice, text, facial expressions, etc., and identifies their emotional state.

[1376] "Analysis" refers to the process of processing received data and extracting and calculating specific information.

[1377] "Advice" is information that serves as a guide for the user's health management and behavior, generated based on the analysis results and emotion data.

[1378] "Feedback" is data that a user inputs into an application, including the results and impressions of actions taken based on the advice provided.

[1379] "Personalization" refers to the process of customizing advice content to each user's specific needs and circumstances.

[1380] To implement this invention, a system must be built that uses a user's smartphone or other device, a server, an emotion engine, and a generative AI model. This system collects the user's daily exercise records and meal photos, and provides personalized health advice based on the analysis results. It can also recognize the user's emotional state and adjust advice accordingly.

[1381] Program Description

[1382] The program of this system is configured as follows:

[1383] 1. Data Collection

[1384] Users use a smartphone application to record their daily exercise and meal photos. The exercise records include the number of steps taken, distance traveled, and calories burned, while the meal photos are images that show the specific meals eaten that day. In addition, data from fitness bands and smartwatches is also synchronized with the application.

[1385] 2. Data Transmission

[1386] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When sending this data to the server, it is securely transmitted using the SSL / TLS protocol.

[1387] 3. Data Analysis

[1388] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[1389] 4. Emotion analysis

[1390] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. Specifically, it uses voice recognition technology to convert voice into text and analyzes the emotion in that text. For example, if the user is feeling stressed, it detects their stress level.

[1391] 5. Generating personalized advice

[1392] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[1393] 6. Advice Notice

[1394] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the device. The device receives the advice sent from the server and notifies the user. The device checks whether the advice has been read and sends that information back to the server.

[1395] 7. Feedback Collection

[1396] The user acts based on the advice provided and provides feedback to the application with the results. For example, the user may input feedback such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner." The device receives the feedback from the user and sends it to the server. The server receives the feedback sent from the device and stores it in a database. The next advice is adjusted based on this feedback.

[1397] Specific examples

[1398] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[1399] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[1400] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[1401] Examples of prompts include:

[1402] The user exercises less. Their stress level is high, so recommend 30 minutes of yoga to relax. Recommend adding more vegetables to dinner.

[1403] As a result, the present invention can provide individual advice based on the user's exercise record, health condition, dietary information, and emotional data, thereby more effectively supporting the user's health management.

[1404] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1405] Step 1:

[1406] Users launch a smartphone application and input their daily exercise records and meal photos. The input exercise records include the number of steps taken, distance traveled, and calorie consumption, and the meal photos are images showing the meals eaten that day. The input data is temporarily saved in local storage.

[1407] Input: Exercise record data, meal photos

[1408] Output: Exercise record data and meal photos saved in local storage

[1409] Step 2:

[1410] The device then sends the received exercise record data and meal photos to a server. Data transmission is secure using the SSL / TLS protocol.

[1411] Input: Exercise record data and meal photos stored in local storage

[1412] Output: Exercise record data and meal photos sent to the server

[1413] Step 3:

[1414] The server stores the exercise record data and meal photos received from the device in a database. It then uses image recognition technology to analyze the meal photos and extract information about the menu, nutrients, and calories. It also calculates calories burned and the amount of exercise from the exercise record data.

[1415] Input: Exercise record data and meal photos sent to the server

[1416] Output: Analysis results (menu, nutrients, calorie information, calories burned, exercise amount)

[1417] Step 4:

[1418] The server collects the user's emotional state from voice and text and performs analysis using an emotion engine. For example, it analyzes text messages and voice data entered by the user within the application and classifies them into emotional categories such as positive, negative, and neutral.

[1419] Input: Audio data, text data

[1420] Output: Parsed emotional state data

[1421] Step 5:

[1422] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it might generate advice like, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[1423] Input: Analysis results (exercise records, dietary data, emotional data)

[1424] Output: Generated healthcare advice

[1425] Step 6:

[1426] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1427] Input: Generated healthcare advice, metadata

[1428] Output: Advice and metadata sent to the device

[1429] Step 7:

[1430] The device receives the advice sent from the server and notifies the user. The user checks the notification and acts based on the advice. The device returns the advice read / implemented status to the server.

[1431] Input: Advice and metadata sent to the device

[1432] Output: Notification to user, read / implemented status returned to server

[1433] Step 8:

[1434] The user acts according to the advice provided and inputs the results into the application as feedback, such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner," and sends this information from the device to the server.

[1435] Input: User feedback data

[1436] Output: Feedback data sent to the server

[1437] Step 9:

[1438] The server receives the feedback sent from the device and stores it in a database, which then adjusts the next advice based on the feedback and provides the most appropriate advice to meet the user's individual needs.

[1439] Input: Feedback data sent to the server

[1440] Output: Adjusted next advice

[1441] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1442] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1443] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1444] [Fourth embodiment]

[1445] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1446] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1447] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1448] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1449] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1451] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1452] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1453] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1454] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1455] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1456] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1457] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1458] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[1459] System Overview

[1460] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[1461] Specific processing of the program

[1462] 1. Data Collection Phase

[1463] User

[1464] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[1465] Terminal

[1466] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[1467] 2. Data analysis phase

[1468] server

[1469] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[1470] 3. Individual advice creation phase

[1471] server

[1472] The generative AI model on the server creates personalized healthcare advice based on the analyzed data. For example, it might generate specific advice such as, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[1473] 4. Advice sending phase

[1474] server

[1475] The server sends the generated advice to the device, along with metadata for managing the advice's read status and implementation status.

[1476] Terminal

[1477] The device notifies the user of the advice sent from the server. It displays the advice "walk for 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[1478] 5. Feedback Phase

[1479] User

[1480] The user acts based on the advice provided and inputs the results into the application, for example, reporting, "I walked for 30 minutes today."

[1481] Terminal

[1482] The terminal receives feedback from the user and transmits it to the server.

[1483] server

[1484] The server uses the received feedback to generate the next piece of advice, allowing the advice to be continuously optimized.

[1485] Example

[1486] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "eat less carbohydrates and more vegetables tomorrow. It is recommended that you jog for at least 30 minutes." The generated advice is sent to the user's device and notified to the user.

[1487] The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[1488] As described above, the present invention is a system that provides individualized advice based on a user's exercise records, health status, and dietary information, and effectively supports the user's health management.

[1489] The processing flow will be explained below.

[1490] Step 1:

[1491] User

[1492] Users use the application to enter photos of their daily exercise records and meals. Exercise records include walking distance, time, calorie consumption, etc. Meal photos are images that show the details of the meals they ate that day.

[1493] Step 2:

[1494] Terminal

[1495] The device receives user-entered exercise records and meal photos, organizes them according to format, and temporarily stores them in local storage. The device also syncs data from devices such as fitness bands and smartwatches.

[1496] Step 3:

[1497] Terminal

[1498] The device generates packets to send the organized data to the server, securely transmitting it using the SSL / TLS protocol, and displays a notification to the user if the data transmission is successful.

[1499] Step 4:

[1500] server

[1501] The server receives the exercise records and meal photos sent from the device, and the received data is stored in a database for use in the next analysis step.

[1502] Step 5:

[1503] server

[1504] The server analyzes exercise records and meal photos. For meal photos, image recognition technology is used to extract information on ingredients, menu items, nutrients, and calories. For exercise records, the server calculates the user's calorie expenditure and exercise volume from the data.

[1505] Step 6:

[1506] server

[1507] The server uses the generative AI model based on the analysis results to generate personalized healthcare advice for the user. For example, if the user's exercise volume is low, the server generates advice recommending additional exercise. If the user's diet is unbalanced, the server recommends a balanced diet.

[1508] Step 7:

[1509] server

[1510] The server converts the generated advice into packets and sends them to the device, along with metadata for tracking whether the advice has been read or implemented.

[1511] Step 8:

[1512] Terminal

[1513] The device receives the advice packet sent from the server and notifies the user of its contents. The user can check the advice by opening the notification.

[1514] Step 9:

[1515] User

[1516] The user acts according to the advice. For example, if a 30-minute jog is recommended, the user performs the exercise. Furthermore, the user also improves their diet based on the advice.

[1517] Step 10:

[1518] User

[1519] The user provides feedback to the application about the results of their actions, such as "I went jogging for 30 minutes" or "I ate a lot of vegetables at dinner."

[1520] Step 11:

[1521] Terminal

[1522] The terminal receives the feedback from the user and transmits it to the server, and notifies the user if the transmission is successful.

[1523] Step 12:

[1524] server

[1525] The server receives the feedback sent from the device, stores it in a database, and takes the feedback into account when generating the next advice, allowing for continuous optimization of personalized advice.

[1526] Example 1

[1527] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1528] As modern lifestyles become more diverse, it is becoming increasingly difficult to provide individually optimized healthcare advice. Existing systems often require cumbersome data collection and analysis from users, resulting in uniform and suboptimal advice. Furthermore, insufficient feedback collection, necessary to verify the effectiveness of advice, reduces the accuracy and effectiveness of the advice. To address this issue, it is necessary to effectively collect and analyze data from users, provide individually tailored advice, and further optimize the advice by incorporating user feedback.

[1529] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1530] In this invention, the server includes means for receiving exercise records and meal photos from a user, means for transmitting the received data to a network device, means for analyzing the exercise records and meal photos in the network device and generating personalized advice based on the analysis results, means for transmitting the generated advice to a terminal, means for notifying the user of the sent advice in the terminal, means for receiving feedback from the user and transmitting it to the network device, means for adjusting next advice based on the received feedback in the network device, and means for generating personalized advice using a generative AI model, thereby enabling the provision of healthcare advice optimized for each user.

[1531] "User" refers to an individual who uses the system to input exercise records, health status, and dietary information and receive healthcare advice.

[1532] "Exercise record" refers to information including the type of exercise performed by the user, the time, distance, calories burned, etc.

[1533] "Meal photos" refer to digital images that are photographs of the meals a user has eaten.

[1534] "Network device" refers to a computer system that has the capability to analyze data received from a user terminal and generate and send personalized advice.

[1535] "Generative AI Model" refers to the artificial intelligence algorithm used to generate personalized healthcare advice based on received and analyzed data.

[1536] "Feedback" refers to information including the results of a user's actions and evaluation of the advice provided.

[1537] "Advice" refers to health management instructions and recommendations created by a generative AI model based on a user's exercise records and dietary information.

[1538] "Analysis" refers to the act of performing calculations and processing on the received exercise record and meal photo data to extract useful information and patterns.

[1539] MODE FOR CARRYING OUT THE INVENTION

[1540] This invention is a system that collects a user's daily exercise records, health status, and dietary information and uses a generative AI model to provide personalized healthcare advice.

[1541] System Overview

[1542] The system receives exercise records and meal photos entered by the user on the device and sends the data to the server. The server analyzes the received data and generates personalized advice based on the results. This advice is sent to the device and notified to the user. The user acts based on the advice and enters the results into the device. The device sends this feedback to the server, which then adjusts the next advice based on the feedback.

[1543] Specific hardware and software

[1544] User device:

[1545] The user uses a smartphone, tablet, or other device. An application is installed on the device to input and save exercise records and meal photos. This application also works in conjunction with a fitness band or smartwatch to collect data.

[1546] server:

[1547] A server responsible for analyzing the data and generating healthcare advice. The server has a database (e.g., MySQL) and an image analysis tool (e.g., Google Cloud Vision API) installed. It also requires computing resources to run the generative AI model (e.g., ChatGPT).

[1548] Specific processing of the program

[1549] Data collection phase:

[1550] Users enter their daily exercise records into the application. They also take photos of their meals and upload them to the application. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[1551] Data transmission phase:

[1552] The device receives input data from users and temporarily stores it before sending it to the server, and also syncs data from fitness bands and smartwatches.

[1553] Data analysis phase:

[1554] The server receives and analyzes the data sent from the device. For example, the server uses image recognition technology (e.g., Google Cloud Vision API) to analyze a photo of a meal and convert it into menu, nutrient, and calorie information. This allows it to extract information such as, "Dinner is about 600 kcal, and is particularly high in carbohydrates."

[1555] Personalized advice creation phase:

[1556] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, reduce the carbohydrates in your dinner a little and eat more vegetables."

[1557] Advice sending phase:

[1558] The server sends the generated advice to the device. It also sends metadata to manage whether the advice has been read or implemented. The device notifies the user of the advice sent from the server. The device displays the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" to the user and checks whether the notification has been read.

[1559] Feedback Phase:

[1560] The user acts based on the advice provided and inputs the results into the application. For example, the user reports, "I walked for 30 minutes today." The device receives the feedback from the user and sends it to the server. The server uses the received feedback to generate the next piece of advice. This allows the advice to be continuously optimized.

[1561] Examples of specific examples and prompts

[1562] As an example, a user uploads a photo of themselves eating "chicken, salad, and bread" after dinner to the app, and enters "3km jogging" as their exercise record for that day. This information is sent from the device to the server, which analyzes it. The server analyzes the data and determines that "dinner had about 700 kcal and a high carbohydrate intake," and generates advice such as "Tomorrow, eat less carbohydrates and more vegetables. It is recommended that you jog for at least 30 minutes."

[1563] The generated advice is sent to the user's device and notified. The next day, the user acts according to the advice provided, and as a result, enters feedback into the app, such as "I went for a 30-minute jog and ate more vegetables for dinner." This feedback is sent from the device to the server, and the server uses this information to adjust the next advice.

[1564] Example prompt sentence:

[1565] User:

[1566] "I ran 5km today. I had salad, chicken, and bread for dinner."

[1567] server:

[1568] "The calorie analysis of your meal shows that it is approximately 600 kcal. Taking into consideration the amount of exercise required, we recommend that you walk for 30 minutes tomorrow. It would also be a good idea to reduce your carbohydrate intake at dinner and eat more vegetables."

[1569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1570] Step 1: Data collection

[1571] Users enter their daily exercise records and meal photos through the application. Specifically, users send text and images such as "I ran 5km today" or "I had salad, chicken, and bread for dinner" to the app. Exercise record data and meal photo data are provided as input, and these data are temporarily saved on the device as output.

[1572] Step 2: Send data

[1573] The device sends the exercise record and meal photo data collected from the user to the server via HTTPS protocol. The temporarily stored data is used as input, and the output is securely transmitted to the server via the network.

[1574] Step 3: Receiving and storing data

[1575] The server receives the exercise record and meal photo data sent from the device and stores them in a database (e.g., MySQL). It receives the data sent from the device as input and stores this data in the database as output.

[1576] Step 4: Data analysis

[1577] The server analyzes the received data. Specifically, it uses Google Cloud Vision API to perform image analysis on food photos and converts them into menu, nutrient, and calorie information. It also uses natural language processing tools to perform text analysis on exercise records. It uses data obtained from the database as input and obtains the analysis results as output.

[1578] Step 5: Create personalized advice

[1579] A generative AI model (e.g., ChatGPT) on the server creates personalized healthcare advice based on the analyzed data. For example, specific advice might be generated such as, "You haven't been exercising much today, so we recommend walking for 30 minutes tomorrow. Also, it would be good to eat a little less carbohydrates and more vegetables at dinner." The analysis results above are used as input, and advice is generated as output.

[1580] Step 6: Send advice

[1581] The server sends the generated advice to the device. The advice also includes metadata for managing read status and implementation status. The generated advice is used as input and sent to the device as output.

[1582] Step 7: Advice Notification

[1583] The device receives advice sent from the server and displays advice such as "walk 30 minutes tomorrow" or "reduce carbohydrates" to the user via push notification. The device receives advice data from the server as input and notifies the user as output.

[1584] Step 8: Gather feedback

[1585] The user acts based on the advice provided and reports the results to the application. For example, they may provide feedback such as, "I walked for 30 minutes today." The user provides the action data they took as input, and this data is temporarily saved on the device as output.

[1586] Step 9: Submit your feedback

[1587] The terminal receives feedback data from the user and transmits it to the server, using the feedback data from the user as input and transmission to the server as output.

[1588] Step 10: Feedback analysis and next-time advice adjustment

[1589] The server reflects the received feedback in the next advice generation, which allows advice to be continuously optimized. Feedback data is received as input, and data that will be reflected in the next advice is generated as output.

[1590] (Application example 1)

[1591] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1592] Conventional health management systems only required users to input data and receive advice, which lacked real-timeness and convenience. Furthermore, there was no mechanism for dynamically providing individual advice when selecting food in a physical store. This made it difficult for users to receive health support in a physical store.

[1593] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1594] In this invention, the server includes a means for receiving exercise records and meal photos, a means for performing food recognition and nutritional analysis, and a means for generating personalized advice based on the analysis results, thereby enabling real-time health advice in a physical store.

[1595] A "user" is an individual who uses this system and provides exercise records and dietary information.

[1596] The "exercise record" is data relating to the exercise performed by the user on a daily basis, and includes information such as distance, time, and type of exercise.

[1597] "Meal photos" are image data that record meals consumed by the user and are used for subsequent analysis.

[1598] The "receiving means" is a function that allows the terminal to receive exercise records and meal photos from the user.

[1599] A "terminal" is a smartphone, smart glasses, or other electronic device used by a user, and is a device for inputting data and receiving advice.

[1600] A "server" is a computer system that aggregates received data and performs analysis and advice generation.

[1601] The "means of analysis" refers to algorithms and software that analyze exercise records and meal photos on a server and extract nutrient and calorie information.

[1602] "Personalized advice" refers to specific health management instructions or recommendations provided to the user based on the analyzed data.

[1603] The "notification means" is a function that allows the terminal to notify the user of the advice that has been generated.

[1604] "Feedback" is information about the actions taken by the user based on the advice and the results of those actions, and is used to optimize future advice.

[1605] The "adjustment means" is a function that allows the server to revise and optimize the advice content from the next time onwards based on feedback from the user.

[1606] "Food recognition" is a technology that uses smart devices in physical stores to take pictures of products and identify their contents.

[1607] "Nutritional analysis" is the process of calculating and analyzing the various nutrients and calories contained in recognized foods.

[1608] "Real-time health advice" is a system that provides on-the-spot health guidance and recommendations when users are choosing food in a physical store.

[1609] A specific embodiment for realizing this invention will be described. This system has the function of collecting and analyzing exercise records and dietary information to provide individualized advice to support the user's health management. The main components of the system are a terminal, a server, and a generative AI model.

[1610] System configuration

[1611] Hardware:

[1612] Devices: Smartphones and smart glasses used by users. These devices input exercise records, take photos of meals, and display and notify advice.

[1613] Server: A computer system located in the cloud that analyzes the received data and generates advice.

[1614] Software used

[1615] software:

[1616] Terminal application: An application that allows users to input exercise records and dietary information and receive advice, which is displayed on the screen in real time.

[1617] Server-side software: Software that analyzes data, runs generative AI models, and incorporates feedback.

[1618] Image analysis algorithm: An algorithm that analyzes food photos to extract menu items, nutrients, and calories.

[1619] System Operation

[1620] 1. Data Collection:

[1621] Users use their smartphones or smart glasses to record their exercise and take photos of their meals. For example, a user might enter information such as "I ran 5km today" or "I had salad, chicken, and bread for dinner."

[1622] 2. Data transmission:

[1623] The terminal transmits the entered exercise records and meal photos to the server.

[1624] 3. Data Analysis:

[1625] The server analyzes the received exercise records and meal photos. It uses an image analysis algorithm to analyze the meal photos and extract nutrient and calorie information. For example, the server uses image recognition technology to analyze a meal photo and obtain information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[1626] 4. Advice Generation:

[1627] The generative AI model on the server generates personalized healthcare advice based on the analyzed data. For example, it might say, "You haven't been exercising much today, so we recommend walking 30 minutes tomorrow. Also, cut down on carbohydrates a little and eat more vegetables at dinner."

[1628] 5. Advice Notice:

[1629] The generated advice is sent to the device and notified to the user. For example, the advice "walk 30 minutes tomorrow" and "reduce carbohydrates" is displayed to the user.

[1630] 6. Feedback received:

[1631] The user acts based on the advice provided and inputs the results into the device, for example, reporting, "I walked for 30 minutes today."

[1632] 7. Data Transmission and Coordination:

[1633] The device sends feedback to the server, which then adjusts the next advice based on the received feedback, allowing for continuous optimization of the advice.

[1634] Use in physical stores

[1635] Examples:

[1636] When a user scans chicken in a physical store, the AI ​​model analyzes it and determines that "this chicken is high in protein but low in fat," and based on the user's exercise history and health status, displays advice such as "This chicken is high in protein, so it fits into your weekly exercise plan. Eat it with a salad."

[1637] Example prompt sentence:

[1638] "User data: Jogging 3 times a week, high protein, low carb diet, good health. Analysis results: Chicken, high protein, low fat. Based on this information, please provide weekly exercise and meal plan suggestions."

[1639] The above is an embodiment of the present invention. This system effectively supports users in managing their health and significantly improves convenience when selecting food items in physical stores.

[1640] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1641] Step 1:

[1642] Data collection and input

[1643] User: The user uses a smartphone or smart glasses to enter exercise records and take photos of meals. For example, "I ran 5km today" or "I had salad, chicken, and bread for dinner." This data is stored on the device and later sent to the server.

[1644] Step 2:

[1645] Data transmission

[1646] Device: The device sends the exercise records and meal photos entered by the user to the server. Here, HTTP requests are used as the protocol for sending input data. The input data includes exercise records and meal photos, and is sent to the server.

[1647] Step 3:

[1648] Data analysis

[1649] Server: The server analyzes the exercise records and meal photos it receives. It uses an image analysis algorithm on the server side to extract information about the menu, nutrients, and calories from the meal photos. The image analysis algorithm receives the photos as input and outputs the menu, nutrients, and calories. For example, it obtains information such as "dinner is about 600 kcal, and is particularly high in carbohydrates."

[1650] Step 4:

[1651] Advice Generation

[1652] Server: The generative AI model on the server generates personalized healthcare advice based on the analyzed data. The generative AI model inputs the analysis results and past user data as prompts and outputs specific advice. For example, the generated advice might be, "Since you haven't exercised much today, we recommend walking for 30 minutes tomorrow. Also, it would be good to reduce carbohydrates a little and eat more vegetables at dinner."

[1653] Step 5:

[1654] Advice Notice

[1655] Device: The advice generated by the server is sent to the device. The device notifies the user of this advice and displays it to them. Possible notification methods include push notifications and in-app notifications. For example, the user might be notified of advice such as "walk 30 minutes tomorrow" and "reduce carbohydrate intake."

[1656] Step 6:

[1657] Feedback Input

[1658] User: The user acts based on the advice provided and inputs the results into the device. The input is feedback information such as "I walked for 30 minutes today." This information is later sent to the server.

[1659] Step 7:

[1660] Feedback submission and processing

[1661] Terminal: Collects feedback information and sends it to the server. Again, a communication protocol such as an HTTP request is used. The input is the feedback data, which is then sent to the server.

[1662] Server: Adjusts and optimizes the next advice based on the received feedback. The server takes past feedback information as input and uses a generative AI model to continuously optimize the advice, making health management more effective for the user.

[1663] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1664] This invention combines a system that collects a user's daily exercise record, health condition, and dietary information and uses a generative AI model to provide personalized healthcare advice, with an emotion engine. The system aims to provide more effective healthcare support by recognizing the user's emotions and adjusting advice based on those emotions.

[1665] System Overview

[1666] The system receives exercise records and meal photo data entered by the user on the device and sends it to the server. The server analyzes the received data and generates individualized advice based on the results. In doing so, it uses an emotion engine to analyze the user's emotional data and reflects it in the advice. The generated advice is sent to the device and notified to the user. In addition, user feedback is collected and used to generate the next advice, aiming for individualization and optimization.

[1667] Specific processing of the program

[1668] 1. Data Collection Phase

[1669] User

[1670] Users use the app to record their daily exercise and meal photos. The exercise records include walking distance, time, and calorie consumption, while the meal photos are images that show the specific meals they ate that day. In addition, data from fitness bands and smartwatches is also synced to the app.

[1671] Terminal

[1672] The device receives the exercise records and meal photos entered by the user, temporarily stores them in local storage, and prepares them to be sent to the server, where they are securely transmitted using the SSL / TLS protocol.

[1673] 2. Data analysis phase

[1674] server

[1675] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[1676] Emotion Engine

[1677] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. For example, if the user is feeling stressed, it detects their stress level.

[1678] 3. Individual advice creation phase

[1679] server

[1680] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[1681] 4. Advice sending phase

[1682] server

[1683] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1684] Terminal

[1685] The terminal receives the advice sent from the server, notifies the user, checks whether the advice has been read, and returns that information to the server.

[1686] 5. Feedback Phase

[1687] User

[1688] The user acts on the advice provided and provides feedback to the application, such as "I did 30 minutes of yoga" or "I ate more vegetables for dinner."

[1689] Terminal

[1690] The terminal receives feedback from the user and transmits it to the server.

[1691] server

[1692] The server receives feedback from the device and stores it in a database. Based on this feedback, the server adjusts the next advice it provides, allowing it to continually provide optimal advice tailored to the user's individual needs.

[1693] Example

[1694] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[1695] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[1696] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[1697] In this way, the present invention is a system that provides individual advice based on emotional data in addition to the user's exercise records, health status, and dietary information, thereby more effectively supporting the user's health management.

[1698] The processing flow will be explained below.

[1699] Step 1:

[1700] User

[1701] Users use the application to record their daily exercise and meal photos. The exercise records include distance traveled, time, and calorie consumption, and the meal photos are taken to show what they ate that day.

[1702] Step 2:

[1703] Terminal

[1704] The device receives the exercise records and meal photos entered by the user and temporarily stores them in local storage, preparing to send the stored data to the server.

[1705] Step 3:

[1706] Terminal

[1707] The device converts the organized data into packets and sends them securely to the server using the SSL / TLS protocol. If the data transmission was successful, the device notifies the user.

[1708] Step 4:

[1709] server

[1710] The server receives the exercise records and meal photos sent from the device, stores them in a database, and then processes the data for analysis.

[1711] Step 5:

[1712] server

[1713] The server analyzes the received data. It uses image recognition technology to analyze food photos and extract menu, nutrient, and calorie information. It also analyzes exercise records to calculate the user's calorie consumption and exercise performance.

[1714] Step 6:

[1715] Emotion Engine

[1716] The server uses an emotion engine to analyze the user's emotions, taking in the user's voice, text, and facial expression data to determine their emotional state, such as stress or happiness.

[1717] Step 7:

[1718] server

[1719] The generative AI model on the server generates personalized healthcare advice for each user based on the analyzed exercise records, dietary data, and emotional data. For example, it generates specific advice such as, "Due to lack of exercise, walk 30 minutes tomorrow. Due to high stress, we recommend doing yoga to relax."

[1720] Step 8:

[1721] server

[1722] The server sends the generated advice and associated metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1723] Step 9:

[1724] Terminal

[1725] The device notifies the user of the advice and metadata received from the server, and the user can check the content of the advice by opening the notification.

[1726] Step 10:

[1727] User

[1728] The user then takes action based on the advice provided, for example, following instructions such as "do 30 minutes of yoga and eat more vegetables for dinner."

[1729] Step 11:

[1730] User

[1731] The user provides feedback to the application about the results of their efforts, such as "I did 30 minutes of yoga today" or "I ate a lot of vegetables for dinner."

[1732] Step 12:

[1733] Terminal

[1734] The terminal receives the feedback from the user and transmits it to the server, and if the transmission is successful, notifies the user.

[1735] Step 13:

[1736] server

[1737] The server receives the feedback sent from the device and stores it in a database, which allows it to take the feedback into account the next time it generates advice and provide individually optimized advice.

[1738] Example

[1739] For example, a user eats "chicken, salad, and bread" for dinner, uploads a photo of it to the app, and records "3km jogging" as an exercise record. This information is sent from the device to the server, which analyzes it. From the photo of the meal, the server identifies "chicken, salad, and bread" and calculates the calories of each. The emotion engine also analyzes the user's voice data and determines that the user is "highly stressed."

[1740] Based on this data, the generative AI model generates specific advice such as "Tomorrow, we recommend 30 minutes of yoga to relax. Eat more vegetables at dinner." The advice is sent to the device and notified to the user. The next day, the user acts on the advice and enters the results into the app. Feedback is sent to the server and reflected in the next advice generation.

[1741] Thus, the present invention is a system that effectively supports the user's health management by providing individualized advice based on the user's exercise records, health status, dietary information, and emotional data.

[1742] Example 2

[1743] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1744] In recent years, interest in lifestyle-related diseases and health management has increased, but there are still limited systems that provide appropriate advice based on individual health and emotional states. Current systems often only collect exercise and dietary data, and do not provide advice that takes into account the emotions of individual users. As a result, there is a challenge in that it is difficult for users to actually achieve effective and sustainable health management.

[1745] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1746] In this invention, the server includes a means for analyzing exercise records and meal photos, a means for generating personalized advice based on the analyzed data and the user's emotional data, and a means for adjusting the next advice based on the received feedback, thereby enabling the provision of more personalized healthcare advice based on multifaceted data such as the user's exercise records, meal information, and even emotional state.

[1747] "Exercise records" are data containing information about the user's daily exercise, such as walking distance, exercise time, and calories burned.

[1748] "Meal photos" are image data of daily meals taken by a user, and include information indicating the type and amount of food and details of ingredients.

[1749] "Terminal" refers to an electronic device such as a smartphone or tablet used by a user, which inputs and sends / receives data through an application.

[1750] "Server" refers to a computer system on which a computing device and software for centrally managing data, performing analysis, and executing necessary processing operates.

[1751] "Analysis" refers to the process in which the server analyzes the exercise records and meal photos received and extracts specific information (e.g., calories, nutrients, etc.).

[1752] A "generative AI model" refers to artificial intelligence technology that automatically creates optimal advice for users based on analyzed data.

[1753] "Emotion data" refers to data that indicates the emotional state of a user, and refers to the analysis results of the emotional state obtained by analyzing information such as voice, text, and facial expressions.

[1754] "Feedback" refers to data that reports to the system the results and impressions of users who follow the advice provided.

[1755] "Personalized advice" refers to optimal health management advice for a user, created by a generative AI model based on the user's specific exercise records, dietary information, and emotional data.

[1756] MODE FOR CARRYING OUT THE INVENTION

[1757] This invention is a system that collects a user's daily exercise record, dietary information, health condition, and emotional data, and provides personalized healthcare advice based on these data. The system aims to realize more effective and personalized healthcare support by recognizing the user's emotions and adjusting the advice content based on those emotions.

[1758] Hardware and software used

[1759] Devices: smartphones, tablets, etc.

[1760] Server: A central system for database management and analysis.

[1761] Software: Applications, generative AI models, data analysis engines, emotion engines

[1762] Specific actions

[1763] User

[1764] Through the application, users can input their daily exercise records and photos of their meals. Exercise records include walking distance, exercise time, and calorie consumption, while meal photos are images that specifically show what they ate that day. Data from fitness bands and smartwatches is also synced to the application. For example, users can input information such as "I ran 3km today" or "I ate chicken, salad, and bread."

[1765] Terminal

[1766] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When the data send button is pressed, the data is securely sent to the server using SSL / TLS protocol. At this stage, the data is encrypted to minimize the risk of leakage.

[1767] server

[1768] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise records and meal photo data. It uses image recognition technology to extract specific menu items, nutrients, and calorie information from the meal photos. For example, it can determine that "a dinner of chicken, salad, and bread contains approximately 700 kcal."

[1769] Emotion Engine

[1770] The emotion engine analyzes the user's voice, text, and facial expression data to identify their emotional state. For example, it may determine that the user is likely feeling stressed.

[1771] Generative AI Models

[1772] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it provides specific advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend that they do 30 minutes of yoga to relax," or "We recommend that they increase the amount of vegetables they eat at dinner to maintain a balanced diet."

[1773] Terminal

[1774] The generated advice is sent from the server to the device and notified to the user via push notification or in-app notification. After the user confirms the advice, its read status is sent back to the server.

[1775] User

[1776] The user acts according to the advice provided and provides feedback to the application with the results. For example, the user might enter information such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner." This feedback is then sent to the server and used to generate advice for the next time.

[1777] Specific examples

[1778] A user uploads a photo of "chicken, salad, and bread" to the app after dinner and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calorie count is approximately 700kcal." Next, an emotion engine analyzes the user's emotions from voice and text messages and determines that "the stress level is high." Based on this information, the generative AI model generates advice such as "I recommend 30 minutes of yoga tomorrow to relax, and I also recommend increasing the amount of vegetables you eat at dinner."

[1779] This advice is sent to the device and notified to the user. The next day, the user acts according to the advice and enters the result into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[1780] Prompt Sentence Examples

[1781] "Please provide personalized advice based on the user's exercise record, dietary content, and emotional data. The user ran 3km today and had chicken, salad, and bread for dinner. Emotion analysis has determined that the user's stress level is high. Please create appropriate healthcare advice based on this."

[1782] In this way, the present invention provides individualized advice based on the user's exercise record, dietary information, health condition, and emotional data, thereby more effectively supporting the user's health management.

[1783] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1784] Step 1: Enter your data

[1785] User

[1786] Through the application, users can input photos of their daily exercise records and meals. For example, they can input information such as "I ran 3km today" or "I ate chicken, salad, and bread." Exercise data from fitness bands and smartwatches is also automatically synchronized. The input data is temporarily stored in the device's local storage.

[1787] Input: Exercise records (text), meal photos (images), wearable device data (calories, steps, etc.)

[1788] Output: Exercise and diet data stored in local storage

[1789] Step 2: Sending data

[1790] Terminal

[1791] The saved exercise records and meal photo data are sent to the server. When sending the data, the SSL / TLS protocol is used to encrypt the data and ensure secure communication. Once the data has been sent to the server, it is deleted from local storage.

[1792] Input: Locally stored exercise and food data

[1793] Output: Data sent to the server

[1794] Step 3: Receiving and storing data

[1795] server

[1796] The server receives exercise records and meal photo data sent from the device. The received data is stored in a database and organized using the user ID as a key. If necessary, filtering is performed to standardize the data format.

[1797] Input: Exercise and diet data sent from the device

[1798] Output: Data to be analyzed stored in a database

[1799] Step 4: Analyze the data

[1800] server

[1801] The server analyzes the received data. It uses image recognition technology to extract menu, nutrient, and calorie information from meal photos. It also calculates calories burned and exercise volume from exercise record data. For example, it might determine that a dinner of chicken, salad, and bread contains 700 kcal.

[1802] Input: Exercise and diet data stored in a database

[1803] Output: Parsed menu, nutritional and calorie information and exercise data

[1804] Step 5: Analyze the sentiment data

[1805] Emotion Engine

[1806] The emotion engine analyzes the user's voice, text, and facial expressions to identify the user's emotional state. For example, it may determine that the user is likely feeling stressed.

[1807] Input: User voice, text, and facial expression data

[1808] Output: Identified emotional state (e.g., high stress level)

[1809] Step 6: Generate personalized advice

[1810] server

[1811] The generative AI model generates personalized healthcare advice based on the analyzed exercise, diet, and emotional data, such as "recommended 30 minutes of yoga to relax" or "recommended to increase the amount of vegetables at dinner to maintain a balanced diet."

[1812] Input: Analyzed exercise data, diet data, emotion data

[1813] Output: personalized advice

[1814] Step 7: Submitting Advice

[1815] server

[1816] The server transmits the generated advice and its metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1817] Input: Generated advice and metadata

[1818] Output: Advice sent to terminal

[1819] Step 8: Notification of advice and confirmation of read receipt

[1820] Terminal

[1821] The device receives the advice and notifies the user via push notification or in-app notification. After the user checks the advice, the device sends back the read status to the server.

[1822] Input: Advice sent by the server

[1823] Output: Advice sent to the user, read status sent back to the server

[1824] Step 9: Enter and submit your feedback

[1825] User

[1826] The user acts on the advice provided and provides feedback to the application with the results, such as "I did yoga for 30 minutes" or "I ate a lot of vegetables for dinner."

[1827] Input: User feedback (results of actions)

[1828] Output: Feedback data

[1829] Terminal

[1830] The device receives the feedback data and transmits it securely to the server using the SSL / TLS protocol.

[1831] Input: User feedback

[1832] Output: Feedback sent to the server

[1833] Step 10: Analyze the feedback and incorporate it into your next advice

[1834] server

[1835] The server analyzes the feedback received from the user and stores it in a database. The AI ​​model adjusts the next advice based on the feedback. For example, it generates appropriate advice based on information such as, "I followed the previous advice and practiced yoga to relax, and as a result, my stress level decreased."

[1836] Input: Feedback received from the user

[1837] Output: Adjustment information to be reflected in the next advice

[1838] (Application example 2)

[1839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1840] Conventional healthcare support systems primarily rely on data such as exercise records and meal photos, and do not take into account the user's emotional state. This poses a challenge: they are unable to provide appropriate advice for health problems that are influenced by the user's emotions. Furthermore, because the advice provided is uniform, it is difficult to provide optimal support tailored to the needs of individual users.

[1841] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1842] In this invention, the server includes means for collecting the user's emotional state and analyzing it using an emotion engine, means for adjusting advice based on the emotional data, and means for analyzing exercise records and meal photos and generating personalized advice based on the analysis results. This enables personalized advice that takes the user's emotions into consideration, making it possible to more effectively support the user's health management.

[1843] The "exercise record" is data on the exercise the user performs each day, including the number of steps taken, distance traveled, calorie consumption, and the like.

[1844] "Meal photos" are data in which the user records the meals they have eaten as photographs.

[1845] "Server" refers to a computer system that receives and analyzes data sent by users and generates personalized advice.

[1846] "Terminal" refers to a device used by a user, such as a smartphone or computer, that has the means to send exercise records and meal photos, and receive and display advice.

[1847] The "emotional state" indicates the mental state of the user and includes emotional categories such as positive, negative, and neutral.

[1848] An "emotion engine" refers to a software or hardware system that analyzes emotions from a user's voice, text, facial expressions, etc., and identifies their emotional state.

[1849] "Analysis" refers to the process of processing received data and extracting and calculating specific information.

[1850] "Advice" is information that serves as a guide for the user's health management and behavior, generated based on the analysis results and emotion data.

[1851] "Feedback" is data that a user inputs into an application, including the results and impressions of actions taken based on the advice provided.

[1852] "Personalization" refers to the process of customizing advice content to each user's specific needs and circumstances.

[1853] To implement this invention, a system must be built that uses a user's smartphone or other device, a server, an emotion engine, and a generative AI model. This system collects the user's daily exercise records and meal photos, and provides personalized health advice based on the analysis results. It can also recognize the user's emotional state and adjust advice accordingly.

[1854] Program Description

[1855] The program of this system is configured as follows:

[1856] 1. Data Collection

[1857] Users use a smartphone application to record their daily exercise and meal photos. The exercise records include the number of steps taken, distance traveled, and calories burned, while the meal photos are images that show the specific meals eaten that day. In addition, data from fitness bands and smartwatches is also synchronized with the application.

[1858] 2. Data Transmission

[1859] The device receives exercise records and meal photos entered by the user and temporarily stores them in local storage. When sending this data to the server, it is securely transmitted using the SSL / TLS protocol.

[1860] 3. Data Analysis

[1861] The server receives the exercise records and meal photos sent from the device and stores them in a database. It then analyzes the exercise record and meal photo data. For example, it uses image recognition technology to extract information about the menu, nutrients, and calories from the meal photos. It also calculates the user's calorie expenditure and exercise volume from the exercise records.

[1862] 4. Emotion analysis

[1863] The server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes data such as the user's voice, text, and facial expressions to identify their emotional state. Specifically, it uses voice recognition technology to convert voice into text and analyzes the emotion in that text. For example, if the user is feeling stressed, it detects their stress level.

[1864] 5. Generating personalized advice

[1865] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it may generate advice such as, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[1866] 6. Advice Notice

[1867] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the device. The device receives the advice sent from the server and notifies the user. The device checks whether the advice has been read and sends that information back to the server.

[1868] 7. Feedback Collection

[1869] The user acts based on the advice provided and provides feedback to the application with the results. For example, the user may input feedback such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner." The device receives the feedback from the user and sends it to the server. The server receives the feedback sent from the device and stores it in a database. The next advice is adjusted based on this feedback.

[1870] Specific examples

[1871] As an example, a user uploads a photo of "chicken, salad, and bread" after dinner to the app and records their exercise as "I ran 3km today." This information is sent from the device to a server, which analyzes the data. Image recognition technology calculates the calories in the chicken, salad, and bread, and determines that "dinner's calories are approximately 700kcal."

[1872] Next, the emotion engine analyzes the user's emotions from voice and text messages and determines that their stress level is high. Based on this information, the generative AI model creates advice such as "Recommend 30 minutes of yoga tomorrow to relax, and increase the amount of vegetables you eat at dinner."

[1873] The generated advice is sent to the device and notified to the user. The next day, the user acts according to the advice, and as a result, enters into the app, "I did 30 minutes of yoga and ate a lot of vegetables for dinner." This feedback is sent to the server and reflected in the next advice generation.

[1874] Examples of prompts include:

[1875] The user exercises less. Their stress level is high, so recommend 30 minutes of yoga to relax. Recommend adding more vegetables to dinner.

[1876] As a result, the present invention can provide individual advice based on the user's exercise record, health condition, dietary information, and emotional data, thereby more effectively supporting the user's health management.

[1877] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1878] Step 1:

[1879] Users launch a smartphone application and input their daily exercise records and meal photos. The input exercise records include the number of steps taken, distance traveled, and calorie consumption, and the meal photos are images showing the meals eaten that day. The input data is temporarily saved in local storage.

[1880] Input: Exercise record data, meal photos

[1881] Output: Exercise record data and meal photos saved in local storage

[1882] Step 2:

[1883] The device then sends the received exercise record data and meal photos to a server. Data transmission is secure using the SSL / TLS protocol.

[1884] Input: Exercise record data and meal photos stored in local storage

[1885] Output: Exercise record data and meal photos sent to the server

[1886] Step 3:

[1887] The server stores the exercise record data and meal photos received from the device in a database. It then uses image recognition technology to analyze the meal photos and extract information about the menu, nutrients, and calories. It also calculates calories burned and the amount of exercise from the exercise record data.

[1888] Input: Exercise record data and meal photos sent to the server

[1889] Output: Analysis results (menu, nutrients, calorie information, calories burned, exercise amount)

[1890] Step 4:

[1891] The server collects the user's emotional state from voice and text and performs analysis using an emotion engine. For example, it analyzes text messages and voice data entered by the user within the application and classifies them into emotional categories such as positive, negative, and neutral.

[1892] Input: Audio data, text data

[1893] Output: Parsed emotional state data

[1894] Step 5:

[1895] The generative AI model on the server generates personalized health care advice based on the analyzed exercise records, dietary data, and emotional data. For example, it might generate advice like, "Because the user's exercise volume is low and their stress level is high, we recommend doing 30 minutes of yoga to relax," or "To ensure a balanced diet, increase the amount of vegetables at dinner."

[1896] Input: Analysis results (exercise records, dietary data, emotional data)

[1897] Output: Generated healthcare advice

[1898] Step 6:

[1899] The server sends the generated advice and metadata (information for tracking whether the advice has been read or implemented) to the terminal.

[1900] Input: Generated healthcare advice, metadata

[1901] Output: Advice and metadata sent to the device

[1902] Step 7:

[1903] The device receives the advice sent from the server and notifies the user. The user checks the notification and acts based on the advice. The device returns the advice read / implemented status to the server.

[1904] Input: Advice and metadata sent to the device

[1905] Output: Notification to user, read / implemented status returned to server

[1906] Step 8:

[1907] The user acts according to the advice provided and inputs the results into the application as feedback, such as "I did 30 minutes of yoga" or "I ate a lot of vegetables for dinner," and sends this information from the device to the server.

[1908] Input: User feedback data

[1909] Output: Feedback data sent to the server

[1910] Step 9:

[1911] The server receives the feedback sent from the device and stores it in a database, which then adjusts the next advice based on the feedback and provides the most appropriate advice to meet the user's individual needs.

[1912] Input: Feedback data sent to the server

[1913] Output: Adjusted next advice

[1914] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1915] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1916] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1917] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1918] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1919] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1920] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1921] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1922] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1923] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1924] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1925] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1926] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1927] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1928] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1929] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1930] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1931] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1932] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1933] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1934] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1935] The following is further disclosed regarding the above embodiment.

[1936] (Claim 1)

[1937] A means for receiving exercise records and meal photos from a user;

[1938] means for transmitting the received data to a server;

[1939] A server analyzes exercise records and meal photos and generates individual advice based on the analysis results;

[1940] means for transmitting the generated advice to a terminal;

[1941] means for notifying the user of the advice sent at the terminal;

[1942] means for receiving feedback from the user and transmitting it to the server;

[1943] a means at the server for adjusting subsequent advice based on the received feedback;

[1944] A system including:

[1945] (Claim 2)

[1946] 10. The system of claim 1, further comprising means for tracking advice reading and implementation.

[1947] (Claim 3)

[1948] 10. The system of claim 1, further comprising means for performing image analysis of the food photographs and converting them into menu, nutrient and calorie information.

[1949] "Example 1"

[1950] (Claim 1)

[1951] A means for receiving exercise records and meal photos from a user;

[1952] means for transmitting the received data to a network device;

[1953] a network device that analyzes exercise records and meal photos and generates personalized advice based on the analysis results;

[1954] means for transmitting the generated advice to a terminal;

[1955] means for notifying the user of the advice sent at the terminal;

[1956] means for receiving and transmitting feedback from a user to a network device;

[1957] means for adjusting subsequent advice based on the received feedback in the network device;

[1958] A means to generate personalized advice using generative AI models; and

[1959] A system including:

[1960] (Claim 2)

[1961] 10. The system of claim 1, further comprising means for tracking advice reading and implementation.

[1962] (Claim 3)

[1963] 10. The system of claim 1, further comprising means for performing image analysis of the food photographs and converting them into menu, nutrient and calorie informat...

Claims

1. A means for receiving exercise records and meal photos from a user; means for transmitting the received data to a server; A server analyzes exercise records and meal photos and generates individual advice based on the analysis results; means for transmitting the generated advice to a terminal; means for notifying the user of the advice sent at the terminal; means for receiving feedback from the user and transmitting it to the server; a means at the server for adjusting subsequent advice based on the received feedback; A system including:

2. 10. The system of claim 1, further comprising means for tracking advice reading and implementation.

3. 10. The system of claim 1, further comprising means for performing image analysis of the food photographs and converting them into menu, nutrient and calorie information.

Images