System

Abstract

A system is provided.SOLUTION: A system, comprising: means for taking a picture of a meal by a user; terminal means for sending the picture data to a server; means for analyzing the picture data and extracting calorie information by the server; means for storing the extracted calorie information in a health database of the user; and means for displaying the calorie information stored in the terminal and providing feedback to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, recording health information is a hassle for many users, and manual input is a particular challenge when it comes to dietary management. While many devices, such as blood pressure monitors and pedometers, can be linked to smartphones, it is necessary to use different apps for each device, making it difficult to centrally manage data. As a result, it is difficult for users to effectively understand their health status and take appropriate health management measures. This situation could hinder users' efforts to maintain and improve their health. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means. First, a means for a user to take a photo of a meal and a terminal means for sending the photo data to a server are used. Next, a means is used in which the server analyzes the photo data and extracts calorie information. A means is provided for storing the extracted calorie information in the user's health database and displaying the calorie information stored on the terminal to provide feedback to the user. A means is also provided in which the user connects a blood pressure monitor and acquires blood pressure data. A means is provided in which the server stores the acquired blood pressure data through the terminal means for sending the acquired blood pressure data to the server and updates the health database. A means is also provided for notifying the user of the updated blood pressure information. Finally, a means is provided in which the server analyzes the collected health data using generative AI technology and generates a detailed health status report and advice based on the analysis results. A means is provided for sending the generated report and advice to the terminal and displaying them to the user. In this way, a system is constructed that allows users to easily understand their health status and manage their health appropriately.

[0006] "User" refers to an individual who uses the system to input data and manage their health.

[0007] "Terminal" refers to a computing device used by a user, such as a smartphone or tablet.

[0008] A "server" refers to a computer system located on the cloud that analyzes and stores data and provides feedback to users.

[0009] "Photo data" refers to image data taken by a user using a terminal.

[0010] An "API request" refers to a communication request made via an interface to send data or request processing from a terminal to a server.

[0011] "Image recognition algorithm" refers to a computational method for analyzing photographic data to identify the type and quantity of food.

[0012] "Calorie information" refers to data that expresses the amount of energy in a food product in numerical form.

[0013] "Health Database" refers to a database that comprehensively manages and stores a user's health-related information.

[0014] "Sphygmomanometer" refers to a medical device for measuring a user's blood pressure.

[0015] "Generative AI technology" refers to a method of analyzing data and generating results using artificial intelligence technology.

[0016] "Report" means a document containing the results of an analysis of the collected data and recommendations.

[0017] "Feedback" refers to information or advice provided to a user. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The embodiments for carrying out the present invention will be described in detail below.

[0040] System configuration

[0041] The system consists of a device on which users take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to users. It also includes a smartphone that can be linked to health devices such as blood pressure monitors.

[0042] Data Entry

[0043] Entering meal data

[0044] User: The user takes a photo of their breakfast with their smartphone camera, which is displayed in the app for review.

[0045] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[0046] Entering blood pressure data

[0047] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0048] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[0049] Data analysis and feedback

[0050] Image analysis and calorie counting

[0051] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result, the identification information and quantities of bread, salad, and yogurt are extracted.

[0052] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0053] Processing blood pressure data

[0054] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[0055] User Notification and Interface

[0056] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0057] Users: Users can check the feedback on their smartphones and use it to manage their health.

[0058] Advanced Services

[0059] Personalized service

[0060] Users: Users can subscribe to paid services and access diet and training plans.

[0061] Server: The server creates a personalized diet plan based on the user's past diet and exercise data, along with advanced analytics and personalized coaching services.

[0062] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0063] Specific examples

[0064] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[0065] This allows users to effortlessly record information about their diet and blood pressure, enabling them to manage their health appropriately.

[0066] The above is a specific embodiment for carrying out the present invention. This system allows users to effectively collect health information and receive advice, making daily health management easier.

[0067] The processing flow will be explained below.

[0068] Dietary data entry and analysis

[0069] Step 1:

[0070] User: Opens the camera app on their smartphone and takes a photo of their meal.

[0071] Step 2:

[0072] On the device: The captured photo is displayed in the app for the user to review.

[0073] Step 3:

[0074] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0075] Step 4:

[0076] Server: Receives the API request and retrieves the photo data.

[0077] Step 5:

[0078] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[0079] Step 6:

[0080] Server: Calculates calories from the extracted food identification information and quantity.

[0081] Step 7:

[0082] Server: Stores the calculated calorie information in the user's health database.

[0083] Step 8:

[0084] Server: Notifies the device that the health database update is complete.

[0085] Step 9:

[0086] Device: Show the user "This morning's meal was balanced."

[0087] Blood pressure data entry and processing

[0088] Step 1:

[0089] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[0090] Step 2:

[0091] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[0092] Step 3:

[0093] User: Measures blood pressure with a blood pressure monitor and records the value.

[0094] Step 4:

[0095] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[0096] Step 5:

[0097] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[0098] Step 6:

[0099] Server: Receives the API request and retrieves the blood pressure data.

[0100] Step 7:

[0101] Server: Stores blood pressure data in a health database.

[0102] Step 8:

[0103] Server: Notifies the device that the health database update is complete.

[0104] Step 9:

[0105] Terminal: Informs the user whether the blood pressure range is normal or not.

[0106] Data analysis and health advice

[0107] Step 1:

[0108] Server: Analyzes all collected data using generative AI technology.

[0109] Step 2:

[0110] Server: Generates detailed health reports and advice.

[0111] Step 3:

[0112] Server: Sends the generated report to the device.

[0113] Step 4:

[0114] Terminal: Display reports and advice in the user interface and notify the user.

[0115] Advanced Services

[0116] Step 1:

[0117] User: Subscribe to the paid service and use the diet and training plans.

[0118] Step 2:

[0119] Server: Creates a personalized plan based on the user's past diet and exercise data.

[0120] Step 3:

[0121] Server: Sends the created plan to the device.

[0122] Step 4:

[0123] On your device: Display your plan and daily goals in the user interface.

[0124] Step 5:

[0125] User: Manage your daily health according to the plan.

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Despite the increasing health consciousness in recent years, it is not easy to centrally manage one's dietary and physiological data and obtain appropriate analysis and feedback in the midst of busy daily life. Conventional systems require manual data entry and analysis, placing a heavy burden on users. Furthermore, it is difficult to provide appropriate personalized advice based on individual health data. This invention aims to solve these problems and provide a system that allows users to easily manage their health.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes a device for a user to take a food image, a mobile terminal that transmits the image data to the network server, a means for the server to analyze the image data using a recognition algorithm and extract nutritional information, a means for storing the extracted nutritional information in the user's health database, and a means for displaying the nutritional information stored in the terminal and providing feedback to the user, thereby enabling users to effortlessly record their own dietary and physiological data, receive analysis and appropriate advice, and effectively manage their health.

[0131] A "user" is an individual who utilizes the system to input dietary and physiological data and receive analysis results and feedback.

[0132] "Food image" refers to image data of the meal taken by a user with a camera.

[0133] "Devices" are devices including cameras for taking pictures of food and health devices for measuring blood pressure.

[0134] A "network server" is a computing device located on the cloud that includes the infrastructure for sending and receiving data.

[0135] A "mobile terminal" is a portable electronic device such as a smartphone or tablet through which data is transmitted to a server.

[0136] A "recognition algorithm" is a computer program that analyzes image data or physiological data and extracts useful information from it.

[0137] "Nutritional information" refers to data on nutrients such as calories, protein, fat, and carbohydrates contained in food.

[0138] "Physiological data" is data that indicates the user's physical condition, such as blood pressure, heart rate, and body temperature.

[0139] The "health database" is a database for centrally managing and storing a user's dietary data and physiological data.

[0140] "Feedback" is information or advice provided to the user based on analyzed data.

[0141] "Machine learning technology" is a technology that allows computers to learn from data, find patterns, and make predictions and decisions.

[0142] System configuration

[0143] The system of this invention includes a device (such as a smartphone with a built-in camera) for users to take photos of their meals, a mobile terminal (such as a smartphone or tablet) for sending the captured photo data to a network server, software for the server to analyze the photo data using a recognition algorithm and extract nutritional information, and a database management system for storing the extracted nutritional information in the user's health database. It also includes integration of the smartphone with a health device (such as a blood pressure monitor) for measuring physiological data (such as blood pressure).

[0144] Data Entry

[0145] Entering meal data

[0146] User: The user takes a photo of their meal using their smartphone camera. For example, if they have bread, salad, and yogurt for breakfast, they take a photo that includes these items.

[0147] Device: The captured photo data is sent to a cloud server. At this time, the user's identification information is included in the API request along with the photo data.

[0148] Entering blood pressure data

[0149] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, when they measure their blood pressure in the morning, the data is recorded on the blood pressure monitor.

[0150] Device: The smartphone automatically acquires blood pressure data from the blood pressure monitor and sends it to a cloud server. Again, the user's identification information is included in the API request.

[0151] Data analysis and feedback

[0152] Image analysis and calorie counting

[0153] Server: Receives photo data in the cloud and analyzes the meal contents using image recognition algorithms. For example, it extracts the identity and quantity of bread, salad, and yogurt from the photo.

[0154] Server: Calculates calories based on the extracted food identification information and quantity, and stores the results in the user's health database.

[0155] Processing blood pressure data

[0156] Server: The cloud server receives the blood pressure data and automatically stores it in a health database.

[0157] User Notification and Interface

[0158] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0159] User: Users can check the latest feedback on their smartphones and use it to manage their health.

[0160] Advanced Services

[0161] Personalized service

[0162] Users: Users can subscribe to paid services to access diet and training plans.

[0163] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[0164] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0165] Specific examples

[0166] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database. A notification is then displayed on the device saying, "This morning's meal was balanced." Similarly, once blood pressure data is sent to the server and stored in the health database, the device is notified whether the blood pressure range is normal.

[0167] Example prompt sentence:

[0168] A user takes a photo of their breakfast with their smartphone camera and sends it to a cloud server, which then uses an image recognition algorithm to analyze the meal and calculate the calories. We are looking for a natural language explanation of the program process for this system.

[0169] This system allows the user to easily manage their health on a daily basis, which is a specific embodiment of the present invention.

[0170] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0171] Step 1:

[0172] Taking photos

[0173] User: The user uses the smartphone camera to take a photo of a meal, for example, bread, salad, and yogurt prepared for breakfast.

[0174] Input: Food image data

[0175] Output: Captured image data

[0176] Step 2:

[0177] Sending photo data

[0178] Device: The smartphone sends the captured photo data to a cloud server, including the image data and user identification information in an API request.

[0179] Input: Captured image data, user identification information

[0180] Output: Image data and user identification information sent to the cloud server

[0181] Step 3:

[0182] Image analysis

[0183] Server: The server analyzes the received photo data in the cloud using an image recognition algorithm (e.g., TensorFlow). The identification information and quantities of bread, salad, and yogurt are extracted as meal contents.

[0184] Input: Image data sent to the cloud server, user identification information

[0185] Output: Identification and quantity of extracted food items

[0186] Step 4:

[0187] Calorie Counting

[0188] Server: Calculates the calories of each food item based on its identification and quantity. The total calorie count is then output.

[0189] Input: Extracted food identities and quantities

[0190] Output: Calculated calorie information

[0191] Step 5:

[0192] Saving calorie information

[0193] Server: Stores the calculated calorie information in the user's health database, which is managed on the cloud.

[0194] Input: Calculated calorie information, user identification information

[0195] Output: Calorie information stored in the user's health database

[0196] Step 6:

[0197] Health data feedback

[0198] Device: Once the health database update is complete, the server notifies the device of the update information, which then displays the calorie information and whether the meal is balanced.

[0199] Input: Updated health data

[0200] Output: Calorie information and feedback displayed on the device

[0201] Step 7:

[0202] Blood pressure data measurement

[0203] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, they measure their morning blood pressure data.

[0204] Input: Physiological data measured by a blood pressure monitor

[0205] Output: Measured blood pressure data

[0206] Step 8:

[0207] Sending blood pressure data

[0208] Device: The smartphone automatically collects data from the blood pressure monitor and sends it to a cloud server. The API request also includes user identification information.

[0209] Input: Blood pressure data measured by a blood pressure monitor, user identification information

[0210] Output: Blood pressure data sent to the cloud server

[0211] Step 9:

[0212] Blood pressure data storage

[0213] Server: The server stores the received blood pressure data in the user's health database.

[0214] Input: Blood pressure data sent to the cloud server, user identification information

[0215] Output: Blood pressure data stored in a health database

[0216] Step 10:

[0217] Blood pressure data feedback

[0218] Terminal: Once the health database update is complete, the server notifies the terminal of the update information, and the terminal displays to the user whether the blood pressure range is normal or not.

[0219] Input: Updated health data

[0220] Output: Blood pressure data and feedback displayed on the device

[0221] Step 11:

[0222] Providing personalized service

[0223] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analytics and personalized coaching services.

[0224] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0225] Input: User's historical data

[0226] Output: Personalized health plan and advice

[0227] (Application example 1)

[0228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] In modern society, users are often too busy to manage their health in their daily lives. In particular, there is a demand for users to easily understand the calories and nutritional information of their meals, as well as their blood pressure status, and receive appropriate feedback and health suggestions. The present invention aims to provide a system that enables such health management to be carried out efficiently and effectively.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0231] In this invention, the server includes: means for a user to take photos of meals; terminal means for transmitting the photo data to the server; means for the server to analyze the photo data and extract calorie information; means for the server to store the extracted calorie information in the user's health database; means for generating health suggestions based on the calorie information stored in the terminal and providing feedback to the user; means for the user to receive health suggestions based on meal selections; means for the user to connect a blood pressure monitor and acquire blood pressure data; terminal means for transmitting the acquired blood pressure data to the server; means for the server to store the blood pressure data and update the health database; means for notifying the user of meal selections and next actions (e.g., drinking water, resting, light exercise) based on the updated blood pressure information; means for the server to analyze the collected data using AI technology; means for generating individual meal suggestions and detailed health status reports and advice based on the analysis results; and means for transmitting the generated reports and advice to the terminal and displaying them to the user. This allows users to easily manage their daily diet and health status while receiving individual health suggestions.

[0232] "User" refers to an individual who uses this system to take photos of their meals and manage their health data.

[0233] "Terminal means" refers to a device (such as a smartphone or tablet) that allows a user to take a photo of their meal and send that data to a server.

[0234] "Server" refers to a computer system located on the cloud that receives, analyzes, and stores photo data and blood pressure data, and provides feedback to users.

[0235] "Calorie information" refers to the calorie content of food analyzed from photos of meals taken by the user.

[0236] "Health database" refers to a database system that centrally manages and stores data related to a user's health (calorie information, blood pressure data, etc.).

[0237] "Feedback" refers to advice and notifications provided to users based on analyzed calorie information and health data.

[0238] "Generative AI technology" refers to artificial intelligence technology used to perform advanced analysis and predictions based on data collected by the server.

[0239] "Health Suggestions" refers to specific dietary choices and next action advice provided based on a user's diet and blood pressure data.

[0240] The embodiments for carrying out the present invention will be described in detail below.

[0241] System configuration

[0242] This system consists of a means for users to take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to the user. It also includes linking the smartphone with health management devices such as blood pressure monitors. Furthermore, the key feature of this novelty is that the server analyzes the collected data using generative AI technology and provides detailed feedback.

[0243] Hardware and Software

[0244] Hardware

[0245] Smartphone (e.g., a general smartphone)

[0246] Tablet (e.g., a general tablet device)

[0247] Blood pressure monitor (e.g., a blood pressure monitor with Bluetooth connectivity)

[0248] software

[0249] Cloud server (e.g. AWS, Google Cloud)

[0250] Image recognition algorithms (e.g. TensorFlow, Keras)

[0251] Health data management systems (e.g., Firebase)

[0252] Data Entry

[0253] Entering meal data

[0254] User: The user takes a photo of their meal using their smartphone camera, which is displayed in the app for review purposes.

[0255] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[0256] Entering blood pressure data

[0257] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0258] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[0259] Data analysis and feedback

[0260] Image analysis and calorie counting

[0261] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result of the analysis, the identification information and quantity of specific foods are extracted.

[0262] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0263] Processing blood pressure data

[0264] Server: The cloud-based server receives the blood pressure data, stores it in the user's health database, and notifies the user about food choices and next actions (e.g., drinking water, taking a break, or doing light exercise) based on the updated blood pressure information.

[0265] User Notification and Interface

[0266] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays calorie information and health suggestions to the user. Specific suggestions include the next meal selection and suggested activities.

[0267] Users: Users can view feedback on their smartphones, tablets, or smart glasses to help manage their health.

[0268] Advanced Services

[0269] Personalized service

[0270] Users: Users can subscribe to the paid service and receive diet and exercise plans, as well as personalized health recommendations based on their food choices.

[0271] Server: The server uses generative AI technology to create a personalized diet plan based on the user's past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[0272] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0273] Specific examples

[0274] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[0275] Prompt Sentence Examples

[0276] "Calculate the calories of a user who eats bread, salad, and yogurt for breakfast, and compare that with their blood pressure data to generate appropriate health suggestions."

[0277] This allows users to easily manage their diet and health status and receive personalized health suggestions.

[0278] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0279] Step 1:

[0280] The user takes a photo of the meal. The user uses the smartphone camera to take a photo of the meal through the application. The input is the photo of the meal taken with the smartphone camera, and the output is the photo data for confirmation that is displayed in the application.

[0281] Step 2:

[0282] The device sends the captured photo data to the server. The device then sends the photo data and user identification information (e.g., user ID) to the cloud server via an API request. The input is the captured photo data and user identification information, and the output is the photo data and user identification information sent to the server.

[0283] Step 3:

[0284] The server analyzes the photo data and extracts calorie information. The server then analyzes the photo data using an image recognition algorithm (such as TensorFlow or Keras) to identify the food contents. The input is the photo data, and the output is the food identification information and its quantity.

[0285] Step 4:

[0286] The server calculates calories based on the extracted food identification information and quantity. The server then references the standard calorie value for each identified food and calculates the total calories for the specific meal. The input is the food identification information and quantity, and the output is the total calorie information.

[0287] Step 5:

[0288] The extracted calorie information is stored in a health database. The server stores the calculated calorie information in the user's health database. The input is the total calorie information and the user identification information, and the output is the updated health database.

[0289] Step 6:

[0290] Health suggestions are generated based on the calorie information stored on the device and feedback is provided to the user. The server uses the stored calorie information and generative AI technology to generate detailed health suggestions and send them to the device. The input is the stored calorie information and past health data, and the output is the generated health suggestions and feedback information.

[0291] Step 7:

[0292] The user acquires blood pressure data using a blood pressure monitor. The user connects the blood pressure monitor to a smartphone via Bluetooth and performs measurements. The input is the blood pressure data obtained from the blood pressure monitor, and the output is the blood pressure data transferred to the smartphone.

[0293] Step 8:

[0294] The device sends blood pressure data to the server. The smartphone sends the blood pressure data and user identification information to a server on the cloud. The input is the blood pressure data and user identification information, and the output is the blood pressure data sent to the server.

[0295] Step 9:

[0296] The server stores the blood pressure data and updates the health database. The cloud server analyzes the received blood pressure data and updates the health database. The input is the blood pressure data and user identification information, and the output is the updated health database.

[0297] Step 10:

[0298] The server suggests and notifies users of meal choices and next actions based on updated calorie and blood pressure information. The server analyzes the data using a generative AI model, generates optimal meal choices and action suggestions (e.g., drinking water, resting, light exercise) for the user, and notifies the device. The input is updated calorie and blood pressure information and past health data, and the output is the generated health suggestions and notification information.

[0299] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0300] The embodiments for carrying out the present invention will be described in detail below.

[0301] System configuration

[0302] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[0303] Data Entry

[0304] Entering meal data

[0305] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[0306] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0307] Entering blood pressure data

[0308] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0309] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[0310] Entering emotion data

[0311] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[0312] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[0313] Data analysis and feedback

[0314] Image analysis and calorie counting

[0315] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm, extracting the type and amount of food as the analysis result.

[0316] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0317] Processing blood pressure data

[0318] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[0319] Emotional Data Analysis

[0320] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[0321] User Notification and Interface

[0322] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0323] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[0324] Advanced Services

[0325] Personalized service

[0326] User: A user subscribes to a paid service and uses diet and training plans.

[0327] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[0328] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0329] Specific examples

[0330] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[0331] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[0332] Furthermore, if the user uses the emotion engine to detect smiles or signs of stress when taking a photo, the emotion data will be sent to the server, and feedback and advice will be tailored accordingly. For example, if it is detected that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[0333] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[0334] The above is a specific embodiment of the present invention. This system allows users to effectively collect health information and emotional data and receive advice, making daily health and mental health management easier.

[0335] The processing flow will be explained below.

[0336] Dietary data entry and analysis

[0337] Step 1:

[0338] User: Opens the camera app on their smartphone and takes a photo of their meal.

[0339] Step 2:

[0340] On the device: The captured photo is displayed in the app for the user to review.

[0341] Step 3:

[0342] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0343] Step 4:

[0344] Server: Receives the API request and retrieves the photo data.

[0345] Step 5:

[0346] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[0347] Step 6:

[0348] Server: Calculates calories from the extracted food identification information and quantity.

[0349] Step 7:

[0350] Server: Stores the calculated calorie information in the user's health database.

[0351] Step 8:

[0352] Server: Notifies the device that the health database update is complete.

[0353] Step 9:

[0354] Device: Show the user "This morning's meal was balanced."

[0355] Blood pressure data entry and processing

[0356] Step 1:

[0357] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[0358] Step 2:

[0359] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[0360] Step 3:

[0361] User: Measures blood pressure with a blood pressure monitor and records the value.

[0362] Step 4:

[0363] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[0364] Step 5:

[0365] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[0366] Step 6:

[0367] Server: Receives the API request and retrieves the blood pressure data.

[0368] Step 7:

[0369] Server: Stores blood pressure data in a health database.

[0370] Step 8:

[0371] Server: Notifies the device that the health database update is complete.

[0372] Step 9:

[0373] Terminal: Informs the user whether the blood pressure range is normal or not.

[0374] Emotion data input and processing

[0375] Step 1:

[0376] User: Take a photo of their facial expression using their smartphone camera.

[0377] Step 2:

[0378] Device: Sends facial expression data to the emotion engine and analyzes the user's emotions.

[0379] Step 3:

[0380] Emotion engine: Acquires analyzed emotion data and sends it to a server on the cloud.

[0381] Step 4:

[0382] Server: Stores emotion data in a health database.

[0383] Step 5:

[0384] Server: Adjusts feedback and advice appropriately based on emotional data.

[0385] Step 6:

[0386] Device: Display emotional feedback and advice in the user interface.

[0387] Data analysis and health advice

[0388] Step 1:

[0389] Server: Analyzes all collected data using generative AI technology.

[0390] Step 2:

[0391] Server: Generates detailed health reports and advice.

[0392] Step 3:

[0393] Server: Sends the generated report to the device.

[0394] Step 4:

[0395] Terminal: Display reports and advice in the user interface and notify the user.

[0396] Advanced Services

[0397] Step 1:

[0398] User: Subscribe to the paid service and use the diet and training plans.

[0399] Step 2:

[0400] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[0401] Step 3:

[0402] Server: Sends the created plan to the device.

[0403] Step 4:

[0404] On your device: Display your plan and daily goals in the user interface.

[0405] Step 5:

[0406] User: Manage your daily health according to the plan.

[0407] Example 2

[0408] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] Conventional health management systems collect dietary and blood pressure data, but are unable to provide feedback that takes into account the user's emotional state. Furthermore, they only analyze the collected data individually, lacking the ability to grasp the user's overall health status and provide appropriate advice to the user. This makes it difficult for users to effectively manage their own health.

[0410] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to take a photo of a meal, a communication means for transmitting the photo data to the server, a means for the server to analyze the photo data using generative AI technology and extract calorie information, a means for storing the extracted calorie information in the user's health database, a means for displaying the calorie information stored on the terminal and providing feedback to the user, a means for analyzing the user's facial expression using an emotion engine and evaluating the user's current emotional state, and a means for adjusting the content of feedback and advice based on the user's emotional data. This allows the user to receive more personalized feedback and advice that takes into account emotional data as well as health information such as diet and blood pressure.

[0411] A "user" is an entity that uses the system to record health information and receive feedback and advice.

[0412] "Means for taking photos of meals" refers to a function that allows a user to capture images of meals using the smartphone camera.

[0413] "Communication means" refers to the technology or protocol used to transmit photograph data and blood pressure data from the terminal to the server.

[0414] "Generative AI technology" is a technology that uses artificial intelligence algorithms executed on a server to analyze and recognize data.

[0415] "Calorie information" is an energy value calculated based on the types and amounts of food consumed by the user.

[0416] A "health database" is a data storage device that stores health-related information such as a user's dietary data, blood pressure data, and emotional data.

[0417] "Means for providing feedback" refers to a function that notifies and displays the results of analysis and advice from the server on the user's device.

[0418] An "emotion engine" is software or algorithms that analyze a user's facial expression data to recognize their emotional state.

[0419] "Means for analyzing facial expressions" refers to technologies and algorithms that analyze the user's facial expression data acquired by a smartphone camera.

[0420] "Emotion data" is information that indicates the user's facial expression and other emotional states.

[0421] The "means for adjusting the content of feedback and advice" is a technique for appropriately changing the content of the feedback and advice provided based on the user's emotional state.

[0422] MODE FOR CARRYING OUT THE INVENTION

[0423] The present invention is a system for comprehensively managing a user's health information and emotional data, and providing feedback and advice. Specific embodiments for carrying out the invention are described in detail below.

[0424] System configuration

[0425] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[0426] Data Entry

[0427] Entering meal data

[0428] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[0429] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0430] Entering blood pressure data

[0431] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0432] Device: Automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[0433] Entering emotion data

[0434] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[0435] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[0436] Data analysis and feedback

[0437] Image analysis and calorie counting

[0438] Server: The server receives the photo data in the cloud and applies image recognition algorithms using generative AI technology. The analysis results include the type and quantity of food.

[0439] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0440] Processing blood pressure data

[0441] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[0442] Emotional Data Analysis

[0443] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[0444] User Notification and Interface

[0445] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0446] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[0447] Advanced Services

[0448] Personalized service

[0449] User: A user subscribes to a paid service and uses diet and training plans.

[0450] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[0451] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0452] Specific examples

[0453] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[0454] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[0455] Furthermore, if the emotion engine detects smiles or signs of stress when taking a photo, the emotion data is sent to the server, and feedback and advice can be tailored accordingly. For example, if it detects that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[0456] Prompt Sentence Examples

[0457] "Take a photo of bread, salad, and yogurt for breakfast and calculate the calories. Use an emotion engine to analyze the user's emotions and generate sentences with tailored feedback based on the results."

[0458] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[0459] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0460] Step 1:

[0461] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[0462] Input: A photo of a meal taken with a smartphone camera

[0463] Output: Photo data displayed in the app

[0464] What it does: The user opens their smartphone camera, takes a photo that captures the entire meal, then opens the app to view the photo.

[0465] Step 2:

[0466] Device: Sends an API request containing the confirmed photo data and user identification information to a server in the cloud.

[0467] Input: Verified photo data and user identification

[0468] Output: API request sent to the server

[0469] Specific operation: When the user checks the photo displayed on the device screen and taps the OK button, an API request containing the photo data and user ID is created and sent to a server in the cloud.

[0470] Step 3:

[0471] Server: The server receives the photo data on the cloud and uses generative AI technology to apply image recognition algorithms to analyze the meal contents.

[0472] Input: Photo data sent to the server

[0473] Output: Analysis results of meal contents (type and amount of food)

[0474] How it works: The received photo data is fed into a generative AI model, which then uses image recognition algorithms to identify food items such as bread, salad, and yogurt from the photo.

[0475] Step 4:

[0476] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0477] Input: Food identity and quantity

[0478] Output: Calorie information stored in the user's health database

[0479] What it does: Calculate the energy value of each food item, add them up to get the total calorie value, and store it in a database.

[0480] Step 5:

[0481] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[0482] Input: Blood pressure data sent from the blood pressure monitor

[0483] Output: Blood pressure information stored in the user's health database

[0484] Specific operation: Receives blood pressure data and records it directly in the database.

[0485] Step 6:

[0486] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[0487] Input: Facial expression data captured with a smartphone

[0488] Output: Facial expression data displayed in the app

[0489] Specific operation: The user uses the smartphone camera to capture their facial expression.

[0490] Step 7:

[0491] Device: Analyzes facial expression data, uses an emotion engine to recognize the user's emotions, and sends the acquired emotion data to a server on the cloud.

[0492] Input: Photographed facial expression data

[0493] Output: Emotion data sent to the server

[0494] What it does: It runs a facial expression recognition algorithm to identify emotions such as smiling or stressed, and sends that data to a server.

[0495] Step 8:

[0496] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[0497] Input: Emotion data sent to the server

[0498] Output: Emotion analysis results

[0499] Specific operation: Analyzes emotion data using analytical algorithms to identify the user's emotional state, such as "happy," "sad," or "stressed."

[0500] Step 9:

[0501] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0502] Input: Health database updates

[0503] Output: A message to inform the user

[0504] Specific operation: When an update notification arrives, a message such as "Calorie information has been updated" or "Blood pressure is within normal range" will be displayed in the notification center of the device.

[0505] Step 10:

[0506] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[0507] Input: Sentiment analysis results

[0508] Output: tailored feedback and advice

[0509] Specific actions: Specific advice such as "Try taking a deep breath to relax" is displayed.

[0510] Step 11:

[0511] User: Subscribe to the paid service and use the diet and training plans.

[0512] Input: Subscription information

[0513] Output: Authorization to use paid services

[0514] Specific behavior: Select a paid service through an in-app purchase and complete the payment.

[0515] Step 12:

[0516] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[0517] Input: Past dietary data, exercise data, emotional data

[0518] Output: Personalized health plan

[0519] What it does: It retrieves past data from a database and uses algorithms to create optimal diet and training plans.

[0520] Step 13:

[0521] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0522] Enter: personalized health plans.

[0523] Output: Display health plan

[0524] Specific behavior: The device notifies you of the plan and displays specific guidelines such as "Today's menu."

[0525] (Application example 2)

[0526] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0527] Conventional dietary management systems have difficulty accurately recording a user's dietary habits and providing feedback based on their health status. Furthermore, they are unable to integrate and analyze multiple health-related data, such as the user's emotional state and blood pressure data, making them inadequate for supporting comprehensive health management. Furthermore, food delivery services also face the challenge of being unable to provide customized meal recommendations based on the user's health and emotional state.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a photo of the meal, terminal means for transmitting the photo data to the server, means for the server to analyze the photo data and extract calorie information, means for storing the extracted calorie information in the user's health database, means for displaying the calorie information stored on the terminal and providing feedback to the user, means for analyzing the meal ordered by the user on the server and extracting calorie information and nutritional components, and means for providing meal suggestions and feedback according to the user's health status based on the extracted information. This enables accurate recording and analysis of the user's meal contents, integrated analysis of multiple health-related data including blood pressure data and emotional data, and customized meal suggestions and feedback based on the user's health status and emotional state.

[0529] The "means for a user to take a photo of a meal" refers to a function or device that allows a user to take a photo of a meal using a smartphone or camera device.

[0530] The "terminal means for transmitting the photographic data to the server" is a terminal having a communication function for transmitting data of the photograph taken to a server on the cloud or a remote server.

[0531] "Means for the server to analyze the photo data and extract calorie information" refers to a function in which the server uses image analysis technology to analyze the photo data received, identify the type and amount of food, and calculate calories.

[0532] The "means for saving extracted calorie information in the user's health database" is a function for recording and saving calorie information obtained by analysis in a user's personal health database.

[0533] "Means for displaying calorie information stored on the device and providing feedback to the user" refers to a function that displays calorie information stored on the device on the user's device and provides feedback on the contents of the meal and calorie consumption.

[0534] "Means for the user to connect a blood pressure monitor and obtain blood pressure data" refers to a function that allows the user to connect a blood pressure monitor via a communication means such as Bluetooth, measure blood pressure, and obtain the data.

[0535] The "terminal means for transmitting acquired blood pressure data to a server" is a communication function of the terminal for transmitting acquired blood pressure data to a server on the cloud.

[0536] "Means for the server to save the blood pressure data and update the health database" refers to a function that allows the server to save the received blood pressure data in the user's health database and update the database based on the latest data.

[0537] The "means for notifying the user of updated blood pressure information" is a function that notifies the user's device of updated blood pressure information and provides feedback according to the situation.

[0538] "Means for analyzing the user's emotional data and adjusting the content of advice based on health management" refers to a function that analyzes the user's facial expressions and other emotional expressions to extract emotional data, and adjusts the feedback and advice content based on that data.

[0539] "Means for the server to analyze collected data using generative AI technology" refers to a function that uses generative AI technology to comprehensively analyze collected dietary, blood pressure, and emotion data.

[0540] "Means for generating detailed reports and advice on health status based on analysis results" is a function that generates detailed reports and specific advice on the user's health status based on the information obtained through analysis.

[0541] The "means for transmitting the generated report and advice to the terminal and displaying it to the user" is a function for transmitting the generated report and advice to the user's device and visually displaying it.

[0542] The "means for providing customized meal suggestions based on the user's emotional state and health data" is a function that makes meal suggestions tailored to the user's situation based on collected emotional data and health data.

[0543] System configuration

[0544] The present invention is a system that allows a user to take a photo of a meal, analyzes the photo data, and extracts calorie information. This system specifically comprises the following components:

[0545] Entering meal data

[0546] 1. Users

[0547] The user takes a photo of the meal using a smartphone or camera device.

[0548] The photo data is sent via the device to a server on the cloud.

[0549] 2. Terminal

[0550] The terminal includes a communication means for transmitting the captured photo data to the server.

[0551] Entering blood pressure data

[0552] 1. Users

[0553] Users connect the blood pressure monitor to their smartphone using wireless communication functions such as Bluetooth to measure their blood pressure.

[0554] 2. Terminal

[0555] The device transmits the blood pressure data obtained from the blood pressure monitor to a server on the cloud.

[0556] Entering emotion data

[0557] 1. Users

[0558] Users take photos of their facial expressions with their smartphone camera while eating or measuring their blood pressure.

[0559] 2. Terminal

[0560] The device uses image analysis technology to analyze facial expression data and recognize the user's emotional state, and the recognized emotional data is sent to a cloud server.

[0561] Data analysis and feedback

[0562] 1. Server

[0563] Image analysis and calorie extraction

[0564] The server receives the photo data in the cloud and uses image analysis algorithms to analyze the meal contents. The analysis results include the type and amount of food, and calorie calculations. The calculation results are stored in the user's health database.

[0565] Processing blood pressure data

[0566] The server receives the blood pressure data and stores it in the user's health database.

[0567] Emotional Data Analysis

[0568] The server analyzes the acquired emotion data using an emotion engine to understand the user's current emotional state.

[0569] 2. Terminal

[0570] Notification and display of information

[0571] Once the health database update is complete, the server notifies the device, which then displays calorie information, blood pressure ranges, and emotional feedback and advice to the user.

[0572] Advanced Services

[0573] 1. Server

[0574] The server uses AI technology to comprehensively analyze the collected data, and automatically generates a detailed health report and advice based on the results.

[0575] 2. Terminal

[0576] The generated reports and advice are sent to the user's device for visual display, and the system also provides customized dietary suggestions based on emotional state and health data.

[0577] Specific examples

[0578] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone and sends it to a server. The server uses an image analysis algorithm to analyze the photo and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal is balanced."

[0579] Similarly, when a user measures their blood pressure, the data is sent to the server and stored in a health database. The device is notified whether the blood pressure range is normal or not. The device also analyzes the user's facial expression data and provides feedback and advice based on their emotional state.

[0580] Prompt Sentence Examples

[0581] "Below is a user-taken photo of a meal. Please analyze this photo to extract the food type and calories:

[0582] bread

[0583] salad

[0584] yogurt

[0585] And the user's blood pressure data is as follows:

[0586] Blood pressure: 135 / 85 mmHg

[0587] The user's emotion data is as follows:

[0588] Emotions: Not much stress

[0589] Use this data to generate health advice and feedback for users.”

[0590] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0591] Step 1:

[0592] A user takes a photo of a meal using a smartphone. At this time, the user checks the photo and saves it on the device. The input data is the photo of the meal, and the output data is the saved photo file.

[0593] Step 2:

[0594] The user's device sends the stored photo data to a server on the cloud. The input data is the photo file of the meal, and the output data is the photo data sent to the server. Specifically, the device uploads the photo data to the server using an HTTP request.

[0595] Step 3:

[0596] The server analyzes the received photo data. The input data is the photo data sent to the server, and the output data is the analyzed data on the type and amount of food. The server uses an image analysis algorithm to identify the type and amount of food.

[0597] Step 4:

[0598] The server calculates calorie information based on the analysis results. The input data is the type and amount of food, and the output data is the calculated calorie information. The server adds up the calorie values ​​for each food item to calculate the total calories.

[0599] Step 5:

[0600] The server saves the calorie information to the user's health database. The input data is the calculated calorie information, and the output data is the updated health database. The server adds the calorie information to the existing database.

[0601] Step 6:

[0602] The user connects a blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. The input data is the user's blood pressure value, and the output data is the blood pressure data recorded on the device. The data from the blood pressure monitor is automatically sent to the smartphone.

[0603] Step 7:

[0604] The device sends blood pressure data to the server. The input data is the blood pressure data recorded on the device, and the output data is the blood pressure data sent to the server. The device uploads the blood pressure data using an HTTP request.

[0605] Step 8:

[0606] The server saves the blood pressure data to the user's health database. The input data is the submitted blood pressure data, and the output data is the updated health database. The server adds the blood pressure data to the existing database.

[0607] Step 9:

[0608] The user captures their facial expression with a smartphone camera to acquire emotional data. The input data is the user's facial expression image, and the output data is analyzed emotional data. The device analyzes the facial expression data in real time.

[0609] Step 10:

[0610] The device sends emotion data to the server. The input data is the analyzed emotion data, and the output data is the emotion data sent to the server. The device uploads the emotion data using an HTTP request.

[0611] Step 11:

[0612] The server analyzes the emotional data and understands the user's emotional state. The input data is the emotional data sent to the server, and the output data is information about the user's current emotional state. The server uses an emotion engine to identify the emotional state.

[0613] Step 12:

[0614] The server integrates all data to evaluate the user's health status and generate feedback. The input data is calorie information, blood pressure data, and emotional data, and the output data is a comprehensive health report and advice. The server analyzes the data using generative AI technology.

[0615] Step 13:

[0616] The server sends the generated report and advice to the user's terminal. The input data are the generated health report and advice, and the output data are the report and advice received by the terminal. The terminal displays a notification to the user.

[0617] Step 14:

[0618] The user checks the report and advice through the terminal and receives feedback on their own health condition. The input data is the report and advice received from the server, and the output data is the feedback information displayed to the user. This allows the user to manage their daily health.

[0619] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0620] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0621] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0622] [Second embodiment]

[0623] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0624] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0625] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0626] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0627] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0628] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0629] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0630] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0631] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0632] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0633] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0634] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0635] The embodiments for carrying out the present invention will be described in detail below.

[0636] System configuration

[0637] The system consists of a device on which users take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to users. It also includes a smartphone that can be linked to health devices such as blood pressure monitors.

[0638] Data Entry

[0639] Entering meal data

[0640] User: The user takes a photo of their breakfast with their smartphone camera, which is displayed in the app for review.

[0641] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[0642] Entering blood pressure data

[0643] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0644] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[0645] Data analysis and feedback

[0646] Image analysis and calorie counting

[0647] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result, the identification information and quantities of bread, salad, and yogurt are extracted.

[0648] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0649] Processing blood pressure data

[0650] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[0651] User Notification and Interface

[0652] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0653] Users: Users can check the feedback on their smartphones and use it to manage their health.

[0654] Advanced Services

[0655] Personalized service

[0656] Users: Users can subscribe to paid services and access diet and training plans.

[0657] Server: The server creates a personalized diet plan based on the user's past diet and exercise data, along with advanced analytics and personalized coaching services.

[0658] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0659] Specific examples

[0660] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[0661] This allows users to effortlessly record information about their diet and blood pressure, enabling them to manage their health appropriately.

[0662] The above is a specific embodiment for carrying out the present invention. This system allows users to effectively collect health information and receive advice, making daily health management easier.

[0663] The processing flow will be explained below.

[0664] Dietary data entry and analysis

[0665] Step 1:

[0666] User: Opens the camera app on their smartphone and takes a photo of their meal.

[0667] Step 2:

[0668] On the device: The captured photo is displayed in the app for the user to review.

[0669] Step 3:

[0670] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0671] Step 4:

[0672] Server: Receives the API request and retrieves the photo data.

[0673] Step 5:

[0674] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[0675] Step 6:

[0676] Server: Calculates calories from the extracted food identification information and quantity.

[0677] Step 7:

[0678] Server: Stores the calculated calorie information in the user's health database.

[0679] Step 8:

[0680] Server: Notifies the device that the health database update is complete.

[0681] Step 9:

[0682] Device: Show the user "This morning's meal was balanced."

[0683] Blood pressure data entry and processing

[0684] Step 1:

[0685] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[0686] Step 2:

[0687] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[0688] Step 3:

[0689] User: Measures blood pressure with a blood pressure monitor and records the value.

[0690] Step 4:

[0691] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[0692] Step 5:

[0693] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[0694] Step 6:

[0695] Server: Receives the API request and retrieves the blood pressure data.

[0696] Step 7:

[0697] Server: Stores blood pressure data in a health database.

[0698] Step 8:

[0699] Server: Notifies the device that the health database update is complete.

[0700] Step 9:

[0701] Terminal: Informs the user whether the blood pressure range is normal or not.

[0702] Data analysis and health advice

[0703] Step 1:

[0704] Server: Analyzes all collected data using generative AI technology.

[0705] Step 2:

[0706] Server: Generates detailed health reports and advice.

[0707] Step 3:

[0708] Server: Sends the generated report to the device.

[0709] Step 4:

[0710] Terminal: Display reports and advice in the user interface and notify the user.

[0711] Advanced Services

[0712] Step 1:

[0713] User: Subscribe to the paid service and use the diet and training plans.

[0714] Step 2:

[0715] Server: Creates a personalized plan based on the user's past diet and exercise data.

[0716] Step 3:

[0717] Server: Sends the created plan to the device.

[0718] Step 4:

[0719] On your device: Display your plan and daily goals in the user interface.

[0720] Step 5:

[0721] User: Manage your daily health according to the plan.

[0722] Example 1

[0723] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Despite the increasing health consciousness in recent years, it is not easy to centrally manage one's dietary and physiological data and obtain appropriate analysis and feedback in the midst of busy daily life. Conventional systems require manual data entry and analysis, placing a heavy burden on users. Furthermore, it is difficult to provide appropriate personalized advice based on individual health data. This invention aims to solve these problems and provide a system that allows users to easily manage their health.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0726] In this invention, the server includes a device for a user to take a food image, a mobile terminal that transmits the image data to the network server, a means for the server to analyze the image data using a recognition algorithm and extract nutritional information, a means for storing the extracted nutritional information in the user's health database, and a means for displaying the nutritional information stored in the terminal and providing feedback to the user, thereby enabling users to effortlessly record their own dietary and physiological data, receive analysis and appropriate advice, and effectively manage their health.

[0727] A "user" is an individual who utilizes the system to input dietary and physiological data and receive analysis results and feedback.

[0728] "Food image" refers to image data of the meal taken by a user with a camera.

[0729] "Devices" are devices including cameras for taking pictures of food and health devices for measuring blood pressure.

[0730] A "network server" is a computing device located on the cloud that includes the infrastructure for sending and receiving data.

[0731] A "mobile terminal" is a portable electronic device such as a smartphone or tablet through which data is transmitted to a server.

[0732] A "recognition algorithm" is a computer program that analyzes image data or physiological data and extracts useful information from it.

[0733] "Nutritional information" refers to data on nutrients such as calories, protein, fat, and carbohydrates contained in food.

[0734] "Physiological data" is data that indicates the user's physical condition, such as blood pressure, heart rate, and body temperature.

[0735] The "health database" is a database for centrally managing and storing a user's dietary data and physiological data.

[0736] "Feedback" is information or advice provided to the user based on analyzed data.

[0737] "Machine learning technology" is a technology that allows computers to learn from data, find patterns, and make predictions and decisions.

[0738] System configuration

[0739] The system of this invention includes a device (such as a smartphone with a built-in camera) for users to take photos of their meals, a mobile terminal (such as a smartphone or tablet) for sending the captured photo data to a network server, software for the server to analyze the photo data using a recognition algorithm and extract nutritional information, and a database management system for storing the extracted nutritional information in the user's health database. It also includes integration of the smartphone with a health device (such as a blood pressure monitor) for measuring physiological data (such as blood pressure).

[0740] Data Entry

[0741] Entering meal data

[0742] User: The user takes a photo of their meal using their smartphone camera. For example, if they have bread, salad, and yogurt for breakfast, they take a photo that includes these items.

[0743] Device: The captured photo data is sent to a cloud server. At this time, the user's identification information is included in the API request along with the photo data.

[0744] Entering blood pressure data

[0745] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, when they measure their blood pressure in the morning, the data is recorded on the blood pressure monitor.

[0746] Device: The smartphone automatically acquires blood pressure data from the blood pressure monitor and sends it to a cloud server. Again, the user's identification information is included in the API request.

[0747] Data analysis and feedback

[0748] Image analysis and calorie counting

[0749] Server: Receives photo data in the cloud and analyzes the meal contents using image recognition algorithms. For example, it extracts the identity and quantity of bread, salad, and yogurt from the photo.

[0750] Server: Calculates calories based on the extracted food identification information and quantity, and stores the results in the user's health database.

[0751] Processing blood pressure data

[0752] Server: The cloud server receives the blood pressure data and automatically stores it in a health database.

[0753] User Notification and Interface

[0754] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0755] User: Users can check the latest feedback on their smartphones and use it to manage their health.

[0756] Advanced Services

[0757] Personalized service

[0758] Users: Users can subscribe to paid services to access diet and training plans.

[0759] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[0760] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0761] Specific examples

[0762] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database. A notification is then displayed on the device saying, "This morning's meal was balanced." Similarly, once blood pressure data is sent to the server and stored in the health database, the device is notified whether the blood pressure range is normal.

[0763] Example prompt sentence:

[0764] A user takes a photo of their breakfast with their smartphone camera and sends it to a cloud server, which then uses an image recognition algorithm to analyze the meal and calculate the calories. We are looking for a natural language explanation of the program process for this system.

[0765] This system allows the user to easily manage their health on a daily basis, which is a specific embodiment of the present invention.

[0766] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0767] Step 1:

[0768] Taking photos

[0769] User: The user uses the smartphone camera to take a photo of a meal, for example, bread, salad, and yogurt prepared for breakfast.

[0770] Input: Food image data

[0771] Output: Captured image data

[0772] Step 2:

[0773] Sending photo data

[0774] Device: The smartphone sends the captured photo data to a cloud server, including the image data and user identification information in an API request.

[0775] Input: Captured image data, user identification information

[0776] Output: Image data and user identification information sent to the cloud server

[0777] Step 3:

[0778] Image analysis

[0779] Server: The server analyzes the received photo data in the cloud using an image recognition algorithm (e.g., TensorFlow). The identification information and quantities of bread, salad, and yogurt are extracted as meal contents.

[0780] Input: Image data sent to the cloud server, user identification information

[0781] Output: Identification and quantity of extracted food items

[0782] Step 4:

[0783] Calorie Counting

[0784] Server: Calculates the calories of each food item based on its identification and quantity. The total calorie count is then output.

[0785] Input: Extracted food identities and quantities

[0786] Output: Calculated calorie information

[0787] Step 5:

[0788] Saving calorie information

[0789] Server: Stores the calculated calorie information in the user's health database, which is managed on the cloud.

[0790] Input: Calculated calorie information, user identification information

[0791] Output: Calorie information stored in the user's health database

[0792] Step 6:

[0793] Health data feedback

[0794] Device: Once the health database update is complete, the server notifies the device of the update information, which then displays the calorie information and whether the meal is balanced.

[0795] Input: Updated health data

[0796] Output: Calorie information and feedback displayed on the device

[0797] Step 7:

[0798] Blood pressure data measurement

[0799] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, they measure their morning blood pressure data.

[0800] Input: Physiological data measured by a blood pressure monitor

[0801] Output: Measured blood pressure data

[0802] Step 8:

[0803] Sending blood pressure data

[0804] Device: The smartphone automatically collects data from the blood pressure monitor and sends it to a cloud server. The API request also includes user identification information.

[0805] Input: Blood pressure data measured by a blood pressure monitor, user identification information

[0806] Output: Blood pressure data sent to the cloud server

[0807] Step 9:

[0808] Blood pressure data storage

[0809] Server: The server stores the received blood pressure data in the user's health database.

[0810] Input: Blood pressure data sent to the cloud server, user identification information

[0811] Output: Blood pressure data stored in a health database

[0812] Step 10:

[0813] Blood pressure data feedback

[0814] Terminal: Once the health database update is complete, the server notifies the terminal of the update information, and the terminal displays to the user whether the blood pressure range is normal or not.

[0815] Input: Updated health data

[0816] Output: Blood pressure data and feedback displayed on the device

[0817] Step 11:

[0818] Providing personalized service

[0819] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analytics and personalized coaching services.

[0820] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0821] Input: User's historical data

[0822] Output: Personalized health plan and advice

[0823] (Application example 1)

[0824] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0825] In modern society, users are often too busy to manage their health in their daily lives. In particular, there is a demand for users to easily understand the calories and nutritional information of their meals, as well as their blood pressure status, and receive appropriate feedback and health suggestions. The present invention aims to provide a system that enables such health management to be carried out efficiently and effectively.

[0826] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0827] In this invention, the server includes: means for a user to take photos of meals; terminal means for transmitting the photo data to the server; means for the server to analyze the photo data and extract calorie information; means for the server to store the extracted calorie information in the user's health database; means for generating health suggestions based on the calorie information stored in the terminal and providing feedback to the user; means for the user to receive health suggestions based on meal selections; means for the user to connect a blood pressure monitor and acquire blood pressure data; terminal means for transmitting the acquired blood pressure data to the server; means for the server to store the blood pressure data and update the health database; means for notifying the user of meal selections and next actions (e.g., drinking water, resting, light exercise) based on the updated blood pressure information; means for the server to analyze the collected data using AI technology; means for generating individual meal suggestions and detailed health status reports and advice based on the analysis results; and means for transmitting the generated reports and advice to the terminal and displaying them to the user. This allows users to easily manage their daily diet and health status while receiving individual health suggestions.

[0828] "User" refers to an individual who uses this system to take photos of their meals and manage their health data.

[0829] "Terminal means" refers to a device (such as a smartphone or tablet) that allows a user to take a photo of their meal and send that data to a server.

[0830] "Server" refers to a computer system located on the cloud that receives, analyzes, and stores photo data and blood pressure data, and provides feedback to users.

[0831] "Calorie information" refers to the calorie content of food analyzed from photos of meals taken by the user.

[0832] "Health database" refers to a database system that centrally manages and stores data related to a user's health (calorie information, blood pressure data, etc.).

[0833] "Feedback" refers to advice and notifications provided to users based on analyzed calorie information and health data.

[0834] "Generative AI technology" refers to artificial intelligence technology used to perform advanced analysis and predictions based on data collected by the server.

[0835] "Health Suggestions" refers to specific dietary choices and next action advice provided based on a user's diet and blood pressure data.

[0836] The embodiments for carrying out the present invention will be described in detail below.

[0837] System configuration

[0838] This system consists of a means for users to take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to the user. It also includes linking the smartphone with health management devices such as blood pressure monitors. Furthermore, the key feature of this novelty is that the server analyzes the collected data using generative AI technology and provides detailed feedback.

[0839] Hardware and Software

[0840] Hardware

[0841] Smartphone (e.g., a general smartphone)

[0842] Tablet (e.g., a general tablet device)

[0843] Blood pressure monitor (e.g., a blood pressure monitor with Bluetooth connectivity)

[0844] software

[0845] Cloud server (e.g. AWS, Google Cloud)

[0846] Image recognition algorithms (e.g. TensorFlow, Keras)

[0847] Health data management systems (e.g., Firebase)

[0848] Data Entry

[0849] Entering meal data

[0850] User: The user takes a photo of their meal using their smartphone camera, which is displayed in the app for review purposes.

[0851] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[0852] Entering blood pressure data

[0853] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0854] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[0855] Data analysis and feedback

[0856] Image analysis and calorie counting

[0857] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result of the analysis, the identification information and quantity of specific foods are extracted.

[0858] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0859] Processing blood pressure data

[0860] Server: The cloud-based server receives the blood pressure data, stores it in the user's health database, and notifies the user about food choices and next actions (e.g., drinking water, taking a break, or doing light exercise) based on the updated blood pressure information.

[0861] User Notification and Interface

[0862] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays calorie information and health suggestions to the user. Specific suggestions include the next meal selection and suggested activities.

[0863] Users: Users can view feedback on their smartphones, tablets, or smart glasses to help manage their health.

[0864] Advanced Services

[0865] Personalized service

[0866] Users: Users can subscribe to the paid service and receive diet and exercise plans, as well as personalized health recommendations based on their food choices.

[0867] Server: The server uses generative AI technology to create a personalized diet plan based on the user's past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[0868] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0869] Specific examples

[0870] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[0871] Prompt Sentence Examples

[0872] "Calculate the calories of a user who eats bread, salad, and yogurt for breakfast, and compare that with their blood pressure data to generate appropriate health suggestions."

[0873] This allows users to easily manage their diet and health status and receive personalized health suggestions.

[0874] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0875] Step 1:

[0876] The user takes a photo of the meal. The user uses the smartphone camera to take a photo of the meal through the application. The input is the photo of the meal taken with the smartphone camera, and the output is the photo data for confirmation that is displayed in the application.

[0877] Step 2:

[0878] The device sends the captured photo data to the server. The device then sends the photo data and user identification information (e.g., user ID) to the cloud server via an API request. The input is the captured photo data and user identification information, and the output is the photo data and user identification information sent to the server.

[0879] Step 3:

[0880] The server analyzes the photo data and extracts calorie information. The server then analyzes the photo data using an image recognition algorithm (such as TensorFlow or Keras) to identify the food contents. The input is the photo data, and the output is the food identification information and its quantity.

[0881] Step 4:

[0882] The server calculates calories based on the extracted food identification information and quantity. The server then references the standard calorie value for each identified food and calculates the total calories for the specific meal. The input is the food identification information and quantity, and the output is the total calorie information.

[0883] Step 5:

[0884] The extracted calorie information is stored in a health database. The server stores the calculated calorie information in the user's health database. The input is the total calorie information and the user identification information, and the output is the updated health database.

[0885] Step 6:

[0886] Health suggestions are generated based on the calorie information stored on the device and feedback is provided to the user. The server uses the stored calorie information and generative AI technology to generate detailed health suggestions and send them to the device. The input is the stored calorie information and past health data, and the output is the generated health suggestions and feedback information.

[0887] Step 7:

[0888] The user acquires blood pressure data using a blood pressure monitor. The user connects the blood pressure monitor to a smartphone via Bluetooth and performs measurements. The input is the blood pressure data obtained from the blood pressure monitor, and the output is the blood pressure data transferred to the smartphone.

[0889] Step 8:

[0890] The device sends blood pressure data to the server. The smartphone sends the blood pressure data and user identification information to a server on the cloud. The input is the blood pressure data and user identification information, and the output is the blood pressure data sent to the server.

[0891] Step 9:

[0892] The server stores the blood pressure data and updates the health database. The cloud server analyzes the received blood pressure data and updates the health database. The input is the blood pressure data and user identification information, and the output is the updated health database.

[0893] Step 10:

[0894] The server suggests and notifies users of meal choices and next actions based on updated calorie and blood pressure information. The server analyzes the data using a generative AI model, generates optimal meal choices and action suggestions (e.g., drinking water, resting, light exercise) for the user, and notifies the device. The input is updated calorie and blood pressure information and past health data, and the output is the generated health suggestions and notification information.

[0895] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0896] The embodiments for carrying out the present invention will be described in detail below.

[0897] System configuration

[0898] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[0899] Data Entry

[0900] Entering meal data

[0901] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[0902] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0903] Entering blood pressure data

[0904] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[0905] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[0906] Entering emotion data

[0907] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[0908] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[0909] Data analysis and feedback

[0910] Image analysis and calorie counting

[0911] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm, extracting the type and amount of food as the analysis result.

[0912] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[0913] Processing blood pressure data

[0914] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[0915] Emotional Data Analysis

[0916] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[0917] User Notification and Interface

[0918] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[0919] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[0920] Advanced Services

[0921] Personalized service

[0922] User: A user subscribes to a paid service and uses diet and training plans.

[0923] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[0924] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[0925] Specific examples

[0926] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[0927] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[0928] Furthermore, if the user uses the emotion engine to detect smiles or signs of stress when taking a photo, the emotion data will be sent to the server, and feedback and advice will be tailored accordingly. For example, if it is detected that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[0929] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[0930] The above is a specific embodiment of the present invention. This system allows users to effectively collect health information and emotional data and receive advice, making daily health and mental health management easier.

[0931] The processing flow will be explained below.

[0932] Dietary data entry and analysis

[0933] Step 1:

[0934] User: Opens the camera app on their smartphone and takes a photo of their meal.

[0935] Step 2:

[0936] On the device: The captured photo is displayed in the app for the user to review.

[0937] Step 3:

[0938] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[0939] Step 4:

[0940] Server: Receives the API request and retrieves the photo data.

[0941] Step 5:

[0942] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[0943] Step 6:

[0944] Server: Calculates calories from the extracted food identification information and quantity.

[0945] Step 7:

[0946] Server: Stores the calculated calorie information in the user's health database.

[0947] Step 8:

[0948] Server: Notifies the device that the health database update is complete.

[0949] Step 9:

[0950] Device: Show the user "This morning's meal was balanced."

[0951] Blood pressure data entry and processing

[0952] Step 1:

[0953] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[0954] Step 2:

[0955] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[0956] Step 3:

[0957] User: Measures blood pressure with a blood pressure monitor and records the value.

[0958] Step 4:

[0959] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[0960] Step 5:

[0961] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[0962] Step 6:

[0963] Server: Receives the API request and retrieves the blood pressure data.

[0964] Step 7:

[0965] Server: Stores blood pressure data in a health database.

[0966] Step 8:

[0967] Server: Notifies the device that the health database update is complete.

[0968] Step 9:

[0969] Terminal: Informs the user whether the blood pressure range is normal or not.

[0970] Emotion data input and processing

[0971] Step 1:

[0972] User: Take a photo of their facial expression using their smartphone camera.

[0973] Step 2:

[0974] Device: Sends facial expression data to the emotion engine and analyzes the user's emotions.

[0975] Step 3:

[0976] Emotion engine: Acquires analyzed emotion data and sends it to a server on the cloud.

[0977] Step 4:

[0978] Server: Stores emotion data in a health database.

[0979] Step 5:

[0980] Server: Adjusts feedback and advice appropriately based on emotional data.

[0981] Step 6:

[0982] Device: Display emotional feedback and advice in the user interface.

[0983] Data analysis and health advice

[0984] Step 1:

[0985] Server: Analyzes all collected data using generative AI technology.

[0986] Step 2:

[0987] Server: Generates detailed health reports and advice.

[0988] Step 3:

[0989] Server: Sends the generated report to the device.

[0990] Step 4:

[0991] Terminal: Display reports and advice in the user interface and notify the user.

[0992] Advanced Services

[0993] Step 1:

[0994] User: Subscribe to the paid service and use the diet and training plans.

[0995] Step 2:

[0996] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[0997] Step 3:

[0998] Server: Sends the created plan to the device.

[0999] Step 4:

[1000] On your device: Display your plan and daily goals in the user interface.

[1001] Step 5:

[1002] User: Manage your daily health according to the plan.

[1003] Example 2

[1004] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1005] Conventional health management systems collect dietary and blood pressure data, but are unable to provide feedback that takes into account the user's emotional state. Furthermore, they only analyze the collected data individually, lacking the ability to grasp the user's overall health status and provide appropriate advice to the user. This makes it difficult for users to effectively manage their own health.

[1006] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to take a photo of a meal, a communication means for transmitting the photo data to the server, a means for the server to analyze the photo data using generative AI technology and extract calorie information, a means for storing the extracted calorie information in the user's health database, a means for displaying the calorie information stored on the terminal and providing feedback to the user, a means for analyzing the user's facial expression using an emotion engine and evaluating the user's current emotional state, and a means for adjusting the content of feedback and advice based on the user's emotional data. This allows the user to receive more personalized feedback and advice that takes into account emotional data as well as health information such as diet and blood pressure.

[1007] A "user" is an entity that uses the system to record health information and receive feedback and advice.

[1008] "Means for taking photos of meals" refers to a function that allows a user to capture images of meals using the smartphone camera.

[1009] "Communication means" refers to the technology or protocol used to transmit photograph data and blood pressure data from the terminal to the server.

[1010] "Generative AI technology" is a technology that uses artificial intelligence algorithms executed on a server to analyze and recognize data.

[1011] "Calorie information" is an energy value calculated based on the types and amounts of food consumed by the user.

[1012] A "health database" is a data storage device that stores health-related information such as a user's dietary data, blood pressure data, and emotional data.

[1013] "Means for providing feedback" refers to a function that notifies and displays the results of analysis and advice from the server on the user's device.

[1014] An "emotion engine" is software or algorithms that analyze a user's facial expression data to recognize their emotional state.

[1015] "Means for analyzing facial expressions" refers to technologies and algorithms that analyze the user's facial expression data acquired by a smartphone camera.

[1016] "Emotion data" is information that indicates the user's facial expression and other emotional states.

[1017] The "means for adjusting the content of feedback and advice" is a technique for appropriately changing the content of the feedback and advice provided based on the user's emotional state.

[1018] MODE FOR CARRYING OUT THE INVENTION

[1019] The present invention is a system for comprehensively managing a user's health information and emotional data, and providing feedback and advice. Specific embodiments for carrying out the invention are described in detail below.

[1020] System configuration

[1021] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[1022] Data Entry

[1023] Entering meal data

[1024] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[1025] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[1026] Entering blood pressure data

[1027] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[1028] Device: Automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[1029] Entering emotion data

[1030] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[1031] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[1032] Data analysis and feedback

[1033] Image analysis and calorie counting

[1034] Server: The server receives the photo data in the cloud and applies image recognition algorithms using generative AI technology. The analysis results include the type and quantity of food.

[1035] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1036] Processing blood pressure data

[1037] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1038] Emotional Data Analysis

[1039] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[1040] User Notification and Interface

[1041] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1042] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[1043] Advanced Services

[1044] Personalized service

[1045] User: A user subscribes to a paid service and uses diet and training plans.

[1046] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[1047] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1048] Specific examples

[1049] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[1050] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[1051] Furthermore, if the emotion engine detects smiles or signs of stress when taking a photo, the emotion data is sent to the server, and feedback and advice can be tailored accordingly. For example, if it detects that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[1052] Prompt Sentence Examples

[1053] "Take a photo of bread, salad, and yogurt for breakfast and calculate the calories. Use an emotion engine to analyze the user's emotions and generate sentences with tailored feedback based on the results."

[1054] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[1055] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1056] Step 1:

[1057] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[1058] Input: A photo of a meal taken with a smartphone camera

[1059] Output: Photo data displayed in the app

[1060] What it does: The user opens their smartphone camera, takes a photo that captures the entire meal, then opens the app to view the photo.

[1061] Step 2:

[1062] Device: Sends an API request containing the confirmed photo data and user identification information to a server in the cloud.

[1063] Input: Verified photo data and user identification

[1064] Output: API request sent to the server

[1065] Specific operation: When the user checks the photo displayed on the device screen and taps the OK button, an API request containing the photo data and user ID is created and sent to a server in the cloud.

[1066] Step 3:

[1067] Server: The server receives the photo data on the cloud and uses generative AI technology to apply image recognition algorithms to analyze the meal contents.

[1068] Input: Photo data sent to the server

[1069] Output: Analysis results of meal contents (type and amount of food)

[1070] How it works: The received photo data is fed into a generative AI model, which then uses image recognition algorithms to identify food items such as bread, salad, and yogurt from the photo.

[1071] Step 4:

[1072] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1073] Input: Food identity and quantity

[1074] Output: Calorie information stored in the user's health database

[1075] What it does: Calculate the energy value of each food item, add them up to get the total calorie value, and store it in a database.

[1076] Step 5:

[1077] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1078] Input: Blood pressure data sent from the blood pressure monitor

[1079] Output: Blood pressure information stored in the user's health database

[1080] Specific operation: Receives blood pressure data and records it directly in the database.

[1081] Step 6:

[1082] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[1083] Input: Facial expression data captured with a smartphone

[1084] Output: Facial expression data displayed in the app

[1085] Specific operation: The user uses the smartphone camera to capture their facial expression.

[1086] Step 7:

[1087] Device: Analyzes facial expression data, uses an emotion engine to recognize the user's emotions, and sends the acquired emotion data to a server on the cloud.

[1088] Input: Photographed facial expression data

[1089] Output: Emotion data sent to the server

[1090] What it does: It runs a facial expression recognition algorithm to identify emotions such as smiling or stressed, and sends that data to a server.

[1091] Step 8:

[1092] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[1093] Input: Emotion data sent to the server

[1094] Output: Emotion analysis results

[1095] Specific operation: Analyzes emotion data using analytical algorithms to identify the user's emotional state, such as "happy," "sad," or "stressed."

[1096] Step 9:

[1097] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1098] Input: Health database updates

[1099] Output: A message to inform the user

[1100] Specific operation: When an update notification arrives, a message such as "Calorie information has been updated" or "Blood pressure is within normal range" will be displayed in the notification center of the device.

[1101] Step 10:

[1102] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[1103] Input: Sentiment analysis results

[1104] Output: tailored feedback and advice

[1105] Specific actions: Specific advice such as "Try taking a deep breath to relax" is displayed.

[1106] Step 11:

[1107] User: Subscribe to the paid service and use the diet and training plans.

[1108] Input: Subscription information

[1109] Output: Authorization to use paid services

[1110] Specific behavior: Select a paid service through an in-app purchase and complete the payment.

[1111] Step 12:

[1112] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[1113] Input: Past dietary data, exercise data, emotional data

[1114] Output: Personalized health plan

[1115] What it does: It retrieves past data from a database and uses algorithms to create optimal diet and training plans.

[1116] Step 13:

[1117] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1118] Enter: personalized health plans.

[1119] Output: Display health plan

[1120] Specific behavior: The device notifies you of the plan and displays specific guidelines such as "Today's menu."

[1121] (Application example 2)

[1122] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1123] Conventional dietary management systems have difficulty accurately recording a user's dietary habits and providing feedback based on their health status. Furthermore, they are unable to integrate and analyze multiple health-related data, such as the user's emotional state and blood pressure data, making them inadequate for supporting comprehensive health management. Furthermore, food delivery services also face the challenge of being unable to provide customized meal recommendations based on the user's health and emotional state.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a photo of the meal, terminal means for transmitting the photo data to the server, means for the server to analyze the photo data and extract calorie information, means for storing the extracted calorie information in the user's health database, means for displaying the calorie information stored on the terminal and providing feedback to the user, means for analyzing the meal ordered by the user on the server and extracting calorie information and nutritional components, and means for providing meal suggestions and feedback according to the user's health status based on the extracted information. This enables accurate recording and analysis of the user's meal contents, integrated analysis of multiple health-related data including blood pressure data and emotional data, and customized meal suggestions and feedback based on the user's health status and emotional state.

[1125] The "means for a user to take a photo of a meal" refers to a function or device that allows a user to take a photo of a meal using a smartphone or camera device.

[1126] The "terminal means for transmitting the photographic data to the server" is a terminal having a communication function for transmitting data of the photograph taken to a server on the cloud or a remote server.

[1127] "Means for the server to analyze the photo data and extract calorie information" refers to a function in which the server uses image analysis technology to analyze the photo data received, identify the type and amount of food, and calculate calories.

[1128] The "means for saving extracted calorie information in the user's health database" is a function for recording and saving calorie information obtained by analysis in a user's personal health database.

[1129] "Means for displaying calorie information stored on the device and providing feedback to the user" refers to a function that displays calorie information stored on the device on the user's device and provides feedback on the contents of the meal and calorie consumption.

[1130] "Means for the user to connect a blood pressure monitor and obtain blood pressure data" refers to a function that allows the user to connect a blood pressure monitor via a communication means such as Bluetooth, measure blood pressure, and obtain the data.

[1131] The "terminal means for transmitting acquired blood pressure data to a server" is a communication function of the terminal for transmitting acquired blood pressure data to a server on the cloud.

[1132] "Means for the server to save the blood pressure data and update the health database" refers to a function that allows the server to save the received blood pressure data in the user's health database and update the database based on the latest data.

[1133] The "means for notifying the user of updated blood pressure information" is a function that notifies the user's device of updated blood pressure information and provides feedback according to the situation.

[1134] "Means for analyzing the user's emotional data and adjusting the content of advice based on health management" refers to a function that analyzes the user's facial expressions and other emotional expressions to extract emotional data, and adjusts the feedback and advice content based on that data.

[1135] "Means for the server to analyze collected data using generative AI technology" refers to a function that uses generative AI technology to comprehensively analyze collected dietary, blood pressure, and emotion data.

[1136] "Means for generating detailed reports and advice on health status based on analysis results" is a function that generates detailed reports and specific advice on the user's health status based on the information obtained through analysis.

[1137] The "means for transmitting the generated report and advice to the terminal and displaying it to the user" is a function for transmitting the generated report and advice to the user's device and visually displaying it.

[1138] The "means for providing customized meal suggestions based on the user's emotional state and health data" is a function that makes meal suggestions tailored to the user's situation based on collected emotional data and health data.

[1139] System configuration

[1140] The present invention is a system that allows a user to take a photo of a meal, analyzes the photo data, and extracts calorie information. This system specifically comprises the following components:

[1141] Entering meal data

[1142] 1. Users

[1143] The user takes a photo of the meal using a smartphone or camera device.

[1144] The photo data is sent via the device to a server on the cloud.

[1145] 2. Terminal

[1146] The terminal includes a communication means for transmitting the captured photo data to the server.

[1147] Entering blood pressure data

[1148] 1. Users

[1149] Users connect the blood pressure monitor to their smartphone using wireless communication functions such as Bluetooth to measure their blood pressure.

[1150] 2. Terminal

[1151] The device transmits the blood pressure data obtained from the blood pressure monitor to a server on the cloud.

[1152] Entering emotion data

[1153] 1. Users

[1154] Users take photos of their facial expressions with their smartphone camera while eating or measuring their blood pressure.

[1155] 2. Terminal

[1156] The device uses image analysis technology to analyze facial expression data and recognize the user's emotional state, and the recognized emotional data is sent to a cloud server.

[1157] Data analysis and feedback

[1158] 1. Server

[1159] Image analysis and calorie extraction

[1160] The server receives the photo data in the cloud and uses image analysis algorithms to analyze the meal contents. The analysis results include the type and amount of food, and calorie calculations. The calculation results are stored in the user's health database.

[1161] Processing blood pressure data

[1162] The server receives the blood pressure data and stores it in the user's health database.

[1163] Emotional Data Analysis

[1164] The server analyzes the acquired emotion data using an emotion engine to understand the user's current emotional state.

[1165] 2. Terminal

[1166] Notification and display of information

[1167] Once the health database update is complete, the server notifies the device, which then displays calorie information, blood pressure ranges, and emotional feedback and advice to the user.

[1168] Advanced Services

[1169] 1. Server

[1170] The server uses AI technology to comprehensively analyze the collected data, and automatically generates a detailed health report and advice based on the results.

[1171] 2. Terminal

[1172] The generated reports and advice are sent to the user's device for visual display, and the system also provides customized dietary suggestions based on emotional state and health data.

[1173] Specific examples

[1174] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone and sends it to a server. The server uses an image analysis algorithm to analyze the photo and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal is balanced."

[1175] Similarly, when a user measures their blood pressure, the data is sent to the server and stored in a health database. The device is notified whether the blood pressure range is normal or not. The device also analyzes the user's facial expression data and provides feedback and advice based on their emotional state.

[1176] Prompt Sentence Examples

[1177] "Below is a user-taken photo of a meal. Please analyze this photo to extract the food type and calories:

[1178] bread

[1179] salad

[1180] yogurt

[1181] And the user's blood pressure data is as follows:

[1182] Blood pressure: 135 / 85 mmHg

[1183] The user's emotion data is as follows:

[1184] Emotions: Not much stress

[1185] Use this data to generate health advice and feedback for users.”

[1186] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1187] Step 1:

[1188] A user takes a photo of a meal using a smartphone. At this time, the user checks the photo and saves it on the device. The input data is the photo of the meal, and the output data is the saved photo file.

[1189] Step 2:

[1190] The user's device sends the stored photo data to a server on the cloud. The input data is the photo file of the meal, and the output data is the photo data sent to the server. Specifically, the device uploads the photo data to the server using an HTTP request.

[1191] Step 3:

[1192] The server analyzes the received photo data. The input data is the photo data sent to the server, and the output data is the analyzed data on the type and amount of food. The server uses an image analysis algorithm to identify the type and amount of food.

[1193] Step 4:

[1194] The server calculates calorie information based on the analysis results. The input data is the type and amount of food, and the output data is the calculated calorie information. The server adds up the calorie values ​​for each food item to calculate the total calories.

[1195] Step 5:

[1196] The server saves the calorie information to the user's health database. The input data is the calculated calorie information, and the output data is the updated health database. The server adds the calorie information to the existing database.

[1197] Step 6:

[1198] The user connects a blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. The input data is the user's blood pressure value, and the output data is the blood pressure data recorded on the device. The data from the blood pressure monitor is automatically sent to the smartphone.

[1199] Step 7:

[1200] The device sends blood pressure data to the server. The input data is the blood pressure data recorded on the device, and the output data is the blood pressure data sent to the server. The device uploads the blood pressure data using an HTTP request.

[1201] Step 8:

[1202] The server saves the blood pressure data to the user's health database. The input data is the submitted blood pressure data, and the output data is the updated health database. The server adds the blood pressure data to the existing database.

[1203] Step 9:

[1204] The user captures their facial expression with a smartphone camera to acquire emotional data. The input data is the user's facial expression image, and the output data is analyzed emotional data. The device analyzes the facial expression data in real time.

[1205] Step 10:

[1206] The device sends emotion data to the server. The input data is the analyzed emotion data, and the output data is the emotion data sent to the server. The device uploads the emotion data using an HTTP request.

[1207] Step 11:

[1208] The server analyzes the emotional data and understands the user's emotional state. The input data is the emotional data sent to the server, and the output data is information about the user's current emotional state. The server uses an emotion engine to identify the emotional state.

[1209] Step 12:

[1210] The server integrates all data to evaluate the user's health status and generate feedback. The input data is calorie information, blood pressure data, and emotional data, and the output data is a comprehensive health report and advice. The server analyzes the data using generative AI technology.

[1211] Step 13:

[1212] The server sends the generated report and advice to the user's terminal. The input data are the generated health report and advice, and the output data are the report and advice received by the terminal. The terminal displays a notification to the user.

[1213] Step 14:

[1214] The user checks the report and advice through the terminal and receives feedback on their own health condition. The input data is the report and advice received from the server, and the output data is the feedback information displayed to the user. This allows the user to manage their daily health.

[1215] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1216] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1217] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1218] [Third embodiment]

[1219] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1220] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1221] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1222] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1223] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1225] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1226] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1227] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1228] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1229] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1230] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1231] The embodiments for carrying out the present invention will be described in detail below.

[1232] System configuration

[1233] The system consists of a device on which users take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to users. It also includes a smartphone that can be linked to health devices such as blood pressure monitors.

[1234] Data Entry

[1235] Entering meal data

[1236] User: The user takes a photo of their breakfast with their smartphone camera, which is displayed in the app for review.

[1237] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[1238] Entering blood pressure data

[1239] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[1240] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[1241] Data analysis and feedback

[1242] Image analysis and calorie counting

[1243] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result, the identification information and quantities of bread, salad, and yogurt are extracted.

[1244] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1245] Processing blood pressure data

[1246] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1247] User Notification and Interface

[1248] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1249] Users: Users can check the feedback on their smartphones and use it to manage their health.

[1250] Advanced Services

[1251] Personalized service

[1252] Users: Users can subscribe to paid services and access diet and training plans.

[1253] Server: The server creates a personalized diet plan based on the user's past diet and exercise data, along with advanced analytics and personalized coaching services.

[1254] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1255] Specific examples

[1256] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[1257] This allows users to effortlessly record information about their diet and blood pressure, enabling them to manage their health appropriately.

[1258] The above is a specific embodiment for carrying out the present invention. This system allows users to effectively collect health information and receive advice, making daily health management easier.

[1259] The processing flow will be explained below.

[1260] Dietary data entry and analysis

[1261] Step 1:

[1262] User: Opens the camera app on their smartphone and takes a photo of their meal.

[1263] Step 2:

[1264] On the device: The captured photo is displayed in the app for the user to review.

[1265] Step 3:

[1266] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[1267] Step 4:

[1268] Server: Receives the API request and retrieves the photo data.

[1269] Step 5:

[1270] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[1271] Step 6:

[1272] Server: Calculates calories from the extracted food identification information and quantity.

[1273] Step 7:

[1274] Server: Stores the calculated calorie information in the user's health database.

[1275] Step 8:

[1276] Server: Notifies the device that the health database update is complete.

[1277] Step 9:

[1278] Device: Show the user "This morning's meal was balanced."

[1279] Blood pressure data entry and processing

[1280] Step 1:

[1281] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[1282] Step 2:

[1283] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[1284] Step 3:

[1285] User: Measures blood pressure with a blood pressure monitor and records the value.

[1286] Step 4:

[1287] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[1288] Step 5:

[1289] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[1290] Step 6:

[1291] Server: Receives the API request and retrieves the blood pressure data.

[1292] Step 7:

[1293] Server: Stores blood pressure data in a health database.

[1294] Step 8:

[1295] Server: Notifies the device that the health database update is complete.

[1296] Step 9:

[1297] Terminal: Informs the user whether the blood pressure range is normal or not.

[1298] Data analysis and health advice

[1299] Step 1:

[1300] Server: Analyzes all collected data using generative AI technology.

[1301] Step 2:

[1302] Server: Generates detailed health reports and advice.

[1303] Step 3:

[1304] Server: Sends the generated report to the device.

[1305] Step 4:

[1306] Terminal: Display reports and advice in the user interface and notify the user.

[1307] Advanced Services

[1308] Step 1:

[1309] User: Subscribe to the paid service and use the diet and training plans.

[1310] Step 2:

[1311] Server: Creates a personalized plan based on the user's past diet and exercise data.

[1312] Step 3:

[1313] Server: Sends the created plan to the device.

[1314] Step 4:

[1315] On your device: Display your plan and daily goals in the user interface.

[1316] Step 5:

[1317] User: Manage your daily health according to the plan.

[1318] Example 1

[1319] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1320] Despite the increasing health consciousness in recent years, it is not easy to centrally manage one's dietary and physiological data and obtain appropriate analysis and feedback in the midst of busy daily life. Conventional systems require manual data entry and analysis, placing a heavy burden on users. Furthermore, it is difficult to provide appropriate personalized advice based on individual health data. This invention aims to solve these problems and provide a system that allows users to easily manage their health.

[1321] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1322] In this invention, the server includes a device for a user to take a food image, a mobile terminal that transmits the image data to the network server, a means for the server to analyze the image data using a recognition algorithm and extract nutritional information, a means for storing the extracted nutritional information in the user's health database, and a means for displaying the nutritional information stored in the terminal and providing feedback to the user, thereby enabling users to effortlessly record their own dietary and physiological data, receive analysis and appropriate advice, and effectively manage their health.

[1323] A "user" is an individual who utilizes the system to input dietary and physiological data and receive analysis results and feedback.

[1324] "Food image" refers to image data of the meal taken by a user with a camera.

[1325] "Devices" are devices including cameras for taking pictures of food and health devices for measuring blood pressure.

[1326] A "network server" is a computing device located on the cloud that includes the infrastructure for sending and receiving data.

[1327] A "mobile terminal" is a portable electronic device such as a smartphone or tablet through which data is transmitted to a server.

[1328] A "recognition algorithm" is a computer program that analyzes image data or physiological data and extracts useful information from it.

[1329] "Nutritional information" refers to data on nutrients such as calories, protein, fat, and carbohydrates contained in food.

[1330] "Physiological data" is data that indicates the user's physical condition, such as blood pressure, heart rate, and body temperature.

[1331] The "health database" is a database for centrally managing and storing a user's dietary data and physiological data.

[1332] "Feedback" is information or advice provided to the user based on analyzed data.

[1333] "Machine learning technology" is a technology that allows computers to learn from data, find patterns, and make predictions and decisions.

[1334] System configuration

[1335] The system of this invention includes a device (such as a smartphone with a built-in camera) for users to take photos of their meals, a mobile terminal (such as a smartphone or tablet) for sending the captured photo data to a network server, software for the server to analyze the photo data using a recognition algorithm and extract nutritional information, and a database management system for storing the extracted nutritional information in the user's health database. It also includes integration of the smartphone with a health device (such as a blood pressure monitor) for measuring physiological data (such as blood pressure).

[1336] Data Entry

[1337] Entering meal data

[1338] User: The user takes a photo of their meal using their smartphone camera. For example, if they have bread, salad, and yogurt for breakfast, they take a photo that includes these items.

[1339] Device: The captured photo data is sent to a cloud server. At this time, the user's identification information is included in the API request along with the photo data.

[1340] Entering blood pressure data

[1341] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, when they measure their blood pressure in the morning, the data is recorded on the blood pressure monitor.

[1342] Device: The smartphone automatically acquires blood pressure data from the blood pressure monitor and sends it to a cloud server. Again, the user's identification information is included in the API request.

[1343] Data analysis and feedback

[1344] Image analysis and calorie counting

[1345] Server: Receives photo data in the cloud and analyzes the meal contents using image recognition algorithms. For example, it extracts the identity and quantity of bread, salad, and yogurt from the photo.

[1346] Server: Calculates calories based on the extracted food identification information and quantity, and stores the results in the user's health database.

[1347] Processing blood pressure data

[1348] Server: The cloud server receives the blood pressure data and automatically stores it in a health database.

[1349] User Notification and Interface

[1350] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1351] User: Users can check the latest feedback on their smartphones and use it to manage their health.

[1352] Advanced Services

[1353] Personalized service

[1354] Users: Users can subscribe to paid services to access diet and training plans.

[1355] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[1356] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1357] Specific examples

[1358] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database. A notification is then displayed on the device saying, "This morning's meal was balanced." Similarly, once blood pressure data is sent to the server and stored in the health database, the device is notified whether the blood pressure range is normal.

[1359] Example prompt sentence:

[1360] A user takes a photo of their breakfast with their smartphone camera and sends it to a cloud server, which then uses an image recognition algorithm to analyze the meal and calculate the calories. We are looking for a natural language explanation of the program process for this system.

[1361] This system allows the user to easily manage their health on a daily basis, which is a specific embodiment of the present invention.

[1362] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1363] Step 1:

[1364] Taking photos

[1365] User: The user uses the smartphone camera to take a photo of a meal, for example, bread, salad, and yogurt prepared for breakfast.

[1366] Input: Food image data

[1367] Output: Captured image data

[1368] Step 2:

[1369] Sending photo data

[1370] Device: The smartphone sends the captured photo data to a cloud server, including the image data and user identification information in an API request.

[1371] Input: Captured image data, user identification information

[1372] Output: Image data and user identification information sent to the cloud server

[1373] Step 3:

[1374] Image analysis

[1375] Server: The server analyzes the received photo data in the cloud using an image recognition algorithm (e.g., TensorFlow). The identification information and quantities of bread, salad, and yogurt are extracted as meal contents.

[1376] Input: Image data sent to the cloud server, user identification information

[1377] Output: Identification and quantity of extracted food items

[1378] Step 4:

[1379] Calorie Counting

[1380] Server: Calculates the calories of each food item based on its identification and quantity. The total calorie count is then output.

[1381] Input: Extracted food identities and quantities

[1382] Output: Calculated calorie information

[1383] Step 5:

[1384] Saving calorie information

[1385] Server: Stores the calculated calorie information in the user's health database, which is managed on the cloud.

[1386] Input: Calculated calorie information, user identification information

[1387] Output: Calorie information stored in the user's health database

[1388] Step 6:

[1389] Health data feedback

[1390] Device: Once the health database update is complete, the server notifies the device of the update information, which then displays the calorie information and whether the meal is balanced.

[1391] Input: Updated health data

[1392] Output: Calorie information and feedback displayed on the device

[1393] Step 7:

[1394] Blood pressure data measurement

[1395] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, they measure their morning blood pressure data.

[1396] Input: Physiological data measured by a blood pressure monitor

[1397] Output: Measured blood pressure data

[1398] Step 8:

[1399] Sending blood pressure data

[1400] Device: The smartphone automatically collects data from the blood pressure monitor and sends it to a cloud server. The API request also includes user identification information.

[1401] Input: Blood pressure data measured by a blood pressure monitor, user identification information

[1402] Output: Blood pressure data sent to the cloud server

[1403] Step 9:

[1404] Blood pressure data storage

[1405] Server: The server stores the received blood pressure data in the user's health database.

[1406] Input: Blood pressure data sent to the cloud server, user identification information

[1407] Output: Blood pressure data stored in a health database

[1408] Step 10:

[1409] Blood pressure data feedback

[1410] Terminal: Once the health database update is complete, the server notifies the terminal of the update information, and the terminal displays to the user whether the blood pressure range is normal or not.

[1411] Input: Updated health data

[1412] Output: Blood pressure data and feedback displayed on the device

[1413] Step 11:

[1414] Providing personalized service

[1415] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analytics and personalized coaching services.

[1416] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1417] Input: User's historical data

[1418] Output: Personalized health plan and advice

[1419] (Application example 1)

[1420] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1421] In modern society, users are often too busy to manage their health in their daily lives. In particular, there is a demand for users to easily understand the calories and nutritional information of their meals, as well as their blood pressure status, and receive appropriate feedback and health suggestions. The present invention aims to provide a system that enables such health management to be carried out efficiently and effectively.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1423] In this invention, the server includes: means for a user to take photos of meals; terminal means for transmitting the photo data to the server; means for the server to analyze the photo data and extract calorie information; means for the server to store the extracted calorie information in the user's health database; means for generating health suggestions based on the calorie information stored in the terminal and providing feedback to the user; means for the user to receive health suggestions based on meal selections; means for the user to connect a blood pressure monitor and acquire blood pressure data; terminal means for transmitting the acquired blood pressure data to the server; means for the server to store the blood pressure data and update the health database; means for notifying the user of meal selections and next actions (e.g., drinking water, resting, light exercise) based on the updated blood pressure information; means for the server to analyze the collected data using AI technology; means for generating individual meal suggestions and detailed health status reports and advice based on the analysis results; and means for transmitting the generated reports and advice to the terminal and displaying them to the user. This allows users to easily manage their daily diet and health status while receiving individual health suggestions.

[1424] "User" refers to an individual who uses this system to take photos of their meals and manage their health data.

[1425] "Terminal means" refers to a device (such as a smartphone or tablet) that allows a user to take a photo of their meal and send that data to a server.

[1426] "Server" refers to a computer system located on the cloud that receives, analyzes, and stores photo data and blood pressure data, and provides feedback to users.

[1427] "Calorie information" refers to the calorie content of food analyzed from photos of meals taken by the user.

[1428] "Health database" refers to a database system that centrally manages and stores data related to a user's health (calorie information, blood pressure data, etc.).

[1429] "Feedback" refers to advice and notifications provided to users based on analyzed calorie information and health data.

[1430] "Generative AI technology" refers to artificial intelligence technology used to perform advanced analysis and predictions based on data collected by the server.

[1431] "Health Suggestions" refers to specific dietary choices and next action advice provided based on a user's diet and blood pressure data.

[1432] The embodiments for carrying out the present invention will be described in detail below.

[1433] System configuration

[1434] This system consists of a means for users to take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to the user. It also includes linking the smartphone with health management devices such as blood pressure monitors. Furthermore, the key feature of this novelty is that the server analyzes the collected data using generative AI technology and provides detailed feedback.

[1435] Hardware and Software

[1436] Hardware

[1437] Smartphone (e.g., a general smartphone)

[1438] Tablet (e.g., a general tablet device)

[1439] Blood pressure monitor (e.g., a blood pressure monitor with Bluetooth connectivity)

[1440] software

[1441] Cloud server (e.g. AWS, Google Cloud)

[1442] Image recognition algorithms (e.g. TensorFlow, Keras)

[1443] Health data management systems (e.g., Firebase)

[1444] Data Entry

[1445] Entering meal data

[1446] User: The user takes a photo of their meal using their smartphone camera, which is displayed in the app for review purposes.

[1447] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[1448] Entering blood pressure data

[1449] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[1450] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[1451] Data analysis and feedback

[1452] Image analysis and calorie counting

[1453] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result of the analysis, the identification information and quantity of specific foods are extracted.

[1454] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1455] Processing blood pressure data

[1456] Server: The cloud-based server receives the blood pressure data, stores it in the user's health database, and notifies the user about food choices and next actions (e.g., drinking water, taking a break, or doing light exercise) based on the updated blood pressure information.

[1457] User Notification and Interface

[1458] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays calorie information and health suggestions to the user. Specific suggestions include the next meal selection and suggested activities.

[1459] Users: Users can view feedback on their smartphones, tablets, or smart glasses to help manage their health.

[1460] Advanced Services

[1461] Personalized service

[1462] Users: Users can subscribe to the paid service and receive diet and exercise plans, as well as personalized health recommendations based on their food choices.

[1463] Server: The server uses generative AI technology to create a personalized diet plan based on the user's past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[1464] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1465] Specific examples

[1466] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[1467] Prompt Sentence Examples

[1468] "Calculate the calories of a user who eats bread, salad, and yogurt for breakfast, and compare that with their blood pressure data to generate appropriate health suggestions."

[1469] This allows users to easily manage their diet and health status and receive personalized health suggestions.

[1470] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1471] Step 1:

[1472] The user takes a photo of the meal. The user uses the smartphone camera to take a photo of the meal through the application. The input is the photo of the meal taken with the smartphone camera, and the output is the photo data for confirmation that is displayed in the application.

[1473] Step 2:

[1474] The device sends the captured photo data to the server. The device then sends the photo data and user identification information (e.g., user ID) to the cloud server via an API request. The input is the captured photo data and user identification information, and the output is the photo data and user identification information sent to the server.

[1475] Step 3:

[1476] The server analyzes the photo data and extracts calorie information. The server then analyzes the photo data using an image recognition algorithm (such as TensorFlow or Keras) to identify the food contents. The input is the photo data, and the output is the food identification information and its quantity.

[1477] Step 4:

[1478] The server calculates calories based on the extracted food identification information and quantity. The server then references the standard calorie value for each identified food and calculates the total calories for the specific meal. The input is the food identification information and quantity, and the output is the total calorie information.

[1479] Step 5:

[1480] The extracted calorie information is stored in a health database. The server stores the calculated calorie information in the user's health database. The input is the total calorie information and the user identification information, and the output is the updated health database.

[1481] Step 6:

[1482] Health suggestions are generated based on the calorie information stored on the device and feedback is provided to the user. The server uses the stored calorie information and generative AI technology to generate detailed health suggestions and send them to the device. The input is the stored calorie information and past health data, and the output is the generated health suggestions and feedback information.

[1483] Step 7:

[1484] The user acquires blood pressure data using a blood pressure monitor. The user connects the blood pressure monitor to a smartphone via Bluetooth and performs measurements. The input is the blood pressure data obtained from the blood pressure monitor, and the output is the blood pressure data transferred to the smartphone.

[1485] Step 8:

[1486] The device sends blood pressure data to the server. The smartphone sends the blood pressure data and user identification information to a server on the cloud. The input is the blood pressure data and user identification information, and the output is the blood pressure data sent to the server.

[1487] Step 9:

[1488] The server stores the blood pressure data and updates the health database. The cloud server analyzes the received blood pressure data and updates the health database. The input is the blood pressure data and user identification information, and the output is the updated health database.

[1489] Step 10:

[1490] The server suggests and notifies users of meal choices and next actions based on updated calorie and blood pressure information. The server analyzes the data using a generative AI model, generates optimal meal choices and action suggestions (e.g., drinking water, resting, light exercise) for the user, and notifies the device. The input is updated calorie and blood pressure information and past health data, and the output is the generated health suggestions and notification information.

[1491] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1492] The embodiments for carrying out the present invention will be described in detail below.

[1493] System configuration

[1494] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[1495] Data Entry

[1496] Entering meal data

[1497] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[1498] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[1499] Entering blood pressure data

[1500] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[1501] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[1502] Entering emotion data

[1503] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[1504] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[1505] Data analysis and feedback

[1506] Image analysis and calorie counting

[1507] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm, extracting the type and amount of food as the analysis result.

[1508] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1509] Processing blood pressure data

[1510] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1511] Emotional Data Analysis

[1512] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[1513] User Notification and Interface

[1514] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1515] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[1516] Advanced Services

[1517] Personalized service

[1518] User: A user subscribes to a paid service and uses diet and training plans.

[1519] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[1520] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1521] Specific examples

[1522] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[1523] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[1524] Furthermore, if the user uses the emotion engine to detect smiles or signs of stress when taking a photo, the emotion data will be sent to the server, and feedback and advice will be tailored accordingly. For example, if it is detected that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[1525] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[1526] The above is a specific embodiment of the present invention. This system allows users to effectively collect health information and emotional data and receive advice, making daily health and mental health management easier.

[1527] The processing flow will be explained below.

[1528] Dietary data entry and analysis

[1529] Step 1:

[1530] User: Opens the camera app on their smartphone and takes a photo of their meal.

[1531] Step 2:

[1532] On the device: The captured photo is displayed in the app for the user to review.

[1533] Step 3:

[1534] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[1535] Step 4:

[1536] Server: Receives the API request and retrieves the photo data.

[1537] Step 5:

[1538] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[1539] Step 6:

[1540] Server: Calculates calories from the extracted food identification information and quantity.

[1541] Step 7:

[1542] Server: Stores the calculated calorie information in the user's health database.

[1543] Step 8:

[1544] Server: Notifies the device that the health database update is complete.

[1545] Step 9:

[1546] Device: Show the user "This morning's meal was balanced."

[1547] Blood pressure data entry and processing

[1548] Step 1:

[1549] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[1550] Step 2:

[1551] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[1552] Step 3:

[1553] User: Measures blood pressure with a blood pressure monitor and records the value.

[1554] Step 4:

[1555] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[1556] Step 5:

[1557] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[1558] Step 6:

[1559] Server: Receives the API request and retrieves the blood pressure data.

[1560] Step 7:

[1561] Server: Stores blood pressure data in a health database.

[1562] Step 8:

[1563] Server: Notifies the device that the health database update is complete.

[1564] Step 9:

[1565] Terminal: Informs the user whether the blood pressure range is normal or not.

[1566] Emotion data input and processing

[1567] Step 1:

[1568] User: Take a photo of their facial expression using their smartphone camera.

[1569] Step 2:

[1570] Device: Sends facial expression data to the emotion engine and analyzes the user's emotions.

[1571] Step 3:

[1572] Emotion engine: Acquires analyzed emotion data and sends it to a server on the cloud.

[1573] Step 4:

[1574] Server: Stores emotion data in a health database.

[1575] Step 5:

[1576] Server: Adjusts feedback and advice appropriately based on emotional data.

[1577] Step 6:

[1578] Device: Display emotional feedback and advice in the user interface.

[1579] Data analysis and health advice

[1580] Step 1:

[1581] Server: Analyzes all collected data using generative AI technology.

[1582] Step 2:

[1583] Server: Generates detailed health reports and advice.

[1584] Step 3:

[1585] Server: Sends the generated report to the device.

[1586] Step 4:

[1587] Terminal: Display reports and advice in the user interface and notify the user.

[1588] Advanced Services

[1589] Step 1:

[1590] User: Subscribe to the paid service and use the diet and training plans.

[1591] Step 2:

[1592] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[1593] Step 3:

[1594] Server: Sends the created plan to the device.

[1595] Step 4:

[1596] On your device: Display your plan and daily goals in the user interface.

[1597] Step 5:

[1598] User: Manage your daily health according to the plan.

[1599] Example 2

[1600] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1601] Conventional health management systems collect dietary and blood pressure data, but are unable to provide feedback that takes into account the user's emotional state. Furthermore, they only analyze the collected data individually, lacking the ability to grasp the user's overall health status and provide appropriate advice to the user. This makes it difficult for users to effectively manage their own health.

[1602] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to take a photo of a meal, a communication means for transmitting the photo data to the server, a means for the server to analyze the photo data using generative AI technology and extract calorie information, a means for storing the extracted calorie information in the user's health database, a means for displaying the calorie information stored on the terminal and providing feedback to the user, a means for analyzing the user's facial expression using an emotion engine and evaluating the user's current emotional state, and a means for adjusting the content of feedback and advice based on the user's emotional data. This allows the user to receive more personalized feedback and advice that takes into account emotional data as well as health information such as diet and blood pressure.

[1603] A "user" is an entity that uses the system to record health information and receive feedback and advice.

[1604] "Means for taking photos of meals" refers to a function that allows a user to capture images of meals using the smartphone camera.

[1605] "Communication means" refers to the technology or protocol used to transmit photograph data and blood pressure data from the terminal to the server.

[1606] "Generative AI technology" is a technology that uses artificial intelligence algorithms executed on a server to analyze and recognize data.

[1607] "Calorie information" is an energy value calculated based on the types and amounts of food consumed by the user.

[1608] A "health database" is a data storage device that stores health-related information such as a user's dietary data, blood pressure data, and emotional data.

[1609] "Means for providing feedback" refers to a function that notifies and displays the results of analysis and advice from the server on the user's device.

[1610] An "emotion engine" is software or algorithms that analyze a user's facial expression data to recognize their emotional state.

[1611] "Means for analyzing facial expressions" refers to technologies and algorithms that analyze the user's facial expression data acquired by a smartphone camera.

[1612] "Emotion data" is information that indicates the user's facial expression and other emotional states.

[1613] The "means for adjusting the content of feedback and advice" is a technique for appropriately changing the content of the feedback and advice provided based on the user's emotional state.

[1614] MODE FOR CARRYING OUT THE INVENTION

[1615] The present invention is a system for comprehensively managing a user's health information and emotional data, and providing feedback and advice. Specific embodiments for carrying out the invention are described in detail below.

[1616] System configuration

[1617] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[1618] Data Entry

[1619] Entering meal data

[1620] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[1621] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[1622] Entering blood pressure data

[1623] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[1624] Device: Automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[1625] Entering emotion data

[1626] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[1627] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[1628] Data analysis and feedback

[1629] Image analysis and calorie counting

[1630] Server: The server receives the photo data in the cloud and applies image recognition algorithms using generative AI technology. The analysis results include the type and quantity of food.

[1631] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1632] Processing blood pressure data

[1633] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1634] Emotional Data Analysis

[1635] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[1636] User Notification and Interface

[1637] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1638] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[1639] Advanced Services

[1640] Personalized service

[1641] User: A user subscribes to a paid service and uses diet and training plans.

[1642] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[1643] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1644] Specific examples

[1645] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[1646] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[1647] Furthermore, if the emotion engine detects smiles or signs of stress when taking a photo, the emotion data is sent to the server, and feedback and advice can be tailored accordingly. For example, if it detects that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[1648] Prompt Sentence Examples

[1649] "Take a photo of bread, salad, and yogurt for breakfast and calculate the calories. Use an emotion engine to analyze the user's emotions and generate sentences with tailored feedback based on the results."

[1650] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[1651] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1652] Step 1:

[1653] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[1654] Input: A photo of a meal taken with a smartphone camera

[1655] Output: Photo data displayed in the app

[1656] What it does: The user opens their smartphone camera, takes a photo that captures the entire meal, then opens the app to view the photo.

[1657] Step 2:

[1658] Device: Sends an API request containing the confirmed photo data and user identification information to a server in the cloud.

[1659] Input: Verified photo data and user identification

[1660] Output: API request sent to the server

[1661] Specific operation: When the user checks the photo displayed on the device screen and taps the OK button, an API request containing the photo data and user ID is created and sent to a server in the cloud.

[1662] Step 3:

[1663] Server: The server receives the photo data on the cloud and uses generative AI technology to apply image recognition algorithms to analyze the meal contents.

[1664] Input: Photo data sent to the server

[1665] Output: Analysis results of meal contents (type and amount of food)

[1666] How it works: The received photo data is fed into a generative AI model, which then uses image recognition algorithms to identify food items such as bread, salad, and yogurt from the photo.

[1667] Step 4:

[1668] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1669] Input: Food identity and quantity

[1670] Output: Calorie information stored in the user's health database

[1671] What it does: Calculate the energy value of each food item, add them up to get the total calorie value, and store it in a database.

[1672] Step 5:

[1673] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1674] Input: Blood pressure data sent from the blood pressure monitor

[1675] Output: Blood pressure information stored in the user's health database

[1676] Specific operation: Receives blood pressure data and records it directly in the database.

[1677] Step 6:

[1678] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[1679] Input: Facial expression data captured with a smartphone

[1680] Output: Facial expression data displayed in the app

[1681] Specific operation: The user uses the smartphone camera to capture their facial expression.

[1682] Step 7:

[1683] Device: Analyzes facial expression data, uses an emotion engine to recognize the user's emotions, and sends the acquired emotion data to a server on the cloud.

[1684] Input: Photographed facial expression data

[1685] Output: Emotion data sent to the server

[1686] What it does: It runs a facial expression recognition algorithm to identify emotions such as smiling or stressed, and sends that data to a server.

[1687] Step 8:

[1688] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[1689] Input: Emotion data sent to the server

[1690] Output: Emotion analysis results

[1691] Specific operation: Analyzes emotion data using analytical algorithms to identify the user's emotional state, such as "happy," "sad," or "stressed."

[1692] Step 9:

[1693] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1694] Input: Health database updates

[1695] Output: A message to inform the user

[1696] Specific operation: When an update notification arrives, a message such as "Calorie information has been updated" or "Blood pressure is within normal range" will be displayed in the notification center of the device.

[1697] Step 10:

[1698] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[1699] Input: Sentiment analysis results

[1700] Output: tailored feedback and advice

[1701] Specific actions: Specific advice such as "Try taking a deep breath to relax" is displayed.

[1702] Step 11:

[1703] User: Subscribe to the paid service and use the diet and training plans.

[1704] Input: Subscription information

[1705] Output: Authorization to use paid services

[1706] Specific behavior: Select a paid service through an in-app purchase and complete the payment.

[1707] Step 12:

[1708] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[1709] Input: Past dietary data, exercise data, emotional data

[1710] Output: Personalized health plan

[1711] What it does: It retrieves past data from a database and uses algorithms to create optimal diet and training plans.

[1712] Step 13:

[1713] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1714] Enter: personalized health plans.

[1715] Output: Display health plan

[1716] Specific behavior: The device notifies you of the plan and displays specific guidelines such as "Today's menu."

[1717] (Application example 2)

[1718] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1719] Conventional dietary management systems have difficulty accurately recording a user's dietary habits and providing feedback based on their health status. Furthermore, they are unable to integrate and analyze multiple health-related data, such as the user's emotional state and blood pressure data, making them inadequate for supporting comprehensive health management. Furthermore, food delivery services also face the challenge of being unable to provide customized meal recommendations based on the user's health and emotional state.

[1720] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a photo of the meal, terminal means for transmitting the photo data to the server, means for the server to analyze the photo data and extract calorie information, means for storing the extracted calorie information in the user's health database, means for displaying the calorie information stored on the terminal and providing feedback to the user, means for analyzing the meal ordered by the user on the server and extracting calorie information and nutritional components, and means for providing meal suggestions and feedback according to the user's health status based on the extracted information. This enables accurate recording and analysis of the user's meal contents, integrated analysis of multiple health-related data including blood pressure data and emotional data, and customized meal suggestions and feedback based on the user's health status and emotional state.

[1721] The "means for a user to take a photo of a meal" refers to a function or device that allows a user to take a photo of a meal using a smartphone or camera device.

[1722] The "terminal means for transmitting the photographic data to the server" is a terminal having a communication function for transmitting data of the photograph taken to a server on the cloud or a remote server.

[1723] "Means for the server to analyze the photo data and extract calorie information" refers to a function in which the server uses image analysis technology to analyze the photo data received, identify the type and amount of food, and calculate calories.

[1724] The "means for saving extracted calorie information in the user's health database" is a function for recording and saving calorie information obtained by analysis in a user's personal health database.

[1725] "Means for displaying calorie information stored on the device and providing feedback to the user" refers to a function that displays calorie information stored on the device on the user's device and provides feedback on the contents of the meal and calorie consumption.

[1726] "Means for the user to connect a blood pressure monitor and obtain blood pressure data" refers to a function that allows the user to connect a blood pressure monitor via a communication means such as Bluetooth, measure blood pressure, and obtain the data.

[1727] The "terminal means for transmitting acquired blood pressure data to a server" is a communication function of the terminal for transmitting acquired blood pressure data to a server on the cloud.

[1728] "Means for the server to save the blood pressure data and update the health database" refers to a function that allows the server to save the received blood pressure data in the user's health database and update the database based on the latest data.

[1729] The "means for notifying the user of updated blood pressure information" is a function that notifies the user's device of updated blood pressure information and provides feedback according to the situation.

[1730] "Means for analyzing the user's emotional data and adjusting the content of advice based on health management" refers to a function that analyzes the user's facial expressions and other emotional expressions to extract emotional data, and adjusts the feedback and advice content based on that data.

[1731] "Means for the server to analyze collected data using generative AI technology" refers to a function that uses generative AI technology to comprehensively analyze collected dietary, blood pressure, and emotion data.

[1732] "Means for generating detailed reports and advice on health status based on analysis results" is a function that generates detailed reports and specific advice on the user's health status based on the information obtained through analysis.

[1733] The "means for transmitting the generated report and advice to the terminal and displaying it to the user" is a function for transmitting the generated report and advice to the user's device and visually displaying it.

[1734] The "means for providing customized meal suggestions based on the user's emotional state and health data" is a function that makes meal suggestions tailored to the user's situation based on collected emotional data and health data.

[1735] System configuration

[1736] The present invention is a system that allows a user to take a photo of a meal, analyzes the photo data, and extracts calorie information. This system specifically comprises the following components:

[1737] Entering meal data

[1738] 1. Users

[1739] The user takes a photo of the meal using a smartphone or camera device.

[1740] The photo data is sent via the device to a server on the cloud.

[1741] 2. Terminal

[1742] The terminal includes a communication means for transmitting the captured photo data to the server.

[1743] Entering blood pressure data

[1744] 1. Users

[1745] Users connect the blood pressure monitor to their smartphone using wireless communication functions such as Bluetooth to measure their blood pressure.

[1746] 2. Terminal

[1747] The device transmits the blood pressure data obtained from the blood pressure monitor to a server on the cloud.

[1748] Entering emotion data

[1749] 1. Users

[1750] Users take photos of their facial expressions with their smartphone camera while eating or measuring their blood pressure.

[1751] 2. Terminal

[1752] The device uses image analysis technology to analyze facial expression data and recognize the user's emotional state, and the recognized emotional data is sent to a cloud server.

[1753] Data analysis and feedback

[1754] 1. Server

[1755] Image analysis and calorie extraction

[1756] The server receives the photo data in the cloud and uses image analysis algorithms to analyze the meal contents. The analysis results include the type and amount of food, and calorie calculations. The calculation results are stored in the user's health database.

[1757] Processing blood pressure data

[1758] The server receives the blood pressure data and stores it in the user's health database.

[1759] Emotional Data Analysis

[1760] The server analyzes the acquired emotion data using an emotion engine to understand the user's current emotional state.

[1761] 2. Terminal

[1762] Notification and display of information

[1763] Once the health database update is complete, the server notifies the device, which then displays calorie information, blood pressure ranges, and emotional feedback and advice to the user.

[1764] Advanced Services

[1765] 1. Server

[1766] The server uses AI technology to comprehensively analyze the collected data, and automatically generates a detailed health report and advice based on the results.

[1767] 2. Terminal

[1768] The generated reports and advice are sent to the user's device for visual display, and the system also provides customized dietary suggestions based on emotional state and health data.

[1769] Specific examples

[1770] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone and sends it to a server. The server uses an image analysis algorithm to analyze the photo and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal is balanced."

[1771] Similarly, when a user measures their blood pressure, the data is sent to the server and stored in a health database. The device is notified whether the blood pressure range is normal or not. The device also analyzes the user's facial expression data and provides feedback and advice based on their emotional state.

[1772] Prompt Sentence Examples

[1773] "Below is a user-taken photo of a meal. Please analyze this photo to extract the food type and calories:

[1774] bread

[1775] salad

[1776] yogurt

[1777] And the user's blood pressure data is as follows:

[1778] Blood pressure: 135 / 85 mmHg

[1779] The user's emotion data is as follows:

[1780] Emotions: Not much stress

[1781] Use this data to generate health advice and feedback for users.”

[1782] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1783] Step 1:

[1784] A user takes a photo of a meal using a smartphone. At this time, the user checks the photo and saves it on the device. The input data is the photo of the meal, and the output data is the saved photo file.

[1785] Step 2:

[1786] The user's device sends the stored photo data to a server on the cloud. The input data is the photo file of the meal, and the output data is the photo data sent to the server. Specifically, the device uploads the photo data to the server using an HTTP request.

[1787] Step 3:

[1788] The server analyzes the received photo data. The input data is the photo data sent to the server, and the output data is the analyzed data on the type and amount of food. The server uses an image analysis algorithm to identify the type and amount of food.

[1789] Step 4:

[1790] The server calculates calorie information based on the analysis results. The input data is the type and amount of food, and the output data is the calculated calorie information. The server adds up the calorie values ​​for each food item to calculate the total calories.

[1791] Step 5:

[1792] The server saves the calorie information to the user's health database. The input data is the calculated calorie information, and the output data is the updated health database. The server adds the calorie information to the existing database.

[1793] Step 6:

[1794] The user connects a blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. The input data is the user's blood pressure value, and the output data is the blood pressure data recorded on the device. The data from the blood pressure monitor is automatically sent to the smartphone.

[1795] Step 7:

[1796] The device sends blood pressure data to the server. The input data is the blood pressure data recorded on the device, and the output data is the blood pressure data sent to the server. The device uploads the blood pressure data using an HTTP request.

[1797] Step 8:

[1798] The server saves the blood pressure data to the user's health database. The input data is the submitted blood pressure data, and the output data is the updated health database. The server adds the blood pressure data to the existing database.

[1799] Step 9:

[1800] The user captures their facial expression with a smartphone camera to acquire emotional data. The input data is the user's facial expression image, and the output data is analyzed emotional data. The device analyzes the facial expression data in real time.

[1801] Step 10:

[1802] The device sends emotion data to the server. The input data is the analyzed emotion data, and the output data is the emotion data sent to the server. The device uploads the emotion data using an HTTP request.

[1803] Step 11:

[1804] The server analyzes the emotional data and understands the user's emotional state. The input data is the emotional data sent to the server, and the output data is information about the user's current emotional state. The server uses an emotion engine to identify the emotional state.

[1805] Step 12:

[1806] The server integrates all data to evaluate the user's health status and generate feedback. The input data is calorie information, blood pressure data, and emotional data, and the output data is a comprehensive health report and advice. The server analyzes the data using generative AI technology.

[1807] Step 13:

[1808] The server sends the generated report and advice to the user's terminal. The input data are the generated health report and advice, and the output data are the report and advice received by the terminal. The terminal displays a notification to the user.

[1809] Step 14:

[1810] The user checks the report and advice through the terminal and receives feedback on their own health condition. The input data is the report and advice received from the server, and the output data is the feedback information displayed to the user. This allows the user to manage their daily health.

[1811] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1813] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1814] [Fourth embodiment]

[1815] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1816] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1817] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1818] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1819] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1820] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1821] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1822] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1823] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1824] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1825] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1826] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1827] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1828] The embodiments for carrying out the present invention will be described in detail below.

[1829] System configuration

[1830] The system consists of a device on which users take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to users. It also includes a smartphone that can be linked to health devices such as blood pressure monitors.

[1831] Data Entry

[1832] Entering meal data

[1833] User: The user takes a photo of their breakfast with their smartphone camera, which is displayed in the app for review.

[1834] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[1835] Entering blood pressure data

[1836] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[1837] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[1838] Data analysis and feedback

[1839] Image analysis and calorie counting

[1840] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result, the identification information and quantities of bread, salad, and yogurt are extracted.

[1841] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[1842] Processing blood pressure data

[1843] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[1844] User Notification and Interface

[1845] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1846] Users: Users can check the feedback on their smartphones and use it to manage their health.

[1847] Advanced Services

[1848] Personalized service

[1849] Users: Users can subscribe to paid services and access diet and training plans.

[1850] Server: The server creates a personalized diet plan based on the user's past diet and exercise data, along with advanced analytics and personalized coaching services.

[1851] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1852] Specific examples

[1853] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[1854] This allows users to effortlessly record information about their diet and blood pressure, enabling them to manage their health appropriately.

[1855] The above is a specific embodiment for carrying out the present invention. This system allows users to effectively collect health information and receive advice, making daily health management easier.

[1856] The processing flow will be explained below.

[1857] Dietary data entry and analysis

[1858] Step 1:

[1859] User: Opens the camera app on their smartphone and takes a photo of their meal.

[1860] Step 2:

[1861] On the device: The captured photo is displayed in the app for the user to review.

[1862] Step 3:

[1863] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[1864] Step 4:

[1865] Server: Receives the API request and retrieves the photo data.

[1866] Step 5:

[1867] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[1868] Step 6:

[1869] Server: Calculates calories from the extracted food identification information and quantity.

[1870] Step 7:

[1871] Server: Stores the calculated calorie information in the user's health database.

[1872] Step 8:

[1873] Server: Notifies the device that the health database update is complete.

[1874] Step 9:

[1875] Device: Show the user "This morning's meal was balanced."

[1876] Blood pressure data entry and processing

[1877] Step 1:

[1878] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[1879] Step 2:

[1880] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[1881] Step 3:

[1882] User: Measures blood pressure with a blood pressure monitor and records the value.

[1883] Step 4:

[1884] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[1885] Step 5:

[1886] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[1887] Step 6:

[1888] Server: Receives the API request and retrieves the blood pressure data.

[1889] Step 7:

[1890] Server: Stores blood pressure data in a health database.

[1891] Step 8:

[1892] Server: Notifies the device that the health database update is complete.

[1893] Step 9:

[1894] Terminal: Informs the user whether the blood pressure range is normal or not.

[1895] Data analysis and health advice

[1896] Step 1:

[1897] Server: Analyzes all collected data using generative AI technology.

[1898] Step 2:

[1899] Server: Generates detailed health reports and advice.

[1900] Step 3:

[1901] Server: Sends the generated report to the device.

[1902] Step 4:

[1903] Terminal: Display reports and advice in the user interface and notify the user.

[1904] Advanced Services

[1905] Step 1:

[1906] User: Subscribe to the paid service and use the diet and training plans.

[1907] Step 2:

[1908] Server: Creates a personalized plan based on the user's past diet and exercise data.

[1909] Step 3:

[1910] Server: Sends the created plan to the device.

[1911] Step 4:

[1912] On your device: Display your plan and daily goals in the user interface.

[1913] Step 5:

[1914] User: Manage your daily health according to the plan.

[1915] Example 1

[1916] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1917] Despite the increasing health consciousness in recent years, it is not easy to centrally manage one's dietary and physiological data and obtain appropriate analysis and feedback in the midst of busy daily life. Conventional systems require manual data entry and analysis, placing a heavy burden on users. Furthermore, it is difficult to provide appropriate personalized advice based on individual health data. This invention aims to solve these problems and provide a system that allows users to easily manage their health.

[1918] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1919] In this invention, the server includes a device for a user to take a food image, a mobile terminal that transmits the image data to the network server, a means for the server to analyze the image data using a recognition algorithm and extract nutritional information, a means for storing the extracted nutritional information in the user's health database, and a means for displaying the nutritional information stored in the terminal and providing feedback to the user, thereby enabling users to effortlessly record their own dietary and physiological data, receive analysis and appropriate advice, and effectively manage their health.

[1920] A "user" is an individual who utilizes the system to input dietary and physiological data and receive analysis results and feedback.

[1921] "Food image" refers to image data of the meal taken by a user with a camera.

[1922] "Devices" are devices including cameras for taking pictures of food and health devices for measuring blood pressure.

[1923] A "network server" is a computing device located on the cloud that includes the infrastructure for sending and receiving data.

[1924] A "mobile terminal" is a portable electronic device such as a smartphone or tablet through which data is transmitted to a server.

[1925] A "recognition algorithm" is a computer program that analyzes image data or physiological data and extracts useful information from it.

[1926] "Nutritional information" refers to data on nutrients such as calories, protein, fat, and carbohydrates contained in food.

[1927] "Physiological data" is data that indicates the user's physical condition, such as blood pressure, heart rate, and body temperature.

[1928] The "health database" is a database for centrally managing and storing a user's dietary data and physiological data.

[1929] "Feedback" is information or advice provided to the user based on analyzed data.

[1930] "Machine learning technology" is a technology that allows computers to learn from data, find patterns, and make predictions and decisions.

[1931] System configuration

[1932] The system of this invention includes a device (such as a smartphone with a built-in camera) for users to take photos of their meals, a mobile terminal (such as a smartphone or tablet) for sending the captured photo data to a network server, software for the server to analyze the photo data using a recognition algorithm and extract nutritional information, and a database management system for storing the extracted nutritional information in the user's health database. It also includes integration of the smartphone with a health device (such as a blood pressure monitor) for measuring physiological data (such as blood pressure).

[1933] Data Entry

[1934] Entering meal data

[1935] User: The user takes a photo of their meal using their smartphone camera. For example, if they have bread, salad, and yogurt for breakfast, they take a photo that includes these items.

[1936] Device: The captured photo data is sent to a cloud server. At this time, the user's identification information is included in the API request along with the photo data.

[1937] Entering blood pressure data

[1938] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, when they measure their blood pressure in the morning, the data is recorded on the blood pressure monitor.

[1939] Device: The smartphone automatically acquires blood pressure data from the blood pressure monitor and sends it to a cloud server. Again, the user's identification information is included in the API request.

[1940] Data analysis and feedback

[1941] Image analysis and calorie counting

[1942] Server: Receives photo data in the cloud and analyzes the meal contents using image recognition algorithms. For example, it extracts the identity and quantity of bread, salad, and yogurt from the photo.

[1943] Server: Calculates calories based on the extracted food identification information and quantity, and stores the results in the user's health database.

[1944] Processing blood pressure data

[1945] Server: The cloud server receives the blood pressure data and automatically stores it in a health database.

[1946] User Notification and Interface

[1947] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[1948] User: Users can check the latest feedback on their smartphones and use it to manage their health.

[1949] Advanced Services

[1950] Personalized service

[1951] Users: Users can subscribe to paid services to access diet and training plans.

[1952] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[1953] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[1954] Specific examples

[1955] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then stores the calculation results in the user's health database. A notification is then displayed on the device saying, "This morning's meal was balanced." Similarly, once blood pressure data is sent to the server and stored in the health database, the device is notified whether the blood pressure range is normal.

[1956] Example prompt sentence:

[1957] A user takes a photo of their breakfast with their smartphone camera and sends it to a cloud server, which then uses an image recognition algorithm to analyze the meal and calculate the calories. We are looking for a natural language explanation of the program process for this system.

[1958] This system allows the user to easily manage their health on a daily basis, which is a specific embodiment of the present invention.

[1959] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1960] Step 1:

[1961] Taking photos

[1962] User: The user uses the smartphone camera to take a photo of a meal, for example, bread, salad, and yogurt prepared for breakfast.

[1963] Input: Food image data

[1964] Output: Captured image data

[1965] Step 2:

[1966] Sending photo data

[1967] Device: The smartphone sends the captured photo data to a cloud server, including the image data and user identification information in an API request.

[1968] Input: Captured image data, user identification information

[1969] Output: Image data and user identification information sent to the cloud server

[1970] Step 3:

[1971] Image analysis

[1972] Server: The server analyzes the received photo data in the cloud using an image recognition algorithm (e.g., TensorFlow). The identification information and quantities of bread, salad, and yogurt are extracted as meal contents.

[1973] Input: Image data sent to the cloud server, user identification information

[1974] Output: Identification and quantity of extracted food items

[1975] Step 4:

[1976] Calorie Counting

[1977] Server: Calculates the calories of each food item based on its identification and quantity. The total calorie count is then output.

[1978] Input: Extracted food identities and quantities

[1979] Output: Calculated calorie information

[1980] Step 5:

[1981] Saving calorie information

[1982] Server: Stores the calculated calorie information in the user's health database, which is managed on the cloud.

[1983] Input: Calculated calorie information, user identification information

[1984] Output: Calorie information stored in the user's health database

[1985] Step 6:

[1986] Health data feedback

[1987] Device: Once the health database update is complete, the server notifies the device of the update information, which then displays the calorie information and whether the meal is balanced.

[1988] Input: Updated health data

[1989] Output: Calorie information and feedback displayed on the device

[1990] Step 7:

[1991] Blood pressure data measurement

[1992] User: The user connects the blood pressure monitor to a smartphone via Bluetooth and measures their blood pressure. For example, they measure their morning blood pressure data.

[1993] Input: Physiological data measured by a blood pressure monitor

[1994] Output: Measured blood pressure data

[1995] Step 8:

[1996] Sending blood pressure data

[1997] Device: The smartphone automatically collects data from the blood pressure monitor and sends it to a cloud server. The API request also includes user identification information.

[1998] Input: Blood pressure data measured by a blood pressure monitor, user identification information

[1999] Output: Blood pressure data sent to the cloud server

[2000] Step 9:

[2001] Blood pressure data storage

[2002] Server: The server stores the received blood pressure data in the user's health database.

[2003] Input: Blood pressure data sent to the cloud server, user identification information

[2004] Output: Blood pressure data stored in a health database

[2005] Step 10:

[2006] Blood pressure data feedback

[2007] Terminal: Once the health database update is complete, the server notifies the terminal of the update information, and the terminal displays to the user whether the blood pressure range is normal or not.

[2008] Input: Updated health data

[2009] Output: Blood pressure data and feedback displayed on the device

[2010] Step 11:

[2011] Providing personalized service

[2012] Server: Generates personalized diet plans based on users' past diet and exercise data, and also provides advanced analytics and personalized coaching services.

[2013] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[2014] Input: User's historical data

[2015] Output: Personalized health plan and advice

[2016] (Application example 1)

[2017] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2018] In modern society, users are often too busy to manage their health in their daily lives. In particular, there is a demand for users to easily understand the calories and nutritional information of their meals, as well as their blood pressure status, and receive appropriate feedback and health suggestions. The present invention aims to provide a system that enables such health management to be carried out efficiently and effectively.

[2019] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2020] In this invention, the server includes: means for a user to take photos of meals; terminal means for transmitting the photo data to the server; means for the server to analyze the photo data and extract calorie information; means for the server to store the extracted calorie information in the user's health database; means for generating health suggestions based on the calorie information stored in the terminal and providing feedback to the user; means for the user to receive health suggestions based on meal selections; means for the user to connect a blood pressure monitor and acquire blood pressure data; terminal means for transmitting the acquired blood pressure data to the server; means for the server to store the blood pressure data and update the health database; means for notifying the user of meal selections and next actions (e.g., drinking water, resting, light exercise) based on the updated blood pressure information; means for the server to analyze the collected data using AI technology; means for generating individual meal suggestions and detailed health status reports and advice based on the analysis results; and means for transmitting the generated reports and advice to the terminal and displaying them to the user. This allows users to easily manage their daily diet and health status while receiving individual health suggestions.

[2021] "User" refers to an individual who uses this system to take photos of their meals and manage their health data.

[2022] "Terminal means" refers to a device (such as a smartphone or tablet) that allows a user to take a photo of their meal and send that data to a server.

[2023] "Server" refers to a computer system located on the cloud that receives, analyzes, and stores photo data and blood pressure data, and provides feedback to users.

[2024] "Calorie information" refers to the calorie content of food analyzed from photos of meals taken by the user.

[2025] "Health database" refers to a database system that centrally manages and stores data related to a user's health (calorie information, blood pressure data, etc.).

[2026] "Feedback" refers to advice and notifications provided to users based on analyzed calorie information and health data.

[2027] "Generative AI technology" refers to artificial intelligence technology used to perform advanced analysis and predictions based on data collected by the server.

[2028] "Health Suggestions" refers to specific dietary choices and next action advice provided based on a user's diet and blood pressure data.

[2029] The embodiments for carrying out the present invention will be described in detail below.

[2030] System configuration

[2031] This system consists of a means for users to take photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that provides the analysis results to the user. It also includes linking the smartphone with health management devices such as blood pressure monitors. Furthermore, the key feature of this novelty is that the server analyzes the collected data using generative AI technology and provides detailed feedback.

[2032] Hardware and Software

[2033] Hardware

[2034] Smartphone (e.g., a general smartphone)

[2035] Tablet (e.g., a general tablet device)

[2036] Blood pressure monitor (e.g., a blood pressure monitor with Bluetooth connectivity)

[2037] software

[2038] Cloud server (e.g. AWS, Google Cloud)

[2039] Image recognition algorithms (e.g. TensorFlow, Keras)

[2040] Health data management systems (e.g., Firebase)

[2041] Data Entry

[2042] Entering meal data

[2043] User: The user takes a photo of their meal using their smartphone camera, which is displayed in the app for review purposes.

[2044] Device: The device sends the captured photo data to a cloud server. The user's identification information is included in the API request along with the photo data.

[2045] Entering blood pressure data

[2046] User: The user connects the blood pressure monitor to their smartphone via Bluetooth and measures their blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[2047] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server in the cloud. Along with the blood pressure data, the user's identification information is included in the API request.

[2048] Data analysis and feedback

[2049] Image analysis and calorie counting

[2050] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm. As a result of the analysis, the identification information and quantity of specific foods are extracted.

[2051] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[2052] Processing blood pressure data

[2053] Server: The cloud-based server receives the blood pressure data, stores it in the user's health database, and notifies the user about food choices and next actions (e.g., drinking water, taking a break, or doing light exercise) based on the updated blood pressure information.

[2054] User Notification and Interface

[2055] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays calorie information and health suggestions to the user. Specific suggestions include the next meal selection and suggested activities.

[2056] Users: Users can view feedback on their smartphones, tablets, or smart glasses to help manage their health.

[2057] Advanced Services

[2058] Personalized service

[2059] Users: Users can subscribe to the paid service and receive diet and exercise plans, as well as personalized health recommendations based on their food choices.

[2060] Server: The server uses generative AI technology to create a personalized diet plan based on the user's past diet and exercise data, and also provides advanced analysis and personalized coaching services.

[2061] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[2062] Specific examples

[2063] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The server then saves the calculation results in the user's health database and notifies the user by displaying a message on the device saying, "This morning's meal was balanced." Similarly, if the user measures their blood pressure, the blood pressure data is sent to the server and saved in the health database. The device is then notified whether their blood pressure is within the normal range.

[2064] Prompt Sentence Examples

[2065] "Calculate the calories of a user who eats bread, salad, and yogurt for breakfast, and compare that with their blood pressure data to generate appropriate health suggestions."

[2066] This allows users to easily manage their diet and health status and receive personalized health suggestions.

[2067] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2068] Step 1:

[2069] The user takes a photo of the meal. The user uses the smartphone camera to take a photo of the meal through the application. The input is the photo of the meal taken with the smartphone camera, and the output is the photo data for confirmation that is displayed in the application.

[2070] Step 2:

[2071] The device sends the captured photo data to the server. The device then sends the photo data and user identification information (e.g., user ID) to the cloud server via an API request. The input is the captured photo data and user identification information, and the output is the photo data and user identification information sent to the server.

[2072] Step 3:

[2073] The server analyzes the photo data and extracts calorie information. The server then analyzes the photo data using an image recognition algorithm (such as TensorFlow or Keras) to identify the food contents. The input is the photo data, and the output is the food identification information and its quantity.

[2074] Step 4:

[2075] The server calculates calories based on the extracted food identification information and quantity. The server then references the standard calorie value for each identified food and calculates the total calories for the specific meal. The input is the food identification information and quantity, and the output is the total calorie information.

[2076] Step 5:

[2077] The extracted calorie information is stored in a health database. The server stores the calculated calorie information in the user's health database. The input is the total calorie information and the user identification information, and the output is the updated health database.

[2078] Step 6:

[2079] Health suggestions are generated based on the calorie information stored on the device and feedback is provided to the user. The server uses the stored calorie information and generative AI technology to generate detailed health suggestions and send them to the device. The input is the stored calorie information and past health data, and the output is the generated health suggestions and feedback information.

[2080] Step 7:

[2081] The user acquires blood pressure data using a blood pressure monitor. The user connects the blood pressure monitor to a smartphone via Bluetooth and performs measurements. The input is the blood pressure data obtained from the blood pressure monitor, and the output is the blood pressure data transferred to the smartphone.

[2082] Step 8:

[2083] The device sends blood pressure data to the server. The smartphone sends the blood pressure data and user identification information to a server on the cloud. The input is the blood pressure data and user identification information, and the output is the blood pressure data sent to the server.

[2084] Step 9:

[2085] The server stores the blood pressure data and updates the health database. The cloud server analyzes the received blood pressure data and updates the health database. The input is the blood pressure data and user identification information, and the output is the updated health database.

[2086] Step 10:

[2087] The server suggests and notifies users of meal choices and next actions based on updated calorie and blood pressure information. The server analyzes the data using a generative AI model, generates optimal meal choices and action suggestions (e.g., drinking water, resting, light exercise) for the user, and notifies the device. The input is updated calorie and blood pressure information and past health data, and the output is the generated health suggestions and notification information.

[2088] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2089] The embodiments for carrying out the present invention will be described in detail below.

[2090] System configuration

[2091] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[2092] Data Entry

[2093] Entering meal data

[2094] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[2095] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[2096] Entering blood pressure data

[2097] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[2098] Device: The device automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[2099] Entering emotion data

[2100] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[2101] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[2102] Data analysis and feedback

[2103] Image analysis and calorie counting

[2104] Server: The server receives the photo data in the cloud and analyzes the meal contents using an image recognition algorithm, extracting the type and amount of food as the analysis result.

[2105] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[2106] Processing blood pressure data

[2107] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[2108] Emotional Data Analysis

[2109] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[2110] User Notification and Interface

[2111] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[2112] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[2113] Advanced Services

[2114] Personalized service

[2115] User: A user subscribes to a paid service and uses diet and training plans.

[2116] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[2117] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[2118] Specific examples

[2119] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[2120] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[2121] Furthermore, if the user uses the emotion engine to detect smiles or signs of stress when taking a photo, the emotion data will be sent to the server, and feedback and advice will be tailored accordingly. For example, if it is detected that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[2122] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[2123] The above is a specific embodiment of the present invention. This system allows users to effectively collect health information and emotional data and receive advice, making daily health and mental health management easier.

[2124] The processing flow will be explained below.

[2125] Dietary data entry and analysis

[2126] Step 1:

[2127] User: Opens the camera app on their smartphone and takes a photo of their meal.

[2128] Step 2:

[2129] On the device: The captured photo is displayed in the app for the user to review.

[2130] Step 3:

[2131] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[2132] Step 4:

[2133] Server: Receives the API request and retrieves the photo data.

[2134] Step 5:

[2135] Server: Analyzes the food in the photo using an image recognition algorithm, and extracts the type and quantity of food as the analysis result.

[2136] Step 6:

[2137] Server: Calculates calories from the extracted food identification information and quantity.

[2138] Step 7:

[2139] Server: Stores the calculated calorie information in the user's health database.

[2140] Step 8:

[2141] Server: Notifies the device that the health database update is complete.

[2142] Step 9:

[2143] Device: Show the user "This morning's meal was balanced."

[2144] Blood pressure data entry and processing

[2145] Step 1:

[2146] User: Connects the blood pressure monitor to a smartphone via Bluetooth.

[2147] Step 2:

[2148] Terminal: Checks the blood pressure monitor connection and notifies the user that it was successful.

[2149] Step 3:

[2150] User: Measures blood pressure with a blood pressure monitor and records the value.

[2151] Step 4:

[2152] Terminal: Automatically acquires values ​​obtained from the blood pressure monitor.

[2153] Step 5:

[2154] Device: Include blood pressure data and user identification information in the API request and send it to a server on the cloud.

[2155] Step 6:

[2156] Server: Receives the API request and retrieves the blood pressure data.

[2157] Step 7:

[2158] Server: Stores blood pressure data in a health database.

[2159] Step 8:

[2160] Server: Notifies the device that the health database update is complete.

[2161] Step 9:

[2162] Terminal: Informs the user whether the blood pressure range is normal or not.

[2163] Emotion data input and processing

[2164] Step 1:

[2165] User: Take a photo of their facial expression using their smartphone camera.

[2166] Step 2:

[2167] Device: Sends facial expression data to the emotion engine and analyzes the user's emotions.

[2168] Step 3:

[2169] Emotion engine: Acquires analyzed emotion data and sends it to a server on the cloud.

[2170] Step 4:

[2171] Server: Stores emotion data in a health database.

[2172] Step 5:

[2173] Server: Adjusts feedback and advice appropriately based on emotional data.

[2174] Step 6:

[2175] Device: Display emotional feedback and advice in the user interface.

[2176] Data analysis and health advice

[2177] Step 1:

[2178] Server: Analyzes all collected data using generative AI technology.

[2179] Step 2:

[2180] Server: Generates detailed health reports and advice.

[2181] Step 3:

[2182] Server: Sends the generated report to the device.

[2183] Step 4:

[2184] Terminal: Display reports and advice in the user interface and notify the user.

[2185] Advanced Services

[2186] Step 1:

[2187] User: Subscribe to the paid service and use the diet and training plans.

[2188] Step 2:

[2189] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[2190] Step 3:

[2191] Server: Sends the created plan to the device.

[2192] Step 4:

[2193] On your device: Display your plan and daily goals in the user interface.

[2194] Step 5:

[2195] User: Manage your daily health according to the plan.

[2196] Example 2

[2197] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2198] Conventional health management systems collect dietary and blood pressure data, but are unable to provide feedback that takes into account the user's emotional state. Furthermore, they only analyze the collected data individually, lacking the ability to grasp the user's overall health status and provide appropriate advice to the user. This makes it difficult for users to effectively manage their own health.

[2199] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for a user to take a photo of a meal, a communication means for transmitting the photo data to the server, a means for the server to analyze the photo data using generative AI technology and extract calorie information, a means for storing the extracted calorie information in the user's health database, a means for displaying the calorie information stored on the terminal and providing feedback to the user, a means for analyzing the user's facial expression using an emotion engine and evaluating the user's current emotional state, and a means for adjusting the content of feedback and advice based on the user's emotional data. This allows the user to receive more personalized feedback and advice that takes into account emotional data as well as health information such as diet and blood pressure.

[2200] A "user" is an entity that uses the system to record health information and receive feedback and advice.

[2201] "Means for taking photos of meals" refers to a function that allows a user to capture images of meals using the smartphone camera.

[2202] "Communication means" refers to the technology or protocol used to transmit photograph data and blood pressure data from the terminal to the server.

[2203] "Generative AI technology" is a technology that uses artificial intelligence algorithms executed on a server to analyze and recognize data.

[2204] "Calorie information" is an energy value calculated based on the types and amounts of food consumed by the user.

[2205] A "health database" is a data storage device that stores health-related information such as a user's dietary data, blood pressure data, and emotional data.

[2206] "Means for providing feedback" refers to a function that notifies and displays the results of analysis and advice from the server on the user's device.

[2207] An "emotion engine" is software or algorithms that analyze a user's facial expression data to recognize their emotional state.

[2208] "Means for analyzing facial expressions" refers to technologies and algorithms that analyze the user's facial expression data acquired by a smartphone camera.

[2209] "Emotion data" is information that indicates the user's facial expression and other emotional states.

[2210] The "means for adjusting the content of feedback and advice" is a technique for appropriately changing the content of the feedback and advice provided based on the user's emotional state.

[2211] MODE FOR CARRYING OUT THE INVENTION

[2212] The present invention is a system for comprehensively managing a user's health information and emotional data, and providing feedback and advice. Specific embodiments for carrying out the invention are described in detail below.

[2213] System configuration

[2214] This system consists of a device on which the user takes photos of their meals, a server that analyzes the photo data and extracts calorie information, and a program that stores the extracted calorie information in the user's health database and provides feedback to the device. It also includes a function to acquire and store blood pressure data using a blood pressure monitor. Furthermore, it also incorporates an emotion engine that recognizes the user's emotions, and has the ability to adjust the content of feedback and advice based on the emotion data.

[2215] Data Entry

[2216] Entering meal data

[2217] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[2218] On the device: The confirmed photo data and user identification information are included in an API request and sent to a server in the cloud.

[2219] Entering blood pressure data

[2220] User: Connects the blood pressure monitor to a smartphone via Bluetooth and measures blood pressure. The measured data is automatically recorded on the blood pressure monitor.

[2221] Device: Automatically acquires blood pressure data from the blood pressure monitor and sends it to a server on the cloud.

[2222] Entering emotion data

[2223] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[2224] Device: Analyzes facial expression data and uses an emotion engine to recognize the user's emotions. The acquired emotion data is sent to a server on the cloud.

[2225] Data analysis and feedback

[2226] Image analysis and calorie counting

[2227] Server: The server receives the photo data in the cloud and applies image recognition algorithms using generative AI technology. The analysis results include the type and quantity of food.

[2228] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[2229] Processing blood pressure data

[2230] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[2231] Emotional Data Analysis

[2232] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[2233] User Notification and Interface

[2234] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[2235] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[2236] Advanced Services

[2237] Personalized service

[2238] User: A user subscribes to a paid service and uses diet and training plans.

[2239] Server: The server creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[2240] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[2241] Specific examples

[2242] For example, consider a user who eats bread, salad, and yogurt for breakfast. The user takes a photo of their breakfast with their smartphone camera and sends the photo data to a server. The server analyzes the photo using an image recognition algorithm and calculates the calories in the bread, salad, and yogurt. The calculation results are stored in the user's health database, and the device notifies the user by displaying a message saying, "This morning's meal was balanced."

[2243] Similarly, when a user measures their blood pressure, the blood pressure data is sent to the server and stored in the health database, and the device is notified whether the blood pressure range is normal or not.

[2244] Furthermore, if the emotion engine detects smiles or signs of stress when taking a photo, the emotion data is sent to the server, and feedback and advice can be tailored accordingly. For example, if it detects that the user is feeling highly stressed, advice on relaxation and stress relief will be provided.

[2245] Prompt Sentence Examples

[2246] "Take a photo of bread, salad, and yogurt for breakfast and calculate the calories. Use an emotion engine to analyze the user's emotions and generate sentences with tailored feedback based on the results."

[2247] This allows users to effortlessly record information about their diet, blood pressure, and emotions, enabling appropriate health management and psychological support.

[2248] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2249] Step 1:

[2250] User: Activates the smartphone camera and takes a photo of the meal. The photo is displayed in the app for the user to review.

[2251] Input: A photo of a meal taken with a smartphone camera

[2252] Output: Photo data displayed in the app

[2253] What it does: The user opens their smartphone camera, takes a photo that captures the entire meal, then opens the app to view the photo.

[2254] Step 2:

[2255] Device: Sends an API request containing the confirmed photo data and user identification information to a server in the cloud.

[2256] Input: Verified photo data and user identification

[2257] Output: API request sent to the server

[2258] Specific operation: When the user checks the photo displayed on the device screen and taps the OK button, an API request containing the photo data and user ID is created and sent to a server in the cloud.

[2259] Step 3:

[2260] Server: The server receives the photo data on the cloud and uses generative AI technology to apply image recognition algorithms to analyze the meal contents.

[2261] Input: Photo data sent to the server

[2262] Output: Analysis results of meal contents (type and amount of food)

[2263] How it works: The received photo data is fed into a generative AI model, which then uses image recognition algorithms to identify food items such as bread, salad, and yogurt from the photo.

[2264] Step 4:

[2265] Server: Calculates calories based on the extracted food identification information and quantity, and stores the calculation results in the user's health database.

[2266] Input: Food identity and quantity

[2267] Output: Calorie information stored in the user's health database

[2268] What it does: Calculate the energy value of each food item, add them up to get the total calorie value, and store it in a database.

[2269] Step 5:

[2270] Server: The server on the cloud receives the blood pressure data and stores it in the user's health database.

[2271] Input: Blood pressure data sent from the blood pressure monitor

[2272] Output: Blood pressure information stored in the user's health database

[2273] Specific operation: Receives blood pressure data and records it directly in the database.

[2274] Step 6:

[2275] User: Uses the smartphone camera to capture their facial expressions while eating or measuring their blood pressure.

[2276] Input: Facial expression data captured with a smartphone

[2277] Output: Facial expression data displayed in the app

[2278] Specific operation: The user uses the smartphone camera to capture their facial expression.

[2279] Step 7:

[2280] Device: Analyzes facial expression data, uses an emotion engine to recognize the user's emotions, and sends the acquired emotion data to a server on the cloud.

[2281] Input: Photographed facial expression data

[2282] Output: Emotion data sent to the server

[2283] What it does: It runs a facial expression recognition algorithm to identify emotions such as smiling or stressed, and sends that data to a server.

[2284] Step 8:

[2285] Server: Analyzes the acquired emotional data using the emotion engine and understands the user's current emotional state.

[2286] Input: Emotion data sent to the server

[2287] Output: Emotion analysis results

[2288] Specific operation: Analyzes emotion data using analytical algorithms to identify the user's emotional state, such as "happy," "sad," or "stressed."

[2289] Step 9:

[2290] Device: Once the health database update is complete, the server notifies the device of the update. The device then displays the calorie information and whether the blood pressure range is normal.

[2291] Input: Health database updates

[2292] Output: A message to inform the user

[2293] Specific operation: When an update notification arrives, a message such as "Calorie information has been updated" or "Blood pressure is within normal range" will be displayed in the notification center of the device.

[2294] Step 10:

[2295] On the device: Based on emotional data, the device can tailor feedback and advice accordingly, for example, offering advice on how to relax if the user is feeling stressed.

[2296] Input: Sentiment analysis results

[2297] Output: tailored feedback and advice

[2298] Specific actions: Specific advice such as "Try taking a deep breath to relax" is displayed.

[2299] Step 11:

[2300] User: Subscribe to the paid service and use the diet and training plans.

[2301] Input: Subscription information

[2302] Output: Authorization to use paid services

[2303] Specific behavior: Select a paid service through an in-app purchase and complete the payment.

[2304] Step 12:

[2305] Server: Creates a personalized plan based on the user's past dietary and exercise data. It also takes into account emotional data to provide a plan that reflects the user's psychological state.

[2306] Input: Past dietary data, exercise data, emotional data

[2307] Output: Personalized health plan

[2308] What it does: It retrieves past data from a database and uses algorithms to create optimal diet and training plans.

[2309] Step 13:

[2310] Device: The created plan is displayed on the device, and the user follows it to manage their daily health.

[2311] Enter: personalized health plans.

[2312] Output: Display health plan

[2313] Specific behavior: The device notifies you of the plan and displays specific guidelines such as "Today's menu."

[2314] (Application example 2)

[2315] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2316] Conventional dietary management systems have difficulty accurately recording a user's dietary habits and providing feedback based on their health status. Furthermore, they are unable to integrate and analyze multiple health-related data, such as the user's emotional state and blood pressure data, making them inadequate for supporting comprehensive health management. Furthermore, food delivery services also face the challenge of being unable to provide customized meal recommendations based on the user's health and emotional state.

[2317] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a photo of the meal, terminal means for transmitting the photo data to the server, means for the server to analyze the photo data and extract calorie information, means for storing the extracted calorie information in the user's health database, means for displaying the calorie information stored on the terminal and providing feedback to the user, means for analyzing the meal ordered by the user on the server and extracting calorie information and nutritional components, and means for providing meal suggestions and feedback according to the user's health status based on the extracted information. This enables accurate recording and analysis of the user's meal contents, integrated analysis of multiple health-related data including blood pressure data and emotional data, and customized meal suggestions and feedback based on the user's health status and emotional state.

[2318] The "means for a user to take a photo of a meal" refers to a function or device that allows a user to take a photo of a meal using a smartphone or camera device.

[2319] The "terminal means for transmitting the photographic data to the server" is a terminal having a communication function for transmitting data of the photograph taken to a server on the cloud or a remote server.

[2320] "Means for the server to analyze the photo data and extract calorie information" refers to a function in which the server uses image analysis technology to analyze the photo data received, identify the type and amount of food, and calculate calories.

[2321] The "means for saving extracted calorie information in the user's health database" is a function for recording and saving calorie information obtained by analysis in a user's personal health database.

[2322] "Means for displaying calorie information stored on the device and providing feedback to the user" refers to a function that displays calorie information stored on the device on the user's device and provides feedback on the contents of the meal and calorie consumption.

[2323] "Means for the user to connect a blood pressure monitor and obtain blood pressure data" refers to a function that allows the user to connect a blood pressure monitor via a communication means such as Bluetooth, measure blood pressure, and obtain the data.

[2324] The "terminal means for transmitting acquired blood pressure data to a server" is a communication function of the terminal for transmitting acquired blood pressure data to a server on the cloud.

[2325] "Means for the server to save the blood pressure data and update the health database" refers to a function that allows the server to save the received blood pressure data in the user's health database and update the database based on the latest data.

[2326] The "means for notifying the user of updated blood pressure information" is a function that notifies the user's device of updated blood pressure information and provides feedback according to the situation.

[2327] "Means for analyzing the user's emotional data and adjusting the content of advice based on health management" refers to a function that analyzes the user's facial expressions and other emotional expressions to extract emotional data, and adjusts the feedback and advice content based on that data.

[2328] "Means for the server to analyze collected data using generative AI technology" refers to a function that uses generative AI technology to comprehensively analyze collected dietary, blood pressure, and emotion data.

[2329] "Means for generating detailed reports and advice on health status based on analysis results" is a function that generates detailed reports and specific advice on the user's health status based on the information obtained through analysis.

[2330] The "means for transmitting the generated report and advice to the terminal and displaying it to the user" is a function for transmitting the generated report and advice to the user's device and visually displaying it.

[2331] The "means for providing customized meal suggestions based on the user's emotional state and health data" is a function that makes meal suggestions tailored to the user's situation based on collected emotional data and health data.

[2332] System configuration

[2333] The present invention is a sys...

Claims

1. a means for a user to take a photo of a meal; a terminal means for transmitting the photograph data to a server; A means for the server to analyze the photo data and extract calorie information; means for storing the extracted calorie information in a user's health database; means for displaying the calorie information stored on the device and providing feedback to the user; A system including:

2. a means for a user to connect a blood pressure monitor and acquire blood pressure data; a terminal means for transmitting the acquired blood pressure data to a server; a server storing the blood pressure data and updating a health database; means for notifying a user of updated blood pressure information; The system of claim 1 , comprising:

3. A means for the server to analyze the collected data using generative AI technology; a means for generating detailed health status reports and advice based on the analysis results; means for transmitting the generated report and advice to a terminal for display to a user; The system of claim 1 , comprising:

Images