System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting health check data; means for taking a picture of each meal and inputting contents of the meal; means for transmitting the health check data, meal data, and step count data to a server; means for analyzing the data in the server, evaluating a health condition, and generating an improvement proposal; means for transmitting the generated improvement proposal and recommended store information to a terminal; and means for notifying a user of the improvement proposal displayed on the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, preventing and improving lifestyle-related diseases and poor health are important issues, but busy people often find it difficult to understand their own health status and take appropriate measures. In particular, accurately managing dietary intake and daily activity levels is difficult, often resulting in a deterioration in health. There is also a need for a method to effectively utilize health checkup results and reflect them in daily life. The purpose of this invention is to solve these problems and provide a specific means for users to efficiently manage their health. [Means for solving the problem]

[0005] The present invention provides a system that evaluates a user's health condition in real time based on the user's health checkup data, dietary data, and step count data, and presents appropriate improvement proposals. Specifically, the problem is solved by the following means.

[0006] a means for inputting health examination data;

[0007] A means to take a photo of each meal and input the meal contents,

[0008] means for transmitting health checkup data, dietary data, and step count data to a server;

[0009] A means for analyzing the data on a server, assessing the health status, and generating improvement proposals;

[0010] means for transmitting the generated improvement plan and recommended store information to the terminal;

[0011] A means of notifying the user of the proposed improvements displayed on the device;

[0012] The system, which includes the above, allows users to easily understand their own health status and take appropriate measures at the appropriate time, thereby realizing effective health management and preventing deterioration of health conditions.

[0013] "Health checkup data" refers to numerical information indicating the physical health status of a user, such as blood pressure, blood sugar level, and cholesterol level, obtained as a result of a health checkup the user underwent at a medical institution.

[0014] "Dietary data" refers to information indicating the contents of the meals consumed by the user, specifically including the type, amount, and nutrient content of the food.

[0015] "Step count data" refers to information about the number of steps a user takes while walking in their daily life, and is recorded using a smartphone or wearable device.

[0016] The "server" is a computer system that receives health checkup data, dietary data, and step count data sent by the user, analyzes them, saves the results, and sends them to the terminal as needed.

[0017] A "terminal" is a device used by a user, such as a smartphone or tablet, that inputs health checkup data and dietary data and communicates with the server.

[0018] "Improvement suggestions" are specific advice and instructions for improving the user's health condition, generated by the server based on the results of the server's assessment of the user's health condition.

[0019] "Recommended store information" is information about the nearest stores and facilities that offer foods and services that help improve health, based on the user's location information.

[0020] "Location information" refers to information that indicates the user's current physical location and is obtained using GPS, communication networks, etc.

[0021] "Notifications" are alerts or messages sent by the device to inform the user of improvement suggestions or recommended store information. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] This invention is a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. It also uses location information to provide recommended store information, allowing users to easily obtain the foods and services they need.

[0044] Program processing

[0045] 1. Initial data registration:

[0046] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0047] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0048] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0049] 2. Data Collection and Transmission:

[0050] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0051] Device: The device receives the meal photos and sends them to the server for analysis. At the same time, it collects step count data and periodically sends it to the server.

[0052] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0053] 3. Health status assessment and generation of improvement proposals:

[0054] Server: The server comprehensively analyzes the collected health checkup data, dietary data, and step count data. It evaluates the user's current health condition and generates specific improvement recommendations. For example, the server may generate an assessment result such as, "Today's salt intake is high, so it is recommended that you choose low-salt foods for your next meal."

[0055] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0056] 4. Notice and Execution:

[0057] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0058] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0059] 5. Gather feedback and suggestions for next improvements:

[0060] Server: The server receives and analyzes the user's behavioral data again. It evaluates the effectiveness of the user's improvement actions and incorporates the feedback into the next improvement plan. This allows for consistent health management.

[0061] Through these processes, users can manage their own health status in real time and take appropriate actions to improve their health. The system continuously monitors the user's health status and provides individually customized health improvement suggestions to help users maintain their health in the long term.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0065] Step 2:

[0066] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0067] Step 3:

[0068] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0069] Step 4:

[0070] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0071] Step 5:

[0072] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0073] Step 6:

[0074] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0075] Step 7:

[0076] Server: The server comprehensively analyzes the health check data, dietary data, and step count data to evaluate the user's current health condition.

[0077] Step 8:

[0078] Server: The server generates specific improvement suggestions based on the evaluation results. For example, it generates advice such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[0079] Step 9:

[0080] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0081] Step 10:

[0082] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0083] Step 11:

[0084] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0085] Step 12:

[0086] Server: The server receives and analyzes the user behavior data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0087] This allows users to consistently manage their health and receive personalized health improvement suggestions, helping them to more effectively maintain and improve their health.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In modern society, individual users need to accurately and quickly grasp a large amount of information in order to effectively manage their own health status in real time and take appropriate measures to improve it. For this reason, there is a demand for systems that can efficiently collect and analyze health checkup data, daily diet data, and exercise data, and provide users with specific improvement suggestions. However, conventional systems have faced challenges, such as cumbersome data collection and insufficient analysis, making it difficult for users to effectively manage their health.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes a means for analyzing the received health checkup data, dietary data, and step count data to evaluate the health condition, a means for generating improvement proposals using a generative AI model, and a means for acquiring recommended store information based on the user's location information, thereby enabling the user to understand their own health condition in real time and obtain specific and effective improvement proposals.

[0093] A "user" is an individual who uses the system to manage their own health status and receive improvement suggestions.

[0094] "Health checkup data" refers to data relating to the user's health condition, such as blood pressure, blood sugar level, and cholesterol level.

[0095] "Dietary data" refers to data that a user records by inputting or taking photos of their daily meals.

[0096] "Step count data" refers to data on the number of steps a user takes in their daily life.

[0097] A "server" is a device that analyzes the received data and provides the user with appropriate improvement suggestions.

[0098] "Analysis" is the process of evaluating health status based on collected data and extracting necessary information.

[0099] A "generative AI model" is a model that uses machine learning technology to generate improvement suggestions based on a user's health condition.

[0100] "Improvement Suggestions" are specific instructions or suggestions for improving the user's health.

[0101] "Location information" refers to the geographic coordinate data of the user's current location.

[0102] "Recommended store information" is information about stores that offer suggested meals and services based on the user's location information.

[0103] "Notifications" are messages or alerts sent from the server to the device, and are a means of conveying information to the user.

[0104] "Feedback" is information that reports to the system the evaluation and results of improvement proposals implemented by the user.

[0105] The present invention provides a system that enables a user to grasp his or her own health condition in real time and assists the user in taking appropriate measures to improve the condition. A specific embodiment of this system will be described below.

[0106] 1. System Configuration

[0107] The system includes a server where users input health checkup data and dietary data, which analyzes this data, a terminal that displays the analysis results, and a generative AI model.

[0108] Hardware and Software

[0109] Device: A mobile device such as a smartphone or tablet, optionally equipped with software that uses OCR technology (e.g., Google Cloud Vision API).

[0110] Server: A high-performance computer for analyzing data and running generative AI models, using database software (e.g., MySQL) and machine learning frameworks (e.g., TensorFlow).

[0111] 2. Initial data registration

[0112] The user launches the app and registers the health check results by entering them or taking a photo. For example, the user enters a blood pressure of 120 / 80 mmHg or a blood sugar level of 90 mg / dL. The device converts the photo into text data using OCR technology as needed and sends the formatted data to the server. The server analyzes the received data and stores it in a basic health database.

[0113] 3. Data Collection and Transmission

[0114] Each time a user eats a meal, they take a photo of it and upload it to the app. For example, they can register an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner. The device receives the meal photos and sends them to a server. The device also collects the user's step count data daily and periodically sends it to the server. The server uses image analysis technology to analyze the meal photos and calculates the key nutrients. It also analyzes the step count data to calculate calories burned.

[0115] 4. Health status assessment and generation of improvement proposals

[0116] The server comprehensively analyzes the accumulated health checkup data, dietary data, and step count data, and uses a generative AI model to evaluate the user's health condition. For example, if the dietary data indicates excessive calorie intake and insufficient exercise, it will generate specific improvement suggestions such as "increase exercise" or "choose low-calorie foods."

[0117] Location information and recommended store information

[0118] The server uses the user's location information to search for recommended stores and restaurants nearby via internet services, obtains information about stores that offer the desired food or service, and sends this information along with improvement suggestions to the user's device.

[0119] 5. Notification and Execution

[0120] The device notifies the user of the improvement suggestions and recommended store information received from the server. For example, a push notification might say, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket." The user can check the notification and take specific action. For example, they might go to a nearby supermarket to purchase low-salt foods or do the suggested exercise.

[0121] 6. Gather feedback and suggest improvements for next time

[0122] After the user completes the action based on the improvement suggestion, they provide feedback on the results. For example, the user might enter feedback such as "I ran for 30 minutes." The server then analyzes this data again and generates the next improvement suggestion based on the feedback. This feedback loop allows the user's health to continuously improve.

[0123] Prompt Sentence Examples

[0124] "The user's health check results show that their blood pressure is 120 / 80 mmHg, their blood sugar is 90 mg / dL, and their cholesterol is 180 mg / dL. Please generate a future health improvement plan based on their dietary and step count data."

[0125] "The user has consumed 2300 kcal and taken 8000 steps today. Please suggest appropriate improvements based on this information."

[0126] As described above, this system comprehensively manages users' health data and provides specific health improvement suggestions based on the analysis results, enabling users to effectively manage their health.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1: Registering initial data

[0129] User: Launches the app and enters their health checkup results, such as blood pressure 120 / 80 mmHg, blood sugar 90 mg / dL, and cholesterol 180 mg / dL, or takes and uploads a photo of their health checkup results.

[0130] Input: Health checkup data such as blood pressure, blood sugar, and cholesterol levels

[0131] Terminal: The received health check data is converted into text data using OCR technology (e.g., Google Cloud Vision API), formatted, and sent to the server in the format "user ID_recording date and time."

[0132] Output: Formatted medical examination data

[0133] Server: Receives formatted health checkup data, evaluates it using an analysis engine, and stores it in a database (e.g., MySQL) as basic health data. It also sets reference and target values.

[0134] Step 2: Collect and send food and exercise data

[0135] User: Takes a photo of each meal and uploads it to the app, for example, an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner.

[0136] Input: Food photos and text data

[0137] Device: Receives photos and sends them to the server. It also collects the user's daily step count data from an activity tracker (e.g., a fitness band) and periodically sends it to the server.

[0138] Output: Meal photos, step count data

[0139] Server: Analyzes food photos using an image analysis engine (e.g., a model using TensorFlow) and calculates macronutrients (calories, protein, carbohydrates, fat, etc.). Also analyzes step count data and calculates calories burned. Integrates this data with basic health data.

[0140] Step 3: Assessing health status and generating improvement proposals

[0141] Server: Comprehensively analyzes health checkup data, dietary data, and step count data to assess the user's health condition. Using a generative AI model, it generates improvement recommendations based on the user's current situation. For example, "Today's calorie intake has exceeded the target, so we recommend choosing low-calorie foods for your next meal."

[0142] Input: Health checkup data, dietary data, step count data

[0143] Output: Health status assessment, improvement suggestions

[0144] Specific actions: To assess health status, compare with past data and baseline values. To generate improvement recommendations, the system aims to provide personalized and specific advice using generative AI models.

[0145] Step 4: Get recommended store information based on location information

[0146] Server: Based on the improvement plan, use the user's location information to search for recommended nearby stores through an internet service (e.g., map API). For example, search for "nearby supermarkets that sell low-salt foods."

[0147] Input: Improvement proposal, user location

[0148] Output: Recommended store information

[0149] Specific operation: The server receives the location data, searches for stores using the API, converts the results into text format, and provides it to the user.

[0150] Step 5: Send and execute notifications

[0151] On the device: The system sends push notifications to the user with the improvement suggestions and recommended store information received from the server. For example, it sends a notification saying, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket."

[0152] Input: Improvement proposals, recommended store information

[0153] Output: User notification

[0154] User: Checks the notification and takes specific action, such as going to a nearby supermarket to buy low-sodium foods.

[0155] Specific behavior: After receiving the notification, the user performs the action as instructed and provides feedback to the app about the results.

[0156] Step 6: Gather feedback and suggest improvements for next time

[0157] User: After completing an action, provide feedback to the app about what you did. For example, you might enter "I ran for 30 minutes."

[0158] Input: Feedback data (results of the action)

[0159] Server: Re-analyzes the received feedback data, evaluates its effectiveness, and incorporates the feedback into the next improvement plan, thereby supporting continuous improvement of health status.

[0160] Output: Next improvement plan

[0161] How it works: Feedback data is collected and evaluated using an analytics engine. Based on the results, the next improvement plan is generated using a generative AI model.

[0162] The above are the specific processing steps and operations of the program for this system.

[0163] (Application example 1)

[0164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] In modern society, individual health management is becoming increasingly important, but it is difficult to effectively maintain health through self-management alone. In particular, there is a lack of methods that comprehensively analyze health checkup results, daily dietary habits, and exercise levels, and provide specific, individually customized health improvement proposals. There are also insufficient methods for users to easily obtain the information necessary to implement improvement proposals (for example, information on recommended stores and services). For this reason, a system that allows users to effectively manage their health on a daily basis is needed.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0167] In this invention, the server includes means for inputting health checkup data, means for taking images of each meal and inputting the meal contents, means for transmitting the health checkup data, meal data, and step count data to the server, means for analyzing the data in the server, evaluating the health condition, and generating improvement suggestions, means for transmitting the generated improvement suggestions and recommended store information to the terminal, means for notifying the user of the improvement suggestions displayed on the terminal, means for accumulating user behavior data and reflecting the data in generating the next improvement suggestion, means for using location information to search for information on the nearest store and provide recommended store information related to meals and exercise, means for analyzing meal contents using image analysis technology and extracting major nutrients, means for evaluating the user's health condition using a generative AI model and generating specific improvement suggestions, and means for notifying the user of the generated improvement suggestions and recommended store information to a smartphone or head-mounted display, thereby enabling the user to manage their own health condition in real time and take appropriate action to improve their health.

[0168] "Health checkup data" refers to test result data that indicates the user's health condition, and includes information such as blood pressure, blood sugar level, and cholesterol level.

[0169] "Meal images" are photographs of meals consumed by a user, providing a visual record of the meal contents.

[0170] "Dietary content" refers to information including the type and amount of food and major nutrients (protein, carbohydrates, fat, etc.).

[0171] "Step count data" is data that records the number of steps a user takes in a day, and indicates the amount of exercise.

[0172] The "server" is a central computer device in the system that receives, analyzes, and evaluates health checkup data, dietary data, and step count data.

[0173] The "means for assessing health status" is a function in which the server analyzes the data received and diagnoses the user's current health status.

[0174] The "means for generating improvement proposals" is a function in which the server creates a specific health improvement plan suited to the user based on the results of the health status evaluation.

[0175] "Recommended store information" refers to information about the nearest store that provides the food or service you need based on your location information and health improvement suggestions.

[0176] A "terminal" refers to a device operated by a user, such as a smartphone or head-mounted display, that receives and displays information from a server.

[0177] "Means of notification" refers to the function that the device uses to inform users of improvement suggestions and recommended store information, and includes push notifications and in-app messages.

[0178] "Behavioral data" refers to data about the actions actually taken by the user, including diet, exercise, purchasing history, etc.

[0179] "Image analysis technology" is a technology that extracts information from image data such as photographs, and is used in health management systems to analyze the contents of meals.

[0180] "Macronutrients" are the important components of food, such as proteins, carbohydrates, and fats, that are necessary for maintaining a healthy human body.

[0181] A "generative AI model" is a model that uses artificial intelligence to analyze a user's health data and automatically generate appropriate health improvement suggestions.

[0182] The present invention provides a system for helping users understand their own health condition in real time and take appropriate measures to improve it. Specific embodiments are described below.

[0183] System configuration:

[0184] 1. Health checkup data input method:

[0185] Users can manually enter their health checkup results using a device (smartphone or head-mounted display) or take a photo and register the data using OCR technology. This data is sent to a server and stored as the user's basic health database.

[0186] 2. Meal content input method:

[0187] Users take photos of each meal and upload them to the app. The device receives the photos and analyzes the meal contents using image analysis technology (such as Google Cloud Vision API). The analysis results are sent to a server as information on macronutrients such as protein, carbohydrates, and fat.

[0188] 3. Step count data collection method:

[0189] The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to collect step count data and periodically transmits it to a server.

[0190] 4. Health status assessment and improvement proposal generation method:

[0191] The server comprehensively analyzes the received health checkup data, dietary data, and step count data. During this process, a generative AI model (using TensorFlow and PyTorch) is used to evaluate the user's health status and generate specific improvement suggestions.

[0192] 5. Recommended store information methods:

[0193] Based on the generated improvement proposal, the server uses the user's location information to search for and obtain recommended store information for the nearest store using Google Maps API, etc. This store information will be related to food and exercise that will help improve the user's health.

[0194] 6. Means of notification:

[0195] The server sends improvement suggestions and recommended store information to the device (smartphone or head-mounted display), which then notifies the user via push notification (Firebase Cloud Messaging) or in-app message.

[0196] Examples:

[0197] 1. Registering health checkup results:

[0198] The user takes an image of the blood pressure result with their smartphone, and OCR technology extracts text data from the image and sends the blood pressure information to the server.

[0199] 2. Enter your meal details:

[0200] Users take a photo of their lunch and upload it to the app, where, using image analysis technology, the server analyzes the macronutrients contained in the meal and adds them to a dietary database.

[0201] 3. Step count data collection:

[0202] The smartphone's built-in pedometer measures the number of steps the user takes each day and periodically sends the data to a server.

[0203] 4. Health status assessment and generation of improvement proposals:

[0204] The server analyzes the collected data using a generative AI model to assess the user's health condition, and generates specific recommendations for improvement, such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[0205] 5. Providing recommended store information:

[0206] Based on the user's location information, the server searches for the nearest supermarket and provides information on stores where low-salt foods can be purchased.

[0207] Example prompt sentence:

[0208] User health data: Blood pressure: 120 / 80 mmHg, Blood glucose: 5.6 mmol / L

[0209] Meal details: Salad, chicken, rice

[0210] Step count data: 7000 steps

[0211] Location information: 35.6895° N, 139.6917° E

[0212] Use this data to assess the user's health status and generate specific recommendations for improvement.

[0213] These components and examples empower users to manage their health in real time and provide the information they need to implement appropriate improvements.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] The user inputs the health checkup data. Using a device (smartphone or head-mounted display), the user can manually enter the health checkup results directly or take a photo of the results and upload it to the app. The device uses OCR technology to extract and format text data from the image data, which then sends the initial health data to the server.

[0217] Input: Medical examination result images or manual input data

[0218] Output: Formatted medical examination data

[0219] Step 2:

[0220] The user takes a photo of each meal and enters the meal contents. The user takes a photo of the meal and uploads it to the app. The device uses image analysis technology (Google Cloud Vision API) to analyze the meal contents and extract macronutrients (protein, carbohydrates, fat, etc.). The analysis results are sent to the server.

[0221] Input: Food image

[0222] Output: Macronutrient information of meal contents

[0223] Step 3:

[0224] The device collects step count data and sends it to a server. The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to measure the number of steps taken each day. The step count data is periodically sent to the server, allowing the amount of daily exercise to be recorded.

[0225] Input: Pedometer data

[0226] Output: Periodically collected step count data

[0227] Step 4:

[0228] The server analyzes health checkup data, dietary data, and step count data to assess the user's health status and generate improvement proposals. The server comprehensively processes and analyzes the received data and evaluates the user's health status using a generative AI model (using TensorFlow or PyTorch). Based on the analysis results, it generates specific improvement proposals.

[0229] Input: Health checkup data, dietary data, step count data

[0230] Output: User's health status assessment results and specific improvement suggestions

[0231] Step 5:

[0232] Based on the generated improvement plan, the server uses location information to obtain recommended store information for the nearest store. The server uses the user's location information to search for and obtain store information that provides the desired food or service using Google Maps API, etc.

[0233] Input: improvement proposal, user location information

[0234] Output: Recommended store information

[0235] Step 6:

[0236] The server sends improvement suggestions and recommended store information to the device, which then notifies the user. The server sends the generated improvement suggestions and recommended store information to the device, which then notifies the user via push notification or in-app message.

[0237] Input: Improvement proposals, recommended store information

[0238] Output: A message to inform the user

[0239] Step 7:

[0240] User behavior data is accumulated and reflected in the generation of the next improvement proposal. The user takes action based on the notification they receive (e.g., shopping at a recommended store). This behavior data is sent back to the server and taken into account in the next data analysis.

[0241] Input: User behavior data

[0242] Output: Data to be reflected in the next improvement plan generation

[0243] This allows the system to manage the user's health status in real time and continue to provide appropriate improvement suggestions.

[0244] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0245] This invention is a system that helps users understand their health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system customizes the improvement suggestions, providing more personalized health management. It also uses location information to provide recommended store information.

[0246] Program processing

[0247] 1. Initial data registration:

[0248] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0249] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0250] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0251] 2. Data Collection and Transmission:

[0252] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0253] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0254] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0255] 3. Adding an Emotion Engine:

[0256] User: The user inputs or automatically collects emotional data through an app equipped with an emotion engine. For example, when a user feels stressed, emotional data can be obtained through facial recognition or voice analysis.

[0257] Device: The device collects emotion data and sends it to the server.

[0258] Server: The server receives the emotion data, integrates it with other health data, and performs analysis.

[0259] 4. Health status assessment and generation of improvement proposals:

[0260] Server: The server comprehensively analyzes the collected health checkup data, dietary data, step count data, and emotional data. It evaluates the user's current health condition and generates specific recommendations for improvement. For example, it generates advice such as, "Today's salt intake was high and your stress level is also high, so choose low-salt foods for your next meal and suggest some exercises to help you relax."

[0261] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0262] 5. Notice and Execution:

[0263] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0264] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0265] 6. Gather feedback and suggestions for next improvements:

[0266] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0267] This allows users to manage their health consistently and receive personalized health improvement suggestions that take their emotional state into account, helping them maintain and improve their health more effectively. The addition of the emotion engine will enable holistic health management that also takes into account the user's mental health.

[0268] The processing flow will be explained below.

[0269] Step 1:

[0270] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0271] Step 2:

[0272] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0273] Step 3:

[0274] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0275] Step 4:

[0276] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0277] Step 5:

[0278] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0279] Step 6:

[0280] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0281] Step 7:

[0282] User: The user uses an app equipped with an emotion engine, and emotion data is input or automatically collected. For example, emotion data is obtained using facial recognition or voice analysis functions when the user feels stressed.

[0283] Step 8:

[0284] Device: The device collects emotion data and sends it to the server.

[0285] Step 9:

[0286] Server: The server receives the emotion data, integrates it with health checkup data, dietary data, and step count data, and performs a comprehensive analysis.

[0287] Step 10:

[0288] Server: The server uses the integrated data to assess the user's current health status and generate specific recommendations for improvement. For example, if the user's stress level is high and their salt intake is high, the server will generate advice such as "Choose low-salt foods for your next meal and do 30 minutes of yoga to relax."

[0289] Step 11:

[0290] Server: Based on the generated improvement suggestions, the server uses the user's location information to search for recommended stores and restaurants nearby, for example, to obtain information on purchasing low-salt foods at a nearby supermarket.

[0291] Step 12:

[0292] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed via push notifications and in-app messages.

[0293] Step 13:

[0294] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or practicing yoga as suggested by the app.

[0295] Step 14:

[0296] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0297] This allows users to consistently manage their health in a way that is individually customized, and helps maintain and improve overall health, taking into account their emotional state.

[0298] Example 2

[0299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] With conventional health management systems, it was difficult for users to accurately understand their own health status in real time and implement appropriate improvement measures. In particular, the data collected was limited, making it impossible to generate customized improvement plans that took into account each user's emotional and behavioral data. In addition, users had no easy way to obtain information about stores offering the foods and services they needed. These issues led to a lack of consistency in users' health management, hindering effective health maintenance and improvement.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0302] In this invention, the server includes: a means for analyzing health checkup data using OCR technology to evaluate health status and generate improvement proposals using a new data source including emotion data; a means for searching for the generated improvement proposals and recommended store information based on location information and transmitting them to the terminal; and a means for reanalyzing the user's behavioral data and emotion data and incorporating the feedback into the next improvement proposal. This makes it possible to comprehensively evaluate each user's health status and provide customized improvement proposals. Furthermore, by utilizing location information to provide information on recommended stores near the user, users can easily obtain the foods and services they need.

[0303] "Health checkup data" is information about the user's physiological health condition, including measurements such as blood pressure, blood sugar level, and cholesterol level.

[0304] "Dietary data" refers to information about the meals a user eats, including photos of the meals, the contents of the meals, and the calories and nutrients consumed.

[0305] "Step data" is information about the number of steps a user takes each day, collected from pedometer apps and devices.

[0306] "Emotional Data" refers to information about a user's emotional state, including stress levels and moods obtained through facial recognition, voice analysis, or manual input.

[0307] "OCR technology" stands for optical character recognition technology, which extracts characters from image data and converts them into text data.

[0308] "Improvement Suggestions" are specific advice or suggestions provided to improve the user's health based on data analyzed by the server.

[0309] "Location information" refers to the user's current physical location, obtained based on GPS and network information.

[0310] "Recommended store information" is information about stores that offer the food or services you need, provided based on your location information.

[0311] "Terminal" means a computing device used by a User, including a smartphone or tablet.

[0312] A "server" is a central computing system that analyzes data and stores and transmits results.

[0313] This invention relates to a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, this system analyzes health checkup data, dietary data, and step count data on a server and notifies the user of specific improvement suggestions, thereby enabling the user to effectively manage their health. In addition, an emotion engine is used to collect user emotion data and customize health management. Location information is also used to provide recommended store information. Specific embodiments of the system are described below.

[0314] Initial data registration

[0315] User: The user launches the smartphone app and enters the results of their health checkup. For example, they manually enter values ​​such as blood pressure, blood sugar, and cholesterol levels. They can also take a photo of their health checkup report and register the data.

[0316] Terminal: The terminal receives the input data and converts the image data into text data using OCR (Optical Character Recognition) technology. The converted data is then formatted into a standard format and sent to the server.

[0317] Server: The server analyzes the health checkup data sent from the device. The analyzed data is stored in the user's basic health database, and health assessment is performed to set baseline and target values. For example, the user's target blood pressure value is set to 125 / 80.

[0318] Data collection and transmission

[0319] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food they eat. They can enter information such as "I had salad and fish for lunch."

[0320] Device: The device receives meal photos and manual input data and sends them to the server. Step count data is also automatically collected from the pedometer app and periodically sent to the server.

[0321] Server: The server analyzes the received food photos and calculates calorie intake and major nutrients (protein, carbohydrates, fat, etc.). It also analyzes the received step count data and calculates energy expenditure. For example, it calculates the calories in a salad from the photo as 100 kcal, and records the energy expenditure of 10,000 steps a day as 400 kcal.

[0322] Adding an Emotion Engine

[0323] User: Users input their emotional data through apps equipped with emotion engines, or the data is collected automatically. For example, when a user feels stressed, emotion data is acquired through facial recognition or voice analysis.

[0324] Device: The device collects emotion data and sends it to the server.

[0325] Server: The server receives the emotional data and analyzes it by combining it with other health data. For example, if the stress level is high, it suggests low-calorie food choices and exercise to relax.

[0326] Evaluating health status and generating improvement proposals

[0327] Server: The server performs a comprehensive analysis based on the collected health checkup data, dietary data, step count data, and emotional data. For example, it may assess that "Your calorie intake is high and your step count is low, so you need to increase your exercise."

[0328] Server: Evaluates the user's current health status and generates specific recommendations for improvement, such as "Today's salt intake is high and your stress level is also high, so choose low-salt foods for your next meal and suggest relaxation exercises."

[0329] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants in the vicinity and obtains store information that offers the desired food or service. For example, "List low-salt foods available at nearby supermarkets."

[0330] Notification and Execution

[0331] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages. For example, "Today's lunch is high in calories, so please choose low-calorie foods for dinner."

[0332] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[0333] Gathering feedback and suggestions for next improvements

[0334] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0335] Prompt Sentence Examples

[0336] "How are you feeling today? Enter your health indicators."

[0337] "Upload a photo of your meal and calculate the calories."

[0338] This system allows users to achieve continuous and individually customized health management, effectively maintaining and improving their overall health.

[0339] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0340] Step 1:

[0341] Enter and submit health checkup data

[0342] User: The user launches the smartphone app and manually enters the results of their health checkup. For example, they enter values ​​such as blood pressure, blood sugar, and cholesterol levels, such as "My blood pressure this morning was 130 / 85."

[0343] Terminal: The terminal receives the input data and can also convert photo data into text data using OCR technology instead of manual input. The converted data is formatted and sent to the server.

[0344] Input: Manual input or photo data.

[0345] Output: Formatted medical examination data.

[0346] Step 2:

[0347] Analysis and storage of health checkup data

[0348] Server: The server receives and analyzes the health checkup data sent from the device, evaluates the user's health status based on the data, and stores it in a basic health database.

[0349] Input: Formatted medical examination data.

[0350] Output: Health status assessment results and baseline health data.

[0351] Step 3:

[0352] Collection and transmission of dietary data

[0353] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0354] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it automatically collects step count data from the pedometer app and periodically sends it to the server.

[0355] Input: Meal photos, manual input data, step count data.

[0356] Output: Data sent to the server.

[0357] Step 4:

[0358] Data analysis

[0359] Server: The server analyzes the received food photos and calculates calorie intake and macronutrients (protein, carbohydrates, fat, etc.). Similarly, it analyzes step count data to calculate energy expenditure.

[0360] Input: Meal photos, manual input data, step count data.

[0361] Output: Calorie intake, macronutrients, energy expenditure.

[0362] Step 5:

[0363] Collecting and transmitting emotional data

[0364] User: Users input or automatically collect emotional data through apps equipped with emotion engines. Emotion data can be obtained through facial recognition and voice analysis.

[0365] Device: The device collects emotion data and sends it to the server.

[0366] Input: Emotion data (facial recognition, voice analysis, manual input).

[0367] Output: Data sent to the server.

[0368] Step 6:

[0369] Overall health assessment

[0370] Server: The server comprehensively analyzes the health check data, dietary data, step count data, and emotional data to evaluate the user's health status, generate specific improvement suggestions, and incorporate the feedback into the user's next actions.

[0371] Input: Health checkup data, dietary data, step count data, emotion data.

[0372] Output: Health status assessment results and improvement recommendations.

[0373] Step 7:

[0374] Use location information and find recommended stores

[0375] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby, and obtains information on stores that offer the desired food or service.

[0376] Input: Improvement suggestion, location information.

[0377] Output: Recommended store information.

[0378] Step 8:

[0379] Notification of improvement proposals and recommended store information

[0380] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0381] Input: Improvement suggestions, recommended store information.

[0382] Output: Notification to the user.

[0383] Step 9:

[0384] Implementing improvement actions

[0385] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[0386] Input: Improvement suggestions, recommended store information.

[0387] Output: Implementation of improvement actions.

[0388] Step 10:

[0389] Gather feedback and generate recommendations for next time improvements

[0390] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluating the effectiveness of the user's improvement actions and incorporating feedback into the next improvement proposal.

[0391] Input: User behavioral data, emotional data.

[0392] Output: Suggested improvements for next time.

[0393] This processing step allows users to receive continuous, personalized health management, consistently maintaining and improving their health effectively.

[0394] (Application example 2)

[0395] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0396] Conventional health management systems could evaluate a user's health status using health checkup data and dietary data, but they were unable to provide specific improvement proposals that took into account the user's emotions or real-time store integration. Furthermore, they lacked a mechanism for providing the products and services users needed in a timely manner when they visited a physical store. This resulted in ineffective health management for users and insufficient support for their mental health.

[0397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0398] In this invention, the server includes a means for inputting health checkup data, a means for taking photos of each meal and inputting the meal contents, a means for transmitting the health checkup data, meal data, and step count data to the server, a means for analyzing the data in the server, evaluating the health status, and generating improvement suggestions, a means for transmitting the generated improvement suggestions and recommended store information to the terminal, a means for notifying the user of the improvement suggestions displayed on the terminal, a means for analyzing facial expressions and voice to collect emotional data, a means for transmitting the emotional data to the server and integrating it with the health data for analysis, and a means for notifying the user of recommended product and service information in real time when the user visits a physical store. This allows the user to understand their health status in real time and receive individually customized improvement suggestions that take their emotional state into account. It also enables the user to obtain desired products and services in a timely manner when visiting a physical store.

[0399] "Health checkup data" refers to medical information such as blood pressure, blood sugar level, and cholesterol level obtained as a result of a health checkup that a user has undergone.

[0400] "Meal data" refers to photos taken by the user for each meal and information about the meal that the user inputs.

[0401] "Step count data" refers to data that records the number of steps a user takes each day.

[0402] The "server" is a data processing system that analyzes collected health checkup data, dietary data, step count data, and emotional data, evaluates health status, and generates improvement proposals.

[0403] "Emotional data" is data that indicates the user's stress level and emotional state obtained by analyzing their facial expressions and voice.

[0404] A "terminal" is an electronic device such as a smartphone or tablet held by the user, which displays improvement suggestions and recommended store information sent from the server.

[0405] "Recommended store information" is information that recommends the nearest suitable store or service based on the user's location information and health condition.

[0406] The "improvement suggestion" is a specific instruction or advice for improving the user's health condition, which is generated by the server by analyzing the health checkup data, dietary data, step count data, and emotional data.

[0407] "Behavioral data" is data that records the history of actions that users actually take and products they purchase.

[0408] "Facial expression and voice analysis" is a process in which a user's facial expressions are captured by a camera and their voice is recorded, and the emotion engine analyzes them to generate emotion data.

[0409] "Real-time notification" is a technology that instantly notifies users of necessary information and recommended products when they visit a physical store.

[0410] The system for implementing this invention collects and analyzes health checkup data, dietary data, step count data, and emotional data in real time. Ultimately, it provides users with health improvement suggestions and recommended store information based on the analysis results. The specific configuration and operation of this system are described below.

[0411] Hardware and Software Configuration

[0412] Hardware

[0413] Device: An electronic device, such as a smartphone or tablet, that users use to enter health data, take photos, and collect emotional data.

[0414] Server: A high-performance computer system installed in a data center that analyzes the received health and emotional data.

[0415] software

[0416] OCR technology: Converts the health check data entered by the user into text data.

[0417] Image analysis technology: Analyzes photos of food and facial expressions to obtain food content and emotional data. A specific example is an emotion recognition model using Keras.

[0418] Data analysis algorithm: Runs on the server and analyzes collected data to assess health status and generate improvement recommendations.

[0419] Data collection and analysis

[0420] Entering health data

[0421] Users can manually enter their health checkup results (such as blood pressure and blood sugar levels) or take a photo and upload it to the app. The device then converts this data into text using OCR technology and sends it to the server, which then creates a basic health database for the user.

[0422] Collection of dietary data

[0423] Users take a photo of their meal each time they eat and upload it to the app, which then sends the photo to a server that uses image analysis technology to calculate calorie and macronutrient intake.

[0424] Collecting step count data

[0425] The user's device uses a pedometer to measure the number of steps taken each day and periodically transmits this data to a server, which then calculates energy consumption.

[0426] Collecting Emotional Data

[0427] When a user feels stressed, emotional data is collected through facial recognition and voice analysis. The device then sends this data to a server, which then combines it with health data for analysis.

[0428] Providing recommended store information

[0429] Based on the analysis results, the server generates personalized health improvement recommendations for the user, such as "Today's salt intake is high, so choose low-salt foods at your next meal."

[0430] The server uses the user's location information to search for recommended stores nearby, and then notifies the user of stores that offer the food or service they need.

[0431] User Notification and Feedback

[0432] Improvement suggestions and recommended store information are sent from the server to the device and displayed as push notifications or in-app messages. Users can check the notification and take specific actions according to the instructions.

[0433] User behavior data and purchase history are also sent to the server and fed back into the analysis results, which are then reflected in future improvement proposals, enabling more advanced customization.

[0434] Prompt Sentence Examples

[0435] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[0436] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0437] Step 1:

[0438] Entering and submitting health data

[0439] Users manually enter their health check results (e.g., blood pressure, blood sugar, cholesterol levels, etc.) or take a photo and upload it to the device, which uses OCR technology to convert this data into text, format it, and send it to the server.

[0440] Input: Medical checkup results (manual entry or photo)

[0441] Output: Formatted text data (sent to server)

[0442] Step 2:

[0443] Collection and transmission of dietary data

[0444] Users take a photo of their meal each time they eat and upload it to the app. The device then sends the photo, along with any manually entered meal information, to a server. The server then uses image analysis technology to analyze the photo and calculate calorie and macronutrient intake.

[0445] Input: Meal photos (and manually entered meal information)

[0446] Output: Calorie intake and macronutrient data (analysis results on the server)

[0447] Step 3:

[0448] Collection and transmission of step count data

[0449] The device uses a built-in pedometer to collect the user's daily step count and periodically transmits this data to a server, which analyzes the step count data and calculates energy consumption.

[0450] Input: Step count data (automatically sent from the device)

[0451] Output: Energy consumption data (analysis results on the server)

[0452] Step 4:

[0453] Acquiring and sending emotion data

[0454] When a user feels stress or other emotions, they take a photo of their face or record their voice. The device sends this data to a server, which then analyzes the data using an emotion recognition model (e.g., a model using Keras) to evaluate the user's emotional state.

[0455] Input: Face photo (or voice data)

[0456] Output: Emotional state data (analysis results on the server)

[0457] Step 5:

[0458] Integrated analysis of health status and generation of improvement proposals

[0459] The server integrates and analyzes the collected health checkup data, dietary data, step count data, and emotional data. Based on the analysis results, it generates customized health improvement recommendations. For example, it creates specific advice such as, "Today's salt intake was high, so choose low-salt foods at your next meal."

[0460] Input: Health checkup data, dietary data, step count data, emotion data

[0461] Output: Health improvement suggestions (specific advice)

[0462] Step 6:

[0463] Generate and notify recommended store information

[0464] The server uses the user's location information to search for nearby recommended store information. The generated improvement suggestions and recommended store information are sent to the device, and the user receives them via push notification or in-app message.

[0465] Input: Location, Health Improvement Suggestion

[0466] Output: Recommended store information (notification to user)

[0467] Step 7:

[0468] Collecting feedback

[0469] The user's behavioral data and purchase history are sent to the server via the device, where the server reanalyzes the data and incorporates the feedback into future improvements, resulting in more accurate and personalized health advice.

[0470] Input: Behavioral data, purchase history

[0471] Output: Feedback data for improvement proposals (analysis results on the server)

[0472] Prompt Sentence Examples

[0473] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[0474] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0476] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0477] [Second embodiment]

[0478] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0479] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0480] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0481] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0482] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0483] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0484] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0485] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0488] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0489] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0490] This invention is a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. It also uses location information to provide recommended store information, allowing users to easily obtain the foods and services they need.

[0491] Program processing

[0492] 1. Initial data registration:

[0493] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0494] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0495] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0496] 2. Data Collection and Transmission:

[0497] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0498] Device: The device receives the meal photos and sends them to the server for analysis. At the same time, it collects step count data and periodically sends it to the server.

[0499] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0500] 3. Health status assessment and generation of improvement proposals:

[0501] Server: The server comprehensively analyzes the collected health checkup data, dietary data, and step count data. It evaluates the user's current health condition and generates specific improvement recommendations. For example, the server may generate an assessment result such as, "Today's salt intake is high, so it is recommended that you choose low-salt foods for your next meal."

[0502] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0503] 4. Notice and Execution:

[0504] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0505] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0506] 5. Gather feedback and suggestions for next improvements:

[0507] Server: The server receives and analyzes the user's behavioral data again. It evaluates the effectiveness of the user's improvement actions and incorporates the feedback into the next improvement plan. This allows for consistent health management.

[0508] Through these processes, users can manage their own health status in real time and take appropriate actions to improve their health. The system continuously monitors the user's health status and provides individually customized health improvement suggestions to help users maintain their health in the long term.

[0509] The processing flow will be explained below.

[0510] Step 1:

[0511] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0512] Step 2:

[0513] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0514] Step 3:

[0515] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0516] Step 4:

[0517] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0518] Step 5:

[0519] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0520] Step 6:

[0521] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0522] Step 7:

[0523] Server: The server comprehensively analyzes the health check data, dietary data, and step count data to evaluate the user's current health condition.

[0524] Step 8:

[0525] Server: The server generates specific improvement suggestions based on the evaluation results. For example, it generates advice such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[0526] Step 9:

[0527] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0528] Step 10:

[0529] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0530] Step 11:

[0531] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0532] Step 12:

[0533] Server: The server receives and analyzes the user behavior data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0534] This allows users to consistently manage their health and receive personalized health improvement suggestions, helping them to more effectively maintain and improve their health.

[0535] Example 1

[0536] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0537] In modern society, individual users need to accurately and quickly grasp a large amount of information in order to effectively manage their own health status in real time and take appropriate measures to improve it. For this reason, there is a demand for systems that can efficiently collect and analyze health checkup data, daily diet data, and exercise data, and provide users with specific improvement suggestions. However, conventional systems have faced challenges, such as cumbersome data collection and insufficient analysis, making it difficult for users to effectively manage their health.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0539] In this invention, the server includes a means for analyzing the received health checkup data, dietary data, and step count data to evaluate the health condition, a means for generating improvement proposals using a generative AI model, and a means for acquiring recommended store information based on the user's location information, thereby enabling the user to understand their own health condition in real time and obtain specific and effective improvement proposals.

[0540] A "user" is an individual who uses the system to manage their own health status and receive improvement suggestions.

[0541] "Health checkup data" refers to data relating to the user's health condition, such as blood pressure, blood sugar level, and cholesterol level.

[0542] "Dietary data" refers to data that a user records by inputting or taking photos of their daily meals.

[0543] "Step count data" refers to data on the number of steps a user takes in their daily life.

[0544] A "server" is a device that analyzes the received data and provides the user with appropriate improvement suggestions.

[0545] "Analysis" is the process of evaluating health status based on collected data and extracting necessary information.

[0546] A "generative AI model" is a model that uses machine learning technology to generate improvement suggestions based on a user's health condition.

[0547] "Improvement Suggestions" are specific instructions or suggestions for improving the user's health.

[0548] "Location information" refers to the geographic coordinate data of the user's current location.

[0549] "Recommended store information" is information about stores that offer suggested meals and services based on the user's location information.

[0550] "Notifications" are messages or alerts sent from the server to the device, and are a means of conveying information to the user.

[0551] "Feedback" is information that reports to the system the evaluation and results of improvement proposals implemented by the user.

[0552] The present invention provides a system that enables a user to grasp his or her own health condition in real time and assists the user in taking appropriate measures to improve the condition. A specific embodiment of this system will be described below.

[0553] 1. System Configuration

[0554] The system includes a server where users input health checkup data and dietary data, which analyzes this data, a terminal that displays the analysis results, and a generative AI model.

[0555] Hardware and Software

[0556] Device: A mobile device such as a smartphone or tablet, optionally equipped with software that uses OCR technology (e.g., Google Cloud Vision API).

[0557] Server: A high-performance computer for analyzing data and running generative AI models, using database software (e.g., MySQL) and machine learning frameworks (e.g., TensorFlow).

[0558] 2. Initial data registration

[0559] The user launches the app and registers the health check results by entering them or taking a photo. For example, the user enters a blood pressure of 120 / 80 mmHg or a blood sugar level of 90 mg / dL. The device converts the photo into text data using OCR technology as needed and sends the formatted data to the server. The server analyzes the received data and stores it in a basic health database.

[0560] 3. Data Collection and Transmission

[0561] Each time a user eats a meal, they take a photo of it and upload it to the app. For example, they can register an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner. The device receives the meal photos and sends them to a server. The device also collects the user's step count data daily and periodically sends it to the server. The server uses image analysis technology to analyze the meal photos and calculates the key nutrients. It also analyzes the step count data to calculate calories burned.

[0562] 4. Health status assessment and generation of improvement proposals

[0563] The server comprehensively analyzes the accumulated health checkup data, dietary data, and step count data, and uses a generative AI model to evaluate the user's health condition. For example, if the dietary data indicates excessive calorie intake and insufficient exercise, it will generate specific improvement suggestions such as "increase exercise" or "choose low-calorie foods."

[0564] Location information and recommended store information

[0565] The server uses the user's location information to search for recommended stores and restaurants nearby via internet services, obtains information about stores that offer the desired food or service, and sends this information along with improvement suggestions to the user's device.

[0566] 5. Notification and Execution

[0567] The device notifies the user of the improvement suggestions and recommended store information received from the server. For example, a push notification might say, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket." The user can check the notification and take specific action. For example, they might go to a nearby supermarket to purchase low-salt foods or do the suggested exercise.

[0568] 6. Gather feedback and suggest improvements for next time

[0569] After the user completes the action based on the improvement suggestion, they provide feedback on the results. For example, the user might enter feedback such as "I ran for 30 minutes." The server then analyzes this data again and generates the next improvement suggestion based on the feedback. This feedback loop allows the user's health to continuously improve.

[0570] Prompt Sentence Examples

[0571] "The user's health check results show that their blood pressure is 120 / 80 mmHg, their blood sugar is 90 mg / dL, and their cholesterol is 180 mg / dL. Please generate a future health improvement plan based on their dietary and step count data."

[0572] "The user has consumed 2300 kcal and taken 8000 steps today. Please suggest appropriate improvements based on this information."

[0573] As described above, this system comprehensively manages users' health data and provides specific health improvement suggestions based on the analysis results, enabling users to effectively manage their health.

[0574] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0575] Step 1: Registering initial data

[0576] User: Launches the app and enters their health checkup results, such as blood pressure 120 / 80 mmHg, blood sugar 90 mg / dL, and cholesterol 180 mg / dL, or takes and uploads a photo of their health checkup results.

[0577] Input: Health checkup data such as blood pressure, blood sugar, and cholesterol levels

[0578] Terminal: The received health check data is converted into text data using OCR technology (e.g., Google Cloud Vision API), formatted, and sent to the server in the format "user ID_recording date and time."

[0579] Output: Formatted medical examination data

[0580] Server: Receives formatted health checkup data, evaluates it using an analysis engine, and stores it in a database (e.g., MySQL) as basic health data. It also sets reference and target values.

[0581] Step 2: Collect and send food and exercise data

[0582] User: Takes a photo of each meal and uploads it to the app, for example, an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner.

[0583] Input: Food photos and text data

[0584] Device: Receives photos and sends them to the server. It also collects the user's daily step count data from an activity tracker (e.g., a fitness band) and periodically sends it to the server.

[0585] Output: Meal photos, step count data

[0586] Server: Analyzes food photos using an image analysis engine (e.g., a model using TensorFlow) and calculates macronutrients (calories, protein, carbohydrates, fat, etc.). Also analyzes step count data and calculates calories burned. Integrates this data with basic health data.

[0587] Step 3: Assessing health status and generating improvement proposals

[0588] Server: Comprehensively analyzes health checkup data, dietary data, and step count data to assess the user's health condition. Using a generative AI model, it generates improvement recommendations based on the user's current situation. For example, "Today's calorie intake has exceeded the target, so we recommend choosing low-calorie foods for your next meal."

[0589] Input: Health checkup data, dietary data, step count data

[0590] Output: Health status assessment, improvement suggestions

[0591] Specific actions: To assess health status, compare with past data and baseline values. To generate improvement recommendations, the system aims to provide personalized and specific advice using generative AI models.

[0592] Step 4: Get recommended store information based on location information

[0593] Server: Based on the improvement plan, use the user's location information to search for recommended nearby stores through an internet service (e.g., map API). For example, search for "nearby supermarkets that sell low-salt foods."

[0594] Input: Improvement proposal, user location

[0595] Output: Recommended store information

[0596] Specific operation: The server receives the location data, searches for stores using the API, converts the results into text format, and provides it to the user.

[0597] Step 5: Send and execute notifications

[0598] On the device: The system sends push notifications to the user with the improvement suggestions and recommended store information received from the server. For example, it sends a notification saying, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket."

[0599] Input: Improvement proposals, recommended store information

[0600] Output: User notification

[0601] User: Checks the notification and takes specific action, such as going to a nearby supermarket to buy low-sodium foods.

[0602] Specific behavior: After receiving the notification, the user performs the action as instructed and provides feedback to the app about the results.

[0603] Step 6: Gather feedback and suggest improvements for next time

[0604] User: After completing an action, provide feedback to the app about what you did. For example, you might enter "I ran for 30 minutes."

[0605] Input: Feedback data (results of the action)

[0606] Server: Re-analyzes the received feedback data, evaluates its effectiveness, and incorporates the feedback into the next improvement plan, thereby supporting continuous improvement of health status.

[0607] Output: Next improvement plan

[0608] How it works: Feedback data is collected and evaluated using an analytics engine. Based on the results, the next improvement plan is generated using a generative AI model.

[0609] The above are the specific processing steps and operations of the program for this system.

[0610] (Application example 1)

[0611] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0612] In modern society, individual health management is becoming increasingly important, but it is difficult to effectively maintain health through self-management alone. In particular, there is a lack of methods that comprehensively analyze health checkup results, daily dietary habits, and exercise levels, and provide specific, individually customized health improvement proposals. There are also insufficient methods for users to easily obtain the information necessary to implement improvement proposals (for example, information on recommended stores and services). For this reason, a system that allows users to effectively manage their health on a daily basis is needed.

[0613] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0614] In this invention, the server includes means for inputting health checkup data, means for taking images of each meal and inputting the meal contents, means for transmitting the health checkup data, meal data, and step count data to the server, means for analyzing the data in the server, evaluating the health condition, and generating improvement suggestions, means for transmitting the generated improvement suggestions and recommended store information to the terminal, means for notifying the user of the improvement suggestions displayed on the terminal, means for accumulating user behavior data and reflecting the data in generating the next improvement suggestion, means for using location information to search for information on the nearest store and provide recommended store information related to meals and exercise, means for analyzing meal contents using image analysis technology and extracting major nutrients, means for evaluating the user's health condition using a generative AI model and generating specific improvement suggestions, and means for notifying the user of the generated improvement suggestions and recommended store information to a smartphone or head-mounted display, thereby enabling the user to manage their own health condition in real time and take appropriate action to improve their health.

[0615] "Health checkup data" refers to test result data that indicates the user's health condition, and includes information such as blood pressure, blood sugar level, and cholesterol level.

[0616] "Meal images" are photographs of meals consumed by a user, providing a visual record of the meal contents.

[0617] "Dietary content" refers to information including the type and amount of food and major nutrients (protein, carbohydrates, fat, etc.).

[0618] "Step count data" is data that records the number of steps a user takes in a day, and indicates the amount of exercise.

[0619] The "server" is a central computer device in the system that receives, analyzes, and evaluates health checkup data, dietary data, and step count data.

[0620] The "means for assessing health status" is a function in which the server analyzes the data received and diagnoses the user's current health status.

[0621] The "means for generating improvement proposals" is a function in which the server creates a specific health improvement plan suited to the user based on the results of the health status evaluation.

[0622] "Recommended store information" refers to information about the nearest store that provides the food or service you need based on your location information and health improvement suggestions.

[0623] A "terminal" refers to a device operated by a user, such as a smartphone or head-mounted display, that receives and displays information from a server.

[0624] "Means of notification" refers to the function that the device uses to inform users of improvement suggestions and recommended store information, and includes push notifications and in-app messages.

[0625] "Behavioral data" refers to data about the actions actually taken by the user, including diet, exercise, purchasing history, etc.

[0626] "Image analysis technology" is a technology that extracts information from image data such as photographs, and is used in health management systems to analyze the contents of meals.

[0627] "Macronutrients" are the important components of food, such as proteins, carbohydrates, and fats, that are necessary for maintaining a healthy human body.

[0628] A "generative AI model" is a model that uses artificial intelligence to analyze a user's health data and automatically generate appropriate health improvement suggestions.

[0629] The present invention provides a system for helping users understand their own health condition in real time and take appropriate measures to improve it. Specific embodiments are described below.

[0630] System configuration:

[0631] 1. Health checkup data input method:

[0632] Users can manually enter their health checkup results using a device (smartphone or head-mounted display) or take a photo and register the data using OCR technology. This data is sent to a server and stored as the user's basic health database.

[0633] 2. Meal content input method:

[0634] Users take photos of each meal and upload them to the app. The device receives the photos and analyzes the meal contents using image analysis technology (such as Google Cloud Vision API). The analysis results are sent to a server as information on macronutrients such as protein, carbohydrates, and fat.

[0635] 3. Step count data collection method:

[0636] The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to collect step count data and periodically transmits it to a server.

[0637] 4. Health status assessment and improvement proposal generation method:

[0638] The server comprehensively analyzes the received health checkup data, dietary data, and step count data. During this process, a generative AI model (using TensorFlow and PyTorch) is used to evaluate the user's health status and generate specific improvement suggestions.

[0639] 5. Recommended store information methods:

[0640] Based on the generated improvement proposal, the server uses the user's location information to search for and obtain recommended store information for the nearest store using Google Maps API, etc. This store information will be related to food and exercise that will help improve the user's health.

[0641] 6. Means of notification:

[0642] The server sends improvement suggestions and recommended store information to the device (smartphone or head-mounted display), which then notifies the user via push notification (Firebase Cloud Messaging) or in-app message.

[0643] Examples:

[0644] 1. Registering health checkup results:

[0645] The user takes an image of the blood pressure result with their smartphone, and OCR technology extracts text data from the image and sends the blood pressure information to the server.

[0646] 2. Enter your meal details:

[0647] Users take a photo of their lunch and upload it to the app, where, using image analysis technology, the server analyzes the macronutrients contained in the meal and adds them to a dietary database.

[0648] 3. Step count data collection:

[0649] The smartphone's built-in pedometer measures the number of steps the user takes each day and periodically sends the data to a server.

[0650] 4. Health status assessment and generation of improvement proposals:

[0651] The server analyzes the collected data using a generative AI model to assess the user's health condition, and generates specific recommendations for improvement, such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[0652] 5. Providing recommended store information:

[0653] Based on the user's location information, the server searches for the nearest supermarket and provides information on stores where low-salt foods can be purchased.

[0654] Example prompt sentence:

[0655] User health data: Blood pressure: 120 / 80 mmHg, Blood glucose: 5.6 mmol / L

[0656] Meal details: Salad, chicken, rice

[0657] Step count data: 7000 steps

[0658] Location information: 35.6895° N, 139.6917° E

[0659] Use this data to assess the user's health status and generate specific recommendations for improvement.

[0660] These components and examples empower users to manage their health in real time and provide the information they need to implement appropriate improvements.

[0661] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0662] Step 1:

[0663] The user inputs the health checkup data. Using a device (smartphone or head-mounted display), the user can manually enter the health checkup results directly or take a photo of the results and upload it to the app. The device uses OCR technology to extract and format text data from the image data, which then sends the initial health data to the server.

[0664] Input: Medical examination result images or manual input data

[0665] Output: Formatted medical examination data

[0666] Step 2:

[0667] The user takes a photo of each meal and enters the meal contents. The user takes a photo of the meal and uploads it to the app. The device uses image analysis technology (Google Cloud Vision API) to analyze the meal contents and extract macronutrients (protein, carbohydrates, fat, etc.). The analysis results are sent to the server.

[0668] Input: Food image

[0669] Output: Macronutrient information of meal contents

[0670] Step 3:

[0671] The device collects step count data and sends it to a server. The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to measure the number of steps taken each day. The step count data is periodically sent to the server, allowing the amount of daily exercise to be recorded.

[0672] Input: Pedometer data

[0673] Output: Periodically collected step count data

[0674] Step 4:

[0675] The server analyzes health checkup data, dietary data, and step count data to assess the user's health status and generate improvement proposals. The server comprehensively processes and analyzes the received data and evaluates the user's health status using a generative AI model (using TensorFlow or PyTorch). Based on the analysis results, it generates specific improvement proposals.

[0676] Input: Health checkup data, dietary data, step count data

[0677] Output: User's health status assessment results and specific improvement suggestions

[0678] Step 5:

[0679] Based on the generated improvement plan, the server uses location information to obtain recommended store information for the nearest store. The server uses the user's location information to search for and obtain store information that provides the desired food or service using Google Maps API, etc.

[0680] Input: improvement proposal, user location information

[0681] Output: Recommended store information

[0682] Step 6:

[0683] The server sends improvement suggestions and recommended store information to the device, which then notifies the user. The server sends the generated improvement suggestions and recommended store information to the device, which then notifies the user via push notification or in-app message.

[0684] Input: Improvement proposals, recommended store information

[0685] Output: A message to inform the user

[0686] Step 7:

[0687] User behavior data is accumulated and reflected in the generation of the next improvement proposal. The user takes action based on the notification they receive (e.g., shopping at a recommended store). This behavior data is sent back to the server and taken into account in the next data analysis.

[0688] Input: User behavior data

[0689] Output: Data to be reflected in the next improvement plan generation

[0690] This allows the system to manage the user's health status in real time and continue to provide appropriate improvement suggestions.

[0691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0692] This invention is a system that helps users understand their health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system customizes the improvement suggestions, providing more personalized health management. It also uses location information to provide recommended store information.

[0693] Program processing

[0694] 1. Initial data registration:

[0695] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0696] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0697] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0698] 2. Data Collection and Transmission:

[0699] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0700] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0701] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0702] 3. Adding an Emotion Engine:

[0703] User: The user inputs or automatically collects emotional data through an app equipped with an emotion engine. For example, when a user feels stressed, emotional data can be obtained through facial recognition or voice analysis.

[0704] Device: The device collects emotion data and sends it to the server.

[0705] Server: The server receives the emotion data, integrates it with other health data, and performs analysis.

[0706] 4. Health status assessment and generation of improvement proposals:

[0707] Server: The server comprehensively analyzes the collected health checkup data, dietary data, step count data, and emotional data. It evaluates the user's current health condition and generates specific recommendations for improvement. For example, it generates advice such as, "Today's salt intake was high and your stress level is also high, so choose low-salt foods for your next meal and suggest some exercises to help you relax."

[0708] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0709] 5. Notice and Execution:

[0710] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0711] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0712] 6. Gather feedback and suggestions for next improvements:

[0713] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0714] This allows users to manage their health consistently and receive personalized health improvement suggestions that take their emotional state into account, helping them maintain and improve their health more effectively. The addition of the emotion engine will enable holistic health management that also takes into account the user's mental health.

[0715] The processing flow will be explained below.

[0716] Step 1:

[0717] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0718] Step 2:

[0719] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0720] Step 3:

[0721] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0722] Step 4:

[0723] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0724] Step 5:

[0725] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0726] Step 6:

[0727] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0728] Step 7:

[0729] User: The user uses an app equipped with an emotion engine, and emotion data is input or automatically collected. For example, emotion data is obtained using facial recognition or voice analysis functions when the user feels stressed.

[0730] Step 8:

[0731] Device: The device collects emotion data and sends it to the server.

[0732] Step 9:

[0733] Server: The server receives the emotion data, integrates it with health checkup data, dietary data, and step count data, and performs a comprehensive analysis.

[0734] Step 10:

[0735] Server: The server uses the integrated data to assess the user's current health status and generate specific recommendations for improvement. For example, if the user's stress level is high and their salt intake is high, the server will generate advice such as "Choose low-salt foods for your next meal and do 30 minutes of yoga to relax."

[0736] Step 11:

[0737] Server: Based on the generated improvement suggestions, the server uses the user's location information to search for recommended stores and restaurants nearby, for example, to obtain information on purchasing low-salt foods at a nearby supermarket.

[0738] Step 12:

[0739] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed via push notifications and in-app messages.

[0740] Step 13:

[0741] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or practicing yoga as suggested by the app.

[0742] Step 14:

[0743] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0744] This allows users to consistently manage their health in a way that is individually customized, and helps maintain and improve overall health, taking into account their emotional state.

[0745] Example 2

[0746] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] With conventional health management systems, it was difficult for users to accurately understand their own health status in real time and implement appropriate improvement measures. In particular, the data collected was limited, making it impossible to generate customized improvement plans that took into account each user's emotional and behavioral data. In addition, users had no easy way to obtain information about stores offering the foods and services they needed. These issues led to a lack of consistency in users' health management, hindering effective health maintenance and improvement.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0749] In this invention, the server includes: a means for analyzing health checkup data using OCR technology to evaluate health status and generate improvement proposals using a new data source including emotion data; a means for searching for the generated improvement proposals and recommended store information based on location information and transmitting them to the terminal; and a means for reanalyzing the user's behavioral data and emotion data and incorporating the feedback into the next improvement proposal. This makes it possible to comprehensively evaluate each user's health status and provide customized improvement proposals. Furthermore, by utilizing location information to provide information on recommended stores near the user, users can easily obtain the foods and services they need.

[0750] "Health checkup data" is information about the user's physiological health condition, including measurements such as blood pressure, blood sugar level, and cholesterol level.

[0751] "Dietary data" refers to information about the meals a user eats, including photos of the meals, the contents of the meals, and the calories and nutrients consumed.

[0752] "Step data" is information about the number of steps a user takes each day, collected from pedometer apps and devices.

[0753] "Emotional Data" refers to information about a user's emotional state, including stress levels and moods obtained through facial recognition, voice analysis, or manual input.

[0754] "OCR technology" stands for optical character recognition technology, which extracts characters from image data and converts them into text data.

[0755] "Improvement Suggestions" are specific advice or suggestions provided to improve the user's health based on data analyzed by the server.

[0756] "Location information" refers to the user's current physical location, obtained based on GPS and network information.

[0757] "Recommended store information" is information about stores that offer the food or services you need, provided based on your location information.

[0758] "Terminal" means a computing device used by a User, including a smartphone or tablet.

[0759] A "server" is a central computing system that analyzes data and stores and transmits results.

[0760] This invention relates to a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, this system analyzes health checkup data, dietary data, and step count data on a server and notifies the user of specific improvement suggestions, thereby enabling the user to effectively manage their health. In addition, an emotion engine is used to collect user emotion data and customize health management. Location information is also used to provide recommended store information. Specific embodiments of the system are described below.

[0761] Initial data registration

[0762] User: The user launches the smartphone app and enters the results of their health checkup. For example, they manually enter values ​​such as blood pressure, blood sugar, and cholesterol levels. They can also take a photo of their health checkup report and register the data.

[0763] Terminal: The terminal receives the input data and converts the image data into text data using OCR (Optical Character Recognition) technology. The converted data is then formatted into a standard format and sent to the server.

[0764] Server: The server analyzes the health checkup data sent from the device. The analyzed data is stored in the user's basic health database, and health assessment is performed to set baseline and target values. For example, the user's target blood pressure value is set to 125 / 80.

[0765] Data collection and transmission

[0766] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food they eat. They can enter information such as "I had salad and fish for lunch."

[0767] Device: The device receives meal photos and manual input data and sends them to the server. Step count data is also automatically collected from the pedometer app and periodically sent to the server.

[0768] Server: The server analyzes the received food photos and calculates calorie intake and major nutrients (protein, carbohydrates, fat, etc.). It also analyzes the received step count data and calculates energy expenditure. For example, it calculates the calories in a salad from the photo as 100 kcal, and records the energy expenditure of 10,000 steps a day as 400 kcal.

[0769] Adding an Emotion Engine

[0770] User: Users input their emotional data through apps equipped with emotion engines, or the data is collected automatically. For example, when a user feels stressed, emotion data is acquired through facial recognition or voice analysis.

[0771] Device: The device collects emotion data and sends it to the server.

[0772] Server: The server receives the emotional data and analyzes it by combining it with other health data. For example, if the stress level is high, it suggests low-calorie food choices and exercise to relax.

[0773] Evaluating health status and generating improvement proposals

[0774] Server: The server performs a comprehensive analysis based on the collected health checkup data, dietary data, step count data, and emotional data. For example, it may assess that "Your calorie intake is high and your step count is low, so you need to increase your exercise."

[0775] Server: Evaluates the user's current health status and generates specific recommendations for improvement, such as "Today's salt intake is high and your stress level is also high, so choose low-salt foods for your next meal and suggest relaxation exercises."

[0776] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants in the vicinity and obtains store information that offers the desired food or service. For example, "List low-salt foods available at nearby supermarkets."

[0777] Notification and Execution

[0778] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages. For example, "Today's lunch is high in calories, so please choose low-calorie foods for dinner."

[0779] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[0780] Gathering feedback and suggestions for next improvements

[0781] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0782] Prompt Sentence Examples

[0783] "How are you feeling today? Enter your health indicators."

[0784] "Upload a photo of your meal and calculate the calories."

[0785] This system allows users to achieve continuous and individually customized health management, effectively maintaining and improving their overall health.

[0786] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0787] Step 1:

[0788] Enter and submit health checkup data

[0789] User: The user launches the smartphone app and manually enters the results of their health checkup. For example, they enter values ​​such as blood pressure, blood sugar, and cholesterol levels, such as "My blood pressure this morning was 130 / 85."

[0790] Terminal: The terminal receives the input data and can also convert photo data into text data using OCR technology instead of manual input. The converted data is formatted and sent to the server.

[0791] Input: Manual input or photo data.

[0792] Output: Formatted medical examination data.

[0793] Step 2:

[0794] Analysis and storage of health checkup data

[0795] Server: The server receives and analyzes the health checkup data sent from the device, evaluates the user's health status based on the data, and stores it in a basic health database.

[0796] Input: Formatted medical examination data.

[0797] Output: Health status assessment results and baseline health data.

[0798] Step 3:

[0799] Collection and transmission of dietary data

[0800] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0801] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it automatically collects step count data from the pedometer app and periodically sends it to the server.

[0802] Input: Meal photos, manual input data, step count data.

[0803] Output: Data sent to the server.

[0804] Step 4:

[0805] Data analysis

[0806] Server: The server analyzes the received food photos and calculates calorie intake and macronutrients (protein, carbohydrates, fat, etc.). Similarly, it analyzes step count data to calculate energy expenditure.

[0807] Input: Meal photos, manual input data, step count data.

[0808] Output: Calorie intake, macronutrients, energy expenditure.

[0809] Step 5:

[0810] Collecting and transmitting emotional data

[0811] User: Users input or automatically collect emotional data through apps equipped with emotion engines. Emotion data can be obtained through facial recognition and voice analysis.

[0812] Device: The device collects emotion data and sends it to the server.

[0813] Input: Emotion data (facial recognition, voice analysis, manual input).

[0814] Output: Data sent to the server.

[0815] Step 6:

[0816] Overall health assessment

[0817] Server: The server comprehensively analyzes the health check data, dietary data, step count data, and emotional data to evaluate the user's health status, generate specific improvement suggestions, and incorporate the feedback into the user's next actions.

[0818] Input: Health checkup data, dietary data, step count data, emotion data.

[0819] Output: Health status assessment results and improvement recommendations.

[0820] Step 7:

[0821] Use location information and find recommended stores

[0822] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby, and obtains information on stores that offer the desired food or service.

[0823] Input: Improvement suggestion, location information.

[0824] Output: Recommended store information.

[0825] Step 8:

[0826] Notification of improvement proposals and recommended store information

[0827] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0828] Input: Improvement suggestions, recommended store information.

[0829] Output: Notification to the user.

[0830] Step 9:

[0831] Implementing improvement actions

[0832] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[0833] Input: Improvement suggestions, recommended store information.

[0834] Output: Implementation of improvement actions.

[0835] Step 10:

[0836] Gather feedback and generate recommendations for next time improvements

[0837] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluating the effectiveness of the user's improvement actions and incorporating feedback into the next improvement proposal.

[0838] Input: User behavioral data, emotional data.

[0839] Output: Suggested improvements for next time.

[0840] This processing step allows users to receive continuous, personalized health management, consistently maintaining and improving their health effectively.

[0841] (Application example 2)

[0842] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0843] Conventional health management systems could evaluate a user's health status using health checkup data and dietary data, but they were unable to provide specific improvement proposals that took into account the user's emotions or real-time store integration. Furthermore, they lacked a mechanism for providing the products and services users needed in a timely manner when they visited a physical store. This resulted in ineffective health management for users and insufficient support for their mental health.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0845] In this invention, the server includes a means for inputting health checkup data, a means for taking photos of each meal and inputting the meal contents, a means for transmitting the health checkup data, meal data, and step count data to the server, a means for analyzing the data in the server, evaluating the health status, and generating improvement suggestions, a means for transmitting the generated improvement suggestions and recommended store information to the terminal, a means for notifying the user of the improvement suggestions displayed on the terminal, a means for analyzing facial expressions and voice to collect emotional data, a means for transmitting the emotional data to the server and integrating it with the health data for analysis, and a means for notifying the user of recommended product and service information in real time when the user visits a physical store. This allows the user to understand their health status in real time and receive individually customized improvement suggestions that take their emotional state into account. It also enables the user to obtain desired products and services in a timely manner when visiting a physical store.

[0846] "Health checkup data" refers to medical information such as blood pressure, blood sugar level, and cholesterol level obtained as a result of a health checkup that a user has undergone.

[0847] "Meal data" refers to photos taken by the user for each meal and information about the meal that the user inputs.

[0848] "Step count data" refers to data that records the number of steps a user takes each day.

[0849] The "server" is a data processing system that analyzes collected health checkup data, dietary data, step count data, and emotional data, evaluates health status, and generates improvement proposals.

[0850] "Emotional data" is data that indicates the user's stress level and emotional state obtained by analyzing their facial expressions and voice.

[0851] A "terminal" is an electronic device such as a smartphone or tablet held by the user, which displays improvement suggestions and recommended store information sent from the server.

[0852] "Recommended store information" is information that recommends the nearest suitable store or service based on the user's location information and health condition.

[0853] The "improvement suggestion" is a specific instruction or advice for improving the user's health condition, which is generated by the server by analyzing the health checkup data, dietary data, step count data, and emotional data.

[0854] "Behavioral data" is data that records the history of actions that users actually take and products they purchase.

[0855] "Facial expression and voice analysis" is a process in which a user's facial expressions are captured by a camera and their voice is recorded, and the emotion engine analyzes them to generate emotion data.

[0856] "Real-time notification" is a technology that instantly notifies users of necessary information and recommended products when they visit a physical store.

[0857] The system for implementing this invention collects and analyzes health checkup data, dietary data, step count data, and emotional data in real time. Ultimately, it provides users with health improvement suggestions and recommended store information based on the analysis results. The specific configuration and operation of this system are described below.

[0858] Hardware and Software Configuration

[0859] Hardware

[0860] Device: An electronic device, such as a smartphone or tablet, that users use to enter health data, take photos, and collect emotional data.

[0861] Server: A high-performance computer system installed in a data center that analyzes the received health and emotional data.

[0862] software

[0863] OCR technology: Converts the health check data entered by the user into text data.

[0864] Image analysis technology: Analyzes photos of food and facial expressions to obtain food content and emotional data. A specific example is an emotion recognition model using Keras.

[0865] Data analysis algorithm: Runs on the server and analyzes collected data to assess health status and generate improvement recommendations.

[0866] Data collection and analysis

[0867] Entering health data

[0868] Users can manually enter their health checkup results (such as blood pressure and blood sugar levels) or take a photo and upload it to the app. The device then converts this data into text using OCR technology and sends it to the server, which then creates a basic health database for the user.

[0869] Collection of dietary data

[0870] Users take a photo of their meal each time they eat and upload it to the app, which then sends the photo to a server that uses image analysis technology to calculate calorie and macronutrient intake.

[0871] Collecting step count data

[0872] The user's device uses a pedometer to measure the number of steps taken each day and periodically transmits this data to a server, which then calculates energy consumption.

[0873] Collecting Emotional Data

[0874] When a user feels stressed, emotional data is collected through facial recognition and voice analysis. The device then sends this data to a server, which then combines it with health data for analysis.

[0875] Providing recommended store information

[0876] Based on the analysis results, the server generates personalized health improvement recommendations for the user, such as "Today's salt intake is high, so choose low-salt foods at your next meal."

[0877] The server uses the user's location information to search for recommended stores nearby, and then notifies the user of stores that offer the food or service they need.

[0878] User Notification and Feedback

[0879] Improvement suggestions and recommended store information are sent from the server to the device and displayed as push notifications or in-app messages. Users can check the notification and take specific actions according to the instructions.

[0880] User behavior data and purchase history are also sent to the server and fed back into the analysis results, which are then reflected in future improvement proposals, enabling more advanced customization.

[0881] Prompt Sentence Examples

[0882] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[0883] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0884] Step 1:

[0885] Entering and submitting health data

[0886] Users manually enter their health check results (e.g., blood pressure, blood sugar, cholesterol levels, etc.) or take a photo and upload it to the device, which uses OCR technology to convert this data into text, format it, and send it to the server.

[0887] Input: Medical checkup results (manual entry or photo)

[0888] Output: Formatted text data (sent to server)

[0889] Step 2:

[0890] Collection and transmission of dietary data

[0891] Users take a photo of their meal each time they eat and upload it to the app. The device then sends the photo, along with any manually entered meal information, to a server. The server then uses image analysis technology to analyze the photo and calculate calorie and macronutrient intake.

[0892] Input: Meal photos (and manually entered meal information)

[0893] Output: Calorie intake and macronutrient data (analysis results on the server)

[0894] Step 3:

[0895] Collection and transmission of step count data

[0896] The device uses a built-in pedometer to collect the user's daily step count and periodically transmits this data to a server, which analyzes the step count data and calculates energy consumption.

[0897] Input: Step count data (automatically sent from the device)

[0898] Output: Energy consumption data (analysis results on the server)

[0899] Step 4:

[0900] Acquiring and sending emotion data

[0901] When a user feels stress or other emotions, they take a photo of their face or record their voice. The device sends this data to a server, which then analyzes the data using an emotion recognition model (e.g., a model using Keras) to evaluate the user's emotional state.

[0902] Input: Face photo (or voice data)

[0903] Output: Emotional state data (analysis results on the server)

[0904] Step 5:

[0905] Integrated analysis of health status and generation of improvement proposals

[0906] The server integrates and analyzes the collected health checkup data, dietary data, step count data, and emotional data. Based on the analysis results, it generates customized health improvement recommendations. For example, it creates specific advice such as, "Today's salt intake was high, so choose low-salt foods at your next meal."

[0907] Input: Health checkup data, dietary data, step count data, emotion data

[0908] Output: Health improvement suggestions (specific advice)

[0909] Step 6:

[0910] Generate and notify recommended store information

[0911] The server uses the user's location information to search for nearby recommended store information. The generated improvement suggestions and recommended store information are sent to the device, and the user receives them via push notification or in-app message.

[0912] Input: Location, Health Improvement Suggestion

[0913] Output: Recommended store information (notification to user)

[0914] Step 7:

[0915] Collecting feedback

[0916] The user's behavioral data and purchase history are sent to the server via the device, where the server reanalyzes the data and incorporates the feedback into future improvements, resulting in more accurate and personalized health advice.

[0917] Input: Behavioral data, purchase history

[0918] Output: Feedback data for improvement proposals (analysis results on the server)

[0919] Prompt Sentence Examples

[0920] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[0921] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0922] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0923] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0924] [Third embodiment]

[0925] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0926] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0927] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0928] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0929] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0930] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0931] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0932] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0933] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0934] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0935] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0936] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0937] This invention is a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. It also uses location information to provide recommended store information, allowing users to easily obtain the foods and services they need.

[0938] Program processing

[0939] 1. Initial data registration:

[0940] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0941] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0942] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0943] 2. Data Collection and Transmission:

[0944] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0945] Device: The device receives the meal photos and sends them to the server for analysis. At the same time, it collects step count data and periodically sends it to the server.

[0946] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0947] 3. Health status assessment and generation of improvement proposals:

[0948] Server: The server comprehensively analyzes the collected health checkup data, dietary data, and step count data. It evaluates the user's current health condition and generates specific improvement recommendations. For example, the server may generate an assessment result such as, "Today's salt intake is high, so it is recommended that you choose low-salt foods for your next meal."

[0949] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0950] 4. Notice and Execution:

[0951] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0952] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0953] 5. Gather feedback and suggestions for next improvements:

[0954] Server: The server receives and analyzes the user's behavioral data again. It evaluates the effectiveness of the user's improvement actions and incorporates the feedback into the next improvement plan. This allows for consistent health management.

[0955] Through these processes, users can manage their own health status in real time and take appropriate actions to improve their health. The system continuously monitors the user's health status and provides individually customized health improvement suggestions to help users maintain their health in the long term.

[0956] The processing flow will be explained below.

[0957] Step 1:

[0958] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[0959] Step 2:

[0960] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[0961] Step 3:

[0962] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[0963] Step 4:

[0964] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[0965] Step 5:

[0966] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[0967] Step 6:

[0968] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[0969] Step 7:

[0970] Server: The server comprehensively analyzes the health check data, dietary data, and step count data to evaluate the user's current health condition.

[0971] Step 8:

[0972] Server: The server generates specific improvement suggestions based on the evaluation results. For example, it generates advice such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[0973] Step 9:

[0974] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[0975] Step 10:

[0976] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[0977] Step 11:

[0978] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[0979] Step 12:

[0980] Server: The server receives and analyzes the user behavior data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[0981] This allows users to consistently manage their health and receive personalized health improvement suggestions, helping them to more effectively maintain and improve their health.

[0982] Example 1

[0983] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0984] In modern society, individual users need to accurately and quickly grasp a large amount of information in order to effectively manage their own health status in real time and take appropriate measures to improve it. For this reason, there is a demand for systems that can efficiently collect and analyze health checkup data, daily diet data, and exercise data, and provide users with specific improvement suggestions. However, conventional systems have faced challenges, such as cumbersome data collection and insufficient analysis, making it difficult for users to effectively manage their health.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0986] In this invention, the server includes a means for analyzing the received health checkup data, dietary data, and step count data to evaluate the health condition, a means for generating improvement proposals using a generative AI model, and a means for acquiring recommended store information based on the user's location information, thereby enabling the user to understand their own health condition in real time and obtain specific and effective improvement proposals.

[0987] A "user" is an individual who uses the system to manage their own health status and receive improvement suggestions.

[0988] "Health checkup data" refers to data relating to the user's health condition, such as blood pressure, blood sugar level, and cholesterol level.

[0989] "Dietary data" refers to data that a user records by inputting or taking photos of their daily meals.

[0990] "Step count data" refers to data on the number of steps a user takes in their daily life.

[0991] A "server" is a device that analyzes the received data and provides the user with appropriate improvement suggestions.

[0992] "Analysis" is the process of evaluating health status based on collected data and extracting necessary information.

[0993] A "generative AI model" is a model that uses machine learning technology to generate improvement suggestions based on a user's health condition.

[0994] "Improvement Suggestions" are specific instructions or suggestions for improving the user's health.

[0995] "Location information" refers to the geographic coordinate data of the user's current location.

[0996] "Recommended store information" is information about stores that offer suggested meals and services based on the user's location information.

[0997] "Notifications" are messages or alerts sent from the server to the device, and are a means of conveying information to the user.

[0998] "Feedback" is information that reports to the system the evaluation and results of improvement proposals implemented by the user.

[0999] The present invention provides a system that enables a user to grasp his or her own health condition in real time and assists the user in taking appropriate measures to improve the condition. A specific embodiment of this system will be described below.

[1000] 1. System Configuration

[1001] The system includes a server where users input health checkup data and dietary data, which analyzes this data, a terminal that displays the analysis results, and a generative AI model.

[1002] Hardware and Software

[1003] Device: A mobile device such as a smartphone or tablet, optionally equipped with software that uses OCR technology (e.g., Google Cloud Vision API).

[1004] Server: A high-performance computer for analyzing data and running generative AI models, using database software (e.g., MySQL) and machine learning frameworks (e.g., TensorFlow).

[1005] 2. Initial data registration

[1006] The user launches the app and registers the health check results by entering them or taking a photo. For example, the user enters a blood pressure of 120 / 80 mmHg or a blood sugar level of 90 mg / dL. The device converts the photo into text data using OCR technology as needed and sends the formatted data to the server. The server analyzes the received data and stores it in a basic health database.

[1007] 3. Data Collection and Transmission

[1008] Each time a user eats a meal, they take a photo of it and upload it to the app. For example, they can register an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner. The device receives the meal photos and sends them to a server. The device also collects the user's step count data daily and periodically sends it to the server. The server uses image analysis technology to analyze the meal photos and calculates the key nutrients. It also analyzes the step count data to calculate calories burned.

[1009] 4. Health status assessment and generation of improvement proposals

[1010] The server comprehensively analyzes the accumulated health checkup data, dietary data, and step count data, and uses a generative AI model to evaluate the user's health condition. For example, if the dietary data indicates excessive calorie intake and insufficient exercise, it will generate specific improvement suggestions such as "increase exercise" or "choose low-calorie foods."

[1011] Location information and recommended store information

[1012] The server uses the user's location information to search for recommended stores and restaurants nearby via internet services, obtains information about stores that offer the desired food or service, and sends this information along with improvement suggestions to the user's device.

[1013] 5. Notification and Execution

[1014] The device notifies the user of the improvement suggestions and recommended store information received from the server. For example, a push notification might say, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket." The user can check the notification and take specific action. For example, they might go to a nearby supermarket to purchase low-salt foods or do the suggested exercise.

[1015] 6. Gather feedback and suggest improvements for next time

[1016] After the user completes the action based on the improvement suggestion, they provide feedback on the results. For example, the user might enter feedback such as "I ran for 30 minutes." The server then analyzes this data again and generates the next improvement suggestion based on the feedback. This feedback loop allows the user's health to continuously improve.

[1017] Prompt Sentence Examples

[1018] "The user's health check results show that their blood pressure is 120 / 80 mmHg, their blood sugar is 90 mg / dL, and their cholesterol is 180 mg / dL. Please generate a future health improvement plan based on their dietary and step count data."

[1019] "The user has consumed 2300 kcal and taken 8000 steps today. Please suggest appropriate improvements based on this information."

[1020] As described above, this system comprehensively manages users' health data and provides specific health improvement suggestions based on the analysis results, enabling users to effectively manage their health.

[1021] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1022] Step 1: Registering initial data

[1023] User: Launches the app and enters their health checkup results, such as blood pressure 120 / 80 mmHg, blood sugar 90 mg / dL, and cholesterol 180 mg / dL, or takes and uploads a photo of their health checkup results.

[1024] Input: Health checkup data such as blood pressure, blood sugar, and cholesterol levels

[1025] Terminal: The received health check data is converted into text data using OCR technology (e.g., Google Cloud Vision API), formatted, and sent to the server in the format "user ID_recording date and time."

[1026] Output: Formatted medical examination data

[1027] Server: Receives formatted health checkup data, evaluates it using an analysis engine, and stores it in a database (e.g., MySQL) as basic health data. It also sets reference and target values.

[1028] Step 2: Collect and send food and exercise data

[1029] User: Takes a photo of each meal and uploads it to the app, for example, an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner.

[1030] Input: Food photos and text data

[1031] Device: Receives photos and sends them to the server. It also collects the user's daily step count data from an activity tracker (e.g., a fitness band) and periodically sends it to the server.

[1032] Output: Meal photos, step count data

[1033] Server: Analyzes food photos using an image analysis engine (e.g., a model using TensorFlow) and calculates macronutrients (calories, protein, carbohydrates, fat, etc.). Also analyzes step count data and calculates calories burned. Integrates this data with basic health data.

[1034] Step 3: Assessing health status and generating improvement proposals

[1035] Server: Comprehensively analyzes health checkup data, dietary data, and step count data to assess the user's health condition. Using a generative AI model, it generates improvement recommendations based on the user's current situation. For example, "Today's calorie intake has exceeded the target, so we recommend choosing low-calorie foods for your next meal."

[1036] Input: Health checkup data, dietary data, step count data

[1037] Output: Health status assessment, improvement suggestions

[1038] Specific actions: To assess health status, compare with past data and baseline values. To generate improvement recommendations, the system aims to provide personalized and specific advice using generative AI models.

[1039] Step 4: Get recommended store information based on location information

[1040] Server: Based on the improvement plan, use the user's location information to search for recommended nearby stores through an internet service (e.g., map API). For example, search for "nearby supermarkets that sell low-salt foods."

[1041] Input: Improvement proposal, user location

[1042] Output: Recommended store information

[1043] Specific operation: The server receives the location data, searches for stores using the API, converts the results into text format, and provides it to the user.

[1044] Step 5: Send and execute notifications

[1045] On the device: The system sends push notifications to the user with the improvement suggestions and recommended store information received from the server. For example, it sends a notification saying, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket."

[1046] Input: Improvement proposals, recommended store information

[1047] Output: User notification

[1048] User: Checks the notification and takes specific action, such as going to a nearby supermarket to buy low-sodium foods.

[1049] Specific behavior: After receiving the notification, the user performs the action as instructed and provides feedback to the app about the results.

[1050] Step 6: Gather feedback and suggest improvements for next time

[1051] User: After completing an action, provide feedback to the app about what you did. For example, you might enter "I ran for 30 minutes."

[1052] Input: Feedback data (results of the action)

[1053] Server: Re-analyzes the received feedback data, evaluates its effectiveness, and incorporates the feedback into the next improvement plan, thereby supporting continuous improvement of health status.

[1054] Output: Next improvement plan

[1055] How it works: Feedback data is collected and evaluated using an analytics engine. Based on the results, the next improvement plan is generated using a generative AI model.

[1056] The above are the specific processing steps and operations of the program for this system.

[1057] (Application example 1)

[1058] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1059] In modern society, individual health management is becoming increasingly important, but it is difficult to effectively maintain health through self-management alone. In particular, there is a lack of methods that comprehensively analyze health checkup results, daily dietary habits, and exercise levels, and provide specific, individually customized health improvement proposals. There are also insufficient methods for users to easily obtain the information necessary to implement improvement proposals (for example, information on recommended stores and services). For this reason, a system that allows users to effectively manage their health on a daily basis is needed.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1061] In this invention, the server includes means for inputting health checkup data, means for taking images of each meal and inputting the meal contents, means for transmitting the health checkup data, meal data, and step count data to the server, means for analyzing the data in the server, evaluating the health condition, and generating improvement suggestions, means for transmitting the generated improvement suggestions and recommended store information to the terminal, means for notifying the user of the improvement suggestions displayed on the terminal, means for accumulating user behavior data and reflecting the data in generating the next improvement suggestion, means for using location information to search for information on the nearest store and provide recommended store information related to meals and exercise, means for analyzing meal contents using image analysis technology and extracting major nutrients, means for evaluating the user's health condition using a generative AI model and generating specific improvement suggestions, and means for notifying the user of the generated improvement suggestions and recommended store information to a smartphone or head-mounted display, thereby enabling the user to manage their own health condition in real time and take appropriate action to improve their health.

[1062] "Health checkup data" refers to test result data that indicates the user's health condition, and includes information such as blood pressure, blood sugar level, and cholesterol level.

[1063] "Meal images" are photographs of meals consumed by a user, providing a visual record of the meal contents.

[1064] "Dietary content" refers to information including the type and amount of food and major nutrients (protein, carbohydrates, fat, etc.).

[1065] "Step count data" is data that records the number of steps a user takes in a day, and indicates the amount of exercise.

[1066] The "server" is a central computer device in the system that receives, analyzes, and evaluates health checkup data, dietary data, and step count data.

[1067] The "means for assessing health status" is a function in which the server analyzes the data received and diagnoses the user's current health status.

[1068] The "means for generating improvement proposals" is a function in which the server creates a specific health improvement plan suited to the user based on the results of the health status evaluation.

[1069] "Recommended store information" refers to information about the nearest store that provides the food or service you need based on your location information and health improvement suggestions.

[1070] A "terminal" refers to a device operated by a user, such as a smartphone or head-mounted display, that receives and displays information from a server.

[1071] "Means of notification" refers to the function that the device uses to inform users of improvement suggestions and recommended store information, and includes push notifications and in-app messages.

[1072] "Behavioral data" refers to data about the actions actually taken by the user, including diet, exercise, purchasing history, etc.

[1073] "Image analysis technology" is a technology that extracts information from image data such as photographs, and is used in health management systems to analyze the contents of meals.

[1074] "Macronutrients" are the important components of food, such as proteins, carbohydrates, and fats, that are necessary for maintaining a healthy human body.

[1075] A "generative AI model" is a model that uses artificial intelligence to analyze a user's health data and automatically generate appropriate health improvement suggestions.

[1076] The present invention provides a system for helping users understand their own health condition in real time and take appropriate measures to improve it. Specific embodiments are described below.

[1077] System configuration:

[1078] 1. Health checkup data input method:

[1079] Users can manually enter their health checkup results using a device (smartphone or head-mounted display) or take a photo and register the data using OCR technology. This data is sent to a server and stored as the user's basic health database.

[1080] 2. Meal content input method:

[1081] Users take photos of each meal and upload them to the app. The device receives the photos and analyzes the meal contents using image analysis technology (such as Google Cloud Vision API). The analysis results are sent to a server as information on macronutrients such as protein, carbohydrates, and fat.

[1082] 3. Step count data collection method:

[1083] The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to collect step count data and periodically transmits it to a server.

[1084] 4. Health status assessment and improvement proposal generation method:

[1085] The server comprehensively analyzes the received health checkup data, dietary data, and step count data. During this process, a generative AI model (using TensorFlow and PyTorch) is used to evaluate the user's health status and generate specific improvement suggestions.

[1086] 5. Recommended store information methods:

[1087] Based on the generated improvement proposal, the server uses the user's location information to search for and obtain recommended store information for the nearest store using Google Maps API, etc. This store information will be related to food and exercise that will help improve the user's health.

[1088] 6. Means of notification:

[1089] The server sends improvement suggestions and recommended store information to the device (smartphone or head-mounted display), which then notifies the user via push notification (Firebase Cloud Messaging) or in-app message.

[1090] Examples:

[1091] 1. Registering health checkup results:

[1092] The user takes an image of the blood pressure result with their smartphone, and OCR technology extracts text data from the image and sends the blood pressure information to the server.

[1093] 2. Enter your meal details:

[1094] Users take a photo of their lunch and upload it to the app, where, using image analysis technology, the server analyzes the macronutrients contained in the meal and adds them to a dietary database.

[1095] 3. Step count data collection:

[1096] The smartphone's built-in pedometer measures the number of steps the user takes each day and periodically sends the data to a server.

[1097] 4. Health status assessment and generation of improvement proposals:

[1098] The server analyzes the collected data using a generative AI model to assess the user's health condition, and generates specific recommendations for improvement, such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[1099] 5. Providing recommended store information:

[1100] Based on the user's location information, the server searches for the nearest supermarket and provides information on stores where low-salt foods can be purchased.

[1101] Example prompt sentence:

[1102] User health data: Blood pressure: 120 / 80 mmHg, Blood glucose: 5.6 mmol / L

[1103] Meal details: Salad, chicken, rice

[1104] Step count data: 7000 steps

[1105] Location information: 35.6895° N, 139.6917° E

[1106] Use this data to assess the user's health status and generate specific recommendations for improvement.

[1107] These components and examples empower users to manage their health in real time and provide the information they need to implement appropriate improvements.

[1108] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1109] Step 1:

[1110] The user inputs the health checkup data. Using a device (smartphone or head-mounted display), the user can manually enter the health checkup results directly or take a photo of the results and upload it to the app. The device uses OCR technology to extract and format text data from the image data, which then sends the initial health data to the server.

[1111] Input: Medical examination result images or manual input data

[1112] Output: Formatted medical examination data

[1113] Step 2:

[1114] The user takes a photo of each meal and enters the meal contents. The user takes a photo of the meal and uploads it to the app. The device uses image analysis technology (Google Cloud Vision API) to analyze the meal contents and extract macronutrients (protein, carbohydrates, fat, etc.). The analysis results are sent to the server.

[1115] Input: Food image

[1116] Output: Macronutrient information of meal contents

[1117] Step 3:

[1118] The device collects step count data and sends it to a server. The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to measure the number of steps taken each day. The step count data is periodically sent to the server, allowing the amount of daily exercise to be recorded.

[1119] Input: Pedometer data

[1120] Output: Periodically collected step count data

[1121] Step 4:

[1122] The server analyzes health checkup data, dietary data, and step count data to assess the user's health status and generate improvement proposals. The server comprehensively processes and analyzes the received data and evaluates the user's health status using a generative AI model (using TensorFlow or PyTorch). Based on the analysis results, it generates specific improvement proposals.

[1123] Input: Health checkup data, dietary data, step count data

[1124] Output: User's health status assessment results and specific improvement suggestions

[1125] Step 5:

[1126] Based on the generated improvement plan, the server uses location information to obtain recommended store information for the nearest store. The server uses the user's location information to search for and obtain store information that provides the desired food or service using Google Maps API, etc.

[1127] Input: improvement proposal, user location information

[1128] Output: Recommended store information

[1129] Step 6:

[1130] The server sends improvement suggestions and recommended store information to the device, which then notifies the user. The server sends the generated improvement suggestions and recommended store information to the device, which then notifies the user via push notification or in-app message.

[1131] Input: Improvement proposals, recommended store information

[1132] Output: A message to inform the user

[1133] Step 7:

[1134] User behavior data is accumulated and reflected in the generation of the next improvement proposal. The user takes action based on the notification they receive (e.g., shopping at a recommended store). This behavior data is sent back to the server and taken into account in the next data analysis.

[1135] Input: User behavior data

[1136] Output: Data to be reflected in the next improvement plan generation

[1137] This allows the system to manage the user's health status in real time and continue to provide appropriate improvement suggestions.

[1138] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1139] This invention is a system that helps users understand their health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system customizes the improvement suggestions, providing more personalized health management. It also uses location information to provide recommended store information.

[1140] Program processing

[1141] 1. Initial data registration:

[1142] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[1143] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[1144] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[1145] 2. Data Collection and Transmission:

[1146] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1147] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[1148] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[1149] 3. Adding an Emotion Engine:

[1150] User: The user inputs or automatically collects emotional data through an app equipped with an emotion engine. For example, when a user feels stressed, emotional data can be obtained through facial recognition or voice analysis.

[1151] Device: The device collects emotion data and sends it to the server.

[1152] Server: The server receives the emotion data, integrates it with other health data, and performs analysis.

[1153] 4. Health status assessment and generation of improvement proposals:

[1154] Server: The server comprehensively analyzes the collected health checkup data, dietary data, step count data, and emotional data. It evaluates the user's current health condition and generates specific recommendations for improvement. For example, it generates advice such as, "Today's salt intake was high and your stress level is also high, so choose low-salt foods for your next meal and suggest some exercises to help you relax."

[1155] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[1156] 5. Notice and Execution:

[1157] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[1158] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[1159] 6. Gather feedback and suggestions for next improvements:

[1160] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1161] This allows users to manage their health consistently and receive personalized health improvement suggestions that take their emotional state into account, helping them maintain and improve their health more effectively. The addition of the emotion engine will enable holistic health management that also takes into account the user's mental health.

[1162] The processing flow will be explained below.

[1163] Step 1:

[1164] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[1165] Step 2:

[1166] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[1167] Step 3:

[1168] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[1169] Step 4:

[1170] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1171] Step 5:

[1172] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[1173] Step 6:

[1174] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[1175] Step 7:

[1176] User: The user uses an app equipped with an emotion engine, and emotion data is input or automatically collected. For example, emotion data is obtained using facial recognition or voice analysis functions when the user feels stressed.

[1177] Step 8:

[1178] Device: The device collects emotion data and sends it to the server.

[1179] Step 9:

[1180] Server: The server receives the emotion data, integrates it with health checkup data, dietary data, and step count data, and performs a comprehensive analysis.

[1181] Step 10:

[1182] Server: The server uses the integrated data to assess the user's current health status and generate specific recommendations for improvement. For example, if the user's stress level is high and their salt intake is high, the server will generate advice such as "Choose low-salt foods for your next meal and do 30 minutes of yoga to relax."

[1183] Step 11:

[1184] Server: Based on the generated improvement suggestions, the server uses the user's location information to search for recommended stores and restaurants nearby, for example, to obtain information on purchasing low-salt foods at a nearby supermarket.

[1185] Step 12:

[1186] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed via push notifications and in-app messages.

[1187] Step 13:

[1188] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or practicing yoga as suggested by the app.

[1189] Step 14:

[1190] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1191] This allows users to consistently manage their health in a way that is individually customized, and helps maintain and improve overall health, taking into account their emotional state.

[1192] Example 2

[1193] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1194] With conventional health management systems, it was difficult for users to accurately understand their own health status in real time and implement appropriate improvement measures. In particular, the data collected was limited, making it impossible to generate customized improvement plans that took into account each user's emotional and behavioral data. In addition, users had no easy way to obtain information about stores offering the foods and services they needed. These issues led to a lack of consistency in users' health management, hindering effective health maintenance and improvement.

[1195] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1196] In this invention, the server includes: a means for analyzing health checkup data using OCR technology to evaluate health status and generate improvement proposals using a new data source including emotion data; a means for searching for the generated improvement proposals and recommended store information based on location information and transmitting them to the terminal; and a means for reanalyzing the user's behavioral data and emotion data and incorporating the feedback into the next improvement proposal. This makes it possible to comprehensively evaluate each user's health status and provide customized improvement proposals. Furthermore, by utilizing location information to provide information on recommended stores near the user, users can easily obtain the foods and services they need.

[1197] "Health checkup data" is information about the user's physiological health condition, including measurements such as blood pressure, blood sugar level, and cholesterol level.

[1198] "Dietary data" refers to information about the meals a user eats, including photos of the meals, the contents of the meals, and the calories and nutrients consumed.

[1199] "Step data" is information about the number of steps a user takes each day, collected from pedometer apps and devices.

[1200] "Emotional Data" refers to information about a user's emotional state, including stress levels and moods obtained through facial recognition, voice analysis, or manual input.

[1201] "OCR technology" stands for optical character recognition technology, which extracts characters from image data and converts them into text data.

[1202] "Improvement Suggestions" are specific advice or suggestions provided to improve the user's health based on data analyzed by the server.

[1203] "Location information" refers to the user's current physical location, obtained based on GPS and network information.

[1204] "Recommended store information" is information about stores that offer the food or services you need, provided based on your location information.

[1205] "Terminal" means a computing device used by a User, including a smartphone or tablet.

[1206] A "server" is a central computing system that analyzes data and stores and transmits results.

[1207] This invention relates to a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, this system analyzes health checkup data, dietary data, and step count data on a server and notifies the user of specific improvement suggestions, thereby enabling the user to effectively manage their health. In addition, an emotion engine is used to collect user emotion data and customize health management. Location information is also used to provide recommended store information. Specific embodiments of the system are described below.

[1208] Initial data registration

[1209] User: The user launches the smartphone app and enters the results of their health checkup. For example, they manually enter values ​​such as blood pressure, blood sugar, and cholesterol levels. They can also take a photo of their health checkup report and register the data.

[1210] Terminal: The terminal receives the input data and converts the image data into text data using OCR (Optical Character Recognition) technology. The converted data is then formatted into a standard format and sent to the server.

[1211] Server: The server analyzes the health checkup data sent from the device. The analyzed data is stored in the user's basic health database, and health assessment is performed to set baseline and target values. For example, the user's target blood pressure value is set to 125 / 80.

[1212] Data collection and transmission

[1213] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food they eat. They can enter information such as "I had salad and fish for lunch."

[1214] Device: The device receives meal photos and manual input data and sends them to the server. Step count data is also automatically collected from the pedometer app and periodically sent to the server.

[1215] Server: The server analyzes the received food photos and calculates calorie intake and major nutrients (protein, carbohydrates, fat, etc.). It also analyzes the received step count data and calculates energy expenditure. For example, it calculates the calories in a salad from the photo as 100 kcal, and records the energy expenditure of 10,000 steps a day as 400 kcal.

[1216] Adding an Emotion Engine

[1217] User: Users input their emotional data through apps equipped with emotion engines, or the data is collected automatically. For example, when a user feels stressed, emotion data is acquired through facial recognition or voice analysis.

[1218] Device: The device collects emotion data and sends it to the server.

[1219] Server: The server receives the emotional data and analyzes it by combining it with other health data. For example, if the stress level is high, it suggests low-calorie food choices and exercise to relax.

[1220] Evaluating health status and generating improvement proposals

[1221] Server: The server performs a comprehensive analysis based on the collected health checkup data, dietary data, step count data, and emotional data. For example, it may assess that "Your calorie intake is high and your step count is low, so you need to increase your exercise."

[1222] Server: Evaluates the user's current health status and generates specific recommendations for improvement, such as "Today's salt intake is high and your stress level is also high, so choose low-salt foods for your next meal and suggest relaxation exercises."

[1223] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants in the vicinity and obtains store information that offers the desired food or service. For example, "List low-salt foods available at nearby supermarkets."

[1224] Notification and Execution

[1225] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages. For example, "Today's lunch is high in calories, so please choose low-calorie foods for dinner."

[1226] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[1227] Gathering feedback and suggestions for next improvements

[1228] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1229] Prompt Sentence Examples

[1230] "How are you feeling today? Enter your health indicators."

[1231] "Upload a photo of your meal and calculate the calories."

[1232] This system allows users to achieve continuous and individually customized health management, effectively maintaining and improving their overall health.

[1233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1234] Step 1:

[1235] Enter and submit health checkup data

[1236] User: The user launches the smartphone app and manually enters the results of their health checkup. For example, they enter values ​​such as blood pressure, blood sugar, and cholesterol levels, such as "My blood pressure this morning was 130 / 85."

[1237] Terminal: The terminal receives the input data and can also convert photo data into text data using OCR technology instead of manual input. The converted data is formatted and sent to the server.

[1238] Input: Manual input or photo data.

[1239] Output: Formatted medical examination data.

[1240] Step 2:

[1241] Analysis and storage of health checkup data

[1242] Server: The server receives and analyzes the health checkup data sent from the device, evaluates the user's health status based on the data, and stores it in a basic health database.

[1243] Input: Formatted medical examination data.

[1244] Output: Health status assessment results and baseline health data.

[1245] Step 3:

[1246] Collection and transmission of dietary data

[1247] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1248] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it automatically collects step count data from the pedometer app and periodically sends it to the server.

[1249] Input: Meal photos, manual input data, step count data.

[1250] Output: Data sent to the server.

[1251] Step 4:

[1252] Data analysis

[1253] Server: The server analyzes the received food photos and calculates calorie intake and macronutrients (protein, carbohydrates, fat, etc.). Similarly, it analyzes step count data to calculate energy expenditure.

[1254] Input: Meal photos, manual input data, step count data.

[1255] Output: Calorie intake, macronutrients, energy expenditure.

[1256] Step 5:

[1257] Collecting and transmitting emotional data

[1258] User: Users input or automatically collect emotional data through apps equipped with emotion engines. Emotion data can be obtained through facial recognition and voice analysis.

[1259] Device: The device collects emotion data and sends it to the server.

[1260] Input: Emotion data (facial recognition, voice analysis, manual input).

[1261] Output: Data sent to the server.

[1262] Step 6:

[1263] Overall health assessment

[1264] Server: The server comprehensively analyzes the health check data, dietary data, step count data, and emotional data to evaluate the user's health status, generate specific improvement suggestions, and incorporate the feedback into the user's next actions.

[1265] Input: Health checkup data, dietary data, step count data, emotion data.

[1266] Output: Health status assessment results and improvement recommendations.

[1267] Step 7:

[1268] Use location information and find recommended stores

[1269] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby, and obtains information on stores that offer the desired food or service.

[1270] Input: Improvement suggestion, location information.

[1271] Output: Recommended store information.

[1272] Step 8:

[1273] Notification of improvement proposals and recommended store information

[1274] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[1275] Input: Improvement suggestions, recommended store information.

[1276] Output: Notification to the user.

[1277] Step 9:

[1278] Implementing improvement actions

[1279] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[1280] Input: Improvement suggestions, recommended store information.

[1281] Output: Implementation of improvement actions.

[1282] Step 10:

[1283] Gather feedback and generate recommendations for next time improvements

[1284] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluating the effectiveness of the user's improvement actions and incorporating feedback into the next improvement proposal.

[1285] Input: User behavioral data, emotional data.

[1286] Output: Suggested improvements for next time.

[1287] This processing step allows users to receive continuous, personalized health management, consistently maintaining and improving their health effectively.

[1288] (Application example 2)

[1289] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1290] Conventional health management systems could evaluate a user's health status using health checkup data and dietary data, but they were unable to provide specific improvement proposals that took into account the user's emotions or real-time store integration. Furthermore, they lacked a mechanism for providing the products and services users needed in a timely manner when they visited a physical store. This resulted in ineffective health management for users and insufficient support for their mental health.

[1291] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1292] In this invention, the server includes a means for inputting health checkup data, a means for taking photos of each meal and inputting the meal contents, a means for transmitting the health checkup data, meal data, and step count data to the server, a means for analyzing the data in the server, evaluating the health status, and generating improvement suggestions, a means for transmitting the generated improvement suggestions and recommended store information to the terminal, a means for notifying the user of the improvement suggestions displayed on the terminal, a means for analyzing facial expressions and voice to collect emotional data, a means for transmitting the emotional data to the server and integrating it with the health data for analysis, and a means for notifying the user of recommended product and service information in real time when the user visits a physical store. This allows the user to understand their health status in real time and receive individually customized improvement suggestions that take their emotional state into account. It also enables the user to obtain desired products and services in a timely manner when visiting a physical store.

[1293] "Health checkup data" refers to medical information such as blood pressure, blood sugar level, and cholesterol level obtained as a result of a health checkup that a user has undergone.

[1294] "Meal data" refers to photos taken by the user for each meal and information about the meal that the user inputs.

[1295] "Step count data" refers to data that records the number of steps a user takes each day.

[1296] The "server" is a data processing system that analyzes collected health checkup data, dietary data, step count data, and emotional data, evaluates health status, and generates improvement proposals.

[1297] "Emotional data" is data that indicates the user's stress level and emotional state obtained by analyzing their facial expressions and voice.

[1298] A "terminal" is an electronic device such as a smartphone or tablet held by the user, which displays improvement suggestions and recommended store information sent from the server.

[1299] "Recommended store information" is information that recommends the nearest suitable store or service based on the user's location information and health condition.

[1300] The "improvement suggestion" is a specific instruction or advice for improving the user's health condition, which is generated by the server by analyzing the health checkup data, dietary data, step count data, and emotional data.

[1301] "Behavioral data" is data that records the history of actions that users actually take and products they purchase.

[1302] "Facial expression and voice analysis" is a process in which a user's facial expressions are captured by a camera and their voice is recorded, and the emotion engine analyzes them to generate emotion data.

[1303] "Real-time notification" is a technology that instantly notifies users of necessary information and recommended products when they visit a physical store.

[1304] The system for implementing this invention collects and analyzes health checkup data, dietary data, step count data, and emotional data in real time. Ultimately, it provides users with health improvement suggestions and recommended store information based on the analysis results. The specific configuration and operation of this system are described below.

[1305] Hardware and Software Configuration

[1306] Hardware

[1307] Device: An electronic device, such as a smartphone or tablet, that users use to enter health data, take photos, and collect emotional data.

[1308] Server: A high-performance computer system installed in a data center that analyzes the received health and emotional data.

[1309] software

[1310] OCR technology: Converts the health check data entered by the user into text data.

[1311] Image analysis technology: Analyzes photos of food and facial expressions to obtain food content and emotional data. A specific example is an emotion recognition model using Keras.

[1312] Data analysis algorithm: Runs on the server and analyzes collected data to assess health status and generate improvement recommendations.

[1313] Data collection and analysis

[1314] Entering health data

[1315] Users can manually enter their health checkup results (such as blood pressure and blood sugar levels) or take a photo and upload it to the app. The device then converts this data into text using OCR technology and sends it to the server, which then creates a basic health database for the user.

[1316] Collection of dietary data

[1317] Users take a photo of their meal each time they eat and upload it to the app, which then sends the photo to a server that uses image analysis technology to calculate calorie and macronutrient intake.

[1318] Collecting step count data

[1319] The user's device uses a pedometer to measure the number of steps taken each day and periodically transmits this data to a server, which then calculates energy consumption.

[1320] Collecting Emotional Data

[1321] When a user feels stressed, emotional data is collected through facial recognition and voice analysis. The device then sends this data to a server, which then combines it with health data for analysis.

[1322] Providing recommended store information

[1323] Based on the analysis results, the server generates personalized health improvement recommendations for the user, such as "Today's salt intake is high, so choose low-salt foods at your next meal."

[1324] The server uses the user's location information to search for recommended stores nearby, and then notifies the user of stores that offer the food or service they need.

[1325] User Notification and Feedback

[1326] Improvement suggestions and recommended store information are sent from the server to the device and displayed as push notifications or in-app messages. Users can check the notification and take specific actions according to the instructions.

[1327] User behavior data and purchase history are also sent to the server and fed back into the analysis results, which are then reflected in future improvement proposals, enabling more advanced customization.

[1328] Prompt Sentence Examples

[1329] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[1330] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1331] Step 1:

[1332] Entering and submitting health data

[1333] Users manually enter their health check results (e.g., blood pressure, blood sugar, cholesterol levels, etc.) or take a photo and upload it to the device, which uses OCR technology to convert this data into text, format it, and send it to the server.

[1334] Input: Medical checkup results (manual entry or photo)

[1335] Output: Formatted text data (sent to server)

[1336] Step 2:

[1337] Collection and transmission of dietary data

[1338] Users take a photo of their meal each time they eat and upload it to the app. The device then sends the photo, along with any manually entered meal information, to a server. The server then uses image analysis technology to analyze the photo and calculate calorie and macronutrient intake.

[1339] Input: Meal photos (and manually entered meal information)

[1340] Output: Calorie intake and macronutrient data (analysis results on the server)

[1341] Step 3:

[1342] Collection and transmission of step count data

[1343] The device uses a built-in pedometer to collect the user's daily step count and periodically transmits this data to a server, which analyzes the step count data and calculates energy consumption.

[1344] Input: Step count data (automatically sent from the device)

[1345] Output: Energy consumption data (analysis results on the server)

[1346] Step 4:

[1347] Acquiring and sending emotion data

[1348] When a user feels stress or other emotions, they take a photo of their face or record their voice. The device sends this data to a server, which then analyzes the data using an emotion recognition model (e.g., a model using Keras) to evaluate the user's emotional state.

[1349] Input: Face photo (or voice data)

[1350] Output: Emotional state data (analysis results on the server)

[1351] Step 5:

[1352] Integrated analysis of health status and generation of improvement proposals

[1353] The server integrates and analyzes the collected health checkup data, dietary data, step count data, and emotional data. Based on the analysis results, it generates customized health improvement recommendations. For example, it creates specific advice such as, "Today's salt intake was high, so choose low-salt foods at your next meal."

[1354] Input: Health checkup data, dietary data, step count data, emotion data

[1355] Output: Health improvement suggestions (specific advice)

[1356] Step 6:

[1357] Generate and notify recommended store information

[1358] The server uses the user's location information to search for nearby recommended store information. The generated improvement suggestions and recommended store information are sent to the device, and the user receives them via push notification or in-app message.

[1359] Input: Location, Health Improvement Suggestion

[1360] Output: Recommended store information (notification to user)

[1361] Step 7:

[1362] Collecting feedback

[1363] The user's behavioral data and purchase history are sent to the server via the device, where the server reanalyzes the data and incorporates the feedback into future improvements, resulting in more accurate and personalized health advice.

[1364] Input: Behavioral data, purchase history

[1365] Output: Feedback data for improvement proposals (analysis results on the server)

[1366] Prompt Sentence Examples

[1367] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[1368] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1370] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1371] [Fourth embodiment]

[1372] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1373] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1375] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1379] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1380] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1381] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1382] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1383] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1384] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] This invention is a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. It also uses location information to provide recommended store information, allowing users to easily obtain the foods and services they need.

[1386] Program processing

[1387] 1. Initial data registration:

[1388] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[1389] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[1390] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[1391] 2. Data Collection and Transmission:

[1392] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1393] Device: The device receives the meal photos and sends them to the server for analysis. At the same time, it collects step count data and periodically sends it to the server.

[1394] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[1395] 3. Health status assessment and generation of improvement proposals:

[1396] Server: The server comprehensively analyzes the collected health checkup data, dietary data, and step count data. It evaluates the user's current health condition and generates specific improvement recommendations. For example, the server may generate an assessment result such as, "Today's salt intake is high, so it is recommended that you choose low-salt foods for your next meal."

[1397] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[1398] 4. Notice and Execution:

[1399] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[1400] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[1401] 5. Gather feedback and suggestions for next improvements:

[1402] Server: The server receives and analyzes the user's behavioral data again. It evaluates the effectiveness of the user's improvement actions and incorporates the feedback into the next improvement plan. This allows for consistent health management.

[1403] Through these processes, users can manage their own health status in real time and take appropriate actions to improve their health. The system continuously monitors the user's health status and provides individually customized health improvement suggestions to help users maintain their health in the long term.

[1404] The processing flow will be explained below.

[1405] Step 1:

[1406] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[1407] Step 2:

[1408] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[1409] Step 3:

[1410] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[1411] Step 4:

[1412] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1413] Step 5:

[1414] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[1415] Step 6:

[1416] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[1417] Step 7:

[1418] Server: The server comprehensively analyzes the health check data, dietary data, and step count data to evaluate the user's current health condition.

[1419] Step 8:

[1420] Server: The server generates specific improvement suggestions based on the evaluation results. For example, it generates advice such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[1421] Step 9:

[1422] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[1423] Step 10:

[1424] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[1425] Step 11:

[1426] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[1427] Step 12:

[1428] Server: The server receives and analyzes the user behavior data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1429] This allows users to consistently manage their health and receive personalized health improvement suggestions, helping them to more effectively maintain and improve their health.

[1430] Example 1

[1431] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] In modern society, individual users need to accurately and quickly grasp a large amount of information in order to effectively manage their own health status in real time and take appropriate measures to improve it. For this reason, there is a demand for systems that can efficiently collect and analyze health checkup data, daily diet data, and exercise data, and provide users with specific improvement suggestions. However, conventional systems have faced challenges, such as cumbersome data collection and insufficient analysis, making it difficult for users to effectively manage their health.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1434] In this invention, the server includes a means for analyzing the received health checkup data, dietary data, and step count data to evaluate the health condition, a means for generating improvement proposals using a generative AI model, and a means for acquiring recommended store information based on the user's location information, thereby enabling the user to understand their own health condition in real time and obtain specific and effective improvement proposals.

[1435] A "user" is an individual who uses the system to manage their own health status and receive improvement suggestions.

[1436] "Health checkup data" refers to data relating to the user's health condition, such as blood pressure, blood sugar level, and cholesterol level.

[1437] "Dietary data" refers to data that a user records by inputting or taking photos of their daily meals.

[1438] "Step count data" refers to data on the number of steps a user takes in their daily life.

[1439] A "server" is a device that analyzes the received data and provides the user with appropriate improvement suggestions.

[1440] "Analysis" is the process of evaluating health status based on collected data and extracting necessary information.

[1441] A "generative AI model" is a model that uses machine learning technology to generate improvement suggestions based on a user's health condition.

[1442] "Improvement Suggestions" are specific instructions or suggestions for improving the user's health.

[1443] "Location information" refers to the geographic coordinate data of the user's current location.

[1444] "Recommended store information" is information about stores that offer suggested meals and services based on the user's location information.

[1445] "Notifications" are messages or alerts sent from the server to the device, and are a means of conveying information to the user.

[1446] "Feedback" is information that reports to the system the evaluation and results of improvement proposals implemented by the user.

[1447] The present invention provides a system that enables a user to grasp his or her own health condition in real time and assists the user in taking appropriate measures to improve the condition. A specific embodiment of this system will be described below.

[1448] 1. System Configuration

[1449] The system includes a server where users input health checkup data and dietary data, which analyzes this data, a terminal that displays the analysis results, and a generative AI model.

[1450] Hardware and Software

[1451] Device: A mobile device such as a smartphone or tablet, optionally equipped with software that uses OCR technology (e.g., Google Cloud Vision API).

[1452] Server: A high-performance computer for analyzing data and running generative AI models, using database software (e.g., MySQL) and machine learning frameworks (e.g., TensorFlow).

[1453] 2. Initial data registration

[1454] The user launches the app and registers the health check results by entering them or taking a photo. For example, the user enters a blood pressure of 120 / 80 mmHg or a blood sugar level of 90 mg / dL. The device converts the photo into text data using OCR technology as needed and sends the formatted data to the server. The server analyzes the received data and stores it in a basic health database.

[1455] 3. Data Collection and Transmission

[1456] Each time a user eats a meal, they take a photo of it and upload it to the app. For example, they can register an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner. The device receives the meal photos and sends them to a server. The device also collects the user's step count data daily and periodically sends it to the server. The server uses image analysis technology to analyze the meal photos and calculates the key nutrients. It also analyzes the step count data to calculate calories burned.

[1457] 4. Health status assessment and generation of improvement proposals

[1458] The server comprehensively analyzes the accumulated health checkup data, dietary data, and step count data, and uses a generative AI model to evaluate the user's health condition. For example, if the dietary data indicates excessive calorie intake and insufficient exercise, it will generate specific improvement suggestions such as "increase exercise" or "choose low-calorie foods."

[1459] Location information and recommended store information

[1460] The server uses the user's location information to search for recommended stores and restaurants nearby via internet services, obtains information about stores that offer the desired food or service, and sends this information along with improvement suggestions to the user's device.

[1461] 5. Notification and Execution

[1462] The device notifies the user of the improvement suggestions and recommended store information received from the server. For example, a push notification might say, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket." The user can check the notification and take specific action. For example, they might go to a nearby supermarket to purchase low-salt foods or do the suggested exercise.

[1463] 6. Gather feedback and suggest improvements for next time

[1464] After the user completes the action based on the improvement suggestion, they provide feedback on the results. For example, the user might enter feedback such as "I ran for 30 minutes." The server then analyzes this data again and generates the next improvement suggestion based on the feedback. This feedback loop allows the user's health to continuously improve.

[1465] Prompt Sentence Examples

[1466] "The user's health check results show that their blood pressure is 120 / 80 mmHg, their blood sugar is 90 mg / dL, and their cholesterol is 180 mg / dL. Please generate a future health improvement plan based on their dietary and step count data."

[1467] "The user has consumed 2300 kcal and taken 8000 steps today. Please suggest appropriate improvements based on this information."

[1468] As described above, this system comprehensively manages users' health data and provides specific health improvement suggestions based on the analysis results, enabling users to effectively manage their health.

[1469] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1470] Step 1: Registering initial data

[1471] User: Launches the app and enters their health checkup results, such as blood pressure 120 / 80 mmHg, blood sugar 90 mg / dL, and cholesterol 180 mg / dL, or takes and uploads a photo of their health checkup results.

[1472] Input: Health checkup data such as blood pressure, blood sugar, and cholesterol levels

[1473] Terminal: The received health check data is converted into text data using OCR technology (e.g., Google Cloud Vision API), formatted, and sent to the server in the format "user ID_recording date and time."

[1474] Output: Formatted medical examination data

[1475] Server: Receives formatted health checkup data, evaluates it using an analysis engine, and stores it in a database (e.g., MySQL) as basic health data. It also sets reference and target values.

[1476] Step 2: Collect and send food and exercise data

[1477] User: Takes a photo of each meal and uploads it to the app, for example, an omelet for breakfast, a sandwich for lunch, and grilled fish for dinner.

[1478] Input: Food photos and text data

[1479] Device: Receives photos and sends them to the server. It also collects the user's daily step count data from an activity tracker (e.g., a fitness band) and periodically sends it to the server.

[1480] Output: Meal photos, step count data

[1481] Server: Analyzes food photos using an image analysis engine (e.g., a model using TensorFlow) and calculates macronutrients (calories, protein, carbohydrates, fat, etc.). Also analyzes step count data and calculates calories burned. Integrates this data with basic health data.

[1482] Step 3: Assessing health status and generating improvement proposals

[1483] Server: Comprehensively analyzes health checkup data, dietary data, and step count data to assess the user's health condition. Using a generative AI model, it generates improvement recommendations based on the user's current situation. For example, "Today's calorie intake has exceeded the target, so we recommend choosing low-calorie foods for your next meal."

[1484] Input: Health checkup data, dietary data, step count data

[1485] Output: Health status assessment, improvement suggestions

[1486] Specific actions: To assess health status, compare with past data and baseline values. To generate improvement recommendations, the system aims to provide personalized and specific advice using generative AI models.

[1487] Step 4: Get recommended store information based on location information

[1488] Server: Based on the improvement plan, use the user's location information to search for recommended nearby stores through an internet service (e.g., map API). For example, search for "nearby supermarkets that sell low-salt foods."

[1489] Input: Improvement proposal, user location

[1490] Output: Recommended store information

[1491] Specific operation: The server receives the location data, searches for stores using the API, converts the results into text format, and provides it to the user.

[1492] Step 5: Send and execute notifications

[1493] On the device: The system sends push notifications to the user with the improvement suggestions and recommended store information received from the server. For example, it sends a notification saying, "Today's meal was high in salt, so we recommend choosing low-salt foods for your next meal. You can purchase these at a nearby supermarket."

[1494] Input: Improvement proposals, recommended store information

[1495] Output: User notification

[1496] User: Checks the notification and takes specific action, such as going to a nearby supermarket to buy low-sodium foods.

[1497] Specific behavior: After receiving the notification, the user performs the action as instructed and provides feedback to the app about the results.

[1498] Step 6: Gather feedback and suggest improvements for next time

[1499] User: After completing an action, provide feedback to the app about what you did. For example, you might enter "I ran for 30 minutes."

[1500] Input: Feedback data (results of the action)

[1501] Server: Re-analyzes the received feedback data, evaluates its effectiveness, and incorporates the feedback into the next improvement plan, thereby supporting continuous improvement of health status.

[1502] Output: Next improvement plan

[1503] How it works: Feedback data is collected and evaluated using an analytics engine. Based on the results, the next improvement plan is generated using a generative AI model.

[1504] The above are the specific processing steps and operations of the program for this system.

[1505] (Application example 1)

[1506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1507] In modern society, individual health management is becoming increasingly important, but it is difficult to effectively maintain health through self-management alone. In particular, there is a lack of methods that comprehensively analyze health checkup results, daily dietary habits, and exercise levels, and provide specific, individually customized health improvement proposals. There are also insufficient methods for users to easily obtain the information necessary to implement improvement proposals (for example, information on recommended stores and services). For this reason, a system that allows users to effectively manage their health on a daily basis is needed.

[1508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1509] In this invention, the server includes means for inputting health checkup data, means for taking images of each meal and inputting the meal contents, means for transmitting the health checkup data, meal data, and step count data to the server, means for analyzing the data in the server, evaluating the health condition, and generating improvement suggestions, means for transmitting the generated improvement suggestions and recommended store information to the terminal, means for notifying the user of the improvement suggestions displayed on the terminal, means for accumulating user behavior data and reflecting the data in generating the next improvement suggestion, means for using location information to search for information on the nearest store and provide recommended store information related to meals and exercise, means for analyzing meal contents using image analysis technology and extracting major nutrients, means for evaluating the user's health condition using a generative AI model and generating specific improvement suggestions, and means for notifying the user of the generated improvement suggestions and recommended store information to a smartphone or head-mounted display, thereby enabling the user to manage their own health condition in real time and take appropriate action to improve their health.

[1510] "Health checkup data" refers to test result data that indicates the user's health condition, and includes information such as blood pressure, blood sugar level, and cholesterol level.

[1511] "Meal images" are photographs of meals consumed by a user, providing a visual record of the meal contents.

[1512] "Dietary content" refers to information including the type and amount of food and major nutrients (protein, carbohydrates, fat, etc.).

[1513] "Step count data" is data that records the number of steps a user takes in a day, and indicates the amount of exercise.

[1514] The "server" is a central computer device in the system that receives, analyzes, and evaluates health checkup data, dietary data, and step count data.

[1515] The "means for assessing health status" is a function in which the server analyzes the data received and diagnoses the user's current health status.

[1516] The "means for generating improvement proposals" is a function in which the server creates a specific health improvement plan suited to the user based on the results of the health status evaluation.

[1517] "Recommended store information" refers to information about the nearest store that provides the food or service you need based on your location information and health improvement suggestions.

[1518] A "terminal" refers to a device operated by a user, such as a smartphone or head-mounted display, that receives and displays information from a server.

[1519] "Means of notification" refers to the function that the device uses to inform users of improvement suggestions and recommended store information, and includes push notifications and in-app messages.

[1520] "Behavioral data" refers to data about the actions actually taken by the user, including diet, exercise, purchasing history, etc.

[1521] "Image analysis technology" is a technology that extracts information from image data such as photographs, and is used in health management systems to analyze the contents of meals.

[1522] "Macronutrients" are the important components of food, such as proteins, carbohydrates, and fats, that are necessary for maintaining a healthy human body.

[1523] A "generative AI model" is a model that uses artificial intelligence to analyze a user's health data and automatically generate appropriate health improvement suggestions.

[1524] The present invention provides a system for helping users understand their own health condition in real time and take appropriate measures to improve it. Specific embodiments are described below.

[1525] System configuration:

[1526] 1. Health checkup data input method:

[1527] Users can manually enter their health checkup results using a device (smartphone or head-mounted display) or take a photo and register the data using OCR technology. This data is sent to a server and stored as the user's basic health database.

[1528] 2. Meal content input method:

[1529] Users take photos of each meal and upload them to the app. The device receives the photos and analyzes the meal contents using image analysis technology (such as Google Cloud Vision API). The analysis results are sent to a server as information on macronutrients such as protein, carbohydrates, and fat.

[1530] 3. Step count data collection method:

[1531] The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to collect step count data and periodically transmits it to a server.

[1532] 4. Health status assessment and improvement proposal generation method:

[1533] The server comprehensively analyzes the received health checkup data, dietary data, and step count data. During this process, a generative AI model (using TensorFlow and PyTorch) is used to evaluate the user's health status and generate specific improvement suggestions.

[1534] 5. Recommended store information methods:

[1535] Based on the generated improvement proposal, the server uses the user's location information to search for and obtain recommended store information for the nearest store using Google Maps API, etc. This store information will be related to food and exercise that will help improve the user's health.

[1536] 6. Means of notification:

[1537] The server sends improvement suggestions and recommended store information to the device (smartphone or head-mounted display), which then notifies the user via push notification (Firebase Cloud Messaging) or in-app message.

[1538] Examples:

[1539] 1. Registering health checkup results:

[1540] The user takes an image of the blood pressure result with their smartphone, and OCR technology extracts text data from the image and sends the blood pressure information to the server.

[1541] 2. Enter your meal details:

[1542] Users take a photo of their lunch and upload it to the app, where, using image analysis technology, the server analyzes the macronutrients contained in the meal and adds them to a dietary database.

[1543] 3. Step count data collection:

[1544] The smartphone's built-in pedometer measures the number of steps the user takes each day and periodically sends the data to a server.

[1545] 4. Health status assessment and generation of improvement proposals:

[1546] The server analyzes the collected data using a generative AI model to assess the user's health condition, and generates specific recommendations for improvement, such as "Today's salt intake is high, so we recommend choosing low-salt foods for your next meal."

[1547] 5. Providing recommended store information:

[1548] Based on the user's location information, the server searches for the nearest supermarket and provides information on stores where low-salt foods can be purchased.

[1549] Example prompt sentence:

[1550] User health data: Blood pressure: 120 / 80 mmHg, Blood glucose: 5.6 mmol / L

[1551] Meal details: Salad, chicken, rice

[1552] Step count data: 7000 steps

[1553] Location information: 35.6895° N, 139.6917° E

[1554] Use this data to assess the user's health status and generate specific recommendations for improvement.

[1555] These components and examples empower users to manage their health in real time and provide the information they need to implement appropriate improvements.

[1556] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1557] Step 1:

[1558] The user inputs the health checkup data. Using a device (smartphone or head-mounted display), the user can manually enter the health checkup results directly or take a photo of the results and upload it to the app. The device uses OCR technology to extract and format text data from the image data, which then sends the initial health data to the server.

[1559] Input: Medical examination result images or manual input data

[1560] Output: Formatted medical examination data

[1561] Step 2:

[1562] The user takes a photo of each meal and enters the meal contents. The user takes a photo of the meal and uploads it to the app. The device uses image analysis technology (Google Cloud Vision API) to analyze the meal contents and extract macronutrients (protein, carbohydrates, fat, etc.). The analysis results are sent to the server.

[1563] Input: Food image

[1564] Output: Macronutrient information of meal contents

[1565] Step 3:

[1566] The device collects step count data and sends it to a server. The user's smartphone or head-mounted display uses a built-in sensor (pedometer) to measure the number of steps taken each day. The step count data is periodically sent to the server, allowing the amount of daily exercise to be recorded.

[1567] Input: Pedometer data

[1568] Output: Periodically collected step count data

[1569] Step 4:

[1570] The server analyzes health checkup data, dietary data, and step count data to assess the user's health status and generate improvement proposals. The server comprehensively processes and analyzes the received data and evaluates the user's health status using a generative AI model (using TensorFlow or PyTorch). Based on the analysis results, it generates specific improvement proposals.

[1571] Input: Health checkup data, dietary data, step count data

[1572] Output: User's health status assessment results and specific improvement suggestions

[1573] Step 5:

[1574] Based on the generated improvement plan, the server uses location information to obtain recommended store information for the nearest store. The server uses the user's location information to search for and obtain store information that provides the desired food or service using Google Maps API, etc.

[1575] Input: improvement proposal, user location information

[1576] Output: Recommended store information

[1577] Step 6:

[1578] The server sends improvement suggestions and recommended store information to the device, which then notifies the user. The server sends the generated improvement suggestions and recommended store information to the device, which then notifies the user via push notification or in-app message.

[1579] Input: Improvement proposals, recommended store information

[1580] Output: A message to inform the user

[1581] Step 7:

[1582] User behavior data is accumulated and reflected in the generation of the next improvement proposal. The user takes action based on the notification they receive (e.g., shopping at a recommended store). This behavior data is sent back to the server and taken into account in the next data analysis.

[1583] Input: User behavior data

[1584] Output: Data to be reflected in the next improvement plan generation

[1585] This allows the system to manage the user's health status in real time and continue to provide appropriate improvement suggestions.

[1586] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1587] This invention is a system that helps users understand their health status in real time and take appropriate measures to improve it. In particular, the system analyzes health checkup data, dietary data, and step count data on a server and notifies users of specific improvement suggestions, enabling them to effectively manage their health. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system customizes the improvement suggestions, providing more personalized health management. It also uses location information to provide recommended store information.

[1588] Program processing

[1589] 1. Initial data registration:

[1590] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[1591] Terminal: The terminal receives this data and converts it to text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[1592] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[1593] 2. Data Collection and Transmission:

[1594] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1595] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[1596] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[1597] 3. Adding an Emotion Engine:

[1598] User: The user inputs or automatically collects emotional data through an app equipped with an emotion engine. For example, when a user feels stressed, emotional data can be obtained through facial recognition or voice analysis.

[1599] Device: The device collects emotion data and sends it to the server.

[1600] Server: The server receives the emotion data, integrates it with other health data, and performs analysis.

[1601] 4. Health status assessment and generation of improvement proposals:

[1602] Server: The server comprehensively analyzes the collected health checkup data, dietary data, step count data, and emotional data. It evaluates the user's current health condition and generates specific recommendations for improvement. For example, it generates advice such as, "Today's salt intake was high and your stress level is also high, so choose low-salt foods for your next meal and suggest some exercises to help you relax."

[1603] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby. Information on stores offering the desired food and services is obtained.

[1604] 5. Notice and Execution:

[1605] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[1606] User: The user reviews the notification and takes specific action as instructed, such as buying low-sodium foods at a nearby supermarket or doing the suggested exercise.

[1607] 6. Gather feedback and suggestions for next improvements:

[1608] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1609] This allows users to manage their health consistently and receive personalized health improvement suggestions that take their emotional state into account, helping them maintain and improve their health more effectively. The addition of the emotion engine will enable holistic health management that also takes into account the user's mental health.

[1610] The processing flow will be explained below.

[1611] Step 1:

[1612] User: The user launches the app and manually enters their health checkup results or takes a photo to register their data, such as blood pressure, blood sugar, and cholesterol levels.

[1613] Step 2:

[1614] Terminal: The terminal receives the health check results and converts them into text data using OCR technology if necessary. The converted data is formatted and sent to the server.

[1615] Step 3:

[1616] Server: The server analyzes the received health checkup data and stores it as a basic health database for the user. It also performs health assessments and sets baseline and target values.

[1617] Step 4:

[1618] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1619] Step 5:

[1620] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it collects step count data and periodically sends it to the server.

[1621] Step 6:

[1622] Server: The server analyzes the received photos of the food and calculates the calorie intake and major nutrients (protein, carbohydrates, fat, etc.). Step count data is also analyzed at the same time to calculate energy expenditure.

[1623] Step 7:

[1624] User: The user uses an app equipped with an emotion engine, and emotion data is input or automatically collected. For example, emotion data is obtained using facial recognition or voice analysis functions when the user feels stressed.

[1625] Step 8:

[1626] Device: The device collects emotion data and sends it to the server.

[1627] Step 9:

[1628] Server: The server receives the emotion data, integrates it with health checkup data, dietary data, and step count data, and performs a comprehensive analysis.

[1629] Step 10:

[1630] Server: The server uses the integrated data to assess the user's current health status and generate specific recommendations for improvement. For example, if the user's stress level is high and their salt intake is high, the server will generate advice such as "Choose low-salt foods for your next meal and do 30 minutes of yoga to relax."

[1631] Step 11:

[1632] Server: Based on the generated improvement suggestions, the server uses the user's location information to search for recommended stores and restaurants nearby, for example, to obtain information on purchasing low-salt foods at a nearby supermarket.

[1633] Step 12:

[1634] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed via push notifications and in-app messages.

[1635] Step 13:

[1636] User: The user reviews the notification and takes the suggested action, such as buying low-sodium foods at a nearby supermarket or practicing yoga as suggested by the app.

[1637] Step 14:

[1638] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1639] This allows users to consistently manage their health in a way that is individually customized, and helps maintain and improve overall health, taking into account their emotional state.

[1640] Example 2

[1641] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1642] With conventional health management systems, it was difficult for users to accurately understand their own health status in real time and implement appropriate improvement measures. In particular, the data collected was limited, making it impossible to generate customized improvement plans that took into account each user's emotional and behavioral data. In addition, users had no easy way to obtain information about stores offering the foods and services they needed. These issues led to a lack of consistency in users' health management, hindering effective health maintenance and improvement.

[1643] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1644] In this invention, the server includes: a means for analyzing health checkup data using OCR technology to evaluate health status and generate improvement proposals using a new data source including emotion data; a means for searching for the generated improvement proposals and recommended store information based on location information and transmitting them to the terminal; and a means for reanalyzing the user's behavioral data and emotion data and incorporating the feedback into the next improvement proposal. This makes it possible to comprehensively evaluate each user's health status and provide customized improvement proposals. Furthermore, by utilizing location information to provide information on recommended stores near the user, users can easily obtain the foods and services they need.

[1645] "Health checkup data" is information about the user's physiological health condition, including measurements such as blood pressure, blood sugar level, and cholesterol level.

[1646] "Dietary data" refers to information about the meals a user eats, including photos of the meals, the contents of the meals, and the calories and nutrients consumed.

[1647] "Step data" is information about the number of steps a user takes each day, collected from pedometer apps and devices.

[1648] "Emotional Data" refers to information about a user's emotional state, including stress levels and moods obtained through facial recognition, voice analysis, or manual input.

[1649] "OCR technology" stands for optical character recognition technology, which extracts characters from image data and converts them into text data.

[1650] "Improvement Suggestions" are specific advice or suggestions provided to improve the user's health based on data analyzed by the server.

[1651] "Location information" refers to the user's current physical location, obtained based on GPS and network information.

[1652] "Recommended store information" is information about stores that offer the food or services you need, provided based on your location information.

[1653] "Terminal" means a computing device used by a User, including a smartphone or tablet.

[1654] A "server" is a central computing system that analyzes data and stores and transmits results.

[1655] This invention relates to a system that helps users understand their own health status in real time and take appropriate measures to improve it. In particular, this system analyzes health checkup data, dietary data, and step count data on a server and notifies the user of specific improvement suggestions, thereby enabling the user to effectively manage their health. In addition, an emotion engine is used to collect user emotion data and customize health management. Location information is also used to provide recommended store information. Specific embodiments of the system are described below.

[1656] Initial data registration

[1657] User: The user launches the smartphone app and enters the results of their health checkup. For example, they manually enter values ​​such as blood pressure, blood sugar, and cholesterol levels. They can also take a photo of their health checkup report and register the data.

[1658] Terminal: The terminal receives the input data and converts the image data into text data using OCR (Optical Character Recognition) technology. The converted data is then formatted into a standard format and sent to the server.

[1659] Server: The server analyzes the health checkup data sent from the device. The analyzed data is stored in the user's basic health database, and health assessment is performed to set baseline and target values. For example, the user's target blood pressure value is set to 125 / 80.

[1660] Data collection and transmission

[1661] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food they eat. They can enter information such as "I had salad and fish for lunch."

[1662] Device: The device receives meal photos and manual input data and sends them to the server. Step count data is also automatically collected from the pedometer app and periodically sent to the server.

[1663] Server: The server analyzes the received food photos and calculates calorie intake and major nutrients (protein, carbohydrates, fat, etc.). It also analyzes the received step count data and calculates energy expenditure. For example, it calculates the calories in a salad from the photo as 100 kcal, and records the energy expenditure of 10,000 steps a day as 400 kcal.

[1664] Adding an Emotion Engine

[1665] User: Users input their emotional data through apps equipped with emotion engines, or the data is collected automatically. For example, when a user feels stressed, emotion data is acquired through facial recognition or voice analysis.

[1666] Device: The device collects emotion data and sends it to the server.

[1667] Server: The server receives the emotional data and analyzes it by combining it with other health data. For example, if the stress level is high, it suggests low-calorie food choices and exercise to relax.

[1668] Evaluating health status and generating improvement proposals

[1669] Server: The server performs a comprehensive analysis based on the collected health checkup data, dietary data, step count data, and emotional data. For example, it may assess that "Your calorie intake is high and your step count is low, so you need to increase your exercise."

[1670] Server: Evaluates the user's current health status and generates specific recommendations for improvement, such as "Today's salt intake is high and your stress level is also high, so choose low-salt foods for your next meal and suggest relaxation exercises."

[1671] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants in the vicinity and obtains store information that offers the desired food or service. For example, "List low-salt foods available at nearby supermarkets."

[1672] Notification and Execution

[1673] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages. For example, "Today's lunch is high in calories, so please choose low-calorie foods for dinner."

[1674] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[1675] Gathering feedback and suggestions for next improvements

[1676] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluates the effectiveness of the user's improvement actions, and incorporates the feedback into the next improvement plan.

[1677] Prompt Sentence Examples

[1678] "How are you feeling today? Enter your health indicators."

[1679] "Upload a photo of your meal and calculate the calories."

[1680] This system allows users to achieve continuous and individually customized health management, effectively maintaining and improving their overall health.

[1681] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1682] Step 1:

[1683] Enter and submit health checkup data

[1684] User: The user launches the smartphone app and manually enters the results of their health checkup. For example, they enter values ​​such as blood pressure, blood sugar, and cholesterol levels, such as "My blood pressure this morning was 130 / 85."

[1685] Terminal: The terminal receives the input data and can also convert photo data into text data using OCR technology instead of manual input. The converted data is formatted and sent to the server.

[1686] Input: Manual input or photo data.

[1687] Output: Formatted medical examination data.

[1688] Step 2:

[1689] Analysis and storage of health checkup data

[1690] Server: The server receives and analyzes the health checkup data sent from the device, evaluates the user's health status based on the data, and stores it in a basic health database.

[1691] Input: Formatted medical examination data.

[1692] Output: Health status assessment results and baseline health data.

[1693] Step 3:

[1694] Collection and transmission of dietary data

[1695] User: Each time a user eats, they take a photo of their meal and upload it to the app. They can also manually enter the type and amount of food.

[1696] Device: The device receives meal photos and manual input data and sends them to the server. At the same time, it automatically collects step count data from the pedometer app and periodically sends it to the server.

[1697] Input: Meal photos, manual input data, step count data.

[1698] Output: Data sent to the server.

[1699] Step 4:

[1700] Data analysis

[1701] Server: The server analyzes the received food photos and calculates calorie intake and macronutrients (protein, carbohydrates, fat, etc.). Similarly, it analyzes step count data to calculate energy expenditure.

[1702] Input: Meal photos, manual input data, step count data.

[1703] Output: Calorie intake, macronutrients, energy expenditure.

[1704] Step 5:

[1705] Collecting and transmitting emotional data

[1706] User: Users input or automatically collect emotional data through apps equipped with emotion engines. Emotion data can be obtained through facial recognition and voice analysis.

[1707] Device: The device collects emotion data and sends it to the server.

[1708] Input: Emotion data (facial recognition, voice analysis, manual input).

[1709] Output: Data sent to the server.

[1710] Step 6:

[1711] Overall health assessment

[1712] Server: The server comprehensively analyzes the health check data, dietary data, step count data, and emotional data to evaluate the user's health status, generate specific improvement suggestions, and incorporate the feedback into the user's next actions.

[1713] Input: Health checkup data, dietary data, step count data, emotion data.

[1714] Output: Health status assessment results and improvement recommendations.

[1715] Step 7:

[1716] Use location information and find recommended stores

[1717] Server: Based on the generated improvement proposal, the server uses the user's location information to search for recommended stores and restaurants nearby, and obtains information on stores that offer the desired food or service.

[1718] Input: Improvement suggestion, location information.

[1719] Output: Recommended store information.

[1720] Step 8:

[1721] Notification of improvement proposals and recommended store information

[1722] Device: The device notifies the user of the improvement suggestions and recommended store information received from the server. Notifications are displayed as push notifications or in-app messages.

[1723] Input: Improvement suggestions, recommended store information.

[1724] Output: Notification to the user.

[1725] Step 9:

[1726] Implementing improvement actions

[1727] User: The user checks the notification and takes specific action, such as buying low-sodium foods at a nearby supermarket or performing the suggested exercise.

[1728] Input: Improvement suggestions, recommended store information.

[1729] Output: Implementation of improvement actions.

[1730] Step 10:

[1731] Gather feedback and generate recommendations for next time improvements

[1732] Server: The server receives and analyzes the user's behavioral and emotional data again, evaluating the effectiveness of the user's improvement actions and incorporating feedback into the next improvement proposal.

[1733] Input: User behavioral data, emotional data.

[1734] Output: Suggested improvements for next time.

[1735] This processing step allows users to receive continuous, personalized health management, consistently maintaining and improving their health effectively.

[1736] (Application example 2)

[1737] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1738] Conventional health management systems could evaluate a user's health status using health checkup data and dietary data, but they were unable to provide specific improvement proposals that took into account the user's emotions or real-time store integration. Furthermore, they lacked a mechanism for providing the products and services users needed in a timely manner when they visited a physical store. This resulted in ineffective health management for users and insufficient support for their mental health.

[1739] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1740] In this invention, the server includes a means for inputting health checkup data, a means for taking photos of each meal and inputting the meal contents, a means for transmitting the health checkup data, meal data, and step count data to the server, a means for analyzing the data in the server, evaluating the health status, and generating improvement suggestions, a means for transmitting the generated improvement suggestions and recommended store information to the terminal, a means for notifying the user of the improvement suggestions displayed on the terminal, a means for analyzing facial expressions and voice to collect emotional data, a means for transmitting the emotional data to the server and integrating it with the health data for analysis, and a means for notifying the user of recommended product and service information in real time when the user visits a physical store. This allows the user to understand their health status in real time and receive individually customized improvement suggestions that take their emotional state into account. It also enables the user to obtain desired products and services in a timely manner when visiting a physical store.

[1741] "Health checkup data" refers to medical information such as blood pressure, blood sugar level, and cholesterol level obtained as a result of a health checkup that a user has undergone.

[1742] "Meal data" refers to photos taken by the user for each meal and information about the meal that the user inputs.

[1743] "Step count data" refers to data that records the number of steps a user takes each day.

[1744] The "server" is a data processing system that analyzes collected health checkup data, dietary data, step count data, and emotional data, evaluates health status, and generates improvement proposals.

[1745] "Emotional data" is data that indicates the user's stress level and emotional state obtained by analyzing their facial expressions and voice.

[1746] A "terminal" is an electronic device such as a smartphone or tablet held by the user, which displays improvement suggestions and recommended store information sent from the server.

[1747] "Recommended store information" is information that recommends the nearest suitable store or service based on the user's location information and health condition.

[1748] The "improvement suggestion" is a specific instruction or advice for improving the user's health condition, which is generated by the server by analyzing the health checkup data, dietary data, step count data, and emotional data.

[1749] "Behavioral data" is data that records the history of actions that users actually take and products they purchase.

[1750] "Facial expression and voice analysis" is a process in which a user's facial expressions are captured by a camera and their voice is recorded, and the emotion engine analyzes them to generate emotion data.

[1751] "Real-time notification" is a technology that instantly notifies users of necessary information and recommended products when they visit a physical store.

[1752] The system for implementing this invention collects and analyzes health checkup data, dietary data, step count data, and emotional data in real time. Ultimately, it provides users with health improvement suggestions and recommended store information based on the analysis results. The specific configuration and operation of this system are described below.

[1753] Hardware and Software Configuration

[1754] Hardware

[1755] Device: An electronic device, such as a smartphone or tablet, that users use to enter health data, take photos, and collect emotional data.

[1756] Server: A high-performance computer system installed in a data center that analyzes the received health and emotional data.

[1757] software

[1758] OCR technology: Converts the health check data entered by the user into text data.

[1759] Image analysis technology: Analyzes photos of food and facial expressions to obtain food content and emotional data. A specific example is an emotion recognition model using Keras.

[1760] Data analysis algorithm: Runs on the server and analyzes collected data to assess health status and generate improvement recommendations.

[1761] Data collection and analysis

[1762] Entering health data

[1763] Users can manually enter their health checkup results (such as blood pressure and blood sugar levels) or take a photo and upload it to the app. The device then converts this data into text using OCR technology and sends it to the server, which then creates a basic health database for the user.

[1764] Collection of dietary data

[1765] Users take a photo of their meal each time they eat and upload it to the app, which then sends the photo to a server that uses image analysis technology to calculate calorie and macronutrient intake.

[1766] Collecting step count data

[1767] The user's device uses a pedometer to measure the number of steps taken each day and periodically transmits this data to a server, which then calculates energy consumption.

[1768] Collecting Emotional Data

[1769] When a user feels stressed, emotional data is collected through facial recognition and voice analysis. The device then sends this data to a server, which then combines it with health data for analysis.

[1770] Providing recommended store information

[1771] Based on the analysis results, the server generates personalized health improvement recommendations for the user, such as "Today's salt intake is high, so choose low-salt foods at your next meal."

[1772] The server uses the user's location information to search for recommended stores nearby, and then notifies the user of stores that offer the food or service they need.

[1773] User Notification and Feedback

[1774] Improvement suggestions and recommended store information are sent from the server to the device and displayed as push notifications or in-app messages. Users can check the notification and take specific actions according to the instructions.

[1775] User behavior data and purchase history are also sent to the server and fed back into the analysis results, which are then reflected in future improvement proposals, enabling more advanced customization.

[1776] Prompt Sentence Examples

[1777] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[1778] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1779] Step 1:

[1780] Entering and submitting health data

[1781] Users manually enter their health check results (e.g., blood pressure, blood sugar, cholesterol levels, etc.) or take a photo and upload it to the device, which uses OCR technology to convert this data into text, format it, and send it to the server.

[1782] Input: Medical checkup results (manual entry or photo)

[1783] Output: Formatted text data (sent to server)

[1784] Step 2:

[1785] Collection and transmission of dietary data

[1786] Users take a photo of their meal each time they eat and upload it to the app. The device then sends the photo, along with any manually entered meal information, to a server. The server then uses image analysis technology to analyze the photo and calculate calorie and macronutrient intake.

[1787] Input: Meal photos (and manually entered meal information)

[1788] Output: Calorie intake and macronutrient data (analysis results on the server)

[1789] Step 3:

[1790] Collection and transmission of step count data

[1791] The device uses a built-in pedometer to collect the user's daily step count and periodically transmits this data to a server, which analyzes the step count data and calculates energy consumption.

[1792] Input: Step count data (automatically sent from the device)

[1793] Output: Energy consumption data (analysis results on the server)

[1794] Step 4:

[1795] Acquiring and sending emotion data

[1796] When a user feels stress or other emotions, they take a photo of their face or record their voice. The device sends this data to a server, which then analyzes the data using an emotion recognition model (e.g., a model using Keras) to evaluate the user's emotional state.

[1797] Input: Face photo (or voice data)

[1798] Output: Emotional state data (analysis results on the server)

[1799] Step 5:

[1800] Integrated analysis of health status and generation of improvement proposals

[1801] The server integrates and analyzes the collected health checkup data, dietary data, step count data, and emotional data. Based on the analysis results, it generates customized health improvement recommendations. For example, it creates specific advice such as, "Today's salt intake was high, so choose low-salt foods at your next meal."

[1802] Input: Health checkup data, dietary data, step count data, emotion data

[1803] Output: Health improvement suggestions (specific advice)

[1804] Step 6:

[1805] Generate and notify recommended store information

[1806] The server uses the user's location information to search for nearby recommended store information. The generated improvement suggestions and recommended store information are sent to the device, and the user receives them via push notification or in-app message.

[1807] Input: Location, Health Improvement Suggestion

[1808] Output: Recommended store information (notification to user)

[1809] Step 7:

[1810] Collecting feedback

[1811] The user's behavioral data and purchase history are sent to the server via the device, where the server reanalyzes the data and incorporates the feedback into future improvements, resulting in more accurate and personalized health advice.

[1812] Input: Behavioral data, purchase history

[1813] Output: Feedback data for improvement proposals (analysis results on the server)

[1814] Prompt Sentence Examples

[1815] Prompt: "Implement an application that uploads a user's health data and generates recommendation information based on emotion analysis in a physical store. Include a function that sends blood pressure, blood sugar, and cholesterol levels to a server as health data, and analyzes stress levels from facial images using an emotion engine."

[1816] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1817] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1818] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1819] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1820] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1821] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1822] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1823] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1824] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1825] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1826] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1827] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1828] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stor...

Claims

1. a means for inputting health examination data; A means to take a photo of each meal and input the meal contents, means for transmitting health checkup data, dietary data, and step count data to a server; A means for analyzing the data on a server, assessing the health status, and generating improvement proposals; means for transmitting the generated improvement plan and recommended store information to the terminal; A means of notifying the user of the proposed improvements displayed on the device; A system including:

2. 2. The system according to claim 1, further comprising means for searching for information on the nearest store using the location information.

3. 10. The system of claim 1, further comprising means for reanalyzing user behavior data and incorporating feedback into subsequent improvement proposals.

Images