System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input life habit, health data and genetic information; means for the server to analyze the data and evaluate the health condition of the user; means for generating a personalized health care plan based on the analysis result; means for the user to continuously input and transmit daily life habit data; means for the server to update the health plan based on new data; and means for the server to generate timely feedback and advice and notify the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional health management systems have difficulty providing customized preventive measures and health plans based on each user's individual lifestyle and health status. It is also difficult to continuously monitor a user's health status and provide appropriate feedback in real time. Furthermore, few systems support a variety of data formats, including text, audio, and images. These challenges make it difficult to achieve effective health management, and users' health risks cannot be properly managed. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means: A means for a user to input lifestyle habits, health data, and genetic information, and a means for transmitting the data to a server. The server further has a means for analyzing the data and evaluating the user's health condition. The system includes a means for generating an individualized health management plan based on the analysis results, and provides a means for transmitting the generated health management plan to a user terminal. The system includes a means for a user to continuously input and transmit daily lifestyle habit data, and a means for the server to update the health plan based on new data. The server also provides a means for generating timely feedback and advice and notifying the user terminal. This enables health management that meets the individual needs of users and makes it possible to efficiently manage health risks.

[0006] "User" refers to an individual who uses the system to input lifestyle, health data, and genetic information.

[0007] "Lifestyle habits" is information about the user's daily actions and habits, and includes data about diet, exercise, sleep, and the like.

[0008] "Health data" refers to information relating to the user's health condition, and includes numerical data such as weight, blood pressure, and blood sugar level, as well as information relating to symptoms.

[0009] "Genetic information" refers to data relating to the user's genetic factors, including genetic analysis results.

[0010] "Data transmission means" refers to a function that transmits data entered by a user using a terminal to a server.

[0011] A "server" refers to a computer system that receives input data, analyzes it, and returns the results.

[0012] "Data analysis means" refers to the processes and algorithms used by the server to assess the user's health status based on the data received.

[0013] A "health management plan" refers to a plan for maintaining or improving health that is individually customized based on the results of data analysis.

[0014] "Health plan transmission means" refers to the function of transmitting the health management plan generated by the server to the user terminal.

[0015] "Continuous input means" refers to a function that allows a user to continuously input daily data into a terminal and transmit that data to a server.

[0016] "Data update means" refers to the process or algorithm by which the server updates an existing health management plan with new data.

[0017] "Feedback generation means" refers to the function of the server to create timely feedback and advice based on the latest data.

[0018] "Notification means" refers to a function that notifies the user of the generated feedback or advice via the terminal. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[0041] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[0042] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0043] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[0044] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[0045] Users continue to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide the most appropriate plan. The server also generates timely feedback and advice and notifies the user. For example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!" can be sent.

[0046] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." The user can take this advice and incorporate it into their diet the next day, achieving effective health management.

[0047] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0048] The processing flow will be explained below.

[0049] Step 1:

[0050] Users input their lifestyle, health data, and genetic information. For example, they use a smartphone app to enter the day's meal details in text and record their exercise history by voice.

[0051] Step 2:

[0052] The device collects the input data and sends it to the server via the communication module. Specifically, it sends text data, voice data, and image data to the server via the endpoint API.

[0053] Step 3:

[0054] The server preprocesses the received data, specifically converting the voice data into text and analyzing the image data to identify the type and quantity of ingredients.

[0055] Step 4:

[0056] The server analyzes the preprocessed data, and uses machine learning algorithms and deep learning models to evaluate the user's health status. Specifically, the current health status is quantified by comparing it with past data.

[0057] Step 5:

[0058] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "recommended 30 minutes of walking every day."

[0059] Step 6:

[0060] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[0061] Step 7:

[0062] The user continuously inputs daily lifestyle data, for example, recording daily meals and exercise details on the device.

[0063] Step 8:

[0064] The device continuously sends newly entered data to the server, which then uses the new data to evaluate the effectiveness of the health management plan and update it as needed.

[0065] Step 9:

[0066] The server generates timely feedback and advice, such as a rating message like "You've achieved 80% of your exercise goal this week" based on the user's latest data.

[0067] Step 10:

[0068] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] Conventional health management systems have limited the format of data that users can enter, making them unable to accommodate a variety of data formats. They also lack the ability to generate personalized health management plans, preventing them from providing specific advice or feedback tailored to the user's health condition. Furthermore, analysis of voice and image data is often performed manually, making efficient health management difficult.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes a voice recognition unit for converting voice data into text, an image analysis unit for analyzing image data to identify the types and amounts of ingredients, and a unit for evaluating the user's health condition using a machine learning or deep learning model based on the analysis results, thereby enabling efficient and personalized health management that can handle a variety of data formats.

[0074] "User" refers to an individual who uses the system to input lifestyle, health data, and genetic information and receive a health management plan.

[0075] A "terminal" is an electronic device, such as a smartphone or PC, that allows users to input and send data.

[0076] The "server" refers to a central processing unit that receives and analyzes data sent from the terminal, generates a health management plan, and provides feedback to the user.

[0077] "Lifestyle data" refers to information about a user's lifestyle, such as diet, exercise records, and sleep patterns.

[0078] "Health data" refers to numerical data that indicates the user's health condition, such as weight, blood pressure, and heart rate.

[0079] "Genetic information" refers to information about a user's genetic makeup, data used to more precisely assess their health status.

[0080] "Speech recognition means" refers to technology that converts voice data into text data.

[0081] "Image analysis means" refers to a technique for analyzing image data and identifying its contents.

[0082] A "machine learning algorithm" is an algorithm that self-learns based on data and makes predictions and evaluations.

[0083] A "deep learning model" is a technology that uses multi-layered neural networks to analyze and predict data.

[0084] A "health management plan" is a personalized plan for diet, exercise, and lifestyle improvement that is generated based on the user's lifestyle and health data.

[0085] "Feedback" refers to real-time advice and evaluations provided to users based on the analysis results.

[0086] "API" stands for Application Program Interface, and is an interface for exchanging data and functions between different software programs.

[0087] "Text data" refers to data expressed in letters and numbers.

[0088] "Image data" refers to data that includes visual information such as photographs and graphs.

[0089] This invention is a health management system that operates in cooperation with three main elements: a user, a terminal, and a server. Specifically, a user inputs lifestyle habits, health data, and genetic information using a terminal, and transmits this data to the server. The server then analyzes the data, evaluates the user's health status, and generates a personalized health management plan.

[0090] First, users use devices such as smartphones or PCs to input lifestyle data, health data, and genetic information. The input data can be in various formats, including text, audio, and images. For example, users can take photos of their meals with their smartphones and record their exercise by voice. This input data is collected by the devices and sent to a server via the Internet.

[0091] The device receives data from the user and sends it to the server using an API (Application Program Interface). For example, the device sends data through a RESTful API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0092] The server plays a central role in data analysis. First, the server uses speech recognition technology to analyze the voice data. A common speech recognition technology is Google's speech recognition API. The voice data is converted into text and stored for further analysis.

[0093] The server then uses image recognition technology to analyze the image data, for example, by using the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. This analysis result is also stored in a database.

[0094] Based on the collected data, the server then applies machine learning algorithms and deep learning models (such as TensorFlow and Scikit-learn) to assess the user's health status. Once the analysis is complete, the server generates a personalized health management plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle improvements.

[0095] The generated health management plan is then sent back to the device, where it can be accessed by the user. The user continuously records their daily lifestyle and health data, which is then sent to the server via the device. The server updates the health management plan based on the new data, generates timely feedback and advice (for example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!"), and notifies the user.

[0096] Examples and prompts

[0097] Specific examples

[0098] For example, when a user records their dinner on a smartphone, they take a photo of the meal and enter the details in text. They also record the time they spent jogging by voice. This data is sent by the device to a server, which uses image analysis technology to identify ingredients and voice recognition technology to convert the exercise record into text. Specific advice is then generated and notified to the user, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." The user can then incorporate this advice into their diet the next day, achieving effective health management.

[0099] Example prompts

[0100] "Build an AI model that can analyze photos of meals, identify the ingredients they contain and their amounts, and provide specific recommendations to users."

[0101] "Please explain the process for converting today's exercise log from audio data to text and generating a health management plan based on that data."

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] Users use their smartphones or PCs (devices) to input lifestyle, health, and genetic information. For example, they might enter text like "Today's dinner is salad and grilled chicken" into an application's input form, take a photo of the meal, and record a voice memo saying they went jogging for 30 minutes. This input data is then saved on the device.

[0105] Input: text data, image data, audio data

[0106] Output: Lifestyle data stored on the device

[0107] Step 2:

[0108] The terminal sends the data entered by the user to the server via the API. Specifically, the terminal converts the data received into JSON format and sends a POST request to the server using the RESTful API.

[0109] Input: Lifestyle data stored on the device

[0110] Output: Data sent to the server

[0111] Step 3:

[0112] The server receives the transmitted data and performs preprocessing for data analysis, such as checking the format of the audio data and converting it to an appropriate format, and resizing the image data.

[0113] Input: Data sent to the server

[0114] Output: Data ready for analysis

[0115] Step 4:

[0116] The server uses speech recognition technology to analyze the voice data, for example, by using the Google Speech-to-Text API to convert the voice data into text, and the speech recognition results are stored in a database.

[0117] Input: Audio data ready for analysis

[0118] Output: Audio data converted to text

[0119] Step 5:

[0120] The server uses image analysis technology to analyze the image data. For example, it uses the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. The results of this analysis are stored in a database.

[0121] Input: Image data ready for analysis

[0122] Output: Analyzed image data (type and amount of ingredients)

[0123] Step 6:

[0124] The server analyzes the collected data using machine learning algorithms and deep learning models to assess the user's health. For example, TensorFlow is used to input dietary and exercise data into the model to obtain a health assessment result, which is then stored in a database.

[0125] Input: Text-converted voice data, analyzed image data

[0126] Output: Health status assessment results

[0127] Step 7:

[0128] The server generates a customized health management plan for each user based on the analysis results, for example, by using Scikit-learn to generate diet and exercise recommendations, and stores the plan in a database.

[0129] Input: Health status assessment results

[0130] Output: A customized health plan

[0131] Step 8:

[0132] The server sends the generated health management plan to the device, for example by sending a real-time push notification to inform the user that a new plan is available.

[0133] Enter: Customized Health Care Plan

[0134] Output: Health management plan sent to the device

[0135] Step 9:

[0136] Users continuously input their daily lifestyle data and send it to the server via their device, which then receives the new data and updates the health management plan.

[0137] Input: Continuously input lifestyle data

[0138] Output: Updated health plan

[0139] Step 10:

[0140] The server generates timely feedback and advice and sends it to the user's device, such as a message saying, "You've achieved 80% of your exercise goal this week. Keep up the great work!"

[0141] Enter: Updated Health Care Plan

[0142] Output: Feedback and advice to the user

[0143] (Application example 1)

[0144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] In modern life, many people face the challenge of managing their health in their busy daily lives. It is also difficult to create specific meal plans tailored to individual health conditions, and there are limited ways to easily obtain such meals. This has resulted in a lack of effective approaches for individuals to maintain their long-term health. In particular, despite the importance of maintaining health through the continuous intake of a nutritionally balanced diet, the lack of individualized support remains a challenge.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0147] In this invention, the server includes: means for a user to input lifestyle habits, health data, and genetic information; means for transmitting the data to the server; means for the server to analyze the data and evaluate the user's health status; means for generating an individualized health management plan based on the analysis results; means for transmitting the generated health management plan to a user terminal; means for the user to continuously input and transmit daily lifestyle habit data; means for the server to update the health plan based on new data; means for the server to generate timely feedback and advice and notify the user terminal; and means for generating an appropriate meal plan based on the generated health management plan, presenting meals based on the plan as a delivery menu, and making them available for ordering. This allows users to easily obtain a meal plan optimized for their health status and obtain those meals through a delivery service.

[0148] "User" refers to an individual who uses the health management system to input lifestyle, health data, and genetic information to improve their health.

[0149] "Lifestyle data" refers to information about the actions and habits of users in their daily lives (e.g., diet, exercise history, sleep patterns, etc.).

[0150] "Health data" refers to numerical information and recorded data that measure a user's health status, such as weight, height, blood pressure, and heart rate.

[0151] "Genetic information" refers to information about a user's genetic background (e.g., family medical history, genetic test results, etc.).

[0152] "Server" refers to a computer system that receives data sent by a user and analyzes and processes it.

[0153] "Analysis" refers to the act of processing the data received by the server to examine it in detail and evaluate patterns and health conditions.

[0154] "Individualized health management plan" refers to a plan or proposal for maintaining health that is customized to suit the individual health condition and goals of each user based on the user's lifestyle data, health data, and genetic information.

[0155] "Feedback" refers to information such as evaluation, advice, and guidance that the server provides to users based on the analysis results.

[0156] A "meal plan" refers to a meal suggestion that takes into consideration nutritional balance and ingredient selection based on the user's health condition.

[0157] "Delivery Menu" means the specific meal options that a User may order based on the generated meal plan.

[0158] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[0159] User Roles

[0160] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[0161] Device Role

[0162] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0163] Server Roles

[0164] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[0165] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[0166] Meal plans and delivery features

[0167] Furthermore, in this invention, the server generates an appropriate meal plan based on the generated health management plan. This meal plan is customized based on nutritional balance and specific health goals. A delivery menu based on the meal plan is then presented, allowing the user to select and order from the menu. This delivery menu is generated using a pre-prepared database and an online meal delivery service API.

[0168] Specific examples

[0169] Consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server. The server analyzes the data and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." Based on this advice, the server then suggests multiple delivery options to the user, from which the user can order the most suitable meal.

[0170] Prompt Sentence Examples

[0171] Below is a concrete example of a prompt sentence.

[0172] Lifestyle: Rice and miso soup for breakfast every day. 30-minute bike ride to work.

[0173] Health stats: I weigh 70kg, am 175cm tall, and go to the gym three times a week.

[0174] Genetic information: High blood pressure runs in the family.

[0175] In this way, users can easily obtain nutritionally balanced meals while achieving effective health management.

[0176] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0177] Step 1:

[0178] Users input their lifestyle, health data, and genetic information into the device in the form of text, voice, or images.

[0179] Specific operations: The user opens the smartphone app, enters lifestyle habits (e.g., diet and exercise records) using text or voice, and takes photos of the meals.

[0180] Input: lifestyle habits, health data, genetic information (text, audio, image)

[0181] Output: Input data stored on the device

[0182] Step 2:

[0183] The device sends the collected data to the server, where the data format is appropriately converted via API.

[0184] Specific operation: The smartphone app calls the API and sends the collected text data, audio data, and image data to the server.

[0185] Input: Input data stored on the device

[0186] Output: The input data sent to the server

[0187] Step 3:

[0188] The server receives the transmitted data and uses voice recognition technology to convert the voice data into text, and image analysis technology to analyze the image data and identify the meal contents.

[0189] What it does: The server converts the audio data into text using a speech recognition system (e.g., Google Speech Recognition API) and then applies an image analysis algorithm (e.g., a food image classification model using TensorFlow).

[0190] Input: Audio data, image data

[0191] Output: Text data (speech recognition results), image analysis results (meal content identification)

[0192] Step 4:

[0193] The server analyzes all data, including the collected text data, and uses machine learning algorithms to assess the user's health status.

[0194] Specific operation: The server applies a machine learning model (e.g., a health prediction model using Scikit-learn) to evaluate health status based on input data.

[0195] Input: Text data, voice recognition results, image analysis results

[0196] Output: User's health status assessment

[0197] Step 5:

[0198] The server generates a personalized health care plan based on the assessment results.

[0199] Specific operation: Based on the evaluation results, the server generates a health management plan using statistical analysis and algorithms (e.g., Python code running in Jupyter Notebook).

[0200] Input: User's health assessment

[0201] Output: personalized health care plan

[0202] Step 6:

[0203] The server transmits the generated health management plan to the user terminal.

[0204] Specific operation: The server sends the generated plan to the user's smartphone app via API.

[0205] Enter: personalized health care plans.

[0206] Output: Health management plan sent to the user's terminal

[0207] Step 7:

[0208] Users continuously input lifestyle habit data based on their health management plan and send it back to the server via their terminal.

[0209] Specific operation: The user re-enters their daily lifestyle and health data into the app, and the device sends it to the server.

[0210] Input: New lifestyle and health data

[0211] Output: New data sent to the server

[0212] Step 8:

[0213] The server updates the health management plan based on new data, generates timely feedback and advice, and notifies the user's device.

[0214] What it does: The server analyzes the new data, uses a generative AI model (e.g., GPT-3) to generate text feedback or advice, and sends a notification to the user's smartphone.

[0215] Input: New lifestyle and health data

[0216] Output: Updated health plan, feedback, and advice

[0217] Step 9:

[0218] The server generates an appropriate meal plan based on the generated health management plan, and presents a delivery menu based on the plan, allowing ordering.

[0219] Specific operation: The server calls the delivery service API (e.g., Uber Eats API) based on the meal plan and presents the user with the optimal delivery menu.

[0220] Enter: personalized health care plans.

[0221] Output: Delivery menu suggestions, available options

[0222] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0223] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0224] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[0225] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0226] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text, and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. It then uses an emotion engine to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[0227] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, a customized health management plan is generated that takes the user's emotions into account. For example, the plan may include exercise plans to reduce stress and suggestions for relaxing meals.

[0228] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message might be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[0229] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[0230] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0231] The processing flow will be explained below.

[0232] Step 1:

[0233] Users input their lifestyle, health, genetic, and emotional data using a smartphone app, taking photos of their meals, recording their exercise records by voice, and recording their mood for the day in text.

[0234] Step 2:

[0235] The device collects input data and sends it to the server via a communication module. Specifically, it sends text data, voice data, image data, and text-based emotion data to the server via an API.

[0236] Step 3:

[0237] The server preprocesses the received data. Specifically, it converts voice data into text using voice recognition technology, analyzes image data using image analysis technology to identify the type and quantity of ingredients, and analyzes the user's emotional data using an emotion engine.

[0238] Step 4:

[0239] The server analyzes the preprocessed data and emotion data. It uses machine learning algorithms and deep learning models to evaluate the user's health status. The analysis results are compared with the user's past data to evaluate the progress of their health status.

[0240] Step 5:

[0241] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "We recommend walking 30 minutes every day" or "Your stress level is high today, so try some relaxation exercises."

[0242] Step 6:

[0243] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[0244] Step 7:

[0245] The user continuously inputs daily lifestyle data and emotional data, for example, recording daily meals, exercise, and moods on the device.

[0246] Step 8:

[0247] The device continuously transmits newly entered data and emotional data to the server, which then evaluates the effectiveness of the health management plan based on the new data and updates the plan as needed.

[0248] Step 9:

[0249] The server generates timely feedback and advice based on the user's latest data, such as "You've achieved 80% of your exercise goal this week" or "If you're feeling stressed, take a deep breath to relax."

[0250] Step 10:

[0251] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[0252] Example 2

[0253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] Conventional health management systems rely on the input of lifestyle, health, and genetic information, but it is difficult to effectively utilize this data individually, making it difficult to provide users with personalized health management plans. In particular, they do not take emotional data into consideration when customizing the system, making it impossible to accurately evaluate the user's overall health status. As a result, users are unable to obtain a health management plan that is suited to them, making it difficult to implement practical health management in their daily lives.

[0255] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the health condition of a user using voice recognition technology and image analysis technology, a means for generating an individualized health management plan using a machine learning algorithm and a deep learning model, and a means for analyzing the user's emotions using an emotion engine. This enables the user to obtain an optimized health management plan based on comprehensive data including lifestyle habits, health data, genetic information, and emotion data, and to perform highly effective health management in real life.

[0256] "User" means a person who uses the System to input lifestyle, health data, and genetic information and receive a personalized health care plan.

[0257] "Lifestyle data" refers to information related to the user's daily life, and is data input in the form of text, audio, images, etc.

[0258] "Health data" refers to information about the user's physical health condition, such as weight, blood pressure, and blood sugar level.

[0259] "Genetic Information" means health-related information based on a user's genes, such as data about genetic risk or genotype.

[0260] A "terminal" is a device that allows users to input lifestyle habits, health data, and genetic information, and includes devices such as smartphones and personal computers.

[0261] The "server" is a device that analyzes the data, evaluates the user's health condition, generates an individualized health management plan, and transmits it to the user's terminal.

[0262] "Speech recognition technology" is a technology that analyzes voice data to understand human speech and convert it into text data.

[0263] "Image analysis technology" refers to the technology of analyzing image data to understand its contents and extract necessary information.

[0264] A "machine learning algorithm" is a method for learning from past data and predicting new data based on the analysis results.

[0265] A "deep learning model" is a technology that uses a multi-layer neural network to learn complex patterns in data and improve the accuracy of analysis.

[0266] An "emotion engine" is a technology that analyzes changes in voice tone and facial expressions to determine the user's emotional state.

[0267] A "health management plan" is a personalized guideline for maintaining and improving health that is generated based on the user's lifestyle, health data, genetic information, and emotional data.

[0268] "Feedback" refers to information generated by the server that provides analysis results and advice to users.

[0269] "Timely feedback" means providing users with the necessary information and advice at the right time.

[0270] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0271] First, users use a device such as a smartphone or personal computer to input their lifestyle, health data, and genetic information. Specifically, they can take photos of their meals, record their exercise records by voice, and enter additional information in text if necessary. This input data is collected by the device and sent to the server.

[0272] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to a server. For example, it sends each piece of data to the server via a REST API and uploads image data to cloud storage, allowing the server to prepare the data for analysis.

[0273] The server acts as a central analyzer of the received data. It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the voice data into text, and image analysis technology (e.g., Google Cloud Vision API) to identify the type and quantity of ingredients from the photo. It also uses an emotion engine to analyze the user's tone of voice and facial expressions. This analysis determines whether the user is stressed or relaxed.

[0274] The server then uses machine learning algorithms (e.g., Scikit-Learn) or deep learning models (e.g., TensorFlow) to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a personalized health management plan. For example, if the stress level is determined to be high, it creates a plan that includes stress-reducing exercise plans and relaxing meal suggestions.

[0275] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message could be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[0276] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day by voice. The device sends this data to a server, which analyzes it. Specific advice is generated, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice, such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[0277] Examples of prompts to input to a generative AI model might include:

[0278] 1. "Identify the types and amounts of ingredients in the dinner picture."

[0279] 2. "Please convert this audio data to text."

[0280] 3. "Judge the stress level from this voice tone."

[0281] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0282] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0283] Step 1:

[0284] Users input their lifestyle habits, health data, and genetic information. Specifically, they open a smartphone app, take photos of their meals, record their exercise by voice, and enter additional information in text format if necessary. The input data includes photos of meals (image data), audio recordings of exercise duration (audio data), and additional information in text format. This data is collected on the device.

[0285] Input: Meal photos, audio recording of exercise time, text data for additional information

[0286] Output: Lifestyle data, health data, and genetic information collected on the device

[0287] Step 2:

[0288] The device processes the collected data and sends it to the server. The device sends image data, audio data, and text data to the server via the REST API. Specifically, it makes an HTTP POST request and uploads the image data to cloud storage.

[0289] Input: Lifestyle data, health data, and genetic information collected on the device

[0290] Output: Data sent to the server

[0291] Step 3:

[0292] The server receives the data sent from the device and prepares it for analysis. It stores the received image files in temporary storage and performs initial checks to ensure data integrity, such as validating the file format and size.

[0293] Input: Data sent to the server

[0294] Output: Analysis-ready data

[0295] Step 4:

[0296] The server converts the voice data into text using speech recognition technology. Specifically, it calls the Google Cloud Speech-to-Text API to convert the voice data into text.

[0297] Input: Audio data

[0298] Output: Audio data converted to text

[0299] Step 5:

[0300] The server uses image analysis technology to identify the types and quantities of ingredients from the photos. Specifically, it uses the Google Cloud Vision API to analyze the photos of the meal entered and identify the types and quantities of ingredients.

[0301] Input: Food image data

[0302] Output: Data identifying the type and amount of ingredients

[0303] Step 6:

[0304] The server uses an emotion engine to analyze voice tones and facial expressions to determine the user's emotions. Specifically, it analyzes voice and image data to detect stress levels and relaxation states.

[0305] Input: Audio data, image data

[0306] Output: User's emotional state data

[0307] Step 7:

[0308] The server uses machine learning algorithms and deep learning models based on the analyzed data to generate personalized health management plans. Specifically, it uses Scikit-Learn and TensorFlow to evaluate the user's health status and create customized health management plans.

[0309] Input: Analyzed data (type and amount of ingredients, speech text, emotional state data)

[0310] Output: A customized health plan

[0311] Step 8:

[0312] The server sends the generated health management plan to the device in JSON format so that the user can view it through a smartphone app.

[0313] Enter: Customized Health Care Plan

[0314] Output: Health management plan sent to the device

[0315] Step 9:

[0316] The user can then check the health management plan and feedback received through the device and put it into action. Specifically, by opening the app, the user can check advice such as "You're not eating enough vegetables today" and reflect it in their meal plan for the next day.

[0317] Input: Health management plan sent to the device

[0318] Output: Confirmation of implemented health management plan and feedback

[0319] Step 10:

[0320] Users continue to record their daily lifestyle and health data and send it to the server via their device. The server then continuously updates the health management plan based on the new data. Specifically, users enter their daily exercise and diet records and send them to the server.

[0321] Input: New lifestyle and health data

[0322] Output: Updated health plan and feedback

[0323] (Application example 2)

[0324] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] Conventional health management systems provide health management plans based on users' lifestyle and health data. However, these systems were unable to take into account the user's emotional state, making it difficult to provide personalized advice based on each user's individual emotions. As a result, they were unable to provide appropriate health management plans or dietary suggestions that responded to stress and emotional changes. Furthermore, they were unable to adequately check whether the proposed plans were being implemented properly and provide timely feedback.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0327] In this invention, the server includes means for analyzing the user's emotions from input data using an emotion engine, means for evaluating the user's health condition based on the analyzed emotion data and generating customized menu suggestions, and means for generating timely feedback and advice and notifying the user terminal. This enables personalized health management plans and meal suggestions that take the user's emotional state into consideration, and makes it possible to provide optimal support in response to changes in the user's stress and emotions.

[0328] "Lifestyle data" refers to data related to the user's daily activities and habits, such as diet, exercise, and sleep.

[0329] "Health data" refers to data relating to a user's physical health, such as weight, blood pressure, and heart rate.

[0330] "Genetic information" refers to data relating to a user's genetic characteristics and physical constitution.

[0331] The "server" is a central computer system that analyzes data collected from users and generates health management plans and feedback.

[0332] The "emotion engine" is a software module that analyzes the user's emotions from input data and customizes health management plans based on the results.

[0333] "Analyzing" means processing collected data using algorithms or specific techniques to understand and evaluate its content.

[0334] An "individualized health care plan" is a health care plan or proposal that is customized to meet a user's specific needs and conditions.

[0335] "Menu suggestion" refers to suggesting the optimal meal menu taking into consideration the user's health and emotional state.

[0336] "Feedback" refers to advice and evaluations generated by the server based on data provided by the user and their daily behavior.

[0337] "Timely feedback" means feedback provided at an appropriate time depending on the situation or condition the user is facing.

[0338] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0339] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[0340] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device transmits this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0341] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text (software used: SpeechRecognition), and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals (software used: OpenCV, PIL, pytesseract). It then uses an emotion engine (software used: emotion_recognition) to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[0342] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a customized health management plan and menu suggestions that take the user's emotions into account. For example, a health management plan that includes exercise plans to reduce stress and meal suggestions that promote relaxation may be generated.

[0343] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message such as "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax" can be sent.

[0344] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and reflect it in their diet and daily activities the next day, achieving more effective health management.

[0345] Example prompts based on generative AI models:

[0346] "I feel like I've been lacking in vegetables this week. I've detected stress in my voice tone, so please suggest a relaxing menu."

[0347] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0348] Step 1:

[0349] The device collects user input data. Users use their smartphones or PCs to input lifestyle habits, health data, and genetic information in text, voice, and image formats. This input data is stored on the device. Input includes photos of meals, exercise records, and daily emotional states.

[0350] Step 2:

[0351] The device sends the collected data to the server. The device uploads the data to the server via API. The input is the user's lifestyle data, health data, and genetic information, and the output is a message confirming successful data transmission to the server.

[0352] Step 3:

[0353] The server analyzes the received data. It uses voice recognition technology (software used: SpeechRecognition) to convert the voice data into text, and image analysis technology (software used: OpenCV, PIL, pytesseract) to identify the types and quantities of ingredients from the meal photos. It also uses an emotion engine (software used: emotion_recognition) to analyze emotions from the voice. The inputs are lifestyle data, health data, genetic information, and emotion data, and the output is analyzed text data, ingredient data, and emotion data.

[0354] Step 4:

[0355] The server generates a personalized health management plan based on the analysis results. The server uses machine learning algorithms and deep learning models to integrate the analyzed text data, food ingredient data, and emotion data to evaluate the user's health status and create a customized plan. The input is the analyzed data, and the output is a personalized health management plan.

[0356] Step 5:

[0357] The server sends the generated health management plan to the terminal. The server sends the generated plan to the terminal so that the user can access it. The input is the health management plan, and the output is a success message for sending the plan to the user terminal.

[0358] Step 6:

[0359] The device provides timely feedback and advice to the user. Based on the health management plan sent from the server, the device notifies the user of the feedback and advice. For example, a message such as "You have achieved 80% of your exercise goal this week" is sent. The input is the data received from the server, and the output is a notification message to the user.

[0360] Step 7:

[0361] The user continuously inputs lifestyle data and sends it to the server via the terminal. The user continues to record their daily lifestyle and emotional state and sends it to the server via the terminal. The input is new lifestyle data, and the output is a message that the data has been successfully sent to the server.

[0362] Step 8:

[0363] The server continuously updates the health management plan based on new data and notifies the user. As soon as the server receives new data, it reevaluates the health management plan and generates a new, optimized plan. The input is new lifestyle data, and the output is the updated health management plan.

[0364] The processing flow is as described above, and the specific operations at each step effectively implement personalized health management that takes into account the user's emotional state.

[0365] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0366] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0367] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0368] [Second embodiment]

[0369] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0370] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0371] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0372] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0373] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0374] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0375] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0376] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0377] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0378] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0379] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0380] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0381] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[0382] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[0383] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0384] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[0385] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[0386] Users continue to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide the most appropriate plan. The server also generates timely feedback and advice and notifies the user. For example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!" can be sent.

[0387] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." The user can take this advice and incorporate it into their diet the next day, achieving effective health management.

[0388] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0389] The processing flow will be explained below.

[0390] Step 1:

[0391] Users input their lifestyle, health data, and genetic information. For example, they use a smartphone app to enter the day's meal details in text and record their exercise history by voice.

[0392] Step 2:

[0393] The device collects the input data and sends it to the server via the communication module. Specifically, it sends text data, voice data, and image data to the server via the endpoint API.

[0394] Step 3:

[0395] The server preprocesses the received data, specifically converting the voice data into text and analyzing the image data to identify the type and quantity of ingredients.

[0396] Step 4:

[0397] The server analyzes the preprocessed data, and uses machine learning algorithms and deep learning models to evaluate the user's health status. Specifically, the current health status is quantified by comparing it with past data.

[0398] Step 5:

[0399] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "recommended 30 minutes of walking every day."

[0400] Step 6:

[0401] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[0402] Step 7:

[0403] The user continuously inputs daily lifestyle data, for example, recording daily meals and exercise details on the device.

[0404] Step 8:

[0405] The device continuously sends newly entered data to the server, which then uses the new data to evaluate the effectiveness of the health management plan and update it as needed.

[0406] Step 9:

[0407] The server generates timely feedback and advice, such as a rating message like "You've achieved 80% of your exercise goal this week" based on the user's latest data.

[0408] Step 10:

[0409] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[0410] Example 1

[0411] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0412] Conventional health management systems have limited the format of data that users can enter, making them unable to accommodate a variety of data formats. They also lack the ability to generate personalized health management plans, preventing them from providing specific advice or feedback tailored to the user's health condition. Furthermore, analysis of voice and image data is often performed manually, making efficient health management difficult.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0414] In this invention, the server includes a voice recognition unit for converting voice data into text, an image analysis unit for analyzing image data to identify the types and amounts of ingredients, and a unit for evaluating the user's health condition using a machine learning or deep learning model based on the analysis results, thereby enabling efficient and personalized health management that can handle a variety of data formats.

[0415] "User" refers to an individual who uses the system to input lifestyle, health data, and genetic information and receive a health management plan.

[0416] A "terminal" is an electronic device, such as a smartphone or PC, that allows users to input and send data.

[0417] The "server" refers to a central processing unit that receives and analyzes data sent from the terminal, generates a health management plan, and provides feedback to the user.

[0418] "Lifestyle data" refers to information about a user's lifestyle, such as diet, exercise records, and sleep patterns.

[0419] "Health data" refers to numerical data that indicates the user's health condition, such as weight, blood pressure, and heart rate.

[0420] "Genetic information" refers to information about a user's genetic makeup, data used to more precisely assess their health status.

[0421] "Speech recognition means" refers to technology that converts voice data into text data.

[0422] "Image analysis means" refers to a technique for analyzing image data and identifying its contents.

[0423] A "machine learning algorithm" is an algorithm that self-learns based on data and makes predictions and evaluations.

[0424] A "deep learning model" is a technology that uses multi-layered neural networks to analyze and predict data.

[0425] A "health management plan" is a personalized plan for diet, exercise, and lifestyle improvement that is generated based on the user's lifestyle and health data.

[0426] "Feedback" refers to real-time advice and evaluations provided to users based on the analysis results.

[0427] "API" stands for Application Program Interface, and is an interface for exchanging data and functions between different software programs.

[0428] "Text data" refers to data expressed in letters and numbers.

[0429] "Image data" refers to data that includes visual information such as photographs and graphs.

[0430] This invention is a health management system that operates in cooperation with three main elements: a user, a terminal, and a server. Specifically, a user inputs lifestyle habits, health data, and genetic information using a terminal, and transmits this data to the server. The server then analyzes the data, evaluates the user's health status, and generates a personalized health management plan.

[0431] First, users use devices such as smartphones or PCs to input lifestyle data, health data, and genetic information. The input data can be in various formats, including text, audio, and images. For example, users can take photos of their meals with their smartphones and record their exercise by voice. This input data is collected by the devices and sent to a server via the Internet.

[0432] The device receives data from the user and sends it to the server using an API (Application Program Interface). For example, the device sends data through a RESTful API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0433] The server plays a central role in data analysis. First, the server uses speech recognition technology to analyze the voice data. A common speech recognition technology is Google's speech recognition API. The voice data is converted into text and stored for further analysis.

[0434] The server then uses image recognition technology to analyze the image data, for example, by using the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. This analysis result is also stored in a database.

[0435] Based on the collected data, the server then applies machine learning algorithms and deep learning models (such as TensorFlow and Scikit-learn) to assess the user's health status. Once the analysis is complete, the server generates a personalized health management plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle improvements.

[0436] The generated health management plan is then sent back to the device, where it can be accessed by the user. The user continuously records their daily lifestyle and health data, which is then sent to the server via the device. The server updates the health management plan based on the new data, generates timely feedback and advice (for example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!"), and notifies the user.

[0437] Examples and prompts

[0438] Specific examples

[0439] For example, when a user records their dinner on a smartphone, they take a photo of the meal and enter the details in text. They also record the time they spent jogging by voice. This data is sent by the device to a server, which uses image analysis technology to identify ingredients and voice recognition technology to convert the exercise record into text. Specific advice is then generated and notified to the user, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." The user can then incorporate this advice into their diet the next day, achieving effective health management.

[0440] Example prompts

[0441] "Build an AI model that can analyze photos of meals, identify the ingredients they contain and their amounts, and provide specific recommendations to users."

[0442] "Please explain the process for converting today's exercise log from audio data to text and generating a health management plan based on that data."

[0443] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0444] Step 1:

[0445] Users use their smartphones or PCs (devices) to input lifestyle, health, and genetic information. For example, they might enter text like "Today's dinner is salad and grilled chicken" into an application's input form, take a photo of the meal, and record a voice memo saying they went jogging for 30 minutes. This input data is then saved on the device.

[0446] Input: text data, image data, audio data

[0447] Output: Lifestyle data stored on the device

[0448] Step 2:

[0449] The terminal sends the data entered by the user to the server via the API. Specifically, the terminal converts the data received into JSON format and sends a POST request to the server using the RESTful API.

[0450] Input: Lifestyle data stored on the device

[0451] Output: Data sent to the server

[0452] Step 3:

[0453] The server receives the transmitted data and performs preprocessing for data analysis, such as checking the format of the audio data and converting it to an appropriate format, and resizing the image data.

[0454] Input: Data sent to the server

[0455] Output: Data ready for analysis

[0456] Step 4:

[0457] The server uses speech recognition technology to analyze the voice data, for example, by using the Google Speech-to-Text API to convert the voice data into text, and the speech recognition results are stored in a database.

[0458] Input: Audio data ready for analysis

[0459] Output: Audio data converted to text

[0460] Step 5:

[0461] The server uses image analysis technology to analyze the image data. For example, it uses the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. The results of this analysis are stored in a database.

[0462] Input: Image data ready for analysis

[0463] Output: Analyzed image data (type and amount of ingredients)

[0464] Step 6:

[0465] The server analyzes the collected data using machine learning algorithms and deep learning models to assess the user's health. For example, TensorFlow is used to input dietary and exercise data into the model to obtain a health assessment result, which is then stored in a database.

[0466] Input: Text-converted voice data, analyzed image data

[0467] Output: Health status assessment results

[0468] Step 7:

[0469] The server generates a customized health management plan for each user based on the analysis results, for example, by using Scikit-learn to generate diet and exercise recommendations, and stores the plan in a database.

[0470] Input: Health status assessment results

[0471] Output: A customized health plan

[0472] Step 8:

[0473] The server sends the generated health management plan to the device, for example by sending a real-time push notification to inform the user that a new plan is available.

[0474] Enter: Customized Health Care Plan

[0475] Output: Health management plan sent to the device

[0476] Step 9:

[0477] Users continuously input their daily lifestyle data and send it to the server via their device, which then receives the new data and updates the health management plan.

[0478] Input: Continuously input lifestyle data

[0479] Output: Updated health plan

[0480] Step 10:

[0481] The server generates timely feedback and advice and sends it to the user's device, such as a message saying, "You've achieved 80% of your exercise goal this week. Keep up the great work!"

[0482] Enter: Updated Health Care Plan

[0483] Output: Feedback and advice to the user

[0484] (Application example 1)

[0485] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0486] In modern life, many people face the challenge of managing their health in their busy daily lives. It is also difficult to create specific meal plans tailored to individual health conditions, and there are limited ways to easily obtain such meals. This has resulted in a lack of effective approaches for individuals to maintain their long-term health. In particular, despite the importance of maintaining health through the continuous intake of a nutritionally balanced diet, the lack of individualized support remains a challenge.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0488] In this invention, the server includes: means for a user to input lifestyle habits, health data, and genetic information; means for transmitting the data to the server; means for the server to analyze the data and evaluate the user's health status; means for generating an individualized health management plan based on the analysis results; means for transmitting the generated health management plan to a user terminal; means for the user to continuously input and transmit daily lifestyle habit data; means for the server to update the health plan based on new data; means for the server to generate timely feedback and advice and notify the user terminal; and means for generating an appropriate meal plan based on the generated health management plan, presenting meals based on the plan as a delivery menu, and making them available for ordering. This allows users to easily obtain a meal plan optimized for their health status and obtain those meals through a delivery service.

[0489] "User" refers to an individual who uses the health management system to input lifestyle, health data, and genetic information to improve their health.

[0490] "Lifestyle data" refers to information about the actions and habits of users in their daily lives (e.g., diet, exercise history, sleep patterns, etc.).

[0491] "Health data" refers to numerical information and recorded data that measure a user's health status, such as weight, height, blood pressure, and heart rate.

[0492] "Genetic information" refers to information about a user's genetic background (e.g., family medical history, genetic test results, etc.).

[0493] "Server" refers to a computer system that receives data sent by a user and analyzes and processes it.

[0494] "Analysis" refers to the act of processing the data received by the server to examine it in detail and evaluate patterns and health conditions.

[0495] "Individualized health management plan" refers to a plan or proposal for maintaining health that is customized to suit the individual health condition and goals of each user based on the user's lifestyle data, health data, and genetic information.

[0496] "Feedback" refers to information such as evaluation, advice, and guidance that the server provides to users based on the analysis results.

[0497] A "meal plan" refers to a meal suggestion that takes into consideration nutritional balance and ingredient selection based on the user's health condition.

[0498] "Delivery Menu" means the specific meal options that a User may order based on the generated meal plan.

[0499] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[0500] User Roles

[0501] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[0502] Device Role

[0503] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0504] Server Roles

[0505] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[0506] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[0507] Meal plans and delivery features

[0508] Furthermore, in this invention, the server generates an appropriate meal plan based on the generated health management plan. This meal plan is customized based on nutritional balance and specific health goals. A delivery menu based on the meal plan is then presented, allowing the user to select and order from the menu. This delivery menu is generated using a pre-prepared database and an online meal delivery service API.

[0509] Specific examples

[0510] Consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server. The server analyzes the data and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." Based on this advice, the server then suggests multiple delivery options to the user, from which the user can order the most suitable meal.

[0511] Prompt Sentence Examples

[0512] Below is a concrete example of a prompt sentence.

[0513] Lifestyle: Rice and miso soup for breakfast every day. 30-minute bike ride to work.

[0514] Health stats: I weigh 70kg, am 175cm tall, and go to the gym three times a week.

[0515] Genetic information: High blood pressure runs in the family.

[0516] In this way, users can easily obtain nutritionally balanced meals while achieving effective health management.

[0517] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0518] Step 1:

[0519] Users input their lifestyle, health data, and genetic information into the device in the form of text, voice, or images.

[0520] Specific operations: The user opens the smartphone app, enters lifestyle habits (e.g., diet and exercise records) using text or voice, and takes photos of the meals.

[0521] Input: lifestyle habits, health data, genetic information (text, audio, image)

[0522] Output: Input data stored on the device

[0523] Step 2:

[0524] The device sends the collected data to the server, where the data format is appropriately converted via API.

[0525] Specific operation: The smartphone app calls the API and sends the collected text data, audio data, and image data to the server.

[0526] Input: Input data stored on the device

[0527] Output: The input data sent to the server

[0528] Step 3:

[0529] The server receives the transmitted data and uses voice recognition technology to convert the voice data into text, and image analysis technology to analyze the image data and identify the meal contents.

[0530] What it does: The server converts the audio data into text using a speech recognition system (e.g., Google Speech Recognition API) and then applies an image analysis algorithm (e.g., a food image classification model using TensorFlow).

[0531] Input: Audio data, image data

[0532] Output: Text data (speech recognition results), image analysis results (meal content identification)

[0533] Step 4:

[0534] The server analyzes all data, including the collected text data, and uses machine learning algorithms to assess the user's health status.

[0535] Specific operation: The server applies a machine learning model (e.g., a health prediction model using Scikit-learn) to evaluate health status based on input data.

[0536] Input: Text data, voice recognition results, image analysis results

[0537] Output: User's health status assessment

[0538] Step 5:

[0539] The server generates a personalized health care plan based on the assessment results.

[0540] Specific operation: Based on the evaluation results, the server generates a health management plan using statistical analysis and algorithms (e.g., Python code running in Jupyter Notebook).

[0541] Input: User's health assessment

[0542] Output: personalized health care plan

[0543] Step 6:

[0544] The server transmits the generated health management plan to the user terminal.

[0545] Specific operation: The server sends the generated plan to the user's smartphone app via API.

[0546] Enter: personalized health care plans.

[0547] Output: Health management plan sent to the user's terminal

[0548] Step 7:

[0549] Users continuously input lifestyle habit data based on their health management plan and send it back to the server via their terminal.

[0550] Specific operation: The user re-enters their daily lifestyle and health data into the app, and the device sends it to the server.

[0551] Input: New lifestyle and health data

[0552] Output: New data sent to the server

[0553] Step 8:

[0554] The server updates the health management plan based on new data, generates timely feedback and advice, and notifies the user's device.

[0555] What it does: The server analyzes the new data, uses a generative AI model (e.g., GPT-3) to generate text feedback or advice, and sends a notification to the user's smartphone.

[0556] Input: New lifestyle and health data

[0557] Output: Updated health plan, feedback, and advice

[0558] Step 9:

[0559] The server generates an appropriate meal plan based on the generated health management plan, and presents a delivery menu based on the plan, allowing ordering.

[0560] Specific operation: The server calls the delivery service API (e.g., Uber Eats API) based on the meal plan and presents the user with the optimal delivery menu.

[0561] Enter: personalized health care plans.

[0562] Output: Delivery menu suggestions, available options

[0563] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0564] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0565] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[0566] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0567] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text, and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. It then uses an emotion engine to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[0568] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, a customized health management plan is generated that takes the user's emotions into account. For example, the plan may include exercise plans to reduce stress and suggestions for relaxing meals.

[0569] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message might be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[0570] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[0571] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0572] The processing flow will be explained below.

[0573] Step 1:

[0574] Users input their lifestyle, health, genetic, and emotional data using a smartphone app, taking photos of their meals, recording their exercise records by voice, and recording their mood for the day in text.

[0575] Step 2:

[0576] The device collects input data and sends it to the server via a communication module. Specifically, it sends text data, voice data, image data, and text-based emotion data to the server via an API.

[0577] Step 3:

[0578] The server preprocesses the received data. Specifically, it converts voice data into text using voice recognition technology, analyzes image data using image analysis technology to identify the type and quantity of ingredients, and analyzes the user's emotional data using an emotion engine.

[0579] Step 4:

[0580] The server analyzes the preprocessed data and emotion data. It uses machine learning algorithms and deep learning models to evaluate the user's health status. The analysis results are compared with the user's past data to evaluate the progress of their health status.

[0581] Step 5:

[0582] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "We recommend walking 30 minutes every day" or "Your stress level is high today, so try some relaxation exercises."

[0583] Step 6:

[0584] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[0585] Step 7:

[0586] The user continuously inputs daily lifestyle data and emotional data, for example, recording daily meals, exercise, and moods on the device.

[0587] Step 8:

[0588] The device continuously transmits newly entered data and emotional data to the server, which then evaluates the effectiveness of the health management plan based on the new data and updates the plan as needed.

[0589] Step 9:

[0590] The server generates timely feedback and advice based on the user's latest data, such as "You've achieved 80% of your exercise goal this week" or "If you're feeling stressed, take a deep breath to relax."

[0591] Step 10:

[0592] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[0593] Example 2

[0594] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Conventional health management systems rely on the input of lifestyle, health, and genetic information, but it is difficult to effectively utilize this data individually, making it difficult to provide users with personalized health management plans. In particular, they do not take emotional data into consideration when customizing the system, making it impossible to accurately evaluate the user's overall health status. As a result, users are unable to obtain a health management plan that is suited to them, making it difficult to implement practical health management in their daily lives.

[0596] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the health condition of a user using voice recognition technology and image analysis technology, a means for generating an individualized health management plan using a machine learning algorithm and a deep learning model, and a means for analyzing the user's emotions using an emotion engine. This enables the user to obtain an optimized health management plan based on comprehensive data including lifestyle habits, health data, genetic information, and emotion data, and to perform highly effective health management in real life.

[0597] "User" means a person who uses the System to input lifestyle, health data, and genetic information and receive a personalized health care plan.

[0598] "Lifestyle data" refers to information related to the user's daily life, and is data input in the form of text, audio, images, etc.

[0599] "Health data" refers to information about the user's physical health condition, such as weight, blood pressure, and blood sugar level.

[0600] "Genetic Information" means health-related information based on a user's genes, such as data about genetic risk or genotype.

[0601] A "terminal" is a device that allows users to input lifestyle habits, health data, and genetic information, and includes devices such as smartphones and personal computers.

[0602] The "server" is a device that analyzes the data, evaluates the user's health condition, generates an individualized health management plan, and transmits it to the user's terminal.

[0603] "Speech recognition technology" is a technology that analyzes voice data to understand human speech and convert it into text data.

[0604] "Image analysis technology" refers to the technology of analyzing image data to understand its contents and extract necessary information.

[0605] A "machine learning algorithm" is a method for learning from past data and predicting new data based on the analysis results.

[0606] A "deep learning model" is a technology that uses a multi-layer neural network to learn complex patterns in data and improve the accuracy of analysis.

[0607] An "emotion engine" is a technology that analyzes changes in voice tone and facial expressions to determine the user's emotional state.

[0608] A "health management plan" is a personalized guideline for maintaining and improving health that is generated based on the user's lifestyle, health data, genetic information, and emotional data.

[0609] "Feedback" refers to information generated by the server that provides analysis results and advice to users.

[0610] "Timely feedback" means providing users with the necessary information and advice at the right time.

[0611] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0612] First, users use a device such as a smartphone or personal computer to input their lifestyle, health data, and genetic information. Specifically, they can take photos of their meals, record their exercise records by voice, and enter additional information in text if necessary. This input data is collected by the device and sent to the server.

[0613] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to a server. For example, it sends each piece of data to the server via a REST API and uploads image data to cloud storage, allowing the server to prepare the data for analysis.

[0614] The server acts as a central analyzer of the received data. It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the voice data into text, and image analysis technology (e.g., Google Cloud Vision API) to identify the type and quantity of ingredients from the photo. It also uses an emotion engine to analyze the user's tone of voice and facial expressions. This analysis determines whether the user is stressed or relaxed.

[0615] The server then uses machine learning algorithms (e.g., Scikit-Learn) or deep learning models (e.g., TensorFlow) to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a personalized health management plan. For example, if the stress level is determined to be high, it creates a plan that includes stress-reducing exercise plans and relaxing meal suggestions.

[0616] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message could be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[0617] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day by voice. The device sends this data to a server, which analyzes it. Specific advice is generated, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice, such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[0618] Examples of prompts to input to a generative AI model might include:

[0619] 1. "Identify the types and amounts of ingredients in the dinner picture."

[0620] 2. "Please convert this audio data to text."

[0621] 3. "Judge the stress level from this voice tone."

[0622] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0623] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] Users input their lifestyle habits, health data, and genetic information. Specifically, they open a smartphone app, take photos of their meals, record their exercise by voice, and enter additional information in text format if necessary. The input data includes photos of meals (image data), audio recordings of exercise duration (audio data), and additional information in text format. This data is collected on the device.

[0626] Input: Meal photos, audio recording of exercise time, text data for additional information

[0627] Output: Lifestyle data, health data, and genetic information collected on the device

[0628] Step 2:

[0629] The device processes the collected data and sends it to the server. The device sends image data, audio data, and text data to the server via the REST API. Specifically, it makes an HTTP POST request and uploads the image data to cloud storage.

[0630] Input: Lifestyle data, health data, and genetic information collected on the device

[0631] Output: Data sent to the server

[0632] Step 3:

[0633] The server receives the data sent from the device and prepares it for analysis. It stores the received image files in temporary storage and performs initial checks to ensure data integrity, such as validating the file format and size.

[0634] Input: Data sent to the server

[0635] Output: Analysis-ready data

[0636] Step 4:

[0637] The server converts the voice data into text using speech recognition technology. Specifically, it calls the Google Cloud Speech-to-Text API to convert the voice data into text.

[0638] Input: Audio data

[0639] Output: Audio data converted to text

[0640] Step 5:

[0641] The server uses image analysis technology to identify the types and quantities of ingredients from the photos. Specifically, it uses the Google Cloud Vision API to analyze the photos of the meal entered and identify the types and quantities of ingredients.

[0642] Input: Food image data

[0643] Output: Data identifying the type and amount of ingredients

[0644] Step 6:

[0645] The server uses an emotion engine to analyze voice tones and facial expressions to determine the user's emotions. Specifically, it analyzes voice and image data to detect stress levels and relaxation states.

[0646] Input: Audio data, image data

[0647] Output: User's emotional state data

[0648] Step 7:

[0649] The server uses machine learning algorithms and deep learning models based on the analyzed data to generate personalized health management plans. Specifically, it uses Scikit-Learn and TensorFlow to evaluate the user's health status and create customized health management plans.

[0650] Input: Analyzed data (type and amount of ingredients, speech text, emotional state data)

[0651] Output: A customized health plan

[0652] Step 8:

[0653] The server sends the generated health management plan to the device in JSON format so that the user can view it through a smartphone app.

[0654] Enter: Customized Health Care Plan

[0655] Output: Health management plan sent to the device

[0656] Step 9:

[0657] The user can then check the health management plan and feedback received through the device and put it into action. Specifically, by opening the app, the user can check advice such as "You're not eating enough vegetables today" and reflect it in their meal plan for the next day.

[0658] Input: Health management plan sent to the device

[0659] Output: Confirmation of implemented health management plan and feedback

[0660] Step 10:

[0661] Users continue to record their daily lifestyle and health data and send it to the server via their device. The server then continuously updates the health management plan based on the new data. Specifically, users enter their daily exercise and diet records and send them to the server.

[0662] Input: New lifestyle and health data

[0663] Output: Updated health plan and feedback

[0664] (Application example 2)

[0665] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0666] Conventional health management systems provide health management plans based on users' lifestyle and health data. However, these systems were unable to take into account the user's emotional state, making it difficult to provide personalized advice based on each user's individual emotions. As a result, they were unable to provide appropriate health management plans or dietary suggestions that responded to stress and emotional changes. Furthermore, they were unable to adequately check whether the proposed plans were being implemented properly and provide timely feedback.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0668] In this invention, the server includes means for analyzing the user's emotions from input data using an emotion engine, means for evaluating the user's health condition based on the analyzed emotion data and generating customized menu suggestions, and means for generating timely feedback and advice and notifying the user terminal. This enables personalized health management plans and meal suggestions that take the user's emotional state into consideration, and makes it possible to provide optimal support in response to changes in the user's stress and emotions.

[0669] "Lifestyle data" refers to data related to the user's daily activities and habits, such as diet, exercise, and sleep.

[0670] "Health data" refers to data relating to a user's physical health, such as weight, blood pressure, and heart rate.

[0671] "Genetic information" refers to data relating to a user's genetic characteristics and physical constitution.

[0672] The "server" is a central computer system that analyzes data collected from users and generates health management plans and feedback.

[0673] The "emotion engine" is a software module that analyzes the user's emotions from input data and customizes health management plans based on the results.

[0674] "Analyzing" means processing collected data using algorithms or specific techniques to understand and evaluate its content.

[0675] An "individualized health care plan" is a health care plan or proposal that is customized to meet a user's specific needs and conditions.

[0676] "Menu suggestion" refers to suggesting the optimal meal menu taking into consideration the user's health and emotional state.

[0677] "Feedback" refers to advice and evaluations generated by the server based on data provided by the user and their daily behavior.

[0678] "Timely feedback" means feedback provided at an appropriate time depending on the situation or condition the user is facing.

[0679] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0680] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[0681] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device transmits this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0682] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text (software used: SpeechRecognition), and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals (software used: OpenCV, PIL, pytesseract). It then uses an emotion engine (software used: emotion_recognition) to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[0683] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a customized health management plan and menu suggestions that take the user's emotions into account. For example, a health management plan that includes exercise plans to reduce stress and meal suggestions that promote relaxation may be generated.

[0684] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message such as "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax" can be sent.

[0685] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and reflect it in their diet and daily activities the next day, achieving more effective health management.

[0686] Example prompts based on generative AI models:

[0687] "I feel like I've been lacking in vegetables this week. I've detected stress in my voice tone, so please suggest a relaxing menu."

[0688] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0689] Step 1:

[0690] The device collects user input data. Users use their smartphones or PCs to input lifestyle habits, health data, and genetic information in text, voice, and image formats. This input data is stored on the device. Input includes photos of meals, exercise records, and daily emotional states.

[0691] Step 2:

[0692] The device sends the collected data to the server. The device uploads the data to the server via API. The input is the user's lifestyle data, health data, and genetic information, and the output is a message confirming successful data transmission to the server.

[0693] Step 3:

[0694] The server analyzes the received data. It uses voice recognition technology (software used: SpeechRecognition) to convert the voice data into text, and image analysis technology (software used: OpenCV, PIL, pytesseract) to identify the types and quantities of ingredients from the meal photos. It also uses an emotion engine (software used: emotion_recognition) to analyze emotions from the voice. The inputs are lifestyle data, health data, genetic information, and emotion data, and the output is analyzed text data, ingredient data, and emotion data.

[0695] Step 4:

[0696] The server generates a personalized health management plan based on the analysis results. The server uses machine learning algorithms and deep learning models to integrate the analyzed text data, food ingredient data, and emotion data to evaluate the user's health status and create a customized plan. The input is the analyzed data, and the output is a personalized health management plan.

[0697] Step 5:

[0698] The server sends the generated health management plan to the terminal. The server sends the generated plan to the terminal so that the user can access it. The input is the health management plan, and the output is a success message for sending the plan to the user terminal.

[0699] Step 6:

[0700] The device provides timely feedback and advice to the user. Based on the health management plan sent from the server, the device notifies the user of the feedback and advice. For example, a message such as "You have achieved 80% of your exercise goal this week" is sent. The input is the data received from the server, and the output is a notification message to the user.

[0701] Step 7:

[0702] The user continuously inputs lifestyle data and sends it to the server via the terminal. The user continues to record their daily lifestyle and emotional state and sends it to the server via the terminal. The input is new lifestyle data, and the output is a message that the data has been successfully sent to the server.

[0703] Step 8:

[0704] The server continuously updates the health management plan based on new data and notifies the user. As soon as the server receives new data, it reevaluates the health management plan and generates a new, optimized plan. The input is new lifestyle data, and the output is the updated health management plan.

[0705] The processing flow is as described above, and the specific operations at each step effectively implement personalized health management that takes into account the user's emotional state.

[0706] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0707] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0708] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0709] [Third embodiment]

[0710] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0711] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0712] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0713] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0714] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0715] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0716] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0717] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0718] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0719] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0720] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0721] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0722] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[0723] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[0724] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0725] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[0726] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[0727] Users continue to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide the most appropriate plan. The server also generates timely feedback and advice and notifies the user. For example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!" can be sent.

[0728] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." The user can take this advice and incorporate it into their diet the next day, achieving effective health management.

[0729] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0730] The processing flow will be explained below.

[0731] Step 1:

[0732] Users input their lifestyle, health data, and genetic information. For example, they use a smartphone app to enter the day's meal details in text and record their exercise history by voice.

[0733] Step 2:

[0734] The device collects the input data and sends it to the server via the communication module. Specifically, it sends text data, voice data, and image data to the server via the endpoint API.

[0735] Step 3:

[0736] The server preprocesses the received data, specifically converting the voice data into text and analyzing the image data to identify the type and quantity of ingredients.

[0737] Step 4:

[0738] The server analyzes the preprocessed data, and uses machine learning algorithms and deep learning models to evaluate the user's health status. Specifically, the current health status is quantified by comparing it with past data.

[0739] Step 5:

[0740] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "recommended 30 minutes of walking every day."

[0741] Step 6:

[0742] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[0743] Step 7:

[0744] The user continuously inputs daily lifestyle data, for example, recording daily meals and exercise details on the device.

[0745] Step 8:

[0746] The device continuously sends newly entered data to the server, which then uses the new data to evaluate the effectiveness of the health management plan and update it as needed.

[0747] Step 9:

[0748] The server generates timely feedback and advice, such as a rating message like "You've achieved 80% of your exercise goal this week" based on the user's latest data.

[0749] Step 10:

[0750] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[0751] Example 1

[0752] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0753] Conventional health management systems have limited the format of data that users can enter, making them unable to accommodate a variety of data formats. They also lack the ability to generate personalized health management plans, preventing them from providing specific advice or feedback tailored to the user's health condition. Furthermore, analysis of voice and image data is often performed manually, making efficient health management difficult.

[0754] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0755] In this invention, the server includes a voice recognition unit for converting voice data into text, an image analysis unit for analyzing image data to identify the types and amounts of ingredients, and a unit for evaluating the user's health condition using a machine learning or deep learning model based on the analysis results, thereby enabling efficient and personalized health management that can handle a variety of data formats.

[0756] "User" refers to an individual who uses the system to input lifestyle, health data, and genetic information and receive a health management plan.

[0757] A "terminal" is an electronic device, such as a smartphone or PC, that allows users to input and send data.

[0758] The "server" refers to a central processing unit that receives and analyzes data sent from the terminal, generates a health management plan, and provides feedback to the user.

[0759] "Lifestyle data" refers to information about a user's lifestyle, such as diet, exercise records, and sleep patterns.

[0760] "Health data" refers to numerical data that indicates the user's health condition, such as weight, blood pressure, and heart rate.

[0761] "Genetic information" refers to information about a user's genetic makeup, data used to more precisely assess their health status.

[0762] "Speech recognition means" refers to technology that converts voice data into text data.

[0763] "Image analysis means" refers to a technique for analyzing image data and identifying its contents.

[0764] A "machine learning algorithm" is an algorithm that self-learns based on data and makes predictions and evaluations.

[0765] A "deep learning model" is a technology that uses multi-layered neural networks to analyze and predict data.

[0766] A "health management plan" is a personalized plan for diet, exercise, and lifestyle improvement that is generated based on the user's lifestyle and health data.

[0767] "Feedback" refers to real-time advice and evaluations provided to users based on the analysis results.

[0768] "API" stands for Application Program Interface, and is an interface for exchanging data and functions between different software programs.

[0769] "Text data" refers to data expressed in letters and numbers.

[0770] "Image data" refers to data that includes visual information such as photographs and graphs.

[0771] This invention is a health management system that operates in cooperation with three main elements: a user, a terminal, and a server. Specifically, a user inputs lifestyle habits, health data, and genetic information using a terminal, and transmits this data to the server. The server then analyzes the data, evaluates the user's health status, and generates a personalized health management plan.

[0772] First, users use devices such as smartphones or PCs to input lifestyle data, health data, and genetic information. The input data can be in various formats, including text, audio, and images. For example, users can take photos of their meals with their smartphones and record their exercise by voice. This input data is collected by the devices and sent to a server via the Internet.

[0773] The device receives data from the user and sends it to the server using an API (Application Program Interface). For example, the device sends data through a RESTful API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0774] The server plays a central role in data analysis. First, the server uses speech recognition technology to analyze the voice data. A common speech recognition technology is Google's speech recognition API. The voice data is converted into text and stored for further analysis.

[0775] The server then uses image recognition technology to analyze the image data, for example, by using the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. This analysis result is also stored in a database.

[0776] Based on the collected data, the server then applies machine learning algorithms and deep learning models (such as TensorFlow and Scikit-learn) to assess the user's health status. Once the analysis is complete, the server generates a personalized health management plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle improvements.

[0777] The generated health management plan is then sent back to the device, where it can be accessed by the user. The user continuously records their daily lifestyle and health data, which is then sent to the server via the device. The server updates the health management plan based on the new data, generates timely feedback and advice (for example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!"), and notifies the user.

[0778] Examples and prompts

[0779] Specific examples

[0780] For example, when a user records their dinner on a smartphone, they take a photo of the meal and enter the details in text. They also record the time they spent jogging by voice. This data is sent by the device to a server, which uses image analysis technology to identify ingredients and voice recognition technology to convert the exercise record into text. Specific advice is then generated and notified to the user, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." The user can then incorporate this advice into their diet the next day, achieving effective health management.

[0781] Example prompts

[0782] "Build an AI model that can analyze photos of meals, identify the ingredients they contain and their amounts, and provide specific recommendations to users."

[0783] "Please explain the process for converting today's exercise log from audio data to text and generating a health management plan based on that data."

[0784] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0785] Step 1:

[0786] Users use their smartphones or PCs (devices) to input lifestyle, health, and genetic information. For example, they might enter text like "Today's dinner is salad and grilled chicken" into an application's input form, take a photo of the meal, and record a voice memo saying they went jogging for 30 minutes. This input data is then saved on the device.

[0787] Input: text data, image data, audio data

[0788] Output: Lifestyle data stored on the device

[0789] Step 2:

[0790] The terminal sends the data entered by the user to the server via the API. Specifically, the terminal converts the data received into JSON format and sends a POST request to the server using the RESTful API.

[0791] Input: Lifestyle data stored on the device

[0792] Output: Data sent to the server

[0793] Step 3:

[0794] The server receives the transmitted data and performs preprocessing for data analysis, such as checking the format of the audio data and converting it to an appropriate format, and resizing the image data.

[0795] Input: Data sent to the server

[0796] Output: Data ready for analysis

[0797] Step 4:

[0798] The server uses speech recognition technology to analyze the voice data, for example, by using the Google Speech-to-Text API to convert the voice data into text, and the speech recognition results are stored in a database.

[0799] Input: Audio data ready for analysis

[0800] Output: Audio data converted to text

[0801] Step 5:

[0802] The server uses image analysis technology to analyze the image data. For example, it uses the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. The results of this analysis are stored in a database.

[0803] Input: Image data ready for analysis

[0804] Output: Analyzed image data (type and amount of ingredients)

[0805] Step 6:

[0806] The server analyzes the collected data using machine learning algorithms and deep learning models to assess the user's health. For example, TensorFlow is used to input dietary and exercise data into the model to obtain a health assessment result, which is then stored in a database.

[0807] Input: Text-converted voice data, analyzed image data

[0808] Output: Health status assessment results

[0809] Step 7:

[0810] The server generates a customized health management plan for each user based on the analysis results, for example, by using Scikit-learn to generate diet and exercise recommendations, and stores the plan in a database.

[0811] Input: Health status assessment results

[0812] Output: A customized health plan

[0813] Step 8:

[0814] The server sends the generated health management plan to the device, for example by sending a real-time push notification to inform the user that a new plan is available.

[0815] Enter: Customized Health Care Plan

[0816] Output: Health management plan sent to the device

[0817] Step 9:

[0818] Users continuously input their daily lifestyle data and send it to the server via their device, which then receives the new data and updates the health management plan.

[0819] Input: Continuously input lifestyle data

[0820] Output: Updated health plan

[0821] Step 10:

[0822] The server generates timely feedback and advice and sends it to the user's device, such as a message saying, "You've achieved 80% of your exercise goal this week. Keep up the great work!"

[0823] Enter: Updated Health Care Plan

[0824] Output: Feedback and advice to the user

[0825] (Application example 1)

[0826] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0827] In modern life, many people face the challenge of managing their health in their busy daily lives. It is also difficult to create specific meal plans tailored to individual health conditions, and there are limited ways to easily obtain such meals. This has resulted in a lack of effective approaches for individuals to maintain their long-term health. In particular, despite the importance of maintaining health through the continuous intake of a nutritionally balanced diet, the lack of individualized support remains a challenge.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0829] In this invention, the server includes: means for a user to input lifestyle habits, health data, and genetic information; means for transmitting the data to the server; means for the server to analyze the data and evaluate the user's health status; means for generating an individualized health management plan based on the analysis results; means for transmitting the generated health management plan to a user terminal; means for the user to continuously input and transmit daily lifestyle habit data; means for the server to update the health plan based on new data; means for the server to generate timely feedback and advice and notify the user terminal; and means for generating an appropriate meal plan based on the generated health management plan, presenting meals based on the plan as a delivery menu, and making them available for ordering. This allows users to easily obtain a meal plan optimized for their health status and obtain those meals through a delivery service.

[0830] "User" refers to an individual who uses the health management system to input lifestyle, health data, and genetic information to improve their health.

[0831] "Lifestyle data" refers to information about the actions and habits of users in their daily lives (e.g., diet, exercise history, sleep patterns, etc.).

[0832] "Health data" refers to numerical information and recorded data that measure a user's health status, such as weight, height, blood pressure, and heart rate.

[0833] "Genetic information" refers to information about a user's genetic background (e.g., family medical history, genetic test results, etc.).

[0834] "Server" refers to a computer system that receives data sent by a user and analyzes and processes it.

[0835] "Analysis" refers to the act of processing the data received by the server to examine it in detail and evaluate patterns and health conditions.

[0836] "Individualized health management plan" refers to a plan or proposal for maintaining health that is customized to suit the individual health condition and goals of each user based on the user's lifestyle data, health data, and genetic information.

[0837] "Feedback" refers to information such as evaluation, advice, and guidance that the server provides to users based on the analysis results.

[0838] A "meal plan" refers to a meal suggestion that takes into consideration nutritional balance and ingredient selection based on the user's health condition.

[0839] "Delivery Menu" means the specific meal options that a User may order based on the generated meal plan.

[0840] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[0841] User Roles

[0842] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[0843] Device Role

[0844] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0845] Server Roles

[0846] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[0847] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[0848] Meal plans and delivery features

[0849] Furthermore, in this invention, the server generates an appropriate meal plan based on the generated health management plan. This meal plan is customized based on nutritional balance and specific health goals. A delivery menu based on the meal plan is then presented, allowing the user to select and order from the menu. This delivery menu is generated using a pre-prepared database and an online meal delivery service API.

[0850] Specific examples

[0851] Consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server. The server analyzes the data and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." Based on this advice, the server then suggests multiple delivery options to the user, from which the user can order the most suitable meal.

[0852] Prompt Sentence Examples

[0853] Below is a concrete example of a prompt sentence.

[0854] Lifestyle: Rice and miso soup for breakfast every day. 30-minute bike ride to work.

[0855] Health stats: I weigh 70kg, am 175cm tall, and go to the gym three times a week.

[0856] Genetic information: High blood pressure runs in the family.

[0857] In this way, users can easily obtain nutritionally balanced meals while achieving effective health management.

[0858] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0859] Step 1:

[0860] Users input their lifestyle, health data, and genetic information into the device in the form of text, voice, or images.

[0861] Specific operations: The user opens the smartphone app, enters lifestyle habits (e.g., diet and exercise records) using text or voice, and takes photos of the meals.

[0862] Input: lifestyle habits, health data, genetic information (text, audio, image)

[0863] Output: Input data stored on the device

[0864] Step 2:

[0865] The device sends the collected data to the server, where the data format is appropriately converted via API.

[0866] Specific operation: The smartphone app calls the API and sends the collected text data, audio data, and image data to the server.

[0867] Input: Input data stored on the device

[0868] Output: The input data sent to the server

[0869] Step 3:

[0870] The server receives the transmitted data and uses voice recognition technology to convert the voice data into text, and image analysis technology to analyze the image data and identify the meal contents.

[0871] What it does: The server converts the audio data into text using a speech recognition system (e.g., Google Speech Recognition API) and then applies an image analysis algorithm (e.g., a food image classification model using TensorFlow).

[0872] Input: Audio data, image data

[0873] Output: Text data (speech recognition results), image analysis results (meal content identification)

[0874] Step 4:

[0875] The server analyzes all data, including the collected text data, and uses machine learning algorithms to assess the user's health status.

[0876] Specific operation: The server applies a machine learning model (e.g., a health prediction model using Scikit-learn) to evaluate health status based on input data.

[0877] Input: Text data, voice recognition results, image analysis results

[0878] Output: User's health status assessment

[0879] Step 5:

[0880] The server generates a personalized health care plan based on the assessment results.

[0881] Specific operation: Based on the evaluation results, the server generates a health management plan using statistical analysis and algorithms (e.g., Python code running in Jupyter Notebook).

[0882] Input: User's health assessment

[0883] Output: personalized health care plan

[0884] Step 6:

[0885] The server transmits the generated health management plan to the user terminal.

[0886] Specific operation: The server sends the generated plan to the user's smartphone app via API.

[0887] Enter: personalized health care plans.

[0888] Output: Health management plan sent to the user's terminal

[0889] Step 7:

[0890] Users continuously input lifestyle habit data based on their health management plan and send it back to the server via their terminal.

[0891] Specific operation: The user re-enters their daily lifestyle and health data into the app, and the device sends it to the server.

[0892] Input: New lifestyle and health data

[0893] Output: New data sent to the server

[0894] Step 8:

[0895] The server updates the health management plan based on new data, generates timely feedback and advice, and notifies the user's device.

[0896] What it does: The server analyzes the new data, uses a generative AI model (e.g., GPT-3) to generate text feedback or advice, and sends a notification to the user's smartphone.

[0897] Input: New lifestyle and health data

[0898] Output: Updated health plan, feedback, and advice

[0899] Step 9:

[0900] The server generates an appropriate meal plan based on the generated health management plan, and presents a delivery menu based on the plan, allowing ordering.

[0901] Specific operation: The server calls the delivery service API (e.g., Uber Eats API) based on the meal plan and presents the user with the optimal delivery menu.

[0902] Enter: personalized health care plans.

[0903] Output: Delivery menu suggestions, available options

[0904] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0905] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0906] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[0907] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[0908] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text, and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. It then uses an emotion engine to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[0909] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, a customized health management plan is generated that takes the user's emotions into account. For example, the plan may include exercise plans to reduce stress and suggestions for relaxing meals.

[0910] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message might be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[0911] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[0912] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0913] The processing flow will be explained below.

[0914] Step 1:

[0915] Users input their lifestyle, health, genetic, and emotional data using a smartphone app, taking photos of their meals, recording their exercise records by voice, and recording their mood for the day in text.

[0916] Step 2:

[0917] The device collects input data and sends it to the server via a communication module. Specifically, it sends text data, voice data, image data, and text-based emotion data to the server via an API.

[0918] Step 3:

[0919] The server preprocesses the received data. Specifically, it converts voice data into text using voice recognition technology, analyzes image data using image analysis technology to identify the type and quantity of ingredients, and analyzes the user's emotional data using an emotion engine.

[0920] Step 4:

[0921] The server analyzes the preprocessed data and emotion data. It uses machine learning algorithms and deep learning models to evaluate the user's health status. The analysis results are compared with the user's past data to evaluate the progress of their health status.

[0922] Step 5:

[0923] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "We recommend walking 30 minutes every day" or "Your stress level is high today, so try some relaxation exercises."

[0924] Step 6:

[0925] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[0926] Step 7:

[0927] The user continuously inputs daily lifestyle data and emotional data, for example, recording daily meals, exercise, and moods on the device.

[0928] Step 8:

[0929] The device continuously transmits newly entered data and emotional data to the server, which then evaluates the effectiveness of the health management plan based on the new data and updates the plan as needed.

[0930] Step 9:

[0931] The server generates timely feedback and advice based on the user's latest data, such as "You've achieved 80% of your exercise goal this week" or "If you're feeling stressed, take a deep breath to relax."

[0932] Step 10:

[0933] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[0934] Example 2

[0935] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0936] Conventional health management systems rely on the input of lifestyle, health, and genetic information, but it is difficult to effectively utilize this data individually, making it difficult to provide users with personalized health management plans. In particular, they do not take emotional data into consideration when customizing the system, making it impossible to accurately evaluate the user's overall health status. As a result, users are unable to obtain a health management plan that is suited to them, making it difficult to implement practical health management in their daily lives.

[0937] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the health condition of a user using voice recognition technology and image analysis technology, a means for generating an individualized health management plan using a machine learning algorithm and a deep learning model, and a means for analyzing the user's emotions using an emotion engine. This enables the user to obtain an optimized health management plan based on comprehensive data including lifestyle habits, health data, genetic information, and emotion data, and to perform highly effective health management in real life.

[0938] "User" means a person who uses the System to input lifestyle, health data, and genetic information and receive a personalized health care plan.

[0939] "Lifestyle data" refers to information related to the user's daily life, and is data input in the form of text, audio, images, etc.

[0940] "Health data" refers to information about the user's physical health condition, such as weight, blood pressure, and blood sugar level.

[0941] "Genetic Information" means health-related information based on a user's genes, such as data about genetic risk or genotype.

[0942] A "terminal" is a device that allows users to input lifestyle habits, health data, and genetic information, and includes devices such as smartphones and personal computers.

[0943] The "server" is a device that analyzes the data, evaluates the user's health condition, generates an individualized health management plan, and transmits it to the user's terminal.

[0944] "Speech recognition technology" is a technology that analyzes voice data to understand human speech and convert it into text data.

[0945] "Image analysis technology" refers to the technology of analyzing image data to understand its contents and extract necessary information.

[0946] A "machine learning algorithm" is a method for learning from past data and predicting new data based on the analysis results.

[0947] A "deep learning model" is a technology that uses a multi-layer neural network to learn complex patterns in data and improve the accuracy of analysis.

[0948] An "emotion engine" is a technology that analyzes changes in voice tone and facial expressions to determine the user's emotional state.

[0949] A "health management plan" is a personalized guideline for maintaining and improving health that is generated based on the user's lifestyle, health data, genetic information, and emotional data.

[0950] "Feedback" refers to information generated by the server that provides analysis results and advice to users.

[0951] "Timely feedback" means providing users with the necessary information and advice at the right time.

[0952] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[0953] First, users use a device such as a smartphone or personal computer to input their lifestyle, health data, and genetic information. Specifically, they can take photos of their meals, record their exercise records by voice, and enter additional information in text if necessary. This input data is collected by the device and sent to the server.

[0954] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to a server. For example, it sends each piece of data to the server via a REST API and uploads image data to cloud storage, allowing the server to prepare the data for analysis.

[0955] The server acts as a central analyzer of the received data. It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the voice data into text, and image analysis technology (e.g., Google Cloud Vision API) to identify the type and quantity of ingredients from the photo. It also uses an emotion engine to analyze the user's tone of voice and facial expressions. This analysis determines whether the user is stressed or relaxed.

[0956] The server then uses machine learning algorithms (e.g., Scikit-Learn) or deep learning models (e.g., TensorFlow) to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a personalized health management plan. For example, if the stress level is determined to be high, it creates a plan that includes stress-reducing exercise plans and relaxing meal suggestions.

[0957] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message could be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[0958] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day by voice. The device sends this data to a server, which analyzes it. Specific advice is generated, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice, such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[0959] Examples of prompts to input to a generative AI model might include:

[0960] 1. "Identify the types and amounts of ingredients in the dinner picture."

[0961] 2. "Please convert this audio data to text."

[0962] 3. "Judge the stress level from this voice tone."

[0963] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[0964] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0965] Step 1:

[0966] Users input their lifestyle habits, health data, and genetic information. Specifically, they open a smartphone app, take photos of their meals, record their exercise by voice, and enter additional information in text format if necessary. The input data includes photos of meals (image data), audio recordings of exercise duration (audio data), and additional information in text format. This data is collected on the device.

[0967] Input: Meal photos, audio recording of exercise time, text data for additional information

[0968] Output: Lifestyle data, health data, and genetic information collected on the device

[0969] Step 2:

[0970] The device processes the collected data and sends it to the server. The device sends image data, audio data, and text data to the server via the REST API. Specifically, it makes an HTTP POST request and uploads the image data to cloud storage.

[0971] Input: Lifestyle data, health data, and genetic information collected on the device

[0972] Output: Data sent to the server

[0973] Step 3:

[0974] The server receives the data sent from the device and prepares it for analysis. It stores the received image files in temporary storage and performs initial checks to ensure data integrity, such as validating the file format and size.

[0975] Input: Data sent to the server

[0976] Output: Analysis-ready data

[0977] Step 4:

[0978] The server converts the voice data into text using speech recognition technology. Specifically, it calls the Google Cloud Speech-to-Text API to convert the voice data into text.

[0979] Input: Audio data

[0980] Output: Audio data converted to text

[0981] Step 5:

[0982] The server uses image analysis technology to identify the types and quantities of ingredients from the photos. Specifically, it uses the Google Cloud Vision API to analyze the photos of the meal entered and identify the types and quantities of ingredients.

[0983] Input: Food image data

[0984] Output: Data identifying the type and amount of ingredients

[0985] Step 6:

[0986] The server uses an emotion engine to analyze voice tones and facial expressions to determine the user's emotions. Specifically, it analyzes voice and image data to detect stress levels and relaxation states.

[0987] Input: Audio data, image data

[0988] Output: User's emotional state data

[0989] Step 7:

[0990] The server uses machine learning algorithms and deep learning models based on the analyzed data to generate personalized health management plans. Specifically, it uses Scikit-Learn and TensorFlow to evaluate the user's health status and create customized health management plans.

[0991] Input: Analyzed data (type and amount of ingredients, speech text, emotional state data)

[0992] Output: A customized health plan

[0993] Step 8:

[0994] The server sends the generated health management plan to the device in JSON format so that the user can view it through a smartphone app.

[0995] Enter: Customized Health Care Plan

[0996] Output: Health management plan sent to the device

[0997] Step 9:

[0998] The user can then check the health management plan and feedback received through the device and put it into action. Specifically, by opening the app, the user can check advice such as "You're not eating enough vegetables today" and reflect it in their meal plan for the next day.

[0999] Input: Health management plan sent to the device

[1000] Output: Confirmation of implemented health management plan and feedback

[1001] Step 10:

[1002] Users continue to record their daily lifestyle and health data and send it to the server via their device. The server then continuously updates the health management plan based on the new data. Specifically, users enter their daily exercise and diet records and send them to the server.

[1003] Input: New lifestyle and health data

[1004] Output: Updated health plan and feedback

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1007] Conventional health management systems provide health management plans based on users' lifestyle and health data. However, these systems were unable to take into account the user's emotional state, making it difficult to provide personalized advice based on each user's individual emotions. As a result, they were unable to provide appropriate health management plans or dietary suggestions that responded to stress and emotional changes. Furthermore, they were unable to adequately check whether the proposed plans were being implemented properly and provide timely feedback.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1009] In this invention, the server includes means for analyzing the user's emotions from input data using an emotion engine, means for evaluating the user's health condition based on the analyzed emotion data and generating customized menu suggestions, and means for generating timely feedback and advice and notifying the user terminal. This enables personalized health management plans and meal suggestions that take the user's emotional state into consideration, and makes it possible to provide optimal support in response to changes in the user's stress and emotions.

[1010] "Lifestyle data" refers to data related to the user's daily activities and habits, such as diet, exercise, and sleep.

[1011] "Health data" refers to data relating to a user's physical health, such as weight, blood pressure, and heart rate.

[1012] "Genetic information" refers to data relating to a user's genetic characteristics and physical constitution.

[1013] The "server" is a central computer system that analyzes data collected from users and generates health management plans and feedback.

[1014] The "emotion engine" is a software module that analyzes the user's emotions from input data and customizes health management plans based on the results.

[1015] "Analyzing" means processing collected data using algorithms or specific techniques to understand and evaluate its content.

[1016] An "individualized health care plan" is a health care plan or proposal that is customized to meet a user's specific needs and conditions.

[1017] "Menu suggestion" refers to suggesting the optimal meal menu taking into consideration the user's health and emotional state.

[1018] "Feedback" refers to advice and evaluations generated by the server based on data provided by the user and their daily behavior.

[1019] "Timely feedback" means feedback provided at an appropriate time depending on the situation or condition the user is facing.

[1020] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[1021] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[1022] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device transmits this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[1023] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text (software used: SpeechRecognition), and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals (software used: OpenCV, PIL, pytesseract). It then uses an emotion engine (software used: emotion_recognition) to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[1024] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a customized health management plan and menu suggestions that take the user's emotions into account. For example, a health management plan that includes exercise plans to reduce stress and meal suggestions that promote relaxation may be generated.

[1025] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message such as "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax" can be sent.

[1026] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and reflect it in their diet and daily activities the next day, achieving more effective health management.

[1027] Example prompts based on generative AI models:

[1028] "I feel like I've been lacking in vegetables this week. I've detected stress in my voice tone, so please suggest a relaxing menu."

[1029] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1030] Step 1:

[1031] The device collects user input data. Users use their smartphones or PCs to input lifestyle habits, health data, and genetic information in text, voice, and image formats. This input data is stored on the device. Input includes photos of meals, exercise records, and daily emotional states.

[1032] Step 2:

[1033] The device sends the collected data to the server. The device uploads the data to the server via API. The input is the user's lifestyle data, health data, and genetic information, and the output is a message confirming successful data transmission to the server.

[1034] Step 3:

[1035] The server analyzes the received data. It uses voice recognition technology (software used: SpeechRecognition) to convert the voice data into text, and image analysis technology (software used: OpenCV, PIL, pytesseract) to identify the types and quantities of ingredients from the meal photos. It also uses an emotion engine (software used: emotion_recognition) to analyze emotions from the voice. The inputs are lifestyle data, health data, genetic information, and emotion data, and the output is analyzed text data, ingredient data, and emotion data.

[1036] Step 4:

[1037] The server generates a personalized health management plan based on the analysis results. The server uses machine learning algorithms and deep learning models to integrate the analyzed text data, food ingredient data, and emotion data to evaluate the user's health status and create a customized plan. The input is the analyzed data, and the output is a personalized health management plan.

[1038] Step 5:

[1039] The server sends the generated health management plan to the terminal. The server sends the generated plan to the terminal so that the user can access it. The input is the health management plan, and the output is a success message for sending the plan to the user terminal.

[1040] Step 6:

[1041] The device provides timely feedback and advice to the user. Based on the health management plan sent from the server, the device notifies the user of the feedback and advice. For example, a message such as "You have achieved 80% of your exercise goal this week" is sent. The input is the data received from the server, and the output is a notification message to the user.

[1042] Step 7:

[1043] The user continuously inputs lifestyle data and sends it to the server via the terminal. The user continues to record their daily lifestyle and emotional state and sends it to the server via the terminal. The input is new lifestyle data, and the output is a message that the data has been successfully sent to the server.

[1044] Step 8:

[1045] The server continuously updates the health management plan based on new data and notifies the user. As soon as the server receives new data, it reevaluates the health management plan and generates a new, optimized plan. The input is new lifestyle data, and the output is the updated health management plan.

[1046] The processing flow is as described above, and the specific operations at each step effectively implement personalized health management that takes into account the user's emotional state.

[1047] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1048] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1049] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1050] [Fourth embodiment]

[1051] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1052] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1053] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1054] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1055] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1056] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1057] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1058] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1059] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1060] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1061] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1062] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1063] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1064] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[1065] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[1066] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[1067] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[1068] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[1069] Users continue to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide the most appropriate plan. The server also generates timely feedback and advice and notifies the user. For example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!" can be sent.

[1070] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." The user can take this advice and incorporate it into their diet the next day, achieving effective health management.

[1071] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[1072] The processing flow will be explained below.

[1073] Step 1:

[1074] Users input their lifestyle, health data, and genetic information. For example, they use a smartphone app to enter the day's meal details in text and record their exercise history by voice.

[1075] Step 2:

[1076] The device collects the input data and sends it to the server via the communication module. Specifically, it sends text data, voice data, and image data to the server via the endpoint API.

[1077] Step 3:

[1078] The server preprocesses the received data, specifically converting the voice data into text and analyzing the image data to identify the type and quantity of ingredients.

[1079] Step 4:

[1080] The server analyzes the preprocessed data, and uses machine learning algorithms and deep learning models to evaluate the user's health status. Specifically, the current health status is quantified by comparing it with past data.

[1081] Step 5:

[1082] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "recommended 30 minutes of walking every day."

[1083] Step 6:

[1084] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[1085] Step 7:

[1086] The user continuously inputs daily lifestyle data, for example, recording daily meals and exercise details on the device.

[1087] Step 8:

[1088] The device continuously sends newly entered data to the server, which then uses the new data to evaluate the effectiveness of the health management plan and update it as needed.

[1089] Step 9:

[1090] The server generates timely feedback and advice, such as a rating message like "You've achieved 80% of your exercise goal this week" based on the user's latest data.

[1091] Step 10:

[1092] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[1093] Example 1

[1094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1095] Conventional health management systems have limited the format of data that users can enter, making them unable to accommodate a variety of data formats. They also lack the ability to generate personalized health management plans, preventing them from providing specific advice or feedback tailored to the user's health condition. Furthermore, analysis of voice and image data is often performed manually, making efficient health management difficult.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1097] In this invention, the server includes a voice recognition unit for converting voice data into text, an image analysis unit for analyzing image data to identify the types and amounts of ingredients, and a unit for evaluating the user's health condition using a machine learning or deep learning model based on the analysis results, thereby enabling efficient and personalized health management that can handle a variety of data formats.

[1098] "User" refers to an individual who uses the system to input lifestyle, health data, and genetic information and receive a health management plan.

[1099] A "terminal" is an electronic device, such as a smartphone or PC, that allows users to input and send data.

[1100] The "server" refers to a central processing unit that receives and analyzes data sent from the terminal, generates a health management plan, and provides feedback to the user.

[1101] "Lifestyle data" refers to information about a user's lifestyle, such as diet, exercise records, and sleep patterns.

[1102] "Health data" refers to numerical data that indicates the user's health condition, such as weight, blood pressure, and heart rate.

[1103] "Genetic information" refers to information about a user's genetic makeup, data used to more precisely assess their health status.

[1104] "Speech recognition means" refers to technology that converts voice data into text data.

[1105] "Image analysis means" refers to a technique for analyzing image data and identifying its contents.

[1106] A "machine learning algorithm" is an algorithm that self-learns based on data and makes predictions and evaluations.

[1107] A "deep learning model" is a technology that uses multi-layered neural networks to analyze and predict data.

[1108] A "health management plan" is a personalized plan for diet, exercise, and lifestyle improvement that is generated based on the user's lifestyle and health data.

[1109] "Feedback" refers to real-time advice and evaluations provided to users based on the analysis results.

[1110] "API" stands for Application Program Interface, and is an interface for exchanging data and functions between different software programs.

[1111] "Text data" refers to data expressed in letters and numbers.

[1112] "Image data" refers to data that includes visual information such as photographs and graphs.

[1113] This invention is a health management system that operates in cooperation with three main elements: a user, a terminal, and a server. Specifically, a user inputs lifestyle habits, health data, and genetic information using a terminal, and transmits this data to the server. The server then analyzes the data, evaluates the user's health status, and generates a personalized health management plan.

[1114] First, users use devices such as smartphones or PCs to input lifestyle data, health data, and genetic information. The input data can be in various formats, including text, audio, and images. For example, users can take photos of their meals with their smartphones and record their exercise by voice. This input data is collected by the devices and sent to a server via the Internet.

[1115] The device receives data from the user and sends it to the server using an API (Application Program Interface). For example, the device sends data through a RESTful API. If the transmission is successful, the server receives the data and prepares it for analysis.

[1116] The server plays a central role in data analysis. First, the server uses speech recognition technology to analyze the voice data. A common speech recognition technology is Google's speech recognition API. The voice data is converted into text and stored for further analysis.

[1117] The server then uses image recognition technology to analyze the image data, for example, by using the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. This analysis result is also stored in a database.

[1118] Based on the collected data, the server then applies machine learning algorithms and deep learning models (such as TensorFlow and Scikit-learn) to assess the user's health status. Once the analysis is complete, the server generates a personalized health management plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle improvements.

[1119] The generated health management plan is then sent back to the device, where it can be accessed by the user. The user continuously records their daily lifestyle and health data, which is then sent to the server via the device. The server updates the health management plan based on the new data, generates timely feedback and advice (for example, a message such as "You've achieved 80% of your exercise goal this week. Keep up the great work!"), and notifies the user.

[1120] Examples and prompts

[1121] Specific examples

[1122] For example, when a user records their dinner on a smartphone, they take a photo of the meal and enter the details in text. They also record the time they spent jogging by voice. This data is sent by the device to a server, which uses image analysis technology to identify ingredients and voice recognition technology to convert the exercise record into text. Specific advice is then generated and notified to the user, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." The user can then incorporate this advice into their diet the next day, achieving effective health management.

[1123] Example prompts

[1124] "Build an AI model that can analyze photos of meals, identify the ingredients they contain and their amounts, and provide specific recommendations to users."

[1125] "Please explain the process for converting today's exercise log from audio data to text and generating a health management plan based on that data."

[1126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1127] Step 1:

[1128] Users use their smartphones or PCs (devices) to input lifestyle, health, and genetic information. For example, they might enter text like "Today's dinner is salad and grilled chicken" into an application's input form, take a photo of the meal, and record a voice memo saying they went jogging for 30 minutes. This input data is then saved on the device.

[1129] Input: text data, image data, audio data

[1130] Output: Lifestyle data stored on the device

[1131] Step 2:

[1132] The terminal sends the data entered by the user to the server via the API. Specifically, the terminal converts the data received into JSON format and sends a POST request to the server using the RESTful API.

[1133] Input: Lifestyle data stored on the device

[1134] Output: Data sent to the server

[1135] Step 3:

[1136] The server receives the transmitted data and performs preprocessing for data analysis, such as checking the format of the audio data and converting it to an appropriate format, and resizing the image data.

[1137] Input: Data sent to the server

[1138] Output: Data ready for analysis

[1139] Step 4:

[1140] The server uses speech recognition technology to analyze the voice data, for example, by using the Google Speech-to-Text API to convert the voice data into text, and the speech recognition results are stored in a database.

[1141] Input: Audio data ready for analysis

[1142] Output: Audio data converted to text

[1143] Step 5:

[1144] The server uses image analysis technology to analyze the image data. For example, it uses the Google Cloud Vision API to identify the types and quantities of ingredients in a photo of a meal. The results of this analysis are stored in a database.

[1145] Input: Image data ready for analysis

[1146] Output: Analyzed image data (type and amount of ingredients)

[1147] Step 6:

[1148] The server analyzes the collected data using machine learning algorithms and deep learning models to assess the user's health. For example, TensorFlow is used to input dietary and exercise data into the model to obtain a health assessment result, which is then stored in a database.

[1149] Input: Text-converted voice data, analyzed image data

[1150] Output: Health status assessment results

[1151] Step 7:

[1152] The server generates a customized health management plan for each user based on the analysis results, for example, by using Scikit-learn to generate diet and exercise recommendations, and stores the plan in a database.

[1153] Input: Health status assessment results

[1154] Output: A customized health plan

[1155] Step 8:

[1156] The server sends the generated health management plan to the device, for example by sending a real-time push notification to inform the user that a new plan is available.

[1157] Enter: Customized Health Care Plan

[1158] Output: Health management plan sent to the device

[1159] Step 9:

[1160] Users continuously input their daily lifestyle data and send it to the server via their device, which then receives the new data and updates the health management plan.

[1161] Input: Continuously input lifestyle data

[1162] Output: Updated health plan

[1163] Step 10:

[1164] The server generates timely feedback and advice and sends it to the user's device, such as a message saying, "You've achieved 80% of your exercise goal this week. Keep up the great work!"

[1165] Enter: Updated Health Care Plan

[1166] Output: Feedback and advice to the user

[1167] (Application example 1)

[1168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1169] In modern life, many people face the challenge of managing their health in their busy daily lives. It is also difficult to create specific meal plans tailored to individual health conditions, and there are limited ways to easily obtain such meals. This has resulted in a lack of effective approaches for individuals to maintain their long-term health. In particular, despite the importance of maintaining health through the continuous intake of a nutritionally balanced diet, the lack of individualized support remains a challenge.

[1170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1171] In this invention, the server includes: means for a user to input lifestyle habits, health data, and genetic information; means for transmitting the data to the server; means for the server to analyze the data and evaluate the user's health status; means for generating an individualized health management plan based on the analysis results; means for transmitting the generated health management plan to a user terminal; means for the user to continuously input and transmit daily lifestyle habit data; means for the server to update the health plan based on new data; means for the server to generate timely feedback and advice and notify the user terminal; and means for generating an appropriate meal plan based on the generated health management plan, presenting meals based on the plan as a delivery menu, and making them available for ordering. This allows users to easily obtain a meal plan optimized for their health status and obtain those meals through a delivery service.

[1172] "User" refers to an individual who uses the health management system to input lifestyle, health data, and genetic information to improve their health.

[1173] "Lifestyle data" refers to information about the actions and habits of users in their daily lives (e.g., diet, exercise history, sleep patterns, etc.).

[1174] "Health data" refers to numerical information and recorded data that measure a user's health status, such as weight, height, blood pressure, and heart rate.

[1175] "Genetic information" refers to information about a user's genetic background (e.g., family medical history, genetic test results, etc.).

[1176] "Server" refers to a computer system that receives data sent by a user and analyzes and processes it.

[1177] "Analysis" refers to the act of processing the data received by the server to examine it in detail and evaluate patterns and health conditions.

[1178] "Individualized health management plan" refers to a plan or proposal for maintaining health that is customized to suit the individual health condition and goals of each user based on the user's lifestyle data, health data, and genetic information.

[1179] "Feedback" refers to information such as evaluation, advice, and guidance that the server provides to users based on the analysis results.

[1180] A "meal plan" refers to a meal suggestion that takes into consideration nutritional balance and ingredient selection based on the user's health condition.

[1181] "Delivery Menu" means the specific meal options that a User may order based on the generated meal plan.

[1182] To implement the present invention, three main elements work in cooperation: the user, the terminal, and the server.

[1183] User Roles

[1184] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to a server.

[1185] Device Role

[1186] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[1187] Server Roles

[1188] The server plays a central role in analyzing the received data. First, it converts the voice data into text using voice recognition technology, then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. Furthermore, it uses machine learning algorithms and deep learning models based on the collected data to assess the user's health status.

[1189] Once the analysis is complete, the server generates a personalized health plan for the user, including specific dietary suggestions, exercise recommendations, and lifestyle changes. The plan is then sent back to the device for the user to access.

[1190] Meal plans and delivery features

[1191] Furthermore, in this invention, the server generates an appropriate meal plan based on the generated health management plan. This meal plan is customized based on nutritional balance and specific health goals. A delivery menu based on the meal plan is then presented, allowing the user to select and order from the menu. This delivery menu is generated using a pre-prepared database and an online meal delivery service API.

[1192] Specific examples

[1193] Consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day using voice recording. The device sends this data to a server. The server analyzes the data and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." Based on this advice, the server then suggests multiple delivery options to the user, from which the user can order the most suitable meal.

[1194] Prompt Sentence Examples

[1195] Below is a concrete example of a prompt sentence.

[1196] Lifestyle: Rice and miso soup for breakfast every day. 30-minute bike ride to work.

[1197] Health stats: I weigh 70kg, am 175cm tall, and go to the gym three times a week.

[1198] Genetic information: High blood pressure runs in the family.

[1199] In this way, users can easily obtain nutritionally balanced meals while achieving effective health management.

[1200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1201] Step 1:

[1202] Users input their lifestyle, health data, and genetic information into the device in the form of text, voice, or images.

[1203] Specific operations: The user opens the smartphone app, enters lifestyle habits (e.g., diet and exercise records) using text or voice, and takes photos of the meals.

[1204] Input: lifestyle habits, health data, genetic information (text, audio, image)

[1205] Output: Input data stored on the device

[1206] Step 2:

[1207] The device sends the collected data to the server, where the data format is appropriately converted via API.

[1208] Specific operation: The smartphone app calls the API and sends the collected text data, audio data, and image data to the server.

[1209] Input: Input data stored on the device

[1210] Output: The input data sent to the server

[1211] Step 3:

[1212] The server receives the transmitted data and uses voice recognition technology to convert the voice data into text, and image analysis technology to analyze the image data and identify the meal contents.

[1213] What it does: The server converts the audio data into text using a speech recognition system (e.g., Google Speech Recognition API) and then applies an image analysis algorithm (e.g., a food image classification model using TensorFlow).

[1214] Input: Audio data, image data

[1215] Output: Text data (speech recognition results), image analysis results (meal content identification)

[1216] Step 4:

[1217] The server analyzes all data, including the collected text data, and uses machine learning algorithms to assess the user's health status.

[1218] Specific operation: The server applies a machine learning model (e.g., a health prediction model using Scikit-learn) to evaluate health status based on input data.

[1219] Input: Text data, voice recognition results, image analysis results

[1220] Output: User's health status assessment

[1221] Step 5:

[1222] The server generates a personalized health care plan based on the assessment results.

[1223] Specific operation: Based on the evaluation results, the server generates a health management plan using statistical analysis and algorithms (e.g., Python code running in Jupyter Notebook).

[1224] Input: User's health assessment

[1225] Output: personalized health care plan

[1226] Step 6:

[1227] The server transmits the generated health management plan to the user terminal.

[1228] Specific operation: The server sends the generated plan to the user's smartphone app via API.

[1229] Enter: personalized health care plans.

[1230] Output: Health management plan sent to the user's terminal

[1231] Step 7:

[1232] Users continuously input lifestyle habit data based on their health management plan and send it back to the server via their terminal.

[1233] Specific operation: The user re-enters their daily lifestyle and health data into the app, and the device sends it to the server.

[1234] Input: New lifestyle and health data

[1235] Output: New data sent to the server

[1236] Step 8:

[1237] The server updates the health management plan based on new data, generates timely feedback and advice, and notifies the user's device.

[1238] What it does: The server analyzes the new data, uses a generative AI model (e.g., GPT-3) to generate text feedback or advice, and sends a notification to the user's smartphone.

[1239] Input: New lifestyle and health data

[1240] Output: Updated health plan, feedback, and advice

[1241] Step 9:

[1242] The server generates an appropriate meal plan based on the generated health management plan, and presents a delivery menu based on the plan, allowing ordering.

[1243] Specific operation: The server calls the delivery service API (e.g., Uber Eats API) based on the meal plan and presents the user with the optimal delivery menu.

[1244] Enter: personalized health care plans.

[1245] Output: Delivery menu suggestions, available options

[1246] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1247] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[1248] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[1249] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device sends this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[1250] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text, and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals. It then uses an emotion engine to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[1251] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, a customized health management plan is generated that takes the user's emotions into account. For example, the plan may include exercise plans to reduce stress and suggestions for relaxing meals.

[1252] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message might be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[1253] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[1254] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[1255] The processing flow will be explained below.

[1256] Step 1:

[1257] Users input their lifestyle, health, genetic, and emotional data using a smartphone app, taking photos of their meals, recording their exercise records by voice, and recording their mood for the day in text.

[1258] Step 2:

[1259] The device collects input data and sends it to the server via a communication module. Specifically, it sends text data, voice data, image data, and text-based emotion data to the server via an API.

[1260] Step 3:

[1261] The server preprocesses the received data. Specifically, it converts voice data into text using voice recognition technology, analyzes image data using image analysis technology to identify the type and quantity of ingredients, and analyzes the user's emotional data using an emotion engine.

[1262] Step 4:

[1263] The server analyzes the preprocessed data and emotion data. It uses machine learning algorithms and deep learning models to evaluate the user's health status. The analysis results are compared with the user's past data to evaluate the progress of their health status.

[1264] Step 5:

[1265] Based on the analysis results, the server generates a customized health management plan for the user, suggesting specific actions such as "We recommend walking 30 minutes every day" or "Your stress level is high today, so try some relaxation exercises."

[1266] Step 6:

[1267] The server sends the generated health management plan to the terminal, which receives the data and displays it on the user interface.

[1268] Step 7:

[1269] The user continuously inputs daily lifestyle data and emotional data, for example, recording daily meals, exercise, and moods on the device.

[1270] Step 8:

[1271] The device continuously transmits newly entered data and emotional data to the server, which then evaluates the effectiveness of the health management plan based on the new data and updates the plan as needed.

[1272] Step 9:

[1273] The server generates timely feedback and advice based on the user's latest data, such as "You've achieved 80% of your exercise goal this week" or "If you're feeling stressed, take a deep breath to relax."

[1274] Step 10:

[1275] The device notifies the user of the generated feedback and advice, for example, via a push notification and allows the user to view detailed advice within the app.

[1276] Example 2

[1277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] Conventional health management systems rely on the input of lifestyle, health, and genetic information, but it is difficult to effectively utilize this data individually, making it difficult to provide users with personalized health management plans. In particular, they do not take emotional data into consideration when customizing the system, making it impossible to accurately evaluate the user's overall health status. As a result, users are unable to obtain a health management plan that is suited to them, making it difficult to implement practical health management in their daily lives.

[1279] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the health condition of a user using voice recognition technology and image analysis technology, a means for generating an individualized health management plan using a machine learning algorithm and a deep learning model, and a means for analyzing the user's emotions using an emotion engine. This enables the user to obtain an optimized health management plan based on comprehensive data including lifestyle habits, health data, genetic information, and emotion data, and to perform highly effective health management in real life.

[1280] "User" means a person who uses the System to input lifestyle, health data, and genetic information and receive a personalized health care plan.

[1281] "Lifestyle data" refers to information related to the user's daily life, and is data input in the form of text, audio, images, etc.

[1282] "Health data" refers to information about the user's physical health condition, such as weight, blood pressure, and blood sugar level.

[1283] "Genetic Information" means health-related information based on a user's genes, such as data about genetic risk or genotype.

[1284] A "terminal" is a device that allows users to input lifestyle habits, health data, and genetic information, and includes devices such as smartphones and personal computers.

[1285] The "server" is a device that analyzes the data, evaluates the user's health condition, generates an individualized health management plan, and transmits it to the user's terminal.

[1286] "Speech recognition technology" is a technology that analyzes voice data to understand human speech and convert it into text data.

[1287] "Image analysis technology" refers to the technology of analyzing image data to understand its contents and extract necessary information.

[1288] A "machine learning algorithm" is a method for learning from past data and predicting new data based on the analysis results.

[1289] A "deep learning model" is a technology that uses a multi-layer neural network to learn complex patterns in data and improve the accuracy of analysis.

[1290] An "emotion engine" is a technology that analyzes changes in voice tone and facial expressions to determine the user's emotional state.

[1291] A "health management plan" is a personalized guideline for maintaining and improving health that is generated based on the user's lifestyle, health data, genetic information, and emotional data.

[1292] "Feedback" refers to information generated by the server that provides analysis results and advice to users.

[1293] "Timely feedback" means providing users with the necessary information and advice at the right time.

[1294] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[1295] First, users use a device such as a smartphone or personal computer to input their lifestyle, health data, and genetic information. Specifically, they can take photos of their meals, record their exercise records by voice, and enter additional information in text if necessary. This input data is collected by the device and sent to the server.

[1296] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to a server. For example, it sends each piece of data to the server via a REST API and uploads image data to cloud storage, allowing the server to prepare the data for analysis.

[1297] The server acts as a central analyzer of the received data. It uses voice recognition technology (e.g., Google Cloud Speech-to-Text) to convert the voice data into text, and image analysis technology (e.g., Google Cloud Vision API) to identify the type and quantity of ingredients from the photo. It also uses an emotion engine to analyze the user's tone of voice and facial expressions. This analysis determines whether the user is stressed or relaxed.

[1298] The server then uses machine learning algorithms (e.g., Scikit-Learn) or deep learning models (e.g., TensorFlow) to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a personalized health management plan. For example, if the stress level is determined to be high, it creates a plan that includes stress-reducing exercise plans and relaxing meal suggestions.

[1299] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message could be sent, such as, "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax."

[1300] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day by voice. The device sends this data to a server, which analyzes it. Specific advice is generated, such as, "You didn't eat enough vegetables today. I recommend you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice, such as, "Try yoga or meditation to relax." The user can take this advice and incorporate it into their diet and daily activities the next day, achieving more effective health management.

[1301] Examples of prompts to input to a generative AI model might include:

[1302] 1. "Identify the types and amounts of ingredients in the dinner picture."

[1303] 2. "Please convert this audio data to text."

[1304] 3. "Judge the stress level from this voice tone."

[1305] As a result, the present invention is able to accommodate a variety of data formats and provide personalized health management that also takes emotional data into consideration, allowing users to obtain specific guidelines for their actions to maintain and improve their health.

[1306] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1307] Step 1:

[1308] Users input their lifestyle habits, health data, and genetic information. Specifically, they open a smartphone app, take photos of their meals, record their exercise by voice, and enter additional information in text format if necessary. The input data includes photos of meals (image data), audio recordings of exercise duration (audio data), and additional information in text format. This data is collected on the device.

[1309] Input: Meal photos, audio recording of exercise time, text data for additional information

[1310] Output: Lifestyle data, health data, and genetic information collected on the device

[1311] Step 2:

[1312] The device processes the collected data and sends it to the server. The device sends image data, audio data, and text data to the server via the REST API. Specifically, it makes an HTTP POST request and uploads the image data to cloud storage.

[1313] Input: Lifestyle data, health data, and genetic information collected on the device

[1314] Output: Data sent to the server

[1315] Step 3:

[1316] The server receives the data sent from the device and prepares it for analysis. It stores the received image files in temporary storage and performs initial checks to ensure data integrity, such as validating the file format and size.

[1317] Input: Data sent to the server

[1318] Output: Analysis-ready data

[1319] Step 4:

[1320] The server converts the voice data into text using speech recognition technology. Specifically, it calls the Google Cloud Speech-to-Text API to convert the voice data into text.

[1321] Input: Audio data

[1322] Output: Audio data converted to text

[1323] Step 5:

[1324] The server uses image analysis technology to identify the types and quantities of ingredients from the photos. Specifically, it uses the Google Cloud Vision API to analyze the photos of the meal entered and identify the types and quantities of ingredients.

[1325] Input: Food image data

[1326] Output: Data identifying the type and amount of ingredients

[1327] Step 6:

[1328] The server uses an emotion engine to analyze voice tones and facial expressions to determine the user's emotions. Specifically, it analyzes voice and image data to detect stress levels and relaxation states.

[1329] Input: Audio data, image data

[1330] Output: User's emotional state data

[1331] Step 7:

[1332] The server uses machine learning algorithms and deep learning models based on the analyzed data to generate personalized health management plans. Specifically, it uses Scikit-Learn and TensorFlow to evaluate the user's health status and create customized health management plans.

[1333] Input: Analyzed data (type and amount of ingredients, speech text, emotional state data)

[1334] Output: A customized health plan

[1335] Step 8:

[1336] The server sends the generated health management plan to the device in JSON format so that the user can view it through a smartphone app.

[1337] Enter: Customized Health Care Plan

[1338] Output: Health management plan sent to the device

[1339] Step 9:

[1340] The user can then check the health management plan and feedback received through the device and put it into action. Specifically, by opening the app, the user can check advice such as "You're not eating enough vegetables today" and reflect it in their meal plan for the next day.

[1341] Input: Health management plan sent to the device

[1342] Output: Confirmation of implemented health management plan and feedback

[1343] Step 10:

[1344] Users continue to record their daily lifestyle and health data and send it to the server via their device. The server then continuously updates the health management plan based on the new data. Specifically, users enter their daily exercise and diet records and send them to the server.

[1345] Input: New lifestyle and health data

[1346] Output: Updated health plan and feedback

[1347] (Application example 2)

[1348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1349] Conventional health management systems provide health management plans based on users' lifestyle and health data. However, these systems were unable to take into account the user's emotional state, making it difficult to provide personalized advice based on each user's individual emotions. As a result, they were unable to provide appropriate health management plans or dietary suggestions that responded to stress and emotional changes. Furthermore, they were unable to adequately check whether the proposed plans were being implemented properly and provide timely feedback.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1351] In this invention, the server includes means for analyzing the user's emotions from input data using an emotion engine, means for evaluating the user's health condition based on the analyzed emotion data and generating customized menu suggestions, and means for generating timely feedback and advice and notifying the user terminal. This enables personalized health management plans and meal suggestions that take the user's emotional state into consideration, and makes it possible to provide optimal support in response to changes in the user's stress and emotions.

[1352] "Lifestyle data" refers to data related to the user's daily activities and habits, such as diet, exercise, and sleep.

[1353] "Health data" refers to data relating to a user's physical health, such as weight, blood pressure, and heart rate.

[1354] "Genetic information" refers to data relating to a user's genetic characteristics and physical constitution.

[1355] The "server" is a central computer system that analyzes data collected from users and generates health management plans and feedback.

[1356] The "emotion engine" is a software module that analyzes the user's emotions from input data and customizes health management plans based on the results.

[1357] "Analyzing" means processing collected data using algorithms or specific techniques to understand and evaluate its content.

[1358] An "individualized health care plan" is a health care plan or proposal that is customized to meet a user's specific needs and conditions.

[1359] "Menu suggestion" refers to suggesting the optimal meal menu taking into consideration the user's health and emotional state.

[1360] "Feedback" refers to advice and evaluations generated by the server based on data provided by the user and their daily behavior.

[1361] "Timely feedback" means feedback provided at an appropriate time depending on the situation or condition the user is facing.

[1362] To implement the present invention, three main elements, a user, a terminal, a server, and an emotion engine, work together.

[1363] First, users use devices such as smartphones or PCs to input their lifestyle, health data, and genetic information. Users can input data in a variety of formats, including text, voice, and images. This input data is collected by the device and sent to the server.

[1364] The device is responsible for transmitting lifestyle data, health data, and genetic information entered by the user to the server. For example, when a user takes a photo of a meal using a smartphone app and records an exercise record using audio, the device transmits this data to the server via an API. If the transmission is successful, the server receives the data and prepares it for analysis.

[1365] The server plays a central role in analyzing the received data. First, it uses voice recognition technology to convert the voice data into text (software used: SpeechRecognition), and then uses image analysis technology to identify the types and quantities of ingredients from photos of meals (software used: OpenCV, PIL, pytesseract). It then uses an emotion engine (software used: emotion_recognition) to analyze the user's emotions from the input data. For example, it detects changes in voice tone and facial expressions to determine whether the user is stressed or relaxed.

[1366] The server uses machine learning algorithms and deep learning models to assess the user's health status based on the analyzed emotional data. Based on the analysis results, it generates a customized health management plan and menu suggestions that take the user's emotions into account. For example, a health management plan that includes exercise plans to reduce stress and meal suggestions that promote relaxation may be generated.

[1367] The generated health management plan is then sent back to the device by the server, where it can be accessed by the user. The user continues to record their daily lifestyle and health data and send it to the server via their device. This allows the server to update the health management plan based on the new data and continuously provide an optimal plan. The server also generates timely feedback and advice and notifies the user. For example, a specific message such as "You have achieved 80% of your exercise goal this week. If you are feeling stressed, take a deep breath to relax" can be sent.

[1368] As a concrete example, consider a scenario in which a user uses a smartphone to record their dinner. In this case, the user takes a photo of the meal and enters the details in text. They also record the amount of time they jogged that day via voice. The device sends this data to a server, which analyzes it and generates specific advice such as, "You didn't eat enough vegetables today. I recommend that you include more vegetables in your meal tomorrow." If the emotion engine detects stress in the user's voice tone, it provides additional advice such as, "Try yoga or meditation to relax." The user can take this advice and reflect it in their diet and daily activities the next day, achieving more effective health management.

[1369] Example prompts based on generative AI models:

[1370] "I feel like I've been lacking in vegetables this week. I've detected stress in my voice tone, so please suggest a relaxing menu."

[1371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1372] Step 1:

[1373] The device collects user input data. Users use their smartphones or PCs to input lifestyle habits, health data, and genetic information in text, voice, and image formats. This input data is stored on the device. Input includes photos of meals, exercise records, and daily emotional states.

[1374] Step 2:

[1375] The device sends the collected data to the server. The device uploads the data to the server via API. The input is the user's lifestyle data, health data, and genetic information, and the output is a message confirming successful data transmission to the server.

[1376] Step 3:

[1377] The server analyzes the received data. It uses voice recognition technology (software used: SpeechRecognition) to convert the voice data into text, and image analysis technology (software used: OpenCV, PIL, pytesseract) to identify the types and quantities of ingredients from the meal photos. It also uses an emotion engine (software used: emotion_recognition) to analyze emotions from the voice. The inputs are lifestyle data, health data, genetic information, and emotion data, and the output is analyzed text data, ingredient data, and emotion data.

[1378] Step 4:

[1379] The server generates a personalized health management plan based on the analysis results. The server uses machine learning algorithms and deep learning models to integrate the analyzed text data, food ingredient data, and emotion data to evaluate the user's health status and create a customized plan. The input is the analyzed data, and the output is a personalized health management plan.

[1380] Step 5:

[1381] The server sends the generated health management plan to the terminal. The server sends the generated plan to the terminal so that the user can access it. The input is the health management plan, and the output is a success message for sending the plan to the user terminal.

[1382] Step 6:

[1383] The device provides timely feedback and advice to the user. Based on the health management plan sent from the server, the device notifies the user of the feedback and advice. For example, a message such as "You have achieved 80% of your exercise goal this week" is sent. The input is the data received from the server, and the output is a notification message to the user.

[1384] Step 7:

[1385] The user continuously inputs lifestyle data and sends it to the server via the terminal. The user continues to record their daily lifestyle and emotional state and sends it to the server via the terminal. The input is new lifestyle data, and the output is a message that the data has been successfully sent to the server.

[1386] Step 8:

[1387] The server continuously updates the health management plan based on new data and notifies the user. As soon as the server receives new data, it reevaluates the health management plan and generates a new, optimized plan. The input is new lifestyle data, and the output is the updated health management plan.

[1388] The processing flow is as described above, and the specific operations at each step effectively implement personalized health management that takes into account the user's emotional state.

[1389] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1391] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1392] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1393] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1394] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1395] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1396] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1397] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1398] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1399] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1400] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1401] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1402] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1403] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1404] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1405] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1406] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1407] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1408] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1409] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1410] The following is further disclosed regarding the above embodiment.

[1411] (Claim 1)

[1412] A means for users to input lifestyle, health data, and genetic information;

[1413] means for transmitting said data to a server;

[1414] A means for the server to analyze the data and evaluate the health status of the user;

[1415] means for generating a personalized health care plan based on the analysis results;

[1416] means for transmitting the generated health management plan to a user terminal;

[1417] A means for users to continuously input and transmit their daily lifestyle data;

[1418] a means for the server to update the health plan based on new data; and

[1419] A means for the server to generate timely feedback and advice and notify the user terminal;

[1420] A system including:

[1421] (Claim 2)

[1422] 2. The system according to claim 1, wherein the lifestyle data is input in the form of text, voice, or image.

[1423] (Claim 3)

[1424] 2. The system according to claim 1, wherein the server processes data using voice recognition and image analysis techniques.

[1425] "Example 1"

[1426] (Claim 1)

[1427] A means for users to input lifestyle, health data, and genetic information;

[1428] means for transmitting said data to a server;

[1429] A means for the server to analyze the data and evaluate the health status of the user;

[1430] means for generating a personalized health care plan based on the analysis results;

[1431] means for transmitting the generated health management plan to a user terminal;

[1432] A means for users to continuously input and transmit their daily lifestyle data;

[1433] a means for the server to update the health plan based on new data; and

[1434] A means for the server to generate timely feedback and advice and notify the user terminal;

[1435] speech recognition means for converting speech data into text;

[1436] image analysis means for analyzing image data to identify the type and amount of ingredients;

[1437] A means for evaluating the health status of the user using a machine learning or deep learning model based on the analysis results;

[1438] A system including:

[1439] (Claim 2)

[1440] 2. The system according to claim 1, wherein the lifestyle data is input in the form of text, voice, or image.

[1441] (Claim 3)

[1442] 2. The system according to claim 1, wherein the server processes data using voice recognition and image analysis techniques.

[1443] "Application Example 1"

[1444] (Claim 1)

[1445] A means for users to input lifestyle, health data, and genetic information;

[1446] means for transmitting said data to a server;

[1447] A means for the server to analyze the data and evaluate the health status of the user;

[1448] means for generating a personalized health care plan based on the analysis results;

[1449] means for transmitting the generated health management plan to a user terminal;

[1450] A means for users to continuously input and transmit their daily lifestyle data;

[1451] a means for the server to update the health plan based on new data; and

[1452] A means for the server to generate timely feedback and advice and notify the user terminal;

[1453] A means for generating an appropriate meal plan based on the generated health management plan, presenting meals based on the plan as a delivery menu, and enabling the meal to be ordered;

[1454] A system including:

[1455] (Claim 2)

[1456] 2. The system according to claim 1, wherein the lifestyle data is input in the form of text, voice, or image.

[1457] (Claim 3)

[1458] 2. The system according to claim 1, wherein the server processes data using voice recognition and image analysis techniques.

[1459] "Example 2: Combining Emotion Engines"

[1460] (Claim 1)

[1461] A means for users to input lifestyle, health data, and genetic information;

[1462] means for transmitting said data to a server;

[1463] A server analyzes the data and evaluates the user's health condition using voice recognition and image analysis techniques;

[1464] a means for generating a personalized health care plan based on the analysis results using machine learning algorithms and deep learning models;

[1465] means for transmitting the generated health management plan to a user terminal;

[1466] A means for users to continuously input and transmit their daily lifestyle data;

[1467] a means for the server to update the health plan based on new data; and

[1468] A means for the server to generate timely feedback and advice and notify the user terminal;

[1469] A system including:

[1470] (Claim 2)

[1471] 2. The system according to claim 1, wherein the lifestyle data is input in the form of text, voice, or image.

[1472] (Claim 3)

[1473] 2. The system according to claim 1, wherein the system analyzes the user's emotions using an emotion engine.

[1474] "Application example 2 when combining emotion engines"

[1475] (Claim 1)

[1476] A means for users to input lifestyle, health data, and genetic information;

[1477] means for transmitting said data to a server;

[1478] A means for the server to analyze the data and evaluate the health status of the user;

[1479] means for generating a personalized health care plan based on the analysis results;

[1480] means for transmitting the generated health management plan to a user terminal;

[1481] A means for users to continuously input and transmit their daily lifestyle data;

[1482] a means for the server to update the health plan based on new data; and

[1483] A means for the server to generate timely feedback and advice and notify the user terminal;

[1484] A means for analyzing user emotions from input data using an emotion engine;

[1485] A means for evaluating the user's health status and generating customized menu suggestions based on the analyzed emotion data;

[1486] A system including:

[1487] (Claim 2)

[1488] 2. The system according to claim 1, wherein the lifestyle data is input in the form of text, voice, or image.

[1489] (Claim 3)

[1490] 2. The system according to claim 1, wherein the server processes data using voice recognition technology and image analysis technology, and analyzes emotion data using emotion analysis technology. [Explanation of symbols]

[1491] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to input lifestyle, health data, and genetic information; means for transmitting said data to a server; A means for the server to analyze the data and evaluate the health status of the user; means for generating a personalized health care plan based on the analysis results; means for transmitting the generated health management plan to a user terminal; A means for users to continuously input and transmit their daily lifestyle data; a means for the server to update the health plan based on new data; and A means for the server to generate timely feedback and advice and notify the user terminal; A system including:

2. 2. The system according to claim 1, wherein the lifestyle data is input in the form of text, voice, or image.

3. 2. The system of claim 1, wherein the server processes data using voice recognition and image analysis techniques.

Images