System

Abstract

To provide a system for enabling a user to receive individually optimized health improvement proposals, and to achieve continuous health improvement.SOLUTION: A system, comprising: means for collecting basic information from a user; means for collecting daily life information of the user in real time; means for storing the collected information in a unified management database; means for analyzing the collected information using a multi-modal generation AI; means for generating a health improvement program customized for each user based on an analysis result; means for transmitting the generated health improvement program to the user; and means for analyzing new information and updating the program.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, health consciousness is on the rise, but it is difficult for individual users to find and maintain appropriate lifestyle improvements. In particular, incorrect information and inconsistent efforts can prevent the expected health improvement effects from being achieved, making continuous healthcare difficult. There is a need to solve this problem and achieve a healthy and sustainable lifestyle by providing health improvement programs optimized for each individual user. [Means for solving the problem]

[0005] The system includes a means for collecting basic information from a user, a means for collecting data on the user's daily life in real time, a means for storing the collected data in a centralized database, a means for analyzing the collected data using multimodal generation AI, a means for generating a health improvement program customized for each user based on the analysis results, a means for sending the generated health improvement program to the user's device and notifying them, a means for recording user feedback and execution status, and a means for analyzing new data and updating the program.This allows users to receive individually optimized health improvement suggestions and achieve sustainable health improvement.

[0006] "Basic Information" refers to basic data about the user, such as the user's name, age, gender, health history, current health status, lifestyle habits, and eating habits.

[0007] "Daily life data" refers to data collected from a user's daily activities, including information such as the number of steps taken, heart rate, sleep patterns, diet, social media posts, and email content.

[0008] A "centralized database" is a database in which all collected data is integrated and stored in a consistent manner.

[0009] "Multimodal generative AI" is an artificial intelligence technology that analyzes multiple different data formats (e.g., text, images, sensor data) and makes comprehensive judgments.

[0010] A "health improvement program" is a specific action plan or proposal for improving health that is customized based on the user's collected data and analysis results.

[0011] "User device" refers to a device used by a user that can receive applications and notifications, such as a smartphone, tablet, or computer.

[0012] "Feedback" refers to reactions and information based on the actions, impressions, and results of a user's actions.

[0013] "Analytical methods" refer to the techniques and processes used to analyze collected data and extract meaningful information.

[0014] "Notification methods" refer to the methods or technologies used to communicate information or offers to users, including push notifications, emails, in-app messages, etc. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[0037] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[0038] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital information such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[0039] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[0040] The server analyzes the data using multimodal generation AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status, and analyzes email content using natural language processing to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0041] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[0042] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[0043] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If it is effective, it will suggest a new exercise or diet program as the next step, and if it is not, it will adopt a different approach. This continuous feedback and program updates allows users to receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[0044] As a specific example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[0045] In this way, the present invention is a system for achieving effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[0049] Step 2:

[0050] The device collects basic information entered by the user and completes the initial setup, which is then sent to the server according to a security protocol.

[0051] Step 3:

[0052] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to collect vital data such as steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[0053] Step 4:

[0054] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[0055] Step 5:

[0056] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[0057] Step 6:

[0058] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it classifies images and videos obtained from social media using a video analysis algorithm to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[0059] Step 7:

[0060] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, meal recipes, exercise programs, etc.) from the web and integrates it with individual data. Specific health programs, such as exercise menus, meal plans, and stress reduction programs, are created.

[0061] Step 8:

[0062] The server sends the generated health improvement program to the user's device and notifies them. New suggestions are provided to the user via push notifications, in-app messages, etc.

[0063] Step 9:

[0064] The user receives a notification and can review the detailed recommendations within the app, where they can choose to implement the suggested exercise and meal plan.

[0065] Step 10:

[0066] The device records the user's feedback and performance in real time and sends it to the server, such as the distance walked, calories burned, and photos of meals.

[0067] Step 11:

[0068] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If the user is close to achieving their goal, it will suggest new exercise and meal plans. If no results are seen, it will suggest a different approach to help the user improve their health.

[0069] Step 12:

[0070] The server generates a new optimized program, sends it back to the user's device, and notifies them, allowing the user to continuously receive and implement the optimal health improvement program.

[0071] Example 1

[0072] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0073] In modern society, there is a demand for health management methods customized for each individual user, but many systems rely on a single data source and limited data analysis, and do not adequately address the diverse lifestyles and health conditions of users. This problem stems from a lack of integrated data collection and analysis technology, as well as a lack of automated individual program generation.

[0074] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0075] In this invention, the server includes means for collecting basic information from a user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for comprehensively analyzing multiple data formats including social media posts and email content to grasp the user's health status and lifestyle trends, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording user feedback and implementation status, and means for acquiring new data based on the analysis results and updating the health improvement program. This makes it possible to provide a comprehensive, automated, and individualized health management program for users' diverse lifestyles and health conditions.

[0076] "User" refers to a user of this system who provides information about personal health conditions and lifestyle habits.

[0077] "Basic information" refers to comprehensive information such as the user's name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[0078] "Daily life data" refers to data including vital information such as the user's number of steps, heart rate, and sleep patterns, as well as social media posts (images, videos, text) and email content (text, attachments).

[0079] "Real-time collection means" refers to systems that use smartphone sensors and other devices to instantly record data from users' daily lives.

[0080] "Centralized database" refers to a database that stores and manages all collected user data in an integrated manner.

[0081] "Multimodal generative AI" refers to artificial intelligence that comprehensively analyzes different types of data, such as text, images, and sensor data.

[0082] "Analysis results" refers to information about the user's health status and lifestyle trends derived from collected data by the multimodal generation AI.

[0083] A "health improvement program" refers to a plan that includes suggestions for exercise menus, meal plans, stress reduction programs, etc., customized for each user based on the analysis results.

[0084] "Means for notifying" refers to a system that has the function of sending the generated health improvement program to the user's terminal and notifying the user.

[0085] "Feedback" refers to information that a user provides to the system, such as their thoughts on the proposed health improvement program and the progress of its implementation.

[0086] "Means for acquiring and updating new data" refers to a system that has the ability to re-analyze the acquired new data and update the health improvement program to the latest version.

[0087] The present invention is a system that aggregates personal information collected from a user's daily life and general health-related information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[0088] First, users install the system's application on their device and create an account. During the registration process, they enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This basic information will serve as an important foundation for subsequent data analysis.

[0089] Next, the device collects real-time data about the user's daily life. Using smartphone sensors, it records vital information such as the number of steps taken, heart rate, and sleep patterns. This data also includes social media posts (images, videos, text) and email content (text and attachments). For example, if a user posts a photo of their meal on a social media account, the image will also be collected.

[0090] The collected data is sent from the device to a server at regular intervals. The data is sent via the Internet, and the server stores the data in a centralized database. This database has the role of integrating and managing all user information.

[0091] When the server detects a data update, it begins analyzing the data using multimodal generation AI. The AI ​​comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it analyzes images and videos obtained from social media to identify dietary details and exercise status. It also analyzes email content using natural language processing to evaluate stress levels and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0092] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information from the web and integrating it with personal data. For example, if a user's steps are low and they eat a lot of fatty foods, the server generates a plan for 30 minutes of walking per day and a low-calorie diet. The generated program is then sent from the server to the user's device, where a notification is sent.

[0093] The user receives a notification, can confirm the suggestions within the app, and then implement them. The device records the user's feedback and progress, and sends them back to the server. This allows the server to periodically analyze new data and evaluate the effectiveness of the health improvement program. If an effect is observed, a new exercise or diet program will be suggested as the next step; if not, a different approach will be adopted, enabling sustainable health improvement.

[0094] As an example of a specific prompt, if the result for a middle-aged man (42 years old, mostly desk-based) is "eating a lot of fatty foods," the suggestion would be something like, "We suggest a plan of 30 minutes of walking per day and a low-calorie diet."

[0095] In this way, the present invention is a system that realizes effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Processing flow

[0098] Step 1: User Registration

[0099] Step 2: Collecting daily data

[0100] Step 3: Send data

[0101] Step 4: Centralize your data

[0102] Step 5: Data analysis

[0103] Step 6: Generate a health improvement program

[0104] Step 7: Notification and Execution

[0105] Step 8: Feedback and Updates

[0106] Explanation of each processing step

[0107] Step 1: User Registration

[0108] Input: User's basic information (name, age, gender, health history, current health condition, lifestyle habits, dietary habits)

[0109] How it works: A user installs the app and enters their basic information on the account creation screen.

[0110] Output: The user's basic information is sent to the server and stored in a database.

[0111] Specific operation: The user downloads the app and follows the on-screen instructions to enter the required information, such as name and age.

[0112] Step 2: Collecting daily data

[0113] Input: Vital information and social media posts from users' daily lives

[0114] How it works: The device collects real-time data on steps taken, heart rate, sleep patterns, and social media posts (images, videos, text) through smartphone sensors and apps.

[0115] Output: The collected daily data is temporarily stored in the device.

[0116] How it works: When a user carries their smartphone, the pedometer and heart rate sensor collect data, and social media posts and emails are also recorded in the background.

[0117] Step 3: Send data

[0118] Input: Daily data collected on the device

[0119] Operation: The device sends the collected data to the server at regular intervals.

[0120] Output: Collected data is sent to the server.

[0121] Specific operation: For example, batch process data every hour and upload it to the server.

[0122] Step 4: Centralize your data

[0123] Input: Daily data and basic information sent to the server

[0124] How it works: The server stores the received data in a centralized database.

[0125] Output: All user data is stored in a centralized database.

[0126] Specific operation: The server automatically integrates newly received data into the existing database.

[0127] Step 5: Data analysis

[0128] Input: User data stored in a centralized database

[0129] How it works: The server uses multimodal generative AI to analyze the collected data. The AI ​​performs text analysis, image recognition, and time-series analysis of sensor data to understand the user's health status and lifestyle trends.

[0130] Output: Analysis results on each user's health status and lifestyle habits are obtained.

[0131] Specific operation: Data is input into the AI ​​model, and image analysis of meal contents from social media posts is performed, and emotional states are analyzed from text data using NLP technology.

[0132] Step 6: Generate a health improvement program

[0133] Input: Analysis results from data analysis

[0134] How it works: Based on the analysis results, the server collects the latest health information from the web and generates a health improvement program customized for each user.

[0135] Output: A personalized health improvement program for each user.

[0136] Specific actions: If the analysis results show that you eat a lot of fatty foods, the suggested program will include "30 minutes of walking per day" and a "low-calorie diet plan."

[0137] Step 7: Notification and Execution

[0138] Input: Generated health improvement program

[0139] Operation: The server sends the generated health improvement program to the user's device and notifies them.

[0140] Output: A health improvement program notification will be displayed on the user's device.

[0141] What it does: The app sends suggestions to the user's device via push notification, and the user can tap the notification to see more details.

[0142] Step 8: Feedback and Updates

[0143] Input: User performance and feedback data

[0144] How it works: The device records the user's progress and feedback and periodically sends it to the server, which retrieves new data, evaluates the program's effectiveness, and updates the health improvement program as needed.

[0145] Output: Updated Health Improvement Program

[0146] Specific actions: When the user implements the suggestions and enters their progress and feedback into the app, the data is sent to the server. Based on this, updates such as "increase walking distance" or "suggest a new approach to stress reduction" are made.

[0147] Through the above processing steps, the present system can provide a health management program customized for each user, thereby achieving effective lifestyle improvements.

[0148] (Application example 1)

[0149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] In modern society, there is a demand for systems that provide lifestyle improvement programs tailored to each individual's health condition. However, existing systems have limitations in efficiently collecting and appropriately analyzing data from individuals' daily lives to propose customized health plans, and they often lack specific lifestyle suggestions in physical stores. As a result, users are unable to receive real-time suggestions for exercise and diet appropriate to their health condition, making it difficult to continuously improve their health.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0152] In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording the user's feedback and implementation status, means for analyzing new data and updating the program, and means for an assistant device in a physical store to make health-related suggestions to the user, thereby enabling the user to receive specific and actionable lifestyle improvement suggestions tailored to their health condition in real time.

[0153] "User" refers to an individual who uses the System to receive a health improvement program.

[0154] "Basic information" includes the user's name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.

[0155] "Daily life data" refers to information collected about a user's daily activities and vital signs, such as the number of steps taken, heart rate, sleep patterns, diet, and exercise status.

[0156] "Real-time" refers to the collection and processing of data immediately, with little or no delay.

[0157] "Centralized database" refers to a database that centrally manages and stores all information collected from users.

[0158] "Multimodal generative AI" refers to AI technology that comprehensively analyzes multiple data formats (text, images, sensor data, etc.) and generates information.

[0159] A "customized health improvement program" is a program that includes suggestions for exercise menus, meal plans, stress reduction programs, and other items optimized for each individual user based on their data.

[0160] "Device" refers to a smartphone, tablet, or other electronic device used by a User.

[0161] "Notification" refers to the act of informing a user's device about a health improvement program or changes thereto.

[0162] "Feedback" refers to information returned to the system about the results and impressions of the health improvement program the user has implemented.

[0163] "Implementation status" is data that indicates to what extent a user is actually implementing a health improvement program.

[0164] "New data" refers to updated data about the user's daily life that is added later to data that has already been collected, analyzed, and stored.

[0165] An "in-store assistant device" is a device that is installed in a store or that can be carried by a user and that provides health-related suggestions to the user.

[0166] The present invention is a system that uses multimodal generation AI to automatically generate a lifestyle improvement program by integrating personal information collected from the user's daily life with general health-related information. Specific embodiments of the present invention are described below.

[0167] First, the user installs the system's application on their device and performs initial registration. They enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and dietary habits. This basic information is important for the user to receive the health improvement program accurately and effectively.

[0168] Secondly, the device uses smartphone sensors and other devices to collect real-time data about the user's daily life, including the number of steps taken, heart rate, sleep patterns, and even social media posts (images, videos, text) and email content (text and attachments). Data collection is essential to gain a detailed understanding of the user's lifestyle.

[0169] The collected data is periodically sent to a server, which stores it in a centralized database, allowing all user information to be managed in an integrated manner and used for analysis.

[0170] The server analyzes the data using multimodal generative AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status. It also uses natural language processing to analyze email content and evaluate the user's stress level and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0171] Based on the analysis, the server generates a personalized health improvement program for each user, a process that involves automatically gathering current health information from the web and integrating it with individual data, such as exercise routines, meal plans, and stress-reducing meditation programs.

[0172] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user can receive the notification, check the suggestions within the app, and implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server. The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and makes new suggestions as necessary. Through this continuous feedback and program updates, users receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[0173] As a concrete example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[0174] Additionally, by using an assistant device installed in a physical store, users can instantly receive health-related suggestions while shopping in the store. For example, a user shopping in a supermarket can be offered low-calorie food suggestions and products that correspond to appropriate exercise levels.

[0175] The hardware used includes smartphone sensors (accelerometer, heart rate sensor, etc.) and assistant devices installed in physical stores, while the software used includes data collection and analysis modules written in Python, natural language processing libraries (spacy and transformers), and databases (MySQL, etc.).

[0176] An example prompt is, "Generate a customized lifestyle improvement program based on the following user's daily life data: steps: 4500, heart rate: 75, calorie intake: 2100. Suggest an appropriate exercise plan and meal plan."

[0177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0178] Step 1:

[0179] The user installs the system application on their device and enters basic information. Specifically, the user enters information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Once this information is collected, the device sends it to the server. Input data: name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Output data: initial registration information.

[0180] Step 2:

[0181] The device collects data on the user's daily life in real time. This includes the number of steps taken, heart rate, sleep patterns, as well as social media posts (images, videos, text) and email content collected using smartphone sensors. The collected data is sent to a server at regular intervals. Input data: number of steps taken, heart rate, sleep patterns, social media posts, email content. Output data: daily life data.

[0182] Step 3:

[0183] The server stores the collected daily life data in a centralized database. The stored data is managed for each user and used for subsequent analysis and generation. Input data: Daily life data. Output data: Data stored in the centralized database.

[0184] Step 4:

[0185] The server analyzes the collected data using multimodal generative AI. Specific analysis operations include identifying dietary content and exercise status through image and video analysis of social media, and evaluating stress levels and emotional state through natural language processing analysis of emails. Data obtained from smartphone sensors is also analyzed over time to determine exercise volume and sleep patterns. Input data: Data from a centralized database. Output data: Analysis results.

[0186] Step 5:

[0187] Based on the analysis results, the server generates a health improvement program customized for each user. This program is generated by automatically collecting the latest health information from the web and integrating it with individual data. Specific actions include creating an exercise menu, meal plan, and meditation program for stress reduction. Input data: analysis results, health information from the web. Output data: customized health improvement program.

[0188] Step 6:

[0189] The server sends the generated health improvement program to the user's device, and the device notifies the user. The user checks the suggestions within the app and implements them. Input data: Customized health improvement program. Output data: Notification sent to the user's device.

[0190] Step 7:

[0191] The device records the user's feedback and execution status. It collects in real time how well the user has implemented the program's suggestions, as well as their thoughts on the suggestions and requests for improvement, and sends this to the server. Input data: user feedback, execution status. Output data: recorded feedback and execution status data.

[0192] Step 8:

[0193] The server analyzes the new data and updates the health improvement program, ensuring that users always receive the latest and most optimized health improvement suggestions. Input data: New daily life data, feedback. Output data: Updated health improvement program.

[0194] Step 9:

[0195] An assistant device in a physical store makes health-related suggestions to users. For example, while shopping at a supermarket, it may suggest low-calorie foods or provide product information based on the amount of exercise. Input data: User's health status, current shopping details. Output data: Health-related suggestions, in-store guidance.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] The present invention relates to a system that aggregates personal information collected from a user's daily life with general health-related information, and combines this information with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[0198] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[0199] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[0200] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[0201] The system also includes an emotion engine that recognizes users' emotions in real time. The emotion engine has the ability to evaluate their emotional state and stress level when analyzing users' social media posts, email content, and even sensor data.

[0202] The server uses multimodal generation AI to analyze data triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition, lifestyle habits, and emotional trends. Specifically, it uses a video analysis algorithm to classify images and videos obtained from social media, identify dietary habits and exercise status, and analyzes email content using natural language processing (NLP) to evaluate stress levels and emotional state. It also analyzes emotional data obtained through an emotion engine to accurately understand the user's psychological state.

[0203] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[0204] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[0205] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the most appropriate program. If an effect is observed, the next step is to suggest a new exercise menu or meal plan. If no effect is observed, a different approach is adopted to support the user's health improvement.

[0206] As a specific example, let's consider the use of an emotion engine. In the case of a middle-aged man (42 years old, mainly desk-bound), in addition to entering basic information, the device collects the user's daily step count, heart rate, and food photos from social media. The system then uses the emotion engine to analyze the user's stress level and emotional state based on the content of their social media posts and emails. The server comprehensively analyzes this data and determines that the user's step count is low, their diet is high, and their stress level is high. The server then recommends 30 minutes of walking per day, a low-calorie diet, and a meditation program to reduce stress, and notifies the user's device. If the user accepts and implements the suggestions, overall health improvement is achieved.

[0207] In this way, the present invention is a system that realizes effective and sustainable lifestyle improvements by analyzing each user's data in detail and providing an individually optimized health improvement program. Furthermore, by combining it with an emotion engine, it is possible to accurately grasp the user's emotional state and make more personalized suggestions.

[0208] The processing flow will be explained below.

[0209] Step 1:

[0210] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[0211] Step 2:

[0212] The terminal collects basic information entered by the user and transmits it to the server according to a security protocol.

[0213] Step 3:

[0214] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[0215] Step 4:

[0216] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[0217] Step 5:

[0218] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[0219] Step 6:

[0220] The device uses an emotion engine to analyze the user's emotional data in real time from social media posts and email content.

[0221] Step 7:

[0222] The terminal transmits the emotion data acquired from the emotion engine to the server.

[0223] Step 8:

[0224] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it analyzes images and videos obtained from social media to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[0225] Step 9:

[0226] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, dietary recipes, exercise programs, etc.) from the web and integrates it with individual data. Specifically, it creates exercise menus, meal plans, stress reduction programs, etc.

[0227] Step 10:

[0228] The server sends the generated health improvement program to the user's device and notifies them, and new suggestions are provided to the user via push notifications and in-app messages.

[0229] Step 11:

[0230] The user receives a notification and sees detailed suggestions within the app, and chooses to follow the suggested exercise and meal plan.

[0231] Step 12:

[0232] The device records the user's feedback and performance status in real time and sends it to the server, such as the distance walked, calories burned, and photos of meal contents.

[0233] Step 13:

[0234] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the optimal program. If the user is close to achieving their goal, it will suggest a new exercise menu or meal plan. If no effect is seen, it will suggest a different approach to help the user improve their health.

[0235] Step 14:

[0236] The server generates a new optimized program, sends it back to the user's device, and notifies the user, allowing the user to continuously receive and implement the optimal health improvement program.

[0237] Example 2

[0238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] Conventional health management systems have struggled to comprehensively collect and analyze users' daily life and emotional data to provide optimal health improvement programs for each individual user. As a result, they were unable to provide effective programs that fully considered the user's actual health and psychological state, limiting the effectiveness of health improvement. Furthermore, the lack of effective analysis of collected data and feedback utilization made it difficult to continuously update and optimize programs.

[0240] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting basic information from the user; means for collecting data on the user's daily life in real time; means for storing the collected data in a centralized database; analysis means using a generative model to analyze the collected data; means for generating a health improvement program customized for each user based on the analysis results; means for transmitting and notifying the user of the generated health improvement program to the user's information terminal; means for recording the user's feedback and execution status; means for analyzing new data and updating the program; analysis means including an emotion engine that analyzes emotion data in real time and evaluates the user's psychological state; and means for further personalizing the program using the emotion data. This makes it possible to comprehensively analyze the user's health and psychological states, provide an individually optimized health improvement program, and continuously update it.

[0241] "User" means an individual who uses the system to improve their own health.

[0242] "Basic information" refers to information about the user, such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[0243] "Daily life data" refers to data related to a user's daily activities and health, including the number of steps taken, heart rate, sleep patterns, social media posts, and email content.

[0244] "Collection methods" refers to the methods and technologies used to collect basic information and daily life data from users.

[0245] A "centralized management database" is a database for managing and storing collected data in an integrated manner.

[0246] "Generative model" refers to artificial intelligence (AI) technology that analyzes multiple data formats (text, images, sensor data, etc.).

[0247] "Analysis means" refers to methods and techniques for analyzing collected data using generative models to assess the user's health and emotional state.

[0248] A "customized health improvement program" is a program that is individually created for each user based on data analysis, and includes exercise menus, meal plans, stress reduction programs, etc.

[0249] "Information terminal" refers to devices such as smartphones and tablets used by users.

[0250] "Notification means" refers to a method or technology for transmitting the generated health improvement program to the user's information terminal and notifying the user.

[0251] "Feedback" refers to response data such as progress and impressions when a user follows a health improvement program.

[0252] "Emotion Engine" refers to artificial intelligence technology that assesses a user's emotional state and stress level from data from their daily life.

[0253] "Personalization" refers to providing suggestions and services that are optimized for each user's individual needs and circumstances.

[0254] "Analysis results" refers to the analysis results of data processed by the generative model or emotion engine.

[0255] "Update tools" refer to methods and techniques for continually reviewing and optimizing health improvement programs based on new data collected.

[0256] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health-related information, and combines this with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[0257] Hardware and Software Configuration

[0258] 1. User Device

[0259] Users use smartphones or tablets as devices. These devices have built-in sensors (pedometers, heart rate sensors, accelerometers, etc.) that collect vital data. The content of user posts and emails is also collected through social networking applications and email applications.

[0260] 2. Server

[0261] The server is equipped with a database system (e.g., MySQL, MongoDB) and a generative model (e.g., OpenAI GPT-4, Google BERT), as well as an emotion engine for data analysis.

[0262] 3. Software

[0263] The application software of this system manages a series of processes, from creating a health improvement program to notifying the results.

[0264] Program processing explanation

[0265] 1. User Registration

[0266] Users first install the system's app on their device and enter basic information (name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.) Once this registration is complete, the information is sent from the device to the server and stored in a centralized database.

[0267] 2. Data Collection

[0268] The device uses built-in sensors to collect vital data such as the number of steps taken, heart rate, and sleep patterns in real time. Furthermore, the device also periodically collects users' social media posts (images, videos, text) and email content (text and attachments). This data is then sent to a server at regular intervals.

[0269] 3. Data analysis and emotion evaluation

[0270] The server stores the received data in a centralized database. It then uses an emotion engine to evaluate the user's emotional state and stress level based on social media posts, email content, and sensor data. It uses a generative model to comprehensively analyze multiple data formats (text, images, and sensor data) to understand the user's health, lifestyle, and emotional trends.

[0271] 4. Generating health improvement programs

[0272] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information and integrating it with personal data. Specific examples of programs that can be generated include exercise routines, meal plans, and meditation programs for stress reduction.

[0273] 5. Program delivery and feedback utilization

[0274] The generated health improvement program is sent from the server to the user's device and notified. The user receives the notification and can review and implement the suggestions within the app. The device records the user's feedback and implementation status in real time and sends it back to the server. The server periodically analyzes the new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary.

[0275] Specific examples

[0276] For example, in the case of a middle-aged man (age 42, mostly desk-bound), the following process is performed:

[0277] 1. Registration: Enter basic information.

[0278] 2. Data collection: Step count, heart rate, and food photos from social media.

[0279] 3. Emotion assessment: Analyzes stress levels and emotional states from social media posts and email content through an emotion engine.

[0280] 4. Data analysis: It was confirmed that the number of steps was low, the amount of fatty food was high, and stress levels were high.

[0281] 5. Program generation: Suggests a 30-minute walk per day, a low-calorie diet, and a meditation program to reduce stress.

[0282] 6. Program distribution: Notification to user's device.

[0283] Example prompts to be input to the generative AI model:

[0284] Create a health improvement program for a 42-year-old man who works a desk job. His daily data shows that he walks few steps, eats a lot of fatty foods, and has high stress levels based on his social media and email activity.

[0285] In this way, the present invention aims to improve the user's health condition efficiently and sustainably by providing an individually optimized health improvement program based on detailed data analysis.

[0286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0287] Step 1:

[0288] The user installs the app and enters basic information. The user registers their name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc. in the app. This information becomes input data and is sent from the device to the server. The server receives this basic information and stores it in a centralized database. This saves the user's basic information in the database and is used in subsequent analysis processes.

[0289] Step 2:

[0290] The device collects data about the user's daily life in real time. As the user goes about their daily life, the device uses its built-in sensors to acquire vital data such as the number of steps taken, heart rate, and sleep patterns. Images, videos, text, and emails posted by the user on social media are also periodically collected. This collected data becomes input data for the device. The device then sends the collected data to a server. The server stores the received data in a centralized database. This allows the user's daily life data to be accumulated in the database.

[0291] Step 3:

[0292] The server analyzes the user's emotional state using an emotion engine. The server receives input data from social media posts, email content, and sensor data stored in a database, and analyzes it using the emotion engine. The emotion engine analyzes text and sensor data to evaluate stress levels and emotional states. The analysis results in data on the user's emotional state. This data is used to generate subsequent health improvement programs.

[0293] Step 4:

[0294] The server performs data analysis using multimodal generative AI. The server comprehensively retrieves multiple data formats, such as vital signs, social media posts, and email content, from the database and inputs them into the generative AI model. The generative AI model analyzes this data to understand the user's health status, lifestyle habits, and emotional trends. As an output, health status assessment data for the user is generated. This assessment data is used to customize health improvement programs.

[0295] Step 5:

[0296] The server generates a health improvement program based on the analysis results. The server uses the user's health status assessment data obtained from the generative AI model as input data, collects and integrates the latest health information, and generates an individually optimized health improvement program. The generated program specifically includes an exercise menu, meal plan, stress reduction program, etc. These programs are customized for each user.

[0297] Step 6:

[0298] The server sends the generated health improvement program to the user's device and notifies them. The server then sends the generated health improvement program to the device, which then notifies the user. The user receives the notification and checks the contents of the proposed program within the app. The effects of the improvement program are realized when the user implements the proposed health improvement program.

[0299] Step 7:

[0300] The user provides feedback, and the device records the execution status. The user runs the proposed program and enters the execution status and impressions within the app. The device collects this feedback in real time and sends the feedback data to a server. The server stores the received feedback data in a database and uses it for analysis.

[0301] Step 8:

[0302] The server analyzes the new data and updates the health improvement program. The server then uses the feedback data and newly collected daily life data to perform another analysis using the generative AI model. Based on the analysis results, the effectiveness of the health improvement program is evaluated and the program is optimized and updated as necessary. This enables continuous health improvement tailored to the user's condition.

[0303] (Application example 2)

[0304] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0305] In modern fitness gyms and wellness centers, it is extremely important to accurately understand the health and emotional state of each user and provide customized health improvement programs based on that information. However, a system that effectively collects and analyzes users' daily life data and emotional state and provides specific health improvement suggestions in real time has not yet been realized. This poses a challenge in providing continuous support for users' health improvement and receiving appropriate feedback.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending and notifying the user of the generated health improvement program, means for collecting the user's implementation conditions and storing them in a database as a record of exercise facility use, means for recording the user's feedback and implementation status, and means for analyzing new data and updating the program. This makes it possible to comprehensively analyze the health and emotional state of each user and continuously provide an optimized health improvement program.

[0307] "Means for collecting basic information from users" refers to devices or methods for obtaining basic information such as the user's name, age, sex, health history, current health condition, lifestyle habits, and eating habits.

[0308] "Means for collecting data on a user's daily life in real time" refers to devices or methods for acquiring vital data such as the number of steps taken by a user, heart rate, and sleep patterns in real time.

[0309] "Means for storing collected data in a unified database" refers to a device or method for storing and managing all collected data in a unified database.

[0310] "Analysis methods using multimodal generation AI to analyze collected data" refers to artificial intelligence devices and methods that comprehensively analyze multiple data formats, such as text data, image data, and sensor data.

[0311] "Means for generating a health improvement program customized for each user based on the analysis results" refers to a device or method that uses analyzed data to automatically generate a health improvement plan optimized for each individual user.

[0312] The "means for transmitting the generated health improvement program to the user's terminal and notifying the user" refers to a device or method for transmitting the generated health improvement program to the user's terminal in real time and notifying the user.

[0313] "Means for collecting user activity conditions and storing them in a database as a record of use of the exercise facility" refers to a device or method for collecting data on the user's activity at the exercise facility and storing it in a database as a record of use.

[0314] "Means for recording user feedback and performance" refers to a device or method for tracking and recording user feedback and performance of a health improvement program.

[0315] "Means for analyzing new data and updating the program" refers to devices and methods that use the latest data for analysis and continuously update and improve the health improvement program.

[0316] "Means for analyzing a user's emotional state" refers to devices or methods for analyzing a user's emotions and stress levels from the content of their social media posts and emails.

[0317] This invention relates to a system that aggregates personal information collected from users' daily lives with general health-related information, and combines it with an emotion engine to generate more accurate health improvement programs for users of fitness gyms and wellness centers.

[0318] System configuration:

[0319] This system consists of the following main components:

[0320] 1. How we collect basic information from you:

[0321] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into a dedicated application.

[0322] 2. Means of collecting real-time data on users' daily lives:

[0323] It uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns.

[0324] 3. How the collected data is stored in a centralized database:

[0325] The collected data is sent from the terminal to a server at regular intervals, and the server stores it in a centralized database.

[0326] 4. Analysis method using multimodal generative AI to analyze collected data:

[0327] The server uses multimodal generative AI to comprehensively analyze the collected text data, image data, and sensor data to understand the user's health condition, lifestyle habits, and emotional trends. At this time, it uses an Emotion Engine to analyze emotions and stress levels from social media posts and email content.

[0328] 5. Means for generating a health improvement program customized for each user based on the analysis results:

[0329] Based on the analysis results, the system generates a health improvement program optimized for each user's individual condition, including exercise routines, meal plans, and meditation programs for stress reduction.

[0330] 6. Means for sending the generated health improvement program to the user's device and notifying them:

[0331] The generated health improvement program is sent to the user's device, where they can view the suggestions within the app.

[0332] 7. Means for collecting user's performance conditions and storing them in a database as a record of use of the sports facility:

[0333] The system collects data on users' activities at fitness gyms and wellness centers (e.g., number of workouts, machine usage history, calories burned, etc.) and stores it in a database.

[0334] 8. Means of recording user feedback and performance:

[0335] When users implement the suggestions, feedback and progress are tracked and recorded, and this data is also sent to the server.

[0336] 9. How to analyze new data and update the program:

[0337] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary. It places particular emphasis on emotional data obtained through the emotion engine, and continually suggests optimal programs that reflect the user's psychological state.

[0338] Examples of hardware and software:

[0339] Smartphone Sensors:

[0340] Collect data about your daily life (step count, heart rate, sleep patterns, etc.).

[0341] Emotion Engine:

[0342] A library for emotion analysis that analyzes stress levels and emotional states from social media posts and email content.

[0343] Health Analyzer:

[0344] A library that analyzes user health data and generates health improvement programs.

[0345] Examples and prompts:

[0346] For example, if a male user named Tanaka visits a fitness gym, the following programs will be offered to him:

[0347] User information: Name: Tanaka, Age: 42, Gender: Male

[0348] Health history: high blood pressure, diabetes

[0349] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[0350] Social media post: I'm tired again today...

[0351] Email content: I feel like I've been feeling a lot of stress at work lately.

[0352] Based on this, the following health improvement program is generated:

[0353] Generate a health improvement program based on the following criteria:

[0354] User information: Name: Tanaka, Age: 42, Gender: Male, Health history: High blood pressure, diabetes

[0355] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[0356] Emotional Data:

[0357] Social media post: I'm tired again today...

[0358] Email content: I feel like I've been feeling a lot of stress at work lately.

[0359] In this way, the optimal program is provided based on the user's health and emotional state, enabling continuous improvement in health.

[0360] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0361] Step 1:

[0362] User Information Collection:

[0363] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into the application, and the device then collects the user's basic information and sends it to the server.

[0364] Input: Basic information from the user

[0365] Output: Basic information stored on the server

[0366] Step 2:

[0367] Real-time collection of daily life data:

[0368] The device uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns, and transmits the collected data to a server at regular intervals.

[0369] Input: Vital data collected from smartphone sensors

[0370] Output: Vital data sent to the server

[0371] Step 3:

[0372] Centralize your data:

[0373] The server receives the basic information and vital data sent from the device and stores them in a centralized database, which manages all user information in an integrated manner.

[0374] Input: Basic information and vital data sent from the device

[0375] Output: Data stored in a centralized database

[0376] Step 4:

[0377] Emotion analysis:

[0378] The server uses the Emotion Engine to analyze users' emotional state and stress levels from social media posts and email content, using text analysis techniques to generate emotion data.

[0379] Input: User's SNS posts and email content

[0380] Output: Emotional state and stress level data

[0381] Step 5:

[0382] Data Analysis:

[0383] The server uses multimodal generative AI to comprehensively analyze basic information, vital data, and emotional data, thereby understanding the user's health condition, lifestyle habits, and emotional tendencies.

[0384] Input: Basic information, vital data, emotional data

[0385] Output: Analysis results of the user's health status, lifestyle habits, and emotional tendencies

[0386] Step 6:

[0387] Generating Health Improvement Programs:

[0388] Based on the analysis, the server generates a customized health improvement program for each user, including exercise routines, meal plans, and meditation programs to reduce stress.

[0389] Input: Analysis results

[0390] Output: A customized health improvement program

[0391] Step 7:

[0392] Health Improvement Program Submissions and Notifications:

[0393] The server then sends the generated health improvement program to the user's device and notifies them, and the user can check the proposed program through the application.

[0394] Input: Generated health improvement program

[0395] Output: Health improvement programs and notifications sent to the user's device

[0396] Step 8:

[0397] Collecting implementation conditions:

[0398] The device collects the user's activity data at the fitness gym or wellness center (for example, number of training sessions, machine usage history, calories burned, etc.) and stores this data in a database.

[0399] Input: Activity data at fitness gyms and wellness centers

[0400] Output: Activity data stored in a database

[0401] Step 9:

[0402] Feedback and performance recording:

[0403] The device tracks and records feedback and progress as users implement the suggestions, and this data is sent back to the server.

[0404] Input: User feedback and execution status

[0405] Output: Feedback and execution status data sent to the server

[0406] Step 10:

[0407] Program updates:

[0408] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates it as necessary, placing particular emphasis on emotional data obtained through the Emotion Engine.

[0409] Input: New data and existing health improvement programs

[0410] Output: Updated Health Improvement Program

[0411] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0413] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0414] [Second embodiment]

[0415] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0416] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0418] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0422] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0425] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0426] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0427] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[0428] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[0429] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital information such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[0430] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[0431] The server analyzes the data using multimodal generation AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status, and analyzes email content using natural language processing to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0432] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[0433] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[0434] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If it is effective, it will suggest a new exercise or diet program as the next step, and if it is not, it will adopt a different approach. This continuous feedback and program updates allows users to receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[0435] As a specific example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[0436] In this way, the present invention is a system for achieving effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[0437] The processing flow will be explained below.

[0438] Step 1:

[0439] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[0440] Step 2:

[0441] The device collects basic information entered by the user and completes the initial setup, which is then sent to the server according to a security protocol.

[0442] Step 3:

[0443] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to collect vital data such as steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[0444] Step 4:

[0445] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[0446] Step 5:

[0447] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[0448] Step 6:

[0449] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it classifies images and videos obtained from social media using a video analysis algorithm to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[0450] Step 7:

[0451] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, meal recipes, exercise programs, etc.) from the web and integrates it with individual data. Specific health programs, such as exercise menus, meal plans, and stress reduction programs, are created.

[0452] Step 8:

[0453] The server sends the generated health improvement program to the user's device and notifies them. New suggestions are provided to the user via push notifications, in-app messages, etc.

[0454] Step 9:

[0455] The user receives a notification and can review the detailed recommendations within the app, where they can choose to implement the suggested exercise and meal plan.

[0456] Step 10:

[0457] The device records the user's feedback and performance in real time and sends it to the server, such as the distance walked, calories burned, and photos of meals.

[0458] Step 11:

[0459] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If the user is close to achieving their goal, it will suggest new exercise and meal plans. If no results are seen, it will suggest a different approach to help the user improve their health.

[0460] Step 12:

[0461] The server generates a new optimized program, sends it back to the user's device, and notifies them, allowing the user to continuously receive and implement the optimal health improvement program.

[0462] Example 1

[0463] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0464] In modern society, there is a demand for health management methods customized for each individual user, but many systems rely on a single data source and limited data analysis, and do not adequately address the diverse lifestyles and health conditions of users. This problem stems from a lack of integrated data collection and analysis technology, as well as a lack of automated individual program generation.

[0465] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0466] In this invention, the server includes means for collecting basic information from a user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for comprehensively analyzing multiple data formats including social media posts and email content to grasp the user's health status and lifestyle trends, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording user feedback and implementation status, and means for acquiring new data based on the analysis results and updating the health improvement program. This makes it possible to provide a comprehensive, automated, and individualized health management program for users' diverse lifestyles and health conditions.

[0467] "User" refers to a user of this system who provides information about personal health conditions and lifestyle habits.

[0468] "Basic information" refers to comprehensive information such as the user's name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[0469] "Daily life data" refers to data including vital information such as the user's number of steps, heart rate, and sleep patterns, as well as social media posts (images, videos, text) and email content (text, attachments).

[0470] "Real-time collection means" refers to systems that use smartphone sensors and other devices to instantly record data from users' daily lives.

[0471] "Centralized database" refers to a database that stores and manages all collected user data in an integrated manner.

[0472] "Multimodal generative AI" refers to artificial intelligence that comprehensively analyzes different types of data, such as text, images, and sensor data.

[0473] "Analysis results" refers to information about the user's health status and lifestyle trends derived from collected data by the multimodal generation AI.

[0474] A "health improvement program" refers to a plan that includes suggestions for exercise menus, meal plans, stress reduction programs, etc., customized for each user based on the analysis results.

[0475] "Means for notifying" refers to a system that has the function of sending the generated health improvement program to the user's terminal and notifying the user.

[0476] "Feedback" refers to information that a user provides to the system, such as their thoughts on the proposed health improvement program and the progress of its implementation.

[0477] "Means for acquiring and updating new data" refers to a system that has the ability to re-analyze the acquired new data and update the health improvement program to the latest version.

[0478] The present invention is a system that aggregates personal information collected from a user's daily life and general health-related information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[0479] First, users install the system's application on their device and create an account. During the registration process, they enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This basic information will serve as an important foundation for subsequent data analysis.

[0480] Next, the device collects real-time data about the user's daily life. Using smartphone sensors, it records vital information such as the number of steps taken, heart rate, and sleep patterns. This data also includes social media posts (images, videos, text) and email content (text and attachments). For example, if a user posts a photo of their meal on a social media account, the image will also be collected.

[0481] The collected data is sent from the device to a server at regular intervals. The data is sent via the Internet, and the server stores the data in a centralized database. This database has the role of integrating and managing all user information.

[0482] When the server detects a data update, it begins analyzing the data using multimodal generation AI. The AI ​​comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it analyzes images and videos obtained from social media to identify dietary details and exercise status. It also analyzes email content using natural language processing to evaluate stress levels and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0483] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information from the web and integrating it with personal data. For example, if a user's steps are low and they eat a lot of fatty foods, the server generates a plan for 30 minutes of walking per day and a low-calorie diet. The generated program is then sent from the server to the user's device, where a notification is sent.

[0484] The user receives a notification, can confirm the suggestions within the app, and then implement them. The device records the user's feedback and progress, and sends them back to the server. This allows the server to periodically analyze new data and evaluate the effectiveness of the health improvement program. If an effect is observed, a new exercise or diet program will be suggested as the next step; if not, a different approach will be adopted, enabling sustainable health improvement.

[0485] As an example of a specific prompt, if the result for a middle-aged man (42 years old, mostly desk-based) is "eating a lot of fatty foods," the suggestion would be something like, "We suggest a plan of 30 minutes of walking per day and a low-calorie diet."

[0486] In this way, the present invention is a system that realizes effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[0487] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0488] Processing flow

[0489] Step 1: User Registration

[0490] Step 2: Collecting daily data

[0491] Step 3: Send data

[0492] Step 4: Centralize your data

[0493] Step 5: Data analysis

[0494] Step 6: Generate a health improvement program

[0495] Step 7: Notification and Execution

[0496] Step 8: Feedback and Updates

[0497] Explanation of each processing step

[0498] Step 1: User Registration

[0499] Input: User's basic information (name, age, gender, health history, current health condition, lifestyle habits, dietary habits)

[0500] How it works: A user installs the app and enters their basic information on the account creation screen.

[0501] Output: The user's basic information is sent to the server and stored in a database.

[0502] Specific operation: The user downloads the app and follows the on-screen instructions to enter the required information, such as name and age.

[0503] Step 2: Collecting daily data

[0504] Input: Vital information and social media posts from users' daily lives

[0505] How it works: The device collects real-time data on steps taken, heart rate, sleep patterns, and social media posts (images, videos, text) through smartphone sensors and apps.

[0506] Output: The collected daily data is temporarily stored in the device.

[0507] How it works: When a user carries their smartphone, the pedometer and heart rate sensor collect data, and social media posts and emails are also recorded in the background.

[0508] Step 3: Send data

[0509] Input: Daily data collected on the device

[0510] Operation: The device sends the collected data to the server at regular intervals.

[0511] Output: Collected data is sent to the server.

[0512] Specific operation: For example, batch process data every hour and upload it to the server.

[0513] Step 4: Centralize your data

[0514] Input: Daily data and basic information sent to the server

[0515] How it works: The server stores the received data in a centralized database.

[0516] Output: All user data is stored in a centralized database.

[0517] Specific operation: The server automatically integrates newly received data into the existing database.

[0518] Step 5: Data analysis

[0519] Input: User data stored in a centralized database

[0520] How it works: The server uses multimodal generative AI to analyze the collected data. The AI ​​performs text analysis, image recognition, and time-series analysis of sensor data to understand the user's health status and lifestyle trends.

[0521] Output: Analysis results on each user's health status and lifestyle habits are obtained.

[0522] Specific operation: Data is input into the AI ​​model, and image analysis of meal contents from social media posts is performed, and emotional states are analyzed from text data using NLP technology.

[0523] Step 6: Generate a health improvement program

[0524] Input: Analysis results from data analysis

[0525] How it works: Based on the analysis results, the server collects the latest health information from the web and generates a health improvement program customized for each user.

[0526] Output: A personalized health improvement program for each user.

[0527] Specific actions: If the analysis results show that you eat a lot of fatty foods, the suggested program will include "30 minutes of walking per day" and a "low-calorie diet plan."

[0528] Step 7: Notification and Execution

[0529] Input: Generated health improvement program

[0530] Operation: The server sends the generated health improvement program to the user's device and notifies them.

[0531] Output: A health improvement program notification will be displayed on the user's device.

[0532] What it does: The app sends suggestions to the user's device via push notification, and the user can tap the notification to see more details.

[0533] Step 8: Feedback and Updates

[0534] Input: User performance and feedback data

[0535] How it works: The device records the user's progress and feedback and periodically sends it to the server, which retrieves new data, evaluates the program's effectiveness, and updates the health improvement program as needed.

[0536] Output: Updated Health Improvement Program

[0537] Specific actions: When the user implements the suggestions and enters their progress and feedback into the app, the data is sent to the server. Based on this, updates such as "increase walking distance" or "suggest a new approach to stress reduction" are made.

[0538] Through the above processing steps, the present system can provide a health management program customized for each user, thereby achieving effective lifestyle improvements.

[0539] (Application example 1)

[0540] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0541] In modern society, there is a demand for systems that provide lifestyle improvement programs tailored to each individual's health condition. However, existing systems have limitations in efficiently collecting and appropriately analyzing data from individuals' daily lives to propose customized health plans, and they often lack specific lifestyle suggestions in physical stores. As a result, users are unable to receive real-time suggestions for exercise and diet appropriate to their health condition, making it difficult to continuously improve their health.

[0542] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0543] In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording the user's feedback and implementation status, means for analyzing new data and updating the program, and means for an assistant device in a physical store to make health-related suggestions to the user, thereby enabling the user to receive specific and actionable lifestyle improvement suggestions tailored to their health condition in real time.

[0544] "User" refers to an individual who uses the System to receive a health improvement program.

[0545] "Basic information" includes the user's name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.

[0546] "Daily life data" refers to information collected about a user's daily activities and vital signs, such as the number of steps taken, heart rate, sleep patterns, diet, and exercise status.

[0547] "Real-time" refers to the collection and processing of data immediately, with little or no delay.

[0548] "Centralized database" refers to a database that centrally manages and stores all information collected from users.

[0549] "Multimodal generative AI" refers to AI technology that comprehensively analyzes multiple data formats (text, images, sensor data, etc.) and generates information.

[0550] A "customized health improvement program" is a program that includes suggestions for exercise menus, meal plans, stress reduction programs, and other items optimized for each individual user based on their data.

[0551] "Device" refers to a smartphone, tablet, or other electronic device used by a User.

[0552] "Notification" refers to the act of informing a user's device about a health improvement program or changes thereto.

[0553] "Feedback" refers to information returned to the system about the results and impressions of the health improvement program the user has implemented.

[0554] "Implementation status" is data that indicates to what extent a user is actually implementing a health improvement program.

[0555] "New data" refers to updated data about the user's daily life that is added later to data that has already been collected, analyzed, and stored.

[0556] An "in-store assistant device" is a device that is installed in a store or that can be carried by a user and that provides health-related suggestions to the user.

[0557] The present invention is a system that uses multimodal generation AI to automatically generate a lifestyle improvement program by integrating personal information collected from the user's daily life with general health-related information. Specific embodiments of the present invention are described below.

[0558] First, the user installs the system's application on their device and performs initial registration. They enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and dietary habits. This basic information is important for the user to receive the health improvement program accurately and effectively.

[0559] Secondly, the device uses smartphone sensors and other devices to collect real-time data about the user's daily life, including the number of steps taken, heart rate, sleep patterns, and even social media posts (images, videos, text) and email content (text and attachments). Data collection is essential to gain a detailed understanding of the user's lifestyle.

[0560] The collected data is periodically sent to a server, which stores it in a centralized database, allowing all user information to be managed in an integrated manner and used for analysis.

[0561] The server analyzes the data using multimodal generative AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status. It also uses natural language processing to analyze email content and evaluate the user's stress level and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0562] Based on the analysis, the server generates a personalized health improvement program for each user, a process that involves automatically gathering current health information from the web and integrating it with individual data, such as exercise routines, meal plans, and stress-reducing meditation programs.

[0563] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user can receive the notification, check the suggestions within the app, and implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server. The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and makes new suggestions as necessary. Through this continuous feedback and program updates, users receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[0564] As a concrete example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[0565] Additionally, by using an assistant device installed in a physical store, users can instantly receive health-related suggestions while shopping in the store. For example, a user shopping in a supermarket can be offered low-calorie food suggestions and products that correspond to appropriate exercise levels.

[0566] The hardware used includes smartphone sensors (accelerometer, heart rate sensor, etc.) and assistant devices installed in physical stores, while the software used includes data collection and analysis modules written in Python, natural language processing libraries (spacy and transformers), and databases (MySQL, etc.).

[0567] An example prompt is, "Generate a customized lifestyle improvement program based on the following user's daily life data: steps: 4500, heart rate: 75, calorie intake: 2100. Suggest an appropriate exercise plan and meal plan."

[0568] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0569] Step 1:

[0570] The user installs the system application on their device and enters basic information. Specifically, the user enters information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Once this information is collected, the device sends it to the server. Input data: name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Output data: initial registration information.

[0571] Step 2:

[0572] The device collects data on the user's daily life in real time. This includes the number of steps taken, heart rate, sleep patterns, as well as social media posts (images, videos, text) and email content collected using smartphone sensors. The collected data is sent to a server at regular intervals. Input data: number of steps taken, heart rate, sleep patterns, social media posts, email content. Output data: daily life data.

[0573] Step 3:

[0574] The server stores the collected daily life data in a centralized database. The stored data is managed for each user and used for subsequent analysis and generation. Input data: Daily life data. Output data: Data stored in the centralized database.

[0575] Step 4:

[0576] The server analyzes the collected data using multimodal generative AI. Specific analysis operations include identifying dietary content and exercise status through image and video analysis of social media, and evaluating stress levels and emotional state through natural language processing analysis of emails. Data obtained from smartphone sensors is also analyzed over time to determine exercise volume and sleep patterns. Input data: Data from a centralized database. Output data: Analysis results.

[0577] Step 5:

[0578] Based on the analysis results, the server generates a health improvement program customized for each user. This program is generated by automatically collecting the latest health information from the web and integrating it with individual data. Specific actions include creating an exercise menu, meal plan, and meditation program for stress reduction. Input data: analysis results, health information from the web. Output data: customized health improvement program.

[0579] Step 6:

[0580] The server sends the generated health improvement program to the user's device, and the device notifies the user. The user checks the suggestions within the app and implements them. Input data: Customized health improvement program. Output data: Notification sent to the user's device.

[0581] Step 7:

[0582] The device records the user's feedback and execution status. It collects in real time how well the user has implemented the program's suggestions, as well as their thoughts on the suggestions and requests for improvement, and sends this to the server. Input data: user feedback, execution status. Output data: recorded feedback and execution status data.

[0583] Step 8:

[0584] The server analyzes the new data and updates the health improvement program, ensuring that users always receive the latest and most optimized health improvement suggestions. Input data: New daily life data, feedback. Output data: Updated health improvement program.

[0585] Step 9:

[0586] An assistant device in a physical store makes health-related suggestions to users. For example, while shopping at a supermarket, it may suggest low-calorie foods or provide product information based on the amount of exercise. Input data: User's health status, current shopping details. Output data: Health-related suggestions, in-store guidance.

[0587] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0588] The present invention relates to a system that aggregates personal information collected from a user's daily life with general health-related information, and combines this information with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[0589] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[0590] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[0591] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[0592] The system also includes an emotion engine that recognizes users' emotions in real time. The emotion engine has the ability to evaluate their emotional state and stress level when analyzing users' social media posts, email content, and even sensor data.

[0593] The server uses multimodal generation AI to analyze data triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition, lifestyle habits, and emotional trends. Specifically, it uses a video analysis algorithm to classify images and videos obtained from social media, identify dietary habits and exercise status, and analyzes email content using natural language processing (NLP) to evaluate stress levels and emotional state. It also analyzes emotional data obtained through an emotion engine to accurately understand the user's psychological state.

[0594] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[0595] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[0596] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the most appropriate program. If an effect is observed, the next step is to suggest a new exercise menu or meal plan. If no effect is observed, a different approach is adopted to support the user's health improvement.

[0597] As a specific example, let's consider the use of an emotion engine. In the case of a middle-aged man (42 years old, mainly desk-bound), in addition to entering basic information, the device collects the user's daily step count, heart rate, and food photos from social media. The system then uses the emotion engine to analyze the user's stress level and emotional state based on the content of their social media posts and emails. The server comprehensively analyzes this data and determines that the user's step count is low, their diet is high, and their stress level is high. The server then recommends 30 minutes of walking per day, a low-calorie diet, and a meditation program to reduce stress, and notifies the user's device. If the user accepts and implements the suggestions, overall health improvement is achieved.

[0598] In this way, the present invention is a system that realizes effective and sustainable lifestyle improvements by analyzing each user's data in detail and providing an individually optimized health improvement program. Furthermore, by combining it with an emotion engine, it is possible to accurately grasp the user's emotional state and make more personalized suggestions.

[0599] The processing flow will be explained below.

[0600] Step 1:

[0601] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[0602] Step 2:

[0603] The terminal collects basic information entered by the user and transmits it to the server according to a security protocol.

[0604] Step 3:

[0605] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[0606] Step 4:

[0607] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[0608] Step 5:

[0609] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[0610] Step 6:

[0611] The device uses an emotion engine to analyze the user's emotional data in real time from social media posts and email content.

[0612] Step 7:

[0613] The terminal transmits the emotion data acquired from the emotion engine to the server.

[0614] Step 8:

[0615] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it analyzes images and videos obtained from social media to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[0616] Step 9:

[0617] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, dietary recipes, exercise programs, etc.) from the web and integrates it with individual data. Specifically, it creates exercise menus, meal plans, stress reduction programs, etc.

[0618] Step 10:

[0619] The server sends the generated health improvement program to the user's device and notifies them, and new suggestions are provided to the user via push notifications and in-app messages.

[0620] Step 11:

[0621] The user receives a notification and sees detailed suggestions within the app, and chooses to follow the suggested exercise and meal plan.

[0622] Step 12:

[0623] The device records the user's feedback and performance status in real time and sends it to the server, such as the distance walked, calories burned, and photos of meal contents.

[0624] Step 13:

[0625] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the optimal program. If the user is close to achieving their goal, it will suggest a new exercise menu or meal plan. If no effect is seen, it will suggest a different approach to help the user improve their health.

[0626] Step 14:

[0627] The server generates a new optimized program, sends it back to the user's device, and notifies the user, allowing the user to continuously receive and implement the optimal health improvement program.

[0628] Example 2

[0629] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] Conventional health management systems have struggled to comprehensively collect and analyze users' daily life and emotional data to provide optimal health improvement programs for each individual user. As a result, they were unable to provide effective programs that fully considered the user's actual health and psychological state, limiting the effectiveness of health improvement. Furthermore, the lack of effective analysis of collected data and feedback utilization made it difficult to continuously update and optimize programs.

[0631] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting basic information from the user; means for collecting data on the user's daily life in real time; means for storing the collected data in a centralized database; analysis means using a generative model to analyze the collected data; means for generating a health improvement program customized for each user based on the analysis results; means for transmitting and notifying the user of the generated health improvement program to the user's information terminal; means for recording the user's feedback and execution status; means for analyzing new data and updating the program; analysis means including an emotion engine that analyzes emotion data in real time and evaluates the user's psychological state; and means for further personalizing the program using the emotion data. This makes it possible to comprehensively analyze the user's health and psychological states, provide an individually optimized health improvement program, and continuously update it.

[0632] "User" means an individual who uses the system to improve their own health.

[0633] "Basic information" refers to information about the user, such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[0634] "Daily life data" refers to data related to a user's daily activities and health, including the number of steps taken, heart rate, sleep patterns, social media posts, and email content.

[0635] "Collection methods" refers to the methods and technologies used to collect basic information and daily life data from users.

[0636] A "centralized management database" is a database for managing and storing collected data in an integrated manner.

[0637] "Generative model" refers to artificial intelligence (AI) technology that analyzes multiple data formats (text, images, sensor data, etc.).

[0638] "Analysis means" refers to methods and techniques for analyzing collected data using generative models to assess the user's health and emotional state.

[0639] A "customized health improvement program" is a program that is individually created for each user based on data analysis, and includes exercise menus, meal plans, stress reduction programs, etc.

[0640] "Information terminal" refers to devices such as smartphones and tablets used by users.

[0641] "Notification means" refers to a method or technology for transmitting the generated health improvement program to the user's information terminal and notifying the user.

[0642] "Feedback" refers to response data such as progress and impressions when a user follows a health improvement program.

[0643] "Emotion Engine" refers to artificial intelligence technology that assesses a user's emotional state and stress level from data from their daily life.

[0644] "Personalization" refers to providing suggestions and services that are optimized for each user's individual needs and circumstances.

[0645] "Analysis results" refers to the analysis results of data processed by the generative model or emotion engine.

[0646] "Update tools" refer to methods and techniques for continually reviewing and optimizing health improvement programs based on new data collected.

[0647] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health-related information, and combines this with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[0648] Hardware and Software Configuration

[0649] 1. User Device

[0650] Users use smartphones or tablets as devices. These devices have built-in sensors (pedometers, heart rate sensors, accelerometers, etc.) that collect vital data. The content of user posts and emails is also collected through social networking applications and email applications.

[0651] 2. Server

[0652] The server is equipped with a database system (e.g., MySQL, MongoDB) and a generative model (e.g., OpenAI GPT-4, Google BERT), as well as an emotion engine for data analysis.

[0653] 3. Software

[0654] The application software of this system manages a series of processes, from creating a health improvement program to notifying the results.

[0655] Program processing explanation

[0656] 1. User Registration

[0657] Users first install the system's app on their device and enter basic information (name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.) Once this registration is complete, the information is sent from the device to the server and stored in a centralized database.

[0658] 2. Data Collection

[0659] The device uses built-in sensors to collect vital data such as the number of steps taken, heart rate, and sleep patterns in real time. Furthermore, the device also periodically collects users' social media posts (images, videos, text) and email content (text and attachments). This data is then sent to a server at regular intervals.

[0660] 3. Data analysis and emotion evaluation

[0661] The server stores the received data in a centralized database. It then uses an emotion engine to evaluate the user's emotional state and stress level based on social media posts, email content, and sensor data. It uses a generative model to comprehensively analyze multiple data formats (text, images, and sensor data) to understand the user's health, lifestyle, and emotional trends.

[0662] 4. Generating health improvement programs

[0663] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information and integrating it with personal data. Specific examples of programs that can be generated include exercise routines, meal plans, and meditation programs for stress reduction.

[0664] 5. Program delivery and feedback utilization

[0665] The generated health improvement program is sent from the server to the user's device and notified. The user receives the notification and can review and implement the suggestions within the app. The device records the user's feedback and implementation status in real time and sends it back to the server. The server periodically analyzes the new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary.

[0666] Specific examples

[0667] For example, in the case of a middle-aged man (age 42, mostly desk-bound), the following process is performed:

[0668] 1. Registration: Enter basic information.

[0669] 2. Data collection: Step count, heart rate, and food photos from social media.

[0670] 3. Emotion assessment: Analyzes stress levels and emotional states from social media posts and email content through an emotion engine.

[0671] 4. Data analysis: It was confirmed that the number of steps was low, the amount of fatty food was high, and stress levels were high.

[0672] 5. Program generation: Suggests a 30-minute walk per day, a low-calorie diet, and a meditation program to reduce stress.

[0673] 6. Program distribution: Notification to user's device.

[0674] Example prompts to be input to the generative AI model:

[0675] Create a health improvement program for a 42-year-old man who works a desk job. His daily data shows that he walks few steps, eats a lot of fatty foods, and has high stress levels based on his social media and email activity.

[0676] In this way, the present invention aims to improve the user's health condition efficiently and sustainably by providing an individually optimized health improvement program based on detailed data analysis.

[0677] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0678] Step 1:

[0679] The user installs the app and enters basic information. The user registers their name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc. in the app. This information becomes input data and is sent from the device to the server. The server receives this basic information and stores it in a centralized database. This saves the user's basic information in the database and is used in subsequent analysis processes.

[0680] Step 2:

[0681] The device collects data about the user's daily life in real time. As the user goes about their daily life, the device uses its built-in sensors to acquire vital data such as the number of steps taken, heart rate, and sleep patterns. Images, videos, text, and emails posted by the user on social media are also periodically collected. This collected data becomes input data for the device. The device then sends the collected data to a server. The server stores the received data in a centralized database. This allows the user's daily life data to be accumulated in the database.

[0682] Step 3:

[0683] The server analyzes the user's emotional state using an emotion engine. The server receives input data from social media posts, email content, and sensor data stored in a database, and analyzes it using the emotion engine. The emotion engine analyzes text and sensor data to evaluate stress levels and emotional states. The analysis results in data on the user's emotional state. This data is used to generate subsequent health improvement programs.

[0684] Step 4:

[0685] The server performs data analysis using multimodal generative AI. The server comprehensively retrieves multiple data formats, such as vital signs, social media posts, and email content, from the database and inputs them into the generative AI model. The generative AI model analyzes this data to understand the user's health status, lifestyle habits, and emotional trends. As an output, health status assessment data for the user is generated. This assessment data is used to customize health improvement programs.

[0686] Step 5:

[0687] The server generates a health improvement program based on the analysis results. The server uses the user's health status assessment data obtained from the generative AI model as input data, collects and integrates the latest health information, and generates an individually optimized health improvement program. The generated program specifically includes an exercise menu, meal plan, stress reduction program, etc. These programs are customized for each user.

[0688] Step 6:

[0689] The server sends the generated health improvement program to the user's device and notifies them. The server then sends the generated health improvement program to the device, which then notifies the user. The user receives the notification and checks the contents of the proposed program within the app. The effects of the improvement program are realized when the user implements the proposed health improvement program.

[0690] Step 7:

[0691] The user provides feedback, and the device records the execution status. The user runs the proposed program and enters the execution status and impressions within the app. The device collects this feedback in real time and sends the feedback data to a server. The server stores the received feedback data in a database and uses it for analysis.

[0692] Step 8:

[0693] The server analyzes the new data and updates the health improvement program. The server then uses the feedback data and newly collected daily life data to perform another analysis using the generative AI model. Based on the analysis results, the effectiveness of the health improvement program is evaluated and the program is optimized and updated as necessary. This enables continuous health improvement tailored to the user's condition.

[0694] (Application example 2)

[0695] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] In modern fitness gyms and wellness centers, it is extremely important to accurately understand the health and emotional state of each user and provide customized health improvement programs based on that information. However, a system that effectively collects and analyzes users' daily life data and emotional state and provides specific health improvement suggestions in real time has not yet been realized. This poses a challenge in providing continuous support for users' health improvement and receiving appropriate feedback.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending and notifying the user of the generated health improvement program, means for collecting the user's implementation conditions and storing them in a database as a record of exercise facility use, means for recording the user's feedback and implementation status, and means for analyzing new data and updating the program. This makes it possible to comprehensively analyze the health and emotional state of each user and continuously provide an optimized health improvement program.

[0698] "Means for collecting basic information from users" refers to devices or methods for obtaining basic information such as the user's name, age, sex, health history, current health condition, lifestyle habits, and eating habits.

[0699] "Means for collecting data on a user's daily life in real time" refers to devices or methods for acquiring vital data such as the number of steps taken by a user, heart rate, and sleep patterns in real time.

[0700] "Means for storing collected data in a unified database" refers to a device or method for storing and managing all collected data in a unified database.

[0701] "Analysis methods using multimodal generation AI to analyze collected data" refers to artificial intelligence devices and methods that comprehensively analyze multiple data formats, such as text data, image data, and sensor data.

[0702] "Means for generating a health improvement program customized for each user based on the analysis results" refers to a device or method that uses analyzed data to automatically generate a health improvement plan optimized for each individual user.

[0703] The "means for transmitting the generated health improvement program to the user's terminal and notifying the user" refers to a device or method for transmitting the generated health improvement program to the user's terminal in real time and notifying the user.

[0704] "Means for collecting user activity conditions and storing them in a database as a record of use of the exercise facility" refers to a device or method for collecting data on the user's activity at the exercise facility and storing it in a database as a record of use.

[0705] "Means for recording user feedback and performance" refers to a device or method for tracking and recording user feedback and performance of a health improvement program.

[0706] "Means for analyzing new data and updating the program" refers to devices and methods that use the latest data for analysis and continuously update and improve the health improvement program.

[0707] "Means for analyzing a user's emotional state" refers to devices or methods for analyzing a user's emotions and stress levels from the content of their social media posts and emails.

[0708] This invention relates to a system that aggregates personal information collected from users' daily lives with general health-related information, and combines it with an emotion engine to generate more accurate health improvement programs for users of fitness gyms and wellness centers.

[0709] System configuration:

[0710] This system consists of the following main components:

[0711] 1. How we collect basic information from you:

[0712] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into a dedicated application.

[0713] 2. Means of collecting real-time data on users' daily lives:

[0714] It uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns.

[0715] 3. How the collected data is stored in a centralized database:

[0716] The collected data is sent from the terminal to a server at regular intervals, and the server stores it in a centralized database.

[0717] 4. Analysis method using multimodal generative AI to analyze collected data:

[0718] The server uses multimodal generative AI to comprehensively analyze the collected text data, image data, and sensor data to understand the user's health condition, lifestyle habits, and emotional trends. At this time, it uses an Emotion Engine to analyze emotions and stress levels from social media posts and email content.

[0719] 5. Means for generating a health improvement program customized for each user based on the analysis results:

[0720] Based on the analysis results, the system generates a health improvement program optimized for each user's individual condition, including exercise routines, meal plans, and meditation programs for stress reduction.

[0721] 6. Means for sending the generated health improvement program to the user's device and notifying them:

[0722] The generated health improvement program is sent to the user's device, where they can view the suggestions within the app.

[0723] 7. Means for collecting user's performance conditions and storing them in a database as a record of use of the sports facility:

[0724] The system collects data on users' activities at fitness gyms and wellness centers (e.g., number of workouts, machine usage history, calories burned, etc.) and stores it in a database.

[0725] 8. Means of recording user feedback and performance:

[0726] When users implement the suggestions, feedback and progress are tracked and recorded, and this data is also sent to the server.

[0727] 9. How to analyze new data and update the program:

[0728] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary. It places particular emphasis on emotional data obtained through the emotion engine, and continually suggests optimal programs that reflect the user's psychological state.

[0729] Examples of hardware and software:

[0730] Smartphone Sensors:

[0731] Collect data about your daily life (step count, heart rate, sleep patterns, etc.).

[0732] Emotion Engine:

[0733] A library for emotion analysis that analyzes stress levels and emotional states from social media posts and email content.

[0734] Health Analyzer:

[0735] A library that analyzes user health data and generates health improvement programs.

[0736] Examples and prompts:

[0737] For example, if a male user named Tanaka visits a fitness gym, the following programs will be offered to him:

[0738] User information: Name: Tanaka, Age: 42, Gender: Male

[0739] Health history: high blood pressure, diabetes

[0740] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[0741] Social media post: I'm tired again today...

[0742] Email content: I feel like I've been feeling a lot of stress at work lately.

[0743] Based on this, the following health improvement program is generated:

[0744] Generate a health improvement program based on the following criteria:

[0745] User information: Name: Tanaka, Age: 42, Gender: Male, Health history: High blood pressure, diabetes

[0746] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[0747] Emotional Data:

[0748] Social media post: I'm tired again today...

[0749] Email content: I feel like I've been feeling a lot of stress at work lately.

[0750] In this way, the optimal program is provided based on the user's health and emotional state, enabling continuous improvement in health.

[0751] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0752] Step 1:

[0753] User Information Collection:

[0754] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into the application, and the device then collects the user's basic information and sends it to the server.

[0755] Input: Basic information from the user

[0756] Output: Basic information stored on the server

[0757] Step 2:

[0758] Real-time collection of daily life data:

[0759] The device uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns, and transmits the collected data to a server at regular intervals.

[0760] Input: Vital data collected from smartphone sensors

[0761] Output: Vital data sent to the server

[0762] Step 3:

[0763] Centralize your data:

[0764] The server receives the basic information and vital data sent from the device and stores them in a centralized database, which manages all user information in an integrated manner.

[0765] Input: Basic information and vital data sent from the device

[0766] Output: Data stored in a centralized database

[0767] Step 4:

[0768] Emotion analysis:

[0769] The server uses the Emotion Engine to analyze users' emotional state and stress levels from social media posts and email content, using text analysis techniques to generate emotion data.

[0770] Input: User's SNS posts and email content

[0771] Output: Emotional state and stress level data

[0772] Step 5:

[0773] Data Analysis:

[0774] The server uses multimodal generative AI to comprehensively analyze basic information, vital data, and emotional data, thereby understanding the user's health condition, lifestyle habits, and emotional tendencies.

[0775] Input: Basic information, vital data, emotional data

[0776] Output: Analysis results of the user's health status, lifestyle habits, and emotional tendencies

[0777] Step 6:

[0778] Generating Health Improvement Programs:

[0779] Based on the analysis, the server generates a customized health improvement program for each user, including exercise routines, meal plans, and meditation programs to reduce stress.

[0780] Input: Analysis results

[0781] Output: A customized health improvement program

[0782] Step 7:

[0783] Health Improvement Program Submissions and Notifications:

[0784] The server then sends the generated health improvement program to the user's device and notifies them, and the user can check the proposed program through the application.

[0785] Input: Generated health improvement program

[0786] Output: Health improvement programs and notifications sent to the user's device

[0787] Step 8:

[0788] Collecting implementation conditions:

[0789] The device collects the user's activity data at the fitness gym or wellness center (for example, number of training sessions, machine usage history, calories burned, etc.) and stores this data in a database.

[0790] Input: Activity data at fitness gyms and wellness centers

[0791] Output: Activity data stored in a database

[0792] Step 9:

[0793] Feedback and performance recording:

[0794] The device tracks and records feedback and progress as users implement the suggestions, and this data is sent back to the server.

[0795] Input: User feedback and execution status

[0796] Output: Feedback and execution status data sent to the server

[0797] Step 10:

[0798] Program updates:

[0799] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates it as necessary, placing particular emphasis on emotional data obtained through the Emotion Engine.

[0800] Input: New data and existing health improvement programs

[0801] Output: Updated Health Improvement Program

[0802] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0803] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0805] [Third embodiment]

[0806] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0807] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0808] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0809] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0810] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0811] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0812] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0813] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0814] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0815] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0816] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0817] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0818] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[0819] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[0820] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital information such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[0821] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[0822] The server analyzes the data using multimodal generation AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status, and analyzes email content using natural language processing to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0823] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[0824] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[0825] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If it is effective, it will suggest a new exercise or diet program as the next step, and if it is not, it will adopt a different approach. This continuous feedback and program updates allows users to receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[0826] As a specific example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[0827] In this way, the present invention is a system for achieving effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[0828] The processing flow will be explained below.

[0829] Step 1:

[0830] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[0831] Step 2:

[0832] The device collects basic information entered by the user and completes the initial setup, which is then sent to the server according to a security protocol.

[0833] Step 3:

[0834] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to collect vital data such as steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[0835] Step 4:

[0836] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[0837] Step 5:

[0838] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[0839] Step 6:

[0840] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it classifies images and videos obtained from social media using a video analysis algorithm to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[0841] Step 7:

[0842] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, meal recipes, exercise programs, etc.) from the web and integrates it with individual data. Specific health programs, such as exercise menus, meal plans, and stress reduction programs, are created.

[0843] Step 8:

[0844] The server sends the generated health improvement program to the user's device and notifies them. New suggestions are provided to the user via push notifications, in-app messages, etc.

[0845] Step 9:

[0846] The user receives a notification and can review the detailed recommendations within the app, where they can choose to implement the suggested exercise and meal plan.

[0847] Step 10:

[0848] The device records the user's feedback and performance in real time and sends it to the server, such as the distance walked, calories burned, and photos of meals.

[0849] Step 11:

[0850] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If the user is close to achieving their goal, it will suggest new exercise and meal plans. If no results are seen, it will suggest a different approach to help the user improve their health.

[0851] Step 12:

[0852] The server generates a new optimized program, sends it back to the user's device, and notifies them, allowing the user to continuously receive and implement the optimal health improvement program.

[0853] Example 1

[0854] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0855] In modern society, there is a demand for health management methods customized for each individual user, but many systems rely on a single data source and limited data analysis, and do not adequately address the diverse lifestyles and health conditions of users. This problem stems from a lack of integrated data collection and analysis technology, as well as a lack of automated individual program generation.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0857] In this invention, the server includes means for collecting basic information from a user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for comprehensively analyzing multiple data formats including social media posts and email content to grasp the user's health status and lifestyle trends, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording user feedback and implementation status, and means for acquiring new data based on the analysis results and updating the health improvement program. This makes it possible to provide a comprehensive, automated, and individualized health management program for users' diverse lifestyles and health conditions.

[0858] "User" refers to a user of this system who provides information about personal health conditions and lifestyle habits.

[0859] "Basic information" refers to comprehensive information such as the user's name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[0860] "Daily life data" refers to data including vital information such as the user's number of steps, heart rate, and sleep patterns, as well as social media posts (images, videos, text) and email content (text, attachments).

[0861] "Real-time collection means" refers to systems that use smartphone sensors and other devices to instantly record data from users' daily lives.

[0862] "Centralized database" refers to a database that stores and manages all collected user data in an integrated manner.

[0863] "Multimodal generative AI" refers to artificial intelligence that comprehensively analyzes different types of data, such as text, images, and sensor data.

[0864] "Analysis results" refers to information about the user's health status and lifestyle trends derived from collected data by the multimodal generation AI.

[0865] A "health improvement program" refers to a plan that includes suggestions for exercise menus, meal plans, stress reduction programs, etc., customized for each user based on the analysis results.

[0866] "Means for notifying" refers to a system that has the function of sending the generated health improvement program to the user's terminal and notifying the user.

[0867] "Feedback" refers to information that a user provides to the system, such as their thoughts on the proposed health improvement program and the progress of its implementation.

[0868] "Means for acquiring and updating new data" refers to a system that has the ability to re-analyze the acquired new data and update the health improvement program to the latest version.

[0869] The present invention is a system that aggregates personal information collected from a user's daily life and general health-related information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[0870] First, users install the system's application on their device and create an account. During the registration process, they enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This basic information will serve as an important foundation for subsequent data analysis.

[0871] Next, the device collects real-time data about the user's daily life. Using smartphone sensors, it records vital information such as the number of steps taken, heart rate, and sleep patterns. This data also includes social media posts (images, videos, text) and email content (text and attachments). For example, if a user posts a photo of their meal on a social media account, the image will also be collected.

[0872] The collected data is sent from the device to a server at regular intervals. The data is sent via the Internet, and the server stores the data in a centralized database. This database has the role of integrating and managing all user information.

[0873] When the server detects a data update, it begins analyzing the data using multimodal generation AI. The AI ​​comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it analyzes images and videos obtained from social media to identify dietary details and exercise status. It also analyzes email content using natural language processing to evaluate stress levels and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0874] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information from the web and integrating it with personal data. For example, if a user's steps are low and they eat a lot of fatty foods, the server generates a plan for 30 minutes of walking per day and a low-calorie diet. The generated program is then sent from the server to the user's device, where a notification is sent.

[0875] The user receives a notification, can confirm the suggestions within the app, and then implement them. The device records the user's feedback and progress, and sends them back to the server. This allows the server to periodically analyze new data and evaluate the effectiveness of the health improvement program. If an effect is observed, a new exercise or diet program will be suggested as the next step; if not, a different approach will be adopted, enabling sustainable health improvement.

[0876] As an example of a specific prompt, if the result for a middle-aged man (42 years old, mostly desk-based) is "eating a lot of fatty foods," the suggestion would be something like, "We suggest a plan of 30 minutes of walking per day and a low-calorie diet."

[0877] In this way, the present invention is a system that realizes effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[0878] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0879] Processing flow

[0880] Step 1: User Registration

[0881] Step 2: Collecting daily data

[0882] Step 3: Send data

[0883] Step 4: Centralize your data

[0884] Step 5: Data analysis

[0885] Step 6: Generate a health improvement program

[0886] Step 7: Notification and Execution

[0887] Step 8: Feedback and Updates

[0888] Explanation of each processing step

[0889] Step 1: User Registration

[0890] Input: User's basic information (name, age, gender, health history, current health condition, lifestyle habits, dietary habits)

[0891] How it works: A user installs the app and enters their basic information on the account creation screen.

[0892] Output: The user's basic information is sent to the server and stored in a database.

[0893] Specific operation: The user downloads the app and follows the on-screen instructions to enter the required information, such as name and age.

[0894] Step 2: Collecting daily data

[0895] Input: Vital information and social media posts from users' daily lives

[0896] How it works: The device collects real-time data on steps taken, heart rate, sleep patterns, and social media posts (images, videos, text) through smartphone sensors and apps.

[0897] Output: The collected daily data is temporarily stored in the device.

[0898] How it works: When a user carries their smartphone, the pedometer and heart rate sensor collect data, and social media posts and emails are also recorded in the background.

[0899] Step 3: Send data

[0900] Input: Daily data collected on the device

[0901] Operation: The device sends the collected data to the server at regular intervals.

[0902] Output: Collected data is sent to the server.

[0903] Specific operation: For example, batch process data every hour and upload it to the server.

[0904] Step 4: Centralize your data

[0905] Input: Daily data and basic information sent to the server

[0906] How it works: The server stores the received data in a centralized database.

[0907] Output: All user data is stored in a centralized database.

[0908] Specific operation: The server automatically integrates newly received data into the existing database.

[0909] Step 5: Data analysis

[0910] Input: User data stored in a centralized database

[0911] How it works: The server uses multimodal generative AI to analyze the collected data. The AI ​​performs text analysis, image recognition, and time-series analysis of sensor data to understand the user's health status and lifestyle trends.

[0912] Output: Analysis results on each user's health status and lifestyle habits are obtained.

[0913] Specific operation: Data is input into the AI ​​model, and image analysis of meal contents from social media posts is performed, and emotional states are analyzed from text data using NLP technology.

[0914] Step 6: Generate a health improvement program

[0915] Input: Analysis results from data analysis

[0916] How it works: Based on the analysis results, the server collects the latest health information from the web and generates a health improvement program customized for each user.

[0917] Output: A personalized health improvement program for each user.

[0918] Specific actions: If the analysis results show that you eat a lot of fatty foods, the suggested program will include "30 minutes of walking per day" and a "low-calorie diet plan."

[0919] Step 7: Notification and Execution

[0920] Input: Generated health improvement program

[0921] Operation: The server sends the generated health improvement program to the user's device and notifies them.

[0922] Output: A health improvement program notification will be displayed on the user's device.

[0923] What it does: The app sends suggestions to the user's device via push notification, and the user can tap the notification to see more details.

[0924] Step 8: Feedback and Updates

[0925] Input: User performance and feedback data

[0926] How it works: The device records the user's progress and feedback and periodically sends it to the server, which retrieves new data, evaluates the program's effectiveness, and updates the health improvement program as needed.

[0927] Output: Updated Health Improvement Program

[0928] Specific actions: When the user implements the suggestions and enters their progress and feedback into the app, the data is sent to the server. Based on this, updates such as "increase walking distance" or "suggest a new approach to stress reduction" are made.

[0929] Through the above processing steps, the present system can provide a health management program customized for each user, thereby achieving effective lifestyle improvements.

[0930] (Application example 1)

[0931] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0932] In modern society, there is a demand for systems that provide lifestyle improvement programs tailored to each individual's health condition. However, existing systems have limitations in efficiently collecting and appropriately analyzing data from individuals' daily lives to propose customized health plans, and they often lack specific lifestyle suggestions in physical stores. As a result, users are unable to receive real-time suggestions for exercise and diet appropriate to their health condition, making it difficult to continuously improve their health.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0934] In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording the user's feedback and implementation status, means for analyzing new data and updating the program, and means for an assistant device in a physical store to make health-related suggestions to the user, thereby enabling the user to receive specific and actionable lifestyle improvement suggestions tailored to their health condition in real time.

[0935] "User" refers to an individual who uses the System to receive a health improvement program.

[0936] "Basic information" includes the user's name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.

[0937] "Daily life data" refers to information collected about a user's daily activities and vital signs, such as the number of steps taken, heart rate, sleep patterns, diet, and exercise status.

[0938] "Real-time" refers to the collection and processing of data immediately, with little or no delay.

[0939] "Centralized database" refers to a database that centrally manages and stores all information collected from users.

[0940] "Multimodal generative AI" refers to AI technology that comprehensively analyzes multiple data formats (text, images, sensor data, etc.) and generates information.

[0941] A "customized health improvement program" is a program that includes suggestions for exercise menus, meal plans, stress reduction programs, and other items optimized for each individual user based on their data.

[0942] "Device" refers to a smartphone, tablet, or other electronic device used by a User.

[0943] "Notification" refers to the act of informing a user's device about a health improvement program or changes thereto.

[0944] "Feedback" refers to information returned to the system about the results and impressions of the health improvement program the user has implemented.

[0945] "Implementation status" is data that indicates to what extent a user is actually implementing a health improvement program.

[0946] "New data" refers to updated data about the user's daily life that is added later to data that has already been collected, analyzed, and stored.

[0947] An "in-store assistant device" is a device that is installed in a store or that can be carried by a user and that provides health-related suggestions to the user.

[0948] The present invention is a system that uses multimodal generation AI to automatically generate a lifestyle improvement program by integrating personal information collected from the user's daily life with general health-related information. Specific embodiments of the present invention are described below.

[0949] First, the user installs the system's application on their device and performs initial registration. They enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and dietary habits. This basic information is important for the user to receive the health improvement program accurately and effectively.

[0950] Secondly, the device uses smartphone sensors and other devices to collect real-time data about the user's daily life, including the number of steps taken, heart rate, sleep patterns, and even social media posts (images, videos, text) and email content (text and attachments). Data collection is essential to gain a detailed understanding of the user's lifestyle.

[0951] The collected data is periodically sent to a server, which stores it in a centralized database, allowing all user information to be managed in an integrated manner and used for analysis.

[0952] The server analyzes the data using multimodal generative AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status. It also uses natural language processing to analyze email content and evaluate the user's stress level and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[0953] Based on the analysis, the server generates a personalized health improvement program for each user, a process that involves automatically gathering current health information from the web and integrating it with individual data, such as exercise routines, meal plans, and stress-reducing meditation programs.

[0954] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user can receive the notification, check the suggestions within the app, and implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server. The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and makes new suggestions as necessary. Through this continuous feedback and program updates, users receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[0955] As a concrete example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[0956] Additionally, by using an assistant device installed in a physical store, users can instantly receive health-related suggestions while shopping in the store. For example, a user shopping in a supermarket can be offered low-calorie food suggestions and products that correspond to appropriate exercise levels.

[0957] The hardware used includes smartphone sensors (accelerometer, heart rate sensor, etc.) and assistant devices installed in physical stores, while the software used includes data collection and analysis modules written in Python, natural language processing libraries (spacy and transformers), and databases (MySQL, etc.).

[0958] An example prompt is, "Generate a customized lifestyle improvement program based on the following user's daily life data: steps: 4500, heart rate: 75, calorie intake: 2100. Suggest an appropriate exercise plan and meal plan."

[0959] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0960] Step 1:

[0961] The user installs the system application on their device and enters basic information. Specifically, the user enters information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Once this information is collected, the device sends it to the server. Input data: name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Output data: initial registration information.

[0962] Step 2:

[0963] The device collects data on the user's daily life in real time. This includes the number of steps taken, heart rate, sleep patterns, as well as social media posts (images, videos, text) and email content collected using smartphone sensors. The collected data is sent to a server at regular intervals. Input data: number of steps taken, heart rate, sleep patterns, social media posts, email content. Output data: daily life data.

[0964] Step 3:

[0965] The server stores the collected daily life data in a centralized database. The stored data is managed for each user and used for subsequent analysis and generation. Input data: Daily life data. Output data: Data stored in the centralized database.

[0966] Step 4:

[0967] The server analyzes the collected data using multimodal generative AI. Specific analysis operations include identifying dietary content and exercise status through image and video analysis of social media, and evaluating stress levels and emotional state through natural language processing analysis of emails. Data obtained from smartphone sensors is also analyzed over time to determine exercise volume and sleep patterns. Input data: Data from a centralized database. Output data: Analysis results.

[0968] Step 5:

[0969] Based on the analysis results, the server generates a health improvement program customized for each user. This program is generated by automatically collecting the latest health information from the web and integrating it with individual data. Specific actions include creating an exercise menu, meal plan, and meditation program for stress reduction. Input data: analysis results, health information from the web. Output data: customized health improvement program.

[0970] Step 6:

[0971] The server sends the generated health improvement program to the user's device, and the device notifies the user. The user checks the suggestions within the app and implements them. Input data: Customized health improvement program. Output data: Notification sent to the user's device.

[0972] Step 7:

[0973] The device records the user's feedback and execution status. It collects in real time how well the user has implemented the program's suggestions, as well as their thoughts on the suggestions and requests for improvement, and sends this to the server. Input data: user feedback, execution status. Output data: recorded feedback and execution status data.

[0974] Step 8:

[0975] The server analyzes the new data and updates the health improvement program, ensuring that users always receive the latest and most optimized health improvement suggestions. Input data: New daily life data, feedback. Output data: Updated health improvement program.

[0976] Step 9:

[0977] An assistant device in a physical store makes health-related suggestions to users. For example, while shopping at a supermarket, it may suggest low-calorie foods or provide product information based on the amount of exercise. Input data: User's health status, current shopping details. Output data: Health-related suggestions, in-store guidance.

[0978] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0979] The present invention relates to a system that aggregates personal information collected from a user's daily life with general health-related information, and combines this information with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[0980] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[0981] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[0982] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[0983] The system also includes an emotion engine that recognizes users' emotions in real time. The emotion engine has the ability to evaluate their emotional state and stress level when analyzing users' social media posts, email content, and even sensor data.

[0984] The server uses multimodal generation AI to analyze data triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition, lifestyle habits, and emotional trends. Specifically, it uses a video analysis algorithm to classify images and videos obtained from social media, identify dietary habits and exercise status, and analyzes email content using natural language processing (NLP) to evaluate stress levels and emotional state. It also analyzes emotional data obtained through an emotion engine to accurately understand the user's psychological state.

[0985] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[0986] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[0987] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the most appropriate program. If an effect is observed, the next step is to suggest a new exercise menu or meal plan. If no effect is observed, a different approach is adopted to support the user's health improvement.

[0988] As a specific example, let's consider the use of an emotion engine. In the case of a middle-aged man (42 years old, mainly desk-bound), in addition to entering basic information, the device collects the user's daily step count, heart rate, and food photos from social media. The system then uses the emotion engine to analyze the user's stress level and emotional state based on the content of their social media posts and emails. The server comprehensively analyzes this data and determines that the user's step count is low, their diet is high, and their stress level is high. The server then recommends 30 minutes of walking per day, a low-calorie diet, and a meditation program to reduce stress, and notifies the user's device. If the user accepts and implements the suggestions, overall health improvement is achieved.

[0989] In this way, the present invention is a system that realizes effective and sustainable lifestyle improvements by analyzing each user's data in detail and providing an individually optimized health improvement program. Furthermore, by combining it with an emotion engine, it is possible to accurately grasp the user's emotional state and make more personalized suggestions.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[0993] Step 2:

[0994] The terminal collects basic information entered by the user and transmits it to the server according to a security protocol.

[0995] Step 3:

[0996] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[0997] Step 4:

[0998] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[0999] Step 5:

[1000] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[1001] Step 6:

[1002] The device uses an emotion engine to analyze the user's emotional data in real time from social media posts and email content.

[1003] Step 7:

[1004] The terminal transmits the emotion data acquired from the emotion engine to the server.

[1005] Step 8:

[1006] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it analyzes images and videos obtained from social media to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[1007] Step 9:

[1008] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, dietary recipes, exercise programs, etc.) from the web and integrates it with individual data. Specifically, it creates exercise menus, meal plans, stress reduction programs, etc.

[1009] Step 10:

[1010] The server sends the generated health improvement program to the user's device and notifies them, and new suggestions are provided to the user via push notifications and in-app messages.

[1011] Step 11:

[1012] The user receives a notification and sees detailed suggestions within the app, and chooses to follow the suggested exercise and meal plan.

[1013] Step 12:

[1014] The device records the user's feedback and performance status in real time and sends it to the server, such as the distance walked, calories burned, and photos of meal contents.

[1015] Step 13:

[1016] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the optimal program. If the user is close to achieving their goal, it will suggest a new exercise menu or meal plan. If no effect is seen, it will suggest a different approach to help the user improve their health.

[1017] Step 14:

[1018] The server generates a new optimized program, sends it back to the user's device, and notifies the user, allowing the user to continuously receive and implement the optimal health improvement program.

[1019] Example 2

[1020] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1021] Conventional health management systems have struggled to comprehensively collect and analyze users' daily life and emotional data to provide optimal health improvement programs for each individual user. As a result, they were unable to provide effective programs that fully considered the user's actual health and psychological state, limiting the effectiveness of health improvement. Furthermore, the lack of effective analysis of collected data and feedback utilization made it difficult to continuously update and optimize programs.

[1022] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting basic information from the user; means for collecting data on the user's daily life in real time; means for storing the collected data in a centralized database; analysis means using a generative model to analyze the collected data; means for generating a health improvement program customized for each user based on the analysis results; means for transmitting and notifying the user of the generated health improvement program to the user's information terminal; means for recording the user's feedback and execution status; means for analyzing new data and updating the program; analysis means including an emotion engine that analyzes emotion data in real time and evaluates the user's psychological state; and means for further personalizing the program using the emotion data. This makes it possible to comprehensively analyze the user's health and psychological states, provide an individually optimized health improvement program, and continuously update it.

[1023] "User" means an individual who uses the system to improve their own health.

[1024] "Basic information" refers to information about the user, such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[1025] "Daily life data" refers to data related to a user's daily activities and health, including the number of steps taken, heart rate, sleep patterns, social media posts, and email content.

[1026] "Collection methods" refers to the methods and technologies used to collect basic information and daily life data from users.

[1027] A "centralized management database" is a database for managing and storing collected data in an integrated manner.

[1028] "Generative model" refers to artificial intelligence (AI) technology that analyzes multiple data formats (text, images, sensor data, etc.).

[1029] "Analysis means" refers to methods and techniques for analyzing collected data using generative models to assess the user's health and emotional state.

[1030] A "customized health improvement program" is a program that is individually created for each user based on data analysis, and includes exercise menus, meal plans, stress reduction programs, etc.

[1031] "Information terminal" refers to devices such as smartphones and tablets used by users.

[1032] "Notification means" refers to a method or technology for transmitting the generated health improvement program to the user's information terminal and notifying the user.

[1033] "Feedback" refers to response data such as progress and impressions when a user follows a health improvement program.

[1034] "Emotion Engine" refers to artificial intelligence technology that assesses a user's emotional state and stress level from data from their daily life.

[1035] "Personalization" refers to providing suggestions and services that are optimized for each user's individual needs and circumstances.

[1036] "Analysis results" refers to the analysis results of data processed by the generative model or emotion engine.

[1037] "Update tools" refer to methods and techniques for continually reviewing and optimizing health improvement programs based on new data collected.

[1038] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health-related information, and combines this with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[1039] Hardware and Software Configuration

[1040] 1. User Device

[1041] Users use smartphones or tablets as devices. These devices have built-in sensors (pedometers, heart rate sensors, accelerometers, etc.) that collect vital data. The content of user posts and emails is also collected through social networking applications and email applications.

[1042] 2. Server

[1043] The server is equipped with a database system (e.g., MySQL, MongoDB) and a generative model (e.g., OpenAI GPT-4, Google BERT), as well as an emotion engine for data analysis.

[1044] 3. Software

[1045] The application software of this system manages a series of processes, from creating a health improvement program to notifying the results.

[1046] Program processing explanation

[1047] 1. User Registration

[1048] Users first install the system's app on their device and enter basic information (name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.) Once this registration is complete, the information is sent from the device to the server and stored in a centralized database.

[1049] 2. Data Collection

[1050] The device uses built-in sensors to collect vital data such as the number of steps taken, heart rate, and sleep patterns in real time. Furthermore, the device also periodically collects users' social media posts (images, videos, text) and email content (text and attachments). This data is then sent to a server at regular intervals.

[1051] 3. Data analysis and emotion evaluation

[1052] The server stores the received data in a centralized database. It then uses an emotion engine to evaluate the user's emotional state and stress level based on social media posts, email content, and sensor data. It uses a generative model to comprehensively analyze multiple data formats (text, images, and sensor data) to understand the user's health, lifestyle, and emotional trends.

[1053] 4. Generating health improvement programs

[1054] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information and integrating it with personal data. Specific examples of programs that can be generated include exercise routines, meal plans, and meditation programs for stress reduction.

[1055] 5. Program delivery and feedback utilization

[1056] The generated health improvement program is sent from the server to the user's device and notified. The user receives the notification and can review and implement the suggestions within the app. The device records the user's feedback and implementation status in real time and sends it back to the server. The server periodically analyzes the new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary.

[1057] Specific examples

[1058] For example, in the case of a middle-aged man (age 42, mostly desk-bound), the following process is performed:

[1059] 1. Registration: Enter basic information.

[1060] 2. Data collection: Step count, heart rate, and food photos from social media.

[1061] 3. Emotion assessment: Analyzes stress levels and emotional states from social media posts and email content through an emotion engine.

[1062] 4. Data analysis: It was confirmed that the number of steps was low, the amount of fatty food was high, and stress levels were high.

[1063] 5. Program generation: Suggests a 30-minute walk per day, a low-calorie diet, and a meditation program to reduce stress.

[1064] 6. Program distribution: Notification to user's device.

[1065] Example prompts to be input to the generative AI model:

[1066] Create a health improvement program for a 42-year-old man who works a desk job. His daily data shows that he walks few steps, eats a lot of fatty foods, and has high stress levels based on his social media and email activity.

[1067] In this way, the present invention aims to improve the user's health condition efficiently and sustainably by providing an individually optimized health improvement program based on detailed data analysis.

[1068] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1069] Step 1:

[1070] The user installs the app and enters basic information. The user registers their name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc. in the app. This information becomes input data and is sent from the device to the server. The server receives this basic information and stores it in a centralized database. This saves the user's basic information in the database and is used in subsequent analysis processes.

[1071] Step 2:

[1072] The device collects data about the user's daily life in real time. As the user goes about their daily life, the device uses its built-in sensors to acquire vital data such as the number of steps taken, heart rate, and sleep patterns. Images, videos, text, and emails posted by the user on social media are also periodically collected. This collected data becomes input data for the device. The device then sends the collected data to a server. The server stores the received data in a centralized database. This allows the user's daily life data to be accumulated in the database.

[1073] Step 3:

[1074] The server analyzes the user's emotional state using an emotion engine. The server receives input data from social media posts, email content, and sensor data stored in a database, and analyzes it using the emotion engine. The emotion engine analyzes text and sensor data to evaluate stress levels and emotional states. The analysis results in data on the user's emotional state. This data is used to generate subsequent health improvement programs.

[1075] Step 4:

[1076] The server performs data analysis using multimodal generative AI. The server comprehensively retrieves multiple data formats, such as vital signs, social media posts, and email content, from the database and inputs them into the generative AI model. The generative AI model analyzes this data to understand the user's health status, lifestyle habits, and emotional trends. As an output, health status assessment data for the user is generated. This assessment data is used to customize health improvement programs.

[1077] Step 5:

[1078] The server generates a health improvement program based on the analysis results. The server uses the user's health status assessment data obtained from the generative AI model as input data, collects and integrates the latest health information, and generates an individually optimized health improvement program. The generated program specifically includes an exercise menu, meal plan, stress reduction program, etc. These programs are customized for each user.

[1079] Step 6:

[1080] The server sends the generated health improvement program to the user's device and notifies them. The server then sends the generated health improvement program to the device, which then notifies the user. The user receives the notification and checks the contents of the proposed program within the app. The effects of the improvement program are realized when the user implements the proposed health improvement program.

[1081] Step 7:

[1082] The user provides feedback, and the device records the execution status. The user runs the proposed program and enters the execution status and impressions within the app. The device collects this feedback in real time and sends the feedback data to a server. The server stores the received feedback data in a database and uses it for analysis.

[1083] Step 8:

[1084] The server analyzes the new data and updates the health improvement program. The server then uses the feedback data and newly collected daily life data to perform another analysis using the generative AI model. Based on the analysis results, the effectiveness of the health improvement program is evaluated and the program is optimized and updated as necessary. This enables continuous health improvement tailored to the user's condition.

[1085] (Application example 2)

[1086] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1087] In modern fitness gyms and wellness centers, it is extremely important to accurately understand the health and emotional state of each user and provide customized health improvement programs based on that information. However, a system that effectively collects and analyzes users' daily life data and emotional state and provides specific health improvement suggestions in real time has not yet been realized. This poses a challenge in providing continuous support for users' health improvement and receiving appropriate feedback.

[1088] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending and notifying the user of the generated health improvement program, means for collecting the user's implementation conditions and storing them in a database as a record of exercise facility use, means for recording the user's feedback and implementation status, and means for analyzing new data and updating the program. This makes it possible to comprehensively analyze the health and emotional state of each user and continuously provide an optimized health improvement program.

[1089] "Means for collecting basic information from users" refers to devices or methods for obtaining basic information such as the user's name, age, sex, health history, current health condition, lifestyle habits, and eating habits.

[1090] "Means for collecting data on a user's daily life in real time" refers to devices or methods for acquiring vital data such as the number of steps taken by a user, heart rate, and sleep patterns in real time.

[1091] "Means for storing collected data in a unified database" refers to a device or method for storing and managing all collected data in a unified database.

[1092] "Analysis methods using multimodal generation AI to analyze collected data" refers to artificial intelligence devices and methods that comprehensively analyze multiple data formats, such as text data, image data, and sensor data.

[1093] "Means for generating a health improvement program customized for each user based on the analysis results" refers to a device or method that uses analyzed data to automatically generate a health improvement plan optimized for each individual user.

[1094] The "means for transmitting the generated health improvement program to the user's terminal and notifying the user" refers to a device or method for transmitting the generated health improvement program to the user's terminal in real time and notifying the user.

[1095] "Means for collecting user activity conditions and storing them in a database as a record of use of the exercise facility" refers to a device or method for collecting data on the user's activity at the exercise facility and storing it in a database as a record of use.

[1096] "Means for recording user feedback and performance" refers to a device or method for tracking and recording user feedback and performance of a health improvement program.

[1097] "Means for analyzing new data and updating the program" refers to devices and methods that use the latest data for analysis and continuously update and improve the health improvement program.

[1098] "Means for analyzing a user's emotional state" refers to devices or methods for analyzing a user's emotions and stress levels from the content of their social media posts and emails.

[1099] This invention relates to a system that aggregates personal information collected from users' daily lives with general health-related information, and combines it with an emotion engine to generate more accurate health improvement programs for users of fitness gyms and wellness centers.

[1100] System configuration:

[1101] This system consists of the following main components:

[1102] 1. How we collect basic information from you:

[1103] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into a dedicated application.

[1104] 2. Means of collecting real-time data on users' daily lives:

[1105] It uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns.

[1106] 3. How the collected data is stored in a centralized database:

[1107] The collected data is sent from the terminal to a server at regular intervals, and the server stores it in a centralized database.

[1108] 4. Analysis method using multimodal generative AI to analyze collected data:

[1109] The server uses multimodal generative AI to comprehensively analyze the collected text data, image data, and sensor data to understand the user's health condition, lifestyle habits, and emotional trends. At this time, it uses an Emotion Engine to analyze emotions and stress levels from social media posts and email content.

[1110] 5. Means for generating a health improvement program customized for each user based on the analysis results:

[1111] Based on the analysis results, the system generates a health improvement program optimized for each user's individual condition, including exercise routines, meal plans, and meditation programs for stress reduction.

[1112] 6. Means for sending the generated health improvement program to the user's device and notifying them:

[1113] The generated health improvement program is sent to the user's device, where they can view the suggestions within the app.

[1114] 7. Means for collecting user's performance conditions and storing them in a database as a record of use of the sports facility:

[1115] The system collects data on users' activities at fitness gyms and wellness centers (e.g., number of workouts, machine usage history, calories burned, etc.) and stores it in a database.

[1116] 8. Means of recording user feedback and performance:

[1117] When users implement the suggestions, feedback and progress are tracked and recorded, and this data is also sent to the server.

[1118] 9. How to analyze new data and update the program:

[1119] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary. It places particular emphasis on emotional data obtained through the emotion engine, and continually suggests optimal programs that reflect the user's psychological state.

[1120] Examples of hardware and software:

[1121] Smartphone Sensors:

[1122] Collect data about your daily life (step count, heart rate, sleep patterns, etc.).

[1123] Emotion Engine:

[1124] A library for emotion analysis that analyzes stress levels and emotional states from social media posts and email content.

[1125] Health Analyzer:

[1126] A library that analyzes user health data and generates health improvement programs.

[1127] Examples and prompts:

[1128] For example, if a male user named Tanaka visits a fitness gym, the following programs will be offered to him:

[1129] User information: Name: Tanaka, Age: 42, Gender: Male

[1130] Health history: high blood pressure, diabetes

[1131] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[1132] Social media post: I'm tired again today...

[1133] Email content: I feel like I've been feeling a lot of stress at work lately.

[1134] Based on this, the following health improvement program is generated:

[1135] Generate a health improvement program based on the following criteria:

[1136] User information: Name: Tanaka, Age: 42, Gender: Male, Health history: High blood pressure, diabetes

[1137] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[1138] Emotional Data:

[1139] Social media post: I'm tired again today...

[1140] Email content: I feel like I've been feeling a lot of stress at work lately.

[1141] In this way, the optimal program is provided based on the user's health and emotional state, enabling continuous improvement in health.

[1142] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1143] Step 1:

[1144] User Information Collection:

[1145] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into the application, and the device then collects the user's basic information and sends it to the server.

[1146] Input: Basic information from the user

[1147] Output: Basic information stored on the server

[1148] Step 2:

[1149] Real-time collection of daily life data:

[1150] The device uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns, and transmits the collected data to a server at regular intervals.

[1151] Input: Vital data collected from smartphone sensors

[1152] Output: Vital data sent to the server

[1153] Step 3:

[1154] Centralize your data:

[1155] The server receives the basic information and vital data sent from the device and stores them in a centralized database, which manages all user information in an integrated manner.

[1156] Input: Basic information and vital data sent from the device

[1157] Output: Data stored in a centralized database

[1158] Step 4:

[1159] Emotion analysis:

[1160] The server uses the Emotion Engine to analyze users' emotional state and stress levels from social media posts and email content, using text analysis techniques to generate emotion data.

[1161] Input: User's SNS posts and email content

[1162] Output: Emotional state and stress level data

[1163] Step 5:

[1164] Data Analysis:

[1165] The server uses multimodal generative AI to comprehensively analyze basic information, vital data, and emotional data, thereby understanding the user's health condition, lifestyle habits, and emotional tendencies.

[1166] Input: Basic information, vital data, emotional data

[1167] Output: Analysis results of the user's health status, lifestyle habits, and emotional tendencies

[1168] Step 6:

[1169] Generating Health Improvement Programs:

[1170] Based on the analysis, the server generates a customized health improvement program for each user, including exercise routines, meal plans, and meditation programs to reduce stress.

[1171] Input: Analysis results

[1172] Output: A customized health improvement program

[1173] Step 7:

[1174] Health Improvement Program Submissions and Notifications:

[1175] The server then sends the generated health improvement program to the user's device and notifies them, and the user can check the proposed program through the application.

[1176] Input: Generated health improvement program

[1177] Output: Health improvement programs and notifications sent to the user's device

[1178] Step 8:

[1179] Collecting implementation conditions:

[1180] The device collects the user's activity data at the fitness gym or wellness center (for example, number of training sessions, machine usage history, calories burned, etc.) and stores this data in a database.

[1181] Input: Activity data at fitness gyms and wellness centers

[1182] Output: Activity data stored in a database

[1183] Step 9:

[1184] Feedback and performance recording:

[1185] The device tracks and records feedback and progress as users implement the suggestions, and this data is sent back to the server.

[1186] Input: User feedback and execution status

[1187] Output: Feedback and execution status data sent to the server

[1188] Step 10:

[1189] Program updates:

[1190] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates it as necessary, placing particular emphasis on emotional data obtained through the Emotion Engine.

[1191] Input: New data and existing health improvement programs

[1192] Output: Updated Health Improvement Program

[1193] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1194] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1195] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1196] [Fourth embodiment]

[1197] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1198] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1199] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1200] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1201] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1202] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1203] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1204] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1205] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1206] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1207] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1208] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1209] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1210] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[1211] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[1212] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital information such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[1213] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[1214] The server analyzes the data using multimodal generation AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status, and analyzes email content using natural language processing to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[1215] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[1216] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[1217] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If it is effective, it will suggest a new exercise or diet program as the next step, and if it is not, it will adopt a different approach. This continuous feedback and program updates allows users to receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[1218] As a specific example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[1219] In this way, the present invention is a system for achieving effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[1220] The processing flow will be explained below.

[1221] Step 1:

[1222] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[1223] Step 2:

[1224] The device collects basic information entered by the user and completes the initial setup, which is then sent to the server according to a security protocol.

[1225] Step 3:

[1226] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to collect vital data such as steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[1227] Step 4:

[1228] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[1229] Step 5:

[1230] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[1231] Step 6:

[1232] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it classifies images and videos obtained from social media using a video analysis algorithm to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[1233] Step 7:

[1234] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, meal recipes, exercise programs, etc.) from the web and integrates it with individual data. Specific health programs, such as exercise menus, meal plans, and stress reduction programs, are created.

[1235] Step 8:

[1236] The server sends the generated health improvement program to the user's device and notifies them. New suggestions are provided to the user via push notifications, in-app messages, etc.

[1237] Step 9:

[1238] The user receives a notification and can review the detailed recommendations within the app, where they can choose to implement the suggested exercise and meal plan.

[1239] Step 10:

[1240] The device records the user's feedback and performance in real time and sends it to the server, such as the distance walked, calories burned, and photos of meals.

[1241] Step 11:

[1242] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. If the user is close to achieving their goal, it will suggest new exercise and meal plans. If no results are seen, it will suggest a different approach to help the user improve their health.

[1243] Step 12:

[1244] The server generates a new optimized program, sends it back to the user's device, and notifies them, allowing the user to continuously receive and implement the optimal health improvement program.

[1245] Example 1

[1246] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1247] In modern society, there is a demand for health management methods customized for each individual user, but many systems rely on a single data source and limited data analysis, and do not adequately address the diverse lifestyles and health conditions of users. This problem stems from a lack of integrated data collection and analysis technology, as well as a lack of automated individual program generation.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1249] In this invention, the server includes means for collecting basic information from a user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for comprehensively analyzing multiple data formats including social media posts and email content to grasp the user's health status and lifestyle trends, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording user feedback and implementation status, and means for acquiring new data based on the analysis results and updating the health improvement program. This makes it possible to provide a comprehensive, automated, and individualized health management program for users' diverse lifestyles and health conditions.

[1250] "User" refers to a user of this system who provides information about personal health conditions and lifestyle habits.

[1251] "Basic information" refers to comprehensive information such as the user's name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[1252] "Daily life data" refers to data including vital information such as the user's number of steps, heart rate, and sleep patterns, as well as social media posts (images, videos, text) and email content (text, attachments).

[1253] "Real-time collection means" refers to systems that use smartphone sensors and other devices to instantly record data from users' daily lives.

[1254] "Centralized database" refers to a database that stores and manages all collected user data in an integrated manner.

[1255] "Multimodal generative AI" refers to artificial intelligence that comprehensively analyzes different types of data, such as text, images, and sensor data.

[1256] "Analysis results" refers to information about the user's health status and lifestyle trends derived from collected data by the multimodal generation AI.

[1257] A "health improvement program" refers to a plan that includes suggestions for exercise menus, meal plans, stress reduction programs, etc., customized for each user based on the analysis results.

[1258] "Means for notifying" refers to a system that has the function of sending the generated health improvement program to the user's terminal and notifying the user.

[1259] "Feedback" refers to information that a user provides to the system, such as their thoughts on the proposed health improvement program and the progress of its implementation.

[1260] "Means for acquiring and updating new data" refers to a system that has the ability to re-analyze the acquired new data and update the health improvement program to the latest version.

[1261] The present invention is a system that aggregates personal information collected from a user's daily life and general health-related information, and automatically generates a lifestyle improvement program using multimodal generation AI. Specific embodiments of the present invention are described below.

[1262] First, users install the system's application on their device and create an account. During the registration process, they enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This basic information will serve as an important foundation for subsequent data analysis.

[1263] Next, the device collects real-time data about the user's daily life. Using smartphone sensors, it records vital information such as the number of steps taken, heart rate, and sleep patterns. This data also includes social media posts (images, videos, text) and email content (text and attachments). For example, if a user posts a photo of their meal on a social media account, the image will also be collected.

[1264] The collected data is sent from the device to a server at regular intervals. The data is sent via the Internet, and the server stores the data in a centralized database. This database has the role of integrating and managing all user information.

[1265] When the server detects a data update, it begins analyzing the data using multimodal generation AI. The AI ​​comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it analyzes images and videos obtained from social media to identify dietary details and exercise status. It also analyzes email content using natural language processing to evaluate stress levels and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[1266] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information from the web and integrating it with personal data. For example, if a user's steps are low and they eat a lot of fatty foods, the server generates a plan for 30 minutes of walking per day and a low-calorie diet. The generated program is then sent from the server to the user's device, where a notification is sent.

[1267] The user receives a notification, can confirm the suggestions within the app, and then implement them. The device records the user's feedback and progress, and sends them back to the server. This allows the server to periodically analyze new data and evaluate the effectiveness of the health improvement program. If an effect is observed, a new exercise or diet program will be suggested as the next step; if not, a different approach will be adopted, enabling sustainable health improvement.

[1268] As an example of a specific prompt, if the result for a middle-aged man (42 years old, mostly desk-based) is "eating a lot of fatty foods," the suggestion would be something like, "We suggest a plan of 30 minutes of walking per day and a low-calorie diet."

[1269] In this way, the present invention is a system that realizes effective lifestyle improvements by analyzing data for each user in detail and providing an individually optimized health improvement program.

[1270] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1271] Processing flow

[1272] Step 1: User Registration

[1273] Step 2: Collecting daily data

[1274] Step 3: Send data

[1275] Step 4: Centralize your data

[1276] Step 5: Data analysis

[1277] Step 6: Generate a health improvement program

[1278] Step 7: Notification and Execution

[1279] Step 8: Feedback and Updates

[1280] Explanation of each processing step

[1281] Step 1: User Registration

[1282] Input: User's basic information (name, age, gender, health history, current health condition, lifestyle habits, dietary habits)

[1283] How it works: A user installs the app and enters their basic information on the account creation screen.

[1284] Output: The user's basic information is sent to the server and stored in a database.

[1285] Specific operation: The user downloads the app and follows the on-screen instructions to enter the required information, such as name and age.

[1286] Step 2: Collecting daily data

[1287] Input: Vital information and social media posts from users' daily lives

[1288] How it works: The device collects real-time data on steps taken, heart rate, sleep patterns, and social media posts (images, videos, text) through smartphone sensors and apps.

[1289] Output: The collected daily data is temporarily stored in the device.

[1290] How it works: When a user carries their smartphone, the pedometer and heart rate sensor collect data, and social media posts and emails are also recorded in the background.

[1291] Step 3: Send data

[1292] Input: Daily data collected on the device

[1293] Operation: The device sends the collected data to the server at regular intervals.

[1294] Output: Collected data is sent to the server.

[1295] Specific operation: For example, batch process data every hour and upload it to the server.

[1296] Step 4: Centralize your data

[1297] Input: Daily data and basic information sent to the server

[1298] How it works: The server stores the received data in a centralized database.

[1299] Output: All user data is stored in a centralized database.

[1300] Specific operation: The server automatically integrates newly received data into the existing database.

[1301] Step 5: Data analysis

[1302] Input: User data stored in a centralized database

[1303] How it works: The server uses multimodal generative AI to analyze the collected data. The AI ​​performs text analysis, image recognition, and time-series analysis of sensor data to understand the user's health status and lifestyle trends.

[1304] Output: Analysis results on each user's health status and lifestyle habits are obtained.

[1305] Specific operation: Data is input into the AI ​​model, and image analysis of meal contents from social media posts is performed, and emotional states are analyzed from text data using NLP technology.

[1306] Step 6: Generate a health improvement program

[1307] Input: Analysis results from data analysis

[1308] How it works: Based on the analysis results, the server collects the latest health information from the web and generates a health improvement program customized for each user.

[1309] Output: A personalized health improvement program for each user.

[1310] Specific actions: If the analysis results show that you eat a lot of fatty foods, the suggested program will include "30 minutes of walking per day" and a "low-calorie diet plan."

[1311] Step 7: Notification and Execution

[1312] Input: Generated health improvement program

[1313] Operation: The server sends the generated health improvement program to the user's device and notifies them.

[1314] Output: A health improvement program notification will be displayed on the user's device.

[1315] What it does: The app sends suggestions to the user's device via push notification, and the user can tap the notification to see more details.

[1316] Step 8: Feedback and Updates

[1317] Input: User performance and feedback data

[1318] How it works: The device records the user's progress and feedback and periodically sends it to the server, which retrieves new data, evaluates the program's effectiveness, and updates the health improvement program as needed.

[1319] Output: Updated Health Improvement Program

[1320] Specific actions: When the user implements the suggestions and enters their progress and feedback into the app, the data is sent to the server. Based on this, updates such as "increase walking distance" or "suggest a new approach to stress reduction" are made.

[1321] Through the above processing steps, the present system can provide a health management program customized for each user, thereby achieving effective lifestyle improvements.

[1322] (Application example 1)

[1323] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1324] In modern society, there is a demand for systems that provide lifestyle improvement programs tailored to each individual's health condition. However, existing systems have limitations in efficiently collecting and appropriately analyzing data from individuals' daily lives to propose customized health plans, and they often lack specific lifestyle suggestions in physical stores. As a result, users are unable to receive real-time suggestions for exercise and diet appropriate to their health condition, making it difficult to continuously improve their health.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1326] In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending the generated health improvement program to the user's device and notifying them, means for recording the user's feedback and implementation status, means for analyzing new data and updating the program, and means for an assistant device in a physical store to make health-related suggestions to the user, thereby enabling the user to receive specific and actionable lifestyle improvement suggestions tailored to their health condition in real time.

[1327] "User" refers to an individual who uses the System to receive a health improvement program.

[1328] "Basic information" includes the user's name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.

[1329] "Daily life data" refers to information collected about a user's daily activities and vital signs, such as the number of steps taken, heart rate, sleep patterns, diet, and exercise status.

[1330] "Real-time" refers to the collection and processing of data immediately, with little or no delay.

[1331] "Centralized database" refers to a database that centrally manages and stores all information collected from users.

[1332] "Multimodal generative AI" refers to AI technology that comprehensively analyzes multiple data formats (text, images, sensor data, etc.) and generates information.

[1333] A "customized health improvement program" is a program that includes suggestions for exercise menus, meal plans, stress reduction programs, and other items optimized for each individual user based on their data.

[1334] "Device" refers to a smartphone, tablet, or other electronic device used by a User.

[1335] "Notification" refers to the act of informing a user's device about a health improvement program or changes thereto.

[1336] "Feedback" refers to information returned to the system about the results and impressions of the health improvement program the user has implemented.

[1337] "Implementation status" is data that indicates to what extent a user is actually implementing a health improvement program.

[1338] "New data" refers to updated data about the user's daily life that is added later to data that has already been collected, analyzed, and stored.

[1339] An "in-store assistant device" is a device that is installed in a store or that can be carried by a user and that provides health-related suggestions to the user.

[1340] The present invention is a system that uses multimodal generation AI to automatically generate a lifestyle improvement program by integrating personal information collected from the user's daily life with general health-related information. Specific embodiments of the present invention are described below.

[1341] First, the user installs the system's application on their device and performs initial registration. They enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and dietary habits. This basic information is important for the user to receive the health improvement program accurately and effectively.

[1342] Secondly, the device uses smartphone sensors and other devices to collect real-time data about the user's daily life, including the number of steps taken, heart rate, sleep patterns, and even social media posts (images, videos, text) and email content (text and attachments). Data collection is essential to gain a detailed understanding of the user's lifestyle.

[1343] The collected data is periodically sent to a server, which stores it in a centralized database, allowing all user information to be managed in an integrated manner and used for analysis.

[1344] The server analyzes the data using multimodal generative AI, triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health status and lifestyle trends. Specifically, it performs video analysis of images and videos obtained from social media to identify dietary details and exercise status. It also uses natural language processing to analyze email content and evaluate the user's stress level and emotional state. It also performs time-series analysis of smartphone sensor data to determine exercise volume and sleep patterns.

[1345] Based on the analysis, the server generates a personalized health improvement program for each user, a process that involves automatically gathering current health information from the web and integrating it with individual data, such as exercise routines, meal plans, and stress-reducing meditation programs.

[1346] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user can receive the notification, check the suggestions within the app, and implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server. The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and makes new suggestions as necessary. Through this continuous feedback and program updates, users receive individually optimized health improvement suggestions, enabling them to achieve sustainable health improvements.

[1347] As a concrete example, consider the case of a middle-aged man (42 years old, mainly working at a desk). The user enters basic information, and the device collects daily steps, heart rate, and photos of meals from social media. The server analyzes this data and determines that the user's steps are low and that their meals are high in fat. The server generates a plan for 30 minutes of walking per day and a low-calorie diet and notifies the user's device. If the user accepts and follows the suggestions, health improvements are achieved.

[1348] Additionally, by using an assistant device installed in a physical store, users can instantly receive health-related suggestions while shopping in the store. For example, a user shopping in a supermarket can be offered low-calorie food suggestions and products that correspond to appropriate exercise levels.

[1349] The hardware used includes smartphone sensors (accelerometer, heart rate sensor, etc.) and assistant devices installed in physical stores, while the software used includes data collection and analysis modules written in Python, natural language processing libraries (spacy and transformers), and databases (MySQL, etc.).

[1350] An example prompt is, "Generate a customized lifestyle improvement program based on the following user's daily life data: steps: 4500, heart rate: 75, calorie intake: 2100. Suggest an appropriate exercise plan and meal plan."

[1351] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1352] Step 1:

[1353] The user installs the system application on their device and enters basic information. Specifically, the user enters information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Once this information is collected, the device sends it to the server. Input data: name, age, gender, health history, current health condition, lifestyle habits, and eating habits. Output data: initial registration information.

[1354] Step 2:

[1355] The device collects data on the user's daily life in real time. This includes the number of steps taken, heart rate, sleep patterns, as well as social media posts (images, videos, text) and email content collected using smartphone sensors. The collected data is sent to a server at regular intervals. Input data: number of steps taken, heart rate, sleep patterns, social media posts, email content. Output data: daily life data.

[1356] Step 3:

[1357] The server stores the collected daily life data in a centralized database. The stored data is managed for each user and used for subsequent analysis and generation. Input data: Daily life data. Output data: Data stored in the centralized database.

[1358] Step 4:

[1359] The server analyzes the collected data using multimodal generative AI. Specific analysis operations include identifying dietary content and exercise status through image and video analysis of social media, and evaluating stress levels and emotional state through natural language processing analysis of emails. Data obtained from smartphone sensors is also analyzed over time to determine exercise volume and sleep patterns. Input data: Data from a centralized database. Output data: Analysis results.

[1360] Step 5:

[1361] Based on the analysis results, the server generates a health improvement program customized for each user. This program is generated by automatically collecting the latest health information from the web and integrating it with individual data. Specific actions include creating an exercise menu, meal plan, and meditation program for stress reduction. Input data: analysis results, health information from the web. Output data: customized health improvement program.

[1362] Step 6:

[1363] The server sends the generated health improvement program to the user's device, and the device notifies the user. The user checks the suggestions within the app and implements them. Input data: Customized health improvement program. Output data: Notification sent to the user's device.

[1364] Step 7:

[1365] The device records the user's feedback and execution status. It collects in real time how well the user has implemented the program's suggestions, as well as their thoughts on the suggestions and requests for improvement, and sends this to the server. Input data: user feedback, execution status. Output data: recorded feedback and execution status data.

[1366] Step 8:

[1367] The server analyzes the new data and updates the health improvement program, ensuring that users always receive the latest and most optimized health improvement suggestions. Input data: New daily life data, feedback. Output data: Updated health improvement program.

[1368] Step 9:

[1369] An assistant device in a physical store makes health-related suggestions to users. For example, while shopping at a supermarket, it may suggest low-calorie foods or provide product information based on the amount of exercise. Input data: User's health status, current shopping details. Output data: Health-related suggestions, in-store guidance.

[1370] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1371] The present invention relates to a system that aggregates personal information collected from a user's daily life with general health-related information, and combines this information with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[1372] Users first install the system's app on their device and register. During this registration process, users enter basic information such as their name, age, gender, health history, current health condition, lifestyle habits, and eating habits. This information will be used in the analysis described below.

[1373] The device then collects real-time data about the user's daily life. This data collection uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. Social media posts (images, videos, text) and email content (text and attachments) are also collected.

[1374] The collected data is sent from the device to a server at regular intervals. The server receives this data and stores it in a centralized database. This database has the role of managing all user information in an integrated manner.

[1375] The system also includes an emotion engine that recognizes users' emotions in real time. The emotion engine has the ability to evaluate their emotional state and stress level when analyzing users' social media posts, email content, and even sensor data.

[1376] The server uses multimodal generation AI to analyze data triggered by data updates. This AI comprehensively analyzes multiple data formats (text, images, sensor data, etc.) to understand the user's health condition, lifestyle habits, and emotional trends. Specifically, it uses a video analysis algorithm to classify images and videos obtained from social media, identify dietary habits and exercise status, and analyzes email content using natural language processing (NLP) to evaluate stress levels and emotional state. It also analyzes emotional data obtained through an emotion engine to accurately understand the user's psychological state.

[1377] Based on the analysis, the server generates a personalized health improvement program for each user, which involves automatically gathering the latest health information from the web and integrating it with personal data. Specific examples of programs include exercise routines, meal plans, and meditation programs for stress reduction.

[1378] The generated health improvement program is sent from the server to the user's device and a notification is sent. The user receives the notification, checks the suggestions within the app, and can then implement them. At this time, the device records the user's feedback and implementation status in real time and sends them back to the server.

[1379] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the most appropriate program. If an effect is observed, the next step is to suggest a new exercise menu or meal plan. If no effect is observed, a different approach is adopted to support the user's health improvement.

[1380] As a specific example, let's consider the use of an emotion engine. In the case of a middle-aged man (42 years old, mainly desk-bound), in addition to entering basic information, the device collects the user's daily step count, heart rate, and food photos from social media. The system then uses the emotion engine to analyze the user's stress level and emotional state based on the content of their social media posts and emails. The server comprehensively analyzes this data and determines that the user's step count is low, their diet is high, and their stress level is high. The server then recommends 30 minutes of walking per day, a low-calorie diet, and a meditation program to reduce stress, and notifies the user's device. If the user accepts and implements the suggestions, overall health improvement is achieved.

[1381] In this way, the present invention is a system that realizes effective and sustainable lifestyle improvements by analyzing each user's data in detail and providing an individually optimized health improvement program. Furthermore, by combining it with an emotion engine, it is possible to accurately grasp the user's emotional state and make more personalized suggestions.

[1382] The processing flow will be explained below.

[1383] Step 1:

[1384] The user installs the app and registers by entering their name, age, gender, and other basic information (health history, lifestyle habits, etc.).

[1385] Step 2:

[1386] The terminal collects basic information entered by the user and transmits it to the server according to a security protocol.

[1387] Step 3:

[1388] The device collects real-time data about daily life. Specifically, it uses smartphone sensors to obtain vital data such as the number of steps taken, heart rate, and sleep patterns. It also collects social media posts (images, videos, text) and email content (text and attachments).

[1389] Step 4:

[1390] The terminal transmits the collected data to the server at regular intervals (for example, at the end of each hour or each day).

[1391] Step 5:

[1392] The server stores the received data in a centralized database, checks the integrity of the data, and updates it to the latest version.

[1393] Step 6:

[1394] The device uses an emotion engine to analyze the user's emotional data in real time from social media posts and email content.

[1395] Step 7:

[1396] The terminal transmits the emotion data acquired from the emotion engine to the server.

[1397] Step 8:

[1398] The server uses multimodal generative AI to analyze data triggered by data updates. Specifically, it analyzes images and videos obtained from social media to identify daily activities and dietary details. It analyzes message content using natural language processing (NLP) to evaluate stress levels and emotional states. It also performs time-series analysis of smartphone sensor data to diagnose exercise volume and sleep patterns.

[1399] Step 9:

[1400] Based on the analysis results, the server generates a customized health improvement program for each user. It automatically collects the latest health information (research articles, dietary recipes, exercise programs, etc.) from the web and integrates it with individual data. Specifically, it creates exercise menus, meal plans, stress reduction programs, etc.

[1401] Step 10:

[1402] The server sends the generated health improvement program to the user's device and notifies them, and new suggestions are provided to the user via push notifications and in-app messages.

[1403] Step 11:

[1404] The user receives a notification and sees detailed suggestions within the app, and chooses to follow the suggested exercise and meal plan.

[1405] Step 12:

[1406] The device records the user's feedback and performance status in real time and sends it to the server, such as the distance walked, calories burned, and photos of meal contents.

[1407] Step 13:

[1408] The server periodically analyzes new data and evaluates the effectiveness of the health improvement program. It places particular emphasis on emotional data obtained through the emotion engine, understanding the user's psychological state and continually recommending the optimal program. If the user is close to achieving their goal, it will suggest a new exercise menu or meal plan. If no effect is seen, it will suggest a different approach to help the user improve their health.

[1409] Step 14:

[1410] The server generates a new optimized program, sends it back to the user's device, and notifies the user, allowing the user to continuously receive and implement the optimal health improvement program.

[1411] Example 2

[1412] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1413] Conventional health management systems have struggled to comprehensively collect and analyze users' daily life and emotional data to provide optimal health improvement programs for each individual user. As a result, they were unable to provide effective programs that fully considered the user's actual health and psychological state, limiting the effectiveness of health improvement. Furthermore, the lack of effective analysis of collected data and feedback utilization made it difficult to continuously update and optimize programs.

[1414] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for collecting basic information from the user; means for collecting data on the user's daily life in real time; means for storing the collected data in a centralized database; analysis means using a generative model to analyze the collected data; means for generating a health improvement program customized for each user based on the analysis results; means for transmitting and notifying the user of the generated health improvement program to the user's information terminal; means for recording the user's feedback and execution status; means for analyzing new data and updating the program; analysis means including an emotion engine that analyzes emotion data in real time and evaluates the user's psychological state; and means for further personalizing the program using the emotion data. This makes it possible to comprehensively analyze the user's health and psychological states, provide an individually optimized health improvement program, and continuously update it.

[1415] "User" means an individual who uses the system to improve their own health.

[1416] "Basic information" refers to information about the user, such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits.

[1417] "Daily life data" refers to data related to a user's daily activities and health, including the number of steps taken, heart rate, sleep patterns, social media posts, and email content.

[1418] "Collection methods" refers to the methods and technologies used to collect basic information and daily life data from users.

[1419] A "centralized management database" is a database for managing and storing collected data in an integrated manner.

[1420] "Generative model" refers to artificial intelligence (AI) technology that analyzes multiple data formats (text, images, sensor data, etc.).

[1421] "Analysis means" refers to methods and techniques for analyzing collected data using generative models to assess the user's health and emotional state.

[1422] A "customized health improvement program" is a program that is individually created for each user based on data analysis, and includes exercise menus, meal plans, stress reduction programs, etc.

[1423] "Information terminal" refers to devices such as smartphones and tablets used by users.

[1424] "Notification means" refers to a method or technology for transmitting the generated health improvement program to the user's information terminal and notifying the user.

[1425] "Feedback" refers to response data such as progress and impressions when a user follows a health improvement program.

[1426] "Emotion Engine" refers to artificial intelligence technology that assesses a user's emotional state and stress level from data from their daily life.

[1427] "Personalization" refers to providing suggestions and services that are optimized for each user's individual needs and circumstances.

[1428] "Analysis results" refers to the analysis results of data processed by the generative model or emotion engine.

[1429] "Update tools" refer to methods and techniques for continually reviewing and optimizing health improvement programs based on new data collected.

[1430] The present invention relates to a system that aggregates personal information collected from a user's daily life and general health-related information, and combines this with an emotion engine to generate a more accurate health improvement program. Specific embodiments of the present invention are described below.

[1431] Hardware and Software Configuration

[1432] 1. User Device

[1433] Users use smartphones or tablets as devices. These devices have built-in sensors (pedometers, heart rate sensors, accelerometers, etc.) that collect vital data. The content of user posts and emails is also collected through social networking applications and email applications.

[1434] 2. Server

[1435] The server is equipped with a database system (e.g., MySQL, MongoDB) and a generative model (e.g., OpenAI GPT-4, Google BERT), as well as an emotion engine for data analysis.

[1436] 3. Software

[1437] The application software of this system manages a series of processes, from creating a health improvement program to notifying the results.

[1438] Program processing explanation

[1439] 1. User Registration

[1440] Users first install the system's app on their device and enter basic information (name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc.) Once this registration is complete, the information is sent from the device to the server and stored in a centralized database.

[1441] 2. Data Collection

[1442] The device uses built-in sensors to collect vital data such as the number of steps taken, heart rate, and sleep patterns in real time. Furthermore, the device also periodically collects users' social media posts (images, videos, text) and email content (text and attachments). This data is then sent to a server at regular intervals.

[1443] 3. Data analysis and emotion evaluation

[1444] The server stores the received data in a centralized database. It then uses an emotion engine to evaluate the user's emotional state and stress level based on social media posts, email content, and sensor data. It uses a generative model to comprehensively analyze multiple data formats (text, images, and sensor data) to understand the user's health, lifestyle, and emotional trends.

[1445] 4. Generating health improvement programs

[1446] Based on the analysis results, the server generates a customized health improvement program for each user. This involves automatically collecting the latest health information and integrating it with personal data. Specific examples of programs that can be generated include exercise routines, meal plans, and meditation programs for stress reduction.

[1447] 5. Program delivery and feedback utilization

[1448] The generated health improvement program is sent from the server to the user's device and notified. The user receives the notification and can review and implement the suggestions within the app. The device records the user's feedback and implementation status in real time and sends it back to the server. The server periodically analyzes the new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary.

[1449] Specific examples

[1450] For example, in the case of a middle-aged man (age 42, mostly desk-bound), the following process is performed:

[1451] 1. Registration: Enter basic information.

[1452] 2. Data collection: Step count, heart rate, and food photos from social media.

[1453] 3. Emotion assessment: Analyzes stress levels and emotional states from social media posts and email content through an emotion engine.

[1454] 4. Data analysis: It was confirmed that the number of steps was low, the amount of fatty food was high, and stress levels were high.

[1455] 5. Program generation: Suggests a 30-minute walk per day, a low-calorie diet, and a meditation program to reduce stress.

[1456] 6. Program distribution: Notification to user's device.

[1457] Example prompts to be input to the generative AI model:

[1458] Create a health improvement program for a 42-year-old man who works a desk job. His daily data shows that he walks few steps, eats a lot of fatty foods, and has high stress levels based on his social media and email activity.

[1459] In this way, the present invention aims to improve the user's health condition efficiently and sustainably by providing an individually optimized health improvement program based on detailed data analysis.

[1460] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1461] Step 1:

[1462] The user installs the app and enters basic information. The user registers their name, age, gender, health history, current health condition, lifestyle habits, eating habits, etc. in the app. This information becomes input data and is sent from the device to the server. The server receives this basic information and stores it in a centralized database. This saves the user's basic information in the database and is used in subsequent analysis processes.

[1463] Step 2:

[1464] The device collects data about the user's daily life in real time. As the user goes about their daily life, the device uses its built-in sensors to acquire vital data such as the number of steps taken, heart rate, and sleep patterns. Images, videos, text, and emails posted by the user on social media are also periodically collected. This collected data becomes input data for the device. The device then sends the collected data to a server. The server stores the received data in a centralized database. This allows the user's daily life data to be accumulated in the database.

[1465] Step 3:

[1466] The server analyzes the user's emotional state using an emotion engine. The server receives input data from social media posts, email content, and sensor data stored in a database, and analyzes it using the emotion engine. The emotion engine analyzes text and sensor data to evaluate stress levels and emotional states. The analysis results in data on the user's emotional state. This data is used to generate subsequent health improvement programs.

[1467] Step 4:

[1468] The server performs data analysis using multimodal generative AI. The server comprehensively retrieves multiple data formats, such as vital signs, social media posts, and email content, from the database and inputs them into the generative AI model. The generative AI model analyzes this data to understand the user's health status, lifestyle habits, and emotional trends. As an output, health status assessment data for the user is generated. This assessment data is used to customize health improvement programs.

[1469] Step 5:

[1470] The server generates a health improvement program based on the analysis results. The server uses the user's health status assessment data obtained from the generative AI model as input data, collects and integrates the latest health information, and generates an individually optimized health improvement program. The generated program specifically includes an exercise menu, meal plan, stress reduction program, etc. These programs are customized for each user.

[1471] Step 6:

[1472] The server sends the generated health improvement program to the user's device and notifies them. The server then sends the generated health improvement program to the device, which then notifies the user. The user receives the notification and checks the contents of the proposed program within the app. The effects of the improvement program are realized when the user implements the proposed health improvement program.

[1473] Step 7:

[1474] The user provides feedback, and the device records the execution status. The user runs the proposed program and enters the execution status and impressions within the app. The device collects this feedback in real time and sends the feedback data to a server. The server stores the received feedback data in a database and uses it for analysis.

[1475] Step 8:

[1476] The server analyzes the new data and updates the health improvement program. The server then uses the feedback data and newly collected daily life data to perform another analysis using the generative AI model. Based on the analysis results, the effectiveness of the health improvement program is evaluated and the program is optimized and updated as necessary. This enables continuous health improvement tailored to the user's condition.

[1477] (Application example 2)

[1478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1479] In modern fitness gyms and wellness centers, it is extremely important to accurately understand the health and emotional state of each user and provide customized health improvement programs based on that information. However, a system that effectively collects and analyzes users' daily life data and emotional state and provides specific health improvement suggestions in real time has not yet been realized. This poses a challenge in providing continuous support for users' health improvement and receiving appropriate feedback.

[1480] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting basic information from the user, means for collecting data on the user's daily life in real time, means for storing the collected data in a centralized database, means for analyzing the collected data using multimodal generation AI, means for generating a health improvement program customized for each user based on the analysis results, means for sending and notifying the user of the generated health improvement program, means for collecting the user's implementation conditions and storing them in a database as a record of exercise facility use, means for recording the user's feedback and implementation status, and means for analyzing new data and updating the program. This makes it possible to comprehensively analyze the health and emotional state of each user and continuously provide an optimized health improvement program.

[1481] "Means for collecting basic information from users" refers to devices or methods for obtaining basic information such as the user's name, age, sex, health history, current health condition, lifestyle habits, and eating habits.

[1482] "Means for collecting data on a user's daily life in real time" refers to devices or methods for acquiring vital data such as the number of steps taken by a user, heart rate, and sleep patterns in real time.

[1483] "Means for storing collected data in a unified database" refers to a device or method for storing and managing all collected data in a unified database.

[1484] "Analysis methods using multimodal generation AI to analyze collected data" refers to artificial intelligence devices and methods that comprehensively analyze multiple data formats, such as text data, image data, and sensor data.

[1485] "Means for generating a health improvement program customized for each user based on the analysis results" refers to a device or method that uses analyzed data to automatically generate a health improvement plan optimized for each individual user.

[1486] The "means for transmitting the generated health improvement program to the user's terminal and notifying the user" refers to a device or method for transmitting the generated health improvement program to the user's terminal in real time and notifying the user.

[1487] "Means for collecting user activity conditions and storing them in a database as a record of use of the exercise facility" refers to a device or method for collecting data on the user's activity at the exercise facility and storing it in a database as a record of use.

[1488] "Means for recording user feedback and performance" refers to a device or method for tracking and recording user feedback and performance of a health improvement program.

[1489] "Means for analyzing new data and updating the program" refers to devices and methods that use the latest data for analysis and continuously update and improve the health improvement program.

[1490] "Means for analyzing a user's emotional state" refers to devices or methods for analyzing a user's emotions and stress levels from the content of their social media posts and emails.

[1491] This invention relates to a system that aggregates personal information collected from users' daily lives with general health-related information, and combines it with an emotion engine to generate more accurate health improvement programs for users of fitness gyms and wellness centers.

[1492] System configuration:

[1493] This system consists of the following main components:

[1494] 1. How we collect basic information from you:

[1495] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into a dedicated application.

[1496] 2. Means of collecting real-time data on users' daily lives:

[1497] It uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns.

[1498] 3. How the collected data is stored in a centralized database:

[1499] The collected data is sent from the terminal to a server at regular intervals, and the server stores it in a centralized database.

[1500] 4. Analysis method using multimodal generative AI to analyze collected data:

[1501] The server uses multimodal generative AI to comprehensively analyze the collected text data, image data, and sensor data to understand the user's health condition, lifestyle habits, and emotional trends. At this time, it uses an Emotion Engine to analyze emotions and stress levels from social media posts and email content.

[1502] 5. Means for generating a health improvement program customized for each user based on the analysis results:

[1503] Based on the analysis results, the system generates a health improvement program optimized for each user's individual condition, including exercise routines, meal plans, and meditation programs for stress reduction.

[1504] 6. Means for sending the generated health improvement program to the user's device and notifying them:

[1505] The generated health improvement program is sent to the user's device, where they can view the suggestions within the app.

[1506] 7. Means for collecting user's performance conditions and storing them in a database as a record of use of the sports facility:

[1507] The system collects data on users' activities at fitness gyms and wellness centers (e.g., number of workouts, machine usage history, calories burned, etc.) and stores it in a database.

[1508] 8. Means of recording user feedback and performance:

[1509] When users implement the suggestions, feedback and progress are tracked and recorded, and this data is also sent to the server.

[1510] 9. How to analyze new data and update the program:

[1511] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates the program as necessary. It places particular emphasis on emotional data obtained through the emotion engine, and continually suggests optimal programs that reflect the user's psychological state.

[1512] Examples of hardware and software:

[1513] Smartphone Sensors:

[1514] Collect data about your daily life (step count, heart rate, sleep patterns, etc.).

[1515] Emotion Engine:

[1516] A library for emotion analysis that analyzes stress levels and emotional states from social media posts and email content.

[1517] Health Analyzer:

[1518] A library that analyzes user health data and generates health improvement programs.

[1519] Examples and prompts:

[1520] For example, if a male user named Tanaka visits a fitness gym, the following programs will be offered to him:

[1521] User information: Name: Tanaka, Age: 42, Gender: Male

[1522] Health history: high blood pressure, diabetes

[1523] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[1524] Social media post: I'm tired again today...

[1525] Email content: I feel like I've been feeling a lot of stress at work lately.

[1526] Based on this, the following health improvement program is generated:

[1527] Generate a health improvement program based on the following criteria:

[1528] User information: Name: Tanaka, Age: 42, Gender: Male, Health history: High blood pressure, diabetes

[1529] Vital data: Steps: 5000, Heart rate: 76, Sleep pattern: 7 hours

[1530] Emotional Data:

[1531] Social media post: I'm tired again today...

[1532] Email content: I feel like I've been feeling a lot of stress at work lately.

[1533] In this way, the optimal program is provided based on the user's health and emotional state, enabling continuous improvement in health.

[1534] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1535] Step 1:

[1536] User Information Collection:

[1537] Users enter basic information such as name, age, gender, health history, current health condition, lifestyle habits, and eating habits into the application, and the device then collects the user's basic information and sends it to the server.

[1538] Input: Basic information from the user

[1539] Output: Basic information stored on the server

[1540] Step 2:

[1541] Real-time collection of daily life data:

[1542] The device uses smartphone sensors to collect real-time vital data such as the user's steps, heart rate, and sleep patterns, and transmits the collected data to a server at regular intervals.

[1543] Input: Vital data collected from smartphone sensors

[1544] Output: Vital data sent to the server

[1545] Step 3:

[1546] Centralize your data:

[1547] The server receives the basic information and vital data sent from the device and stores them in a centralized database, which manages all user information in an integrated manner.

[1548] Input: Basic information and vital data sent from the device

[1549] Output: Data stored in a centralized database

[1550] Step 4:

[1551] Emotion analysis:

[1552] The server uses the Emotion Engine to analyze users' emotional state and stress levels from social media posts and email content, using text analysis techniques to generate emotion data.

[1553] Input: User's SNS posts and email content

[1554] Output: Emotional state and stress level data

[1555] Step 5:

[1556] Data Analysis:

[1557] The server uses multimodal generative AI to comprehensively analyze basic information, vital data, and emotional data, thereby understanding the user's health condition, lifestyle habits, and emotional tendencies.

[1558] Input: Basic information, vital data, emotional data

[1559] Output: Analysis results of the user's health status, lifestyle habits, and emotional tendencies

[1560] Step 6:

[1561] Generating Health Improvement Programs:

[1562] Based on the analysis, the server generates a customized health improvement program for each user, including exercise routines, meal plans, and meditation programs to reduce stress.

[1563] Input: Analysis results

[1564] Output: A customized health improvement program

[1565] Step 7:

[1566] Health Improvement Program Submissions and Notifications:

[1567] The server then sends the generated health improvement program to the user's device and notifies them, and the user can check the proposed program through the application.

[1568] Input: Generated health improvement program

[1569] Output: Health improvement programs and notifications sent to the user's device

[1570] Step 8:

[1571] Collecting implementation conditions:

[1572] The device collects the user's activity data at the fitness gym or wellness center (for example, number of training sessions, machine usage history, calories burned, etc.) and stores this data in a database.

[1573] Input: Activity data at fitness gyms and wellness centers

[1574] Output: Activity data stored in a database

[1575] Step 9:

[1576] Feedback and performance recording:

[1577] The device tracks and records feedback and progress as users implement the suggestions, and this data is sent back to the server.

[1578] Input: User feedback and execution status

[1579] Output: Feedback and execution status data sent to the server

[1580] Step 10:

[1581] Program updates:

[1582] The server periodically analyzes new data, evaluates the effectiveness of the health improvement program, and updates it as necessary, placing particular emphasis on emotional data obtained through the Emotion Engine.

[1583] Input: New data and existing health improvement programs

[1584] Output: Updated Health Improvement Program

[1585] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1586] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1587] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1588] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1589] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1590] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1591] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1592] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1593] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more." ...

Claims

1. a means of collecting basic information from users; A means of collecting data on users' daily lives in real time; a means for storing the collected data in a centralized database; An analytical method using multimodal generation AI to analyze the collected data, A means for generating a health improvement program customized for each user based on the analysis results; a means for transmitting the generated health improvement program to a user's terminal and notifying the user; A means of recording user feedback and performance; A means of analyzing new data and updating the program; A system including:

2. The system of claim 1, wherein data on the user's daily life is collected from smartphone sensors.

3. The system according to claim 1, wherein data on the user's daily life is collected from social media posts.

Images