System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an image of a meal from a user; means for analyzing the image to identify nutrients consumed; means for collecting basic information and health data of the user; means for identifying missing nutrients for the user based on the basic information and health data; means for suggesting optimal supplements based on the missing nutrients; means for extracting supplements based on the suggestions; and means for storing a history of supplement extractions and collecting feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's modern diet, it is difficult to achieve a balanced nutritional intake, and many people often end up with deficiencies or excesses of certain nutrients. It is also difficult to know which supplements to take and to avoid overdose or deficiency. These issues make it difficult for users to properly manage their health. [Means for solving the problem]

[0005] The present invention provides a system that receives images of meals from a user and uses AI to analyze the images to identify the nutrients ingested by the user. Furthermore, the system collects the user's basic information and health data, and based on this, accurately identifies the nutrients the user is lacking. This makes it easier to achieve a balanced nutritional intake by suggesting optimal supplements and providing a means for automatically selecting supplements. Furthermore, the system stores a history of supplement selection and collects user feedback to improve the accuracy of the system and the user experience. This allows users to more easily manage their health and efficiently supplement their necessary nutrients.

[0006] "User" refers to an individual who uses this system and is the entity who uploads food photos and provides personal information to the system.

[0007] "Meal Images" are photographs that users take and upload to the system to record their daily meals.

[0008] "Analysis" refers to the process by which AI processes image data of meals to identify the types, quantities, and nutritional components of food.

[0009] "Nutrients" are components necessary for maintaining health, such as vitamins, minerals, and proteins, that are obtained from food and supplements.

[0010] "Basic information" refers to personal information such as the user's age, gender, weight, and blood pressure.

[0011] "Health data" refers to physiological data such as a user's body composition, number of steps taken, and blood pressure, and includes information collected from IoT devices, etc.

[0012] "Nutrients that are deficient" are nutrients that the user needs but is not consuming in sufficient amounts, as identified from the user's dietary and health data.

[0013] "Optimal supplements" are supplements suggested to fill in the user's nutritional deficiencies.

[0014] "Supplement extraction" refers to the process of extracting and providing a specified amount of supplements containing essential nutrients to the user.

[0015] "History" refers to a record of the supplements a user has taken and the nutrients suggested.

[0016] "Feedback" refers to user feedback on the system, such as evaluations and suggestions for improvement. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system that allows users to take photos of their meals, performs nutritional analysis using AI, and provides optimal supplements to individuals. This system is composed of multiple elements, including a user, a terminal, and a server. Specific embodiments are described in detail below.

[0039] System configuration

[0040] The system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, and a supplement delivery device. The role of each element is explained below.

[0041] User operations

[0042] user:

[0043] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[0044] Users take photos of their daily meals using a mobile device and upload them to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used, and the data is synchronized.

[0045] Terminal handling

[0046] Device:

[0047] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[0048] Collects user profile information and health data and sends it to a server.

[0049] Server Processing

[0050] server:

[0051] The server sends the user's meal image to the AI ​​engine for image analysis, which uses image analysis algorithms to identify the ingredients consumed and their nutritional content.

[0052] It analyzes nutrient deficiencies by combining the user's basic information and health data.

[0053] Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[0054] Based on the suggestion, instructions to extract the supplement are sent to the "supplement providing device."

[0055] Operating the supplement delivery device

[0056] server:

[0057] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[0058] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[0059] Specific examples

[0060] A specific example will be described below.

[0061] Example 1: Breakfast analysis and supplement provision

[0062] 1. User Operation

[0063] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[0064] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[0065] 2. Terminal Processing

[0066] The device receives the image of the meal, stores it in memory, and sends it to the server.

[0067] 3. Server Processing

[0068] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[0069] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[0070] The server calculates the optimal combination of supplements, including vitamin D and calcium, and sends the recommendations to the user.

[0071] The user reviews and accepts the proposal.

[0072] 4. Supplement Extraction

[0073] Vitamin D and calcium cartridges are placed in the "supplement supply device" and the optimal amount is extracted.

[0074] The device notifies the server that it is ready, and the user is notified.

[0075] As described above, this system comprehensively manages the user's dietary and health data, and individually replenishes necessary nutrients, thereby achieving balanced nutritional intake.

[0076] The processing flow will be explained below.

[0077] Step 1:

[0078] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[0079] Step 2:

[0080] Users take photos of their meals using their smartphones and upload the images to the application. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, and synchronize that data with the application.

[0081] Step 3:

[0082] The device receives the meal image sent by the user and temporarily stores it in memory. The image data is then sent to a cloud server, along with the user's profile information and health data.

[0083] Step 4:

[0084] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[0085] Step 5:

[0086] The server analyzes the user's basic information and health data in combination with the AI ​​analysis results, identifies any nutrient deficiencies, and calculates the optimal combination of supplements based on that information.

[0087] Step 6:

[0088] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[0089] Step 7:

[0090] If the user agrees to the suggestion, the server instructs the user to insert the cartridge into the "supplement providing device." The device then checks the inserted cartridge and extracts the optimal amount of the required supplement.

[0091] Step 8:

[0092] The supplement providing device completes the extraction and sends a completion notification to the server, which then notifies the terminal that the supplement is ready.

[0093] Step 9:

[0094] The user receives and takes the supplements. This updates the user's health data and history, leading to improved accuracy of the next recommendations. The server periodically collects user feedback and uses it to improve the system.

[0095] In this way, a system is realized that allows users to efficiently take in the nutrients they need on a daily basis.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] In today's world, it is difficult to properly manage individual nutritional status and quickly and accurately replenish individual nutrient deficiencies. Conventional systems lack the ability to comprehensively analyze users' dietary and health data, limiting manual data entry and supplement selection. Furthermore, there is no system that utilizes IoT devices to automatically collect and comprehensively analyze health data, resulting in the problem of inappropriate individual nutritional supplementation.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for receiving meal images from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for identifying nutrient deficiencies in the user based on the basic information and health data, means for suggesting optimal supplements based on the nutrient deficiencies, means for selecting supplements based on the suggestions, means for storing a history of supplement selection and collecting feedback, means for using a cloud-based AI engine to analyze the meal images, means for using the AI ​​engine to compare the image analysis results with a nutritional information database, means for automatically collecting the user's health data via an IoT device, and means for combining and analyzing the user's basic information, health data, and dietary data using a statistical analysis tool. This allows for integrated management of the user's diet and health data, enabling prompt and appropriate individual nutritional supplementation.

[0101] "User" refers to an individual who uses the System.

[0102] "Meal Image" refers to an image file containing visual data of the meal the user consumes.

[0103] "Means for analyzing images to identify ingested nutrients" refers to technologies and algorithms for recognizing ingredients in images of food and identifying the nutritional components of those ingredients.

[0104] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[0105] "Health data" refers to data that indicates a user's health status, such as the number of steps taken, body fat percentage, and blood pressure.

[0106] "Means for identifying nutrient deficiencies" refers to technology that analyzes and identifies nutrients that are lacking in the user's current diet based on the user's basic information and health data.

[0107] "Means for suggesting optimal supplements" refers to technology that calculates the optimal combination of supplements to make up for missing nutrients and suggests them to users.

[0108] "Means for extracting supplements" refers to the equipment and technology used to prepare and deliver the proposed supplements to the user in the appropriate amounts.

[0109] "Means for storing the history of supplement extraction and collecting feedback" refers to the function of storing the history of supplement use and collecting opinions and experiences from users.

[0110] A "cloud-based AI engine" refers to an artificial intelligence processing system that operates in a cloud environment.

[0111] "Nutrition information database" refers to a database that accumulates data on the nutritional components of various food ingredients.

[0112] "IoT devices" refer to health management devices that are capable of transmitting data via the Internet.

[0113] "Statistical analysis tools" refer to software tools used to analyze various types of data and derive specific patterns or trends.

[0114] This invention relates to a system that allows users to take photos of their meals, analyzes their nutritional intake using AI, and provides optimal supplements for each individual. The invention is composed of multiple elements, including the user, a terminal, and a server.

[0115] System configuration

[0116] This system consists of a user device (such as a smartphone or tablet), a cloud-based server, and a supplement delivery device. Each element is described in detail below.

[0117] User operations

[0118] user:

[0119] 1. Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, and blood pressure.

[0120] 2. Take photos of your daily meals with your smartphone camera and upload them to the application.

[0121] 3. Sync IoT devices such as pedometers, body composition monitors, and blood pressure monitors with your smartphone to collect health data.

[0122] Terminal handling

[0123] Device:

[0124] 1. Receives the food image sent by the user and temporarily stores it in memory. The stored image data is then sent to the cloud server.

[0125] 2. It also collects user profile information and health data and sends them to the server.

[0126] Server Processing

[0127] server:

[0128] 1. The received meal image is sent to a cloud-based AI engine for image analysis. Specifically, an image analysis tool (e.g., Google® Cloud Vision API) is used to identify the ingredients ingested from the image.

[0129] 2. The image analysis results are compared with a nutritional information database to identify the nutritional components of each ingredient.

[0130] 3. Analyze the user's basic information and health data using statistical analysis tools (e.g., Python, R) to identify nutrient deficiencies.

[0131] 4. Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[0132] 5. Insert the required supplement cartridge into the supplement providing device and send instructions to extract the optimal amount.

[0133] Operating the supplement delivery device

[0134] server:

[0135] 1. Place the cartridge into the supplement dispenser and extract the optimal amount of supplement you need.

[0136] 2. Provides extracted supplements in powder form and notifies you when the device is ready.

[0137] Specific examples

[0138] Specific examples are given below:

[0139] Example 1: Breakfast analysis and supplement provision

[0140] 1. User Action:

[0141] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it with their smartphone.

[0142] Photos of meals are uploaded to the application and user profile information is sent to the server.

[0143] 2. Terminal processing:

[0144] The device receives the image of the meal, stores it in memory, and sends it to a cloud server.

[0145] 3. Server processing:

[0146] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[0147] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[0148] The server calculates the optimal combination of supplements containing vitamin D and calcium and sends the recommendation to the user, who then reviews and accepts the recommendation.

[0149] 4. Supplement Extraction:

[0150] Insert vitamin D and calcium cartridges into the supplement dispenser and extract the optimal amount.

[0151] The device notifies the server that it is ready and notifies the user.

[0152] Prompt Sentence Examples

[0153] Examples of prompts to input to a generative AI model might include:

[0154] "If a user wants to take a photo of an omelet, salad, and orange juice and run a nutritional analysis, what steps should they take?"

[0155] By inputting this prompt, the generative AI model will guide the user through the nutritional analysis process of the meal and generate an optimal supplement delivery flow.

[0156] As described above, this system comprehensively manages the user's dietary and health data, enabling prompt and appropriate individual nutritional supplementation.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Step 1: User logs in

[0159] Specific behavior:

[0160] The user launches the app on their smartphone, enters their user ID and password on the login screen, and presses the "Login" button.

[0161] Input: User ID, Password

[0162] Output: User credentials

[0163] Data processing / calculation: The server checks the user ID and password against the database to perform authentication.

[0164] Step 2: User enters profile information

[0165] Specific behavior:

[0166] When logging in for the first time, the user enters personal information such as age, gender, weight, and blood pressure on the profile entry screen that appears, and presses the "Save" button.

[0167] Input: Profile information such as age, gender, weight, blood pressure, etc.

[0168] Output: User profile data

[0169] Data processing / calculation: The terminal formats the input information and sends it to the server.

[0170] Step 3: User takes and uploads a photo of their meal

[0171] Specific behavior:

[0172] Users take a photo of their meal with their smartphone camera and press the "upload" button to upload the image to the app.

[0173] Input: Food image file

[0174] Output: Saved food image data

[0175] Data processing / calculation: The device temporarily stores the image file in memory, compresses it, and then sends it to the cloud server.

[0176] Step 4: User synchronizes data on IoT devices

[0177] Specific behavior:

[0178] The user turns on Bluetooth on each IoT device and smartphone and presses the "Data Sync" button in the app.

[0179] Input: IoT device data (number of steps, body fat percentage, blood pressure, etc.)

[0180] Output: Collected health data

[0181] Data processing / calculation: The terminal receives data from the IoT device and sends it to the server in bulk.

[0182] Step 5: The device temporarily saves the food image and sends it to the server.

[0183] Specific behavior:

[0184] The device temporarily stores the food images sent by the user in memory and sends them to a cloud server.

[0185] Input: Food image data

[0186] Output: Image data stored on a cloud server

[0187] Data processing / calculation: The device compresses the food images and sends them to a cloud server via the network.

[0188] Step 6: The device collects the user's profile information and health data and sends it to the server.

[0189] Specific behavior:

[0190] The device collects the user's profile information and health data from IoT devices and sends it to a cloud server.

[0191] Input: Profile information, health data

[0192] Output: User profile information and health data stored on a cloud server

[0193] Data processing / calculation: The data collected by the terminal is formatted and sent in bulk to the cloud server.

[0194] Step 7: The server sends the food image to the AI ​​engine for image analysis.

[0195] Specific behavior:

[0196] The server sends the received food images to a cloud-based AI engine, which analyzes the images.

[0197] Input: Food image data

[0198] Output: Parsed ingredient information

[0199] Data processing / computation: A cloud-based AI engine (e.g., Google Cloud Vision API) identifies ingredients from images and recognizes their nutritional content.

[0200] Step 8: The server compares the image analysis results with the nutrition information database.

[0201] Specific behavior:

[0202] The server compares the image analysis results with a nutritional information database to identify the nutritional components of each ingredient.

[0203] Input: Ingredient information

[0204] Output: Nutritional information

[0205] Data processing / calculation: The server compares the image analysis results with a database and extracts nutrient data.

[0206] Step 9: The server analyzes the user's basic information and health data to identify nutrient deficiencies.

[0207] Specific behavior:

[0208] The server uses statistical analysis tools (e.g., Python, R) to combine and analyze the user's basic information, health data, and dietary information.

[0209] Input: Basic information, health data, nutritional information

[0210] Output: Missing nutrient information

[0211] Data processing / calculation: Statistical analysis tools analyze the input data and identify nutrient deficiencies.

[0212] Step 10: The server calculates the optimal supplement combination and sends the proposal to the user.

[0213] Specific behavior:

[0214] The server uses an algorithm to generate a list of supplements containing the nutrients to be supplemented and sends suggestions to the user.

[0215] Input: Missing nutrient information

[0216] Output: Supplement suggestion information

[0217] Data processing / calculation: The server calculates the optimal combination of supplements based on the nutrients that are lacking and generates recommendations.

[0218] Step 11: The server sends an instruction to the supplement providing device to extract the supplement.

[0219] Specific behavior:

[0220] The server sets the necessary supplement cartridge in the supplement providing device and sends instructions to extract the optimal amount.

[0221] Input: Supplement suggestion information

[0222] Output: Extracted supplement

[0223] Data processing / calculation: The server sends commands to the provider based on the suggested information and extracts the appropriate supplements.

[0224] The above are the specific processing steps of this system.

[0225] (Application example 1)

[0226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0227] As health consciousness grows, there is a demand for personalized nutritional management and optimal nutritional supplements. However, conventional systems have made it difficult to provide individually tailored nutritional supplements in physical stores, and they have been unable to respond in real time based on users' health data. This has made it difficult for users to quickly obtain the optimal nutritional supplements tailored to their health condition.

[0228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0229] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for extracting optimal amounts of nutritional supplements using a device in a physical store, means used in the physical store for providing the nutritional supplements, and means for storing a history of the provision of nutritional supplements and collecting feedback, thereby enabling users to quickly receive optimal nutritional supplements based on their own health condition at the physical store.

[0230] A "user" is a person who uses the system to take photos of meals and perform nutritional analysis.

[0231] "Meal images" are photographic data that visually record the contents of meals consumed by the user.

[0232] "Nutrients" are components such as vitamins, minerals, and proteins obtained from the food a user consumes.

[0233] "Basic information" refers to personal data such as the user's age, gender, weight, height, and health condition.

[0234] "Health data" refers to data that indicates the user's current health status, such as the number of steps taken, blood pressure, and body composition.

[0235] "Dietary supplements" are foods such as supplements and energy drinks that are suggested or offered to users to supplement nutrients that they are lacking.

[0236] A "brick and mortar store" is a physical store where a user can visit and receive nutritional supplements.

[0237] A "server" is a computer system that receives, analyzes, and processes data from users.

[0238] "Suggestions" are recommendations to provide optimal nutritional supplements to users based on their nutrient deficiencies.

[0239] "Extraction" is the act of extracting a specific amount of a dietary supplement needed based on a proposal.

[0240] "Device" refers to a device that is installed in a physical store and provides nutritional supplements in accordance with instructions from the server.

[0241] "History" refers to the record of dietary supplements provided to a user and their feedback data.

[0242] "Feedback" is information provided by users through comments and ratings about the effectiveness and satisfaction of a dietary supplement.

[0243] MODE FOR CARRYING OUT THE INVENTION

[0244] This invention is a system that allows users to take photos of their meals, upload them to the cloud, perform nutritional analysis, and provide optimal nutritional supplements to individuals in physical stores. Specific embodiments are described below.

[0245] System configuration

[0246] The system consists of a device used by users, such as a smartphone or tablet, a cloud-based server, and a nutritional supplement supply device in a physical store. The role of each element is explained below.

[0247] User operations

[0248] user:

[0249] Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, height, and blood pressure.

[0250] Users take photos of their daily meals with their smartphones and upload them to the app. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data and synchronize the data.

[0251] Terminal handling

[0252] Device:

[0253] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[0254] Collects user profile information and health data and sends it to a server.

[0255] Server Processing

[0256] server:

[0257] The server receives the user's meal images and analyzes the meal contents using a generative AI model, leveraging machine learning libraries such as TENSORFLOW® and PyTorch.

[0258] The system analyzes nutrient deficiencies by combining the user's basic information and health data. It uses Python to process the health data and compare it with known nutritional data stored in a database to identify nutrient deficiencies.

[0259] The server then suggests optimal nutritional supplements based on the nutrients that are lacking and notifies the user, who is then sent a notification to their smartphone.

[0260] Operating a nutritional supplement delivery device

[0261] Device:

[0262] The nutritional supplement supply device installed in the physical store receives instructions from the server and extracts the optimal nutritional supplements. This process is carried out using IoT control software using Node.js and Python.

[0263] The device will then be notified that the correct amount of nutritional supplement is ready to be dispensed to the user.

[0264] Specific examples

[0265] Example: Breakfast analysis and nutritional supplement provision

[0266] 1. User Operation

[0267] The user eats a sandwich, salad, and orange juice for breakfast and takes a photo of it.

[0268] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[0269] 2. Terminal Processing

[0270] The device receives the image of the meal, stores it in memory, and sends it to the server.

[0271] Profile information and health data are also sent to the server.

[0272] 3. Server Processing

[0273] The server sends the images to a generative AI model that analyzes the ingredients (e.g., protein in a sandwich, vitamins in a salad, vitamin C in orange juice).

[0274] This is combined with the user's basic information and health check data to identify vitamin D and calcium deficiencies.

[0275] The server calculates the optimal combination of nutritional supplements containing vitamin D and calcium and sends the proposal to the user, who then reviews and accepts the proposal.

[0276] 4. Nutritional supplements provided

[0277] Vitamin D and calcium ingredients are placed in a dispenser in a physical store, and the optimal amount is extracted.

[0278] The device notifies the server that it is ready, and the user is notified.

[0279] Example prompts to input to a generative AI model:

[0280] Meal image: "https: / / example.com / user_images / breakfast.jpg"

[0281] User profile: Age 30, Gender female, Weight 60kg, Height 160cm, Recent problems: Tired easily

[0282] Health data: Steps 8000, Blood pressure 120 / 80, Body fat 22%

[0283] This prompt text is imported into the server and analyzed to create a system that provides appropriate nutritional supplements.

[0284] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0285] Step 1:

[0286] user:

[0287] The user takes a picture of their meal, such as breakfast, with their smartphone, and the image is saved in an application on their smartphone.

[0288] Input: Food image

[0289] Output: Image data stored on the smartphone

[0290] Specific operation: Take a photo of your meal using the smartphone's camera app, and the image will be automatically saved in the application folder.

[0291] Step 2:

[0292] user:

[0293] Launch the application and enter your login information to log in to the system. The user will then confirm and enter their profile information (age, gender, weight, etc.) and sync their health data (step count, blood pressure, etc.).

[0294] Input: Login information, profile information, health data

[0295] Output: User data stored in the application

[0296] Specific operation: Enter login information on the login screen, confirm and enter profile information, and sync health data from IoT devices to the application.

[0297] Step 3:

[0298] Device:

[0299] The images of the meal taken by the user, along with the profile information and health data entered, are sent to a cloud server.

[0300] Input: Food images, profile information, health data

[0301] Output: Data sent to the cloud server

[0302] Specific operation: Use the application's upload function to send the selected images and synchronized data to the cloud server.

[0303] Step 4:

[0304] server:

[0305] The cloud server inputs the received image data into the generative AI model and performs a nutritional analysis of the meal contents, using libraries such as TensorFlow and PyTorch.

[0306] Input: Food image

[0307] Output: Nutritional data for meals

[0308] How it works: The generative AI model analyzes image data and identifies the nutritional content of each ingredient. For example, it calculates the protein content of an omelet or the vitamin content of a salad.

[0309] Step 5:

[0310] server:

[0311] Using the user's basic information and health data, Python is used to run an algorithm to identify nutrient deficiencies.

[0312] Input: Profile information, health data, dietary nutrient data

[0313] Output: List of nutrients that are lacking

[0314] What it does: It compares the received data with an existing nutrition database and calculates the nutrient deficiencies the user may have (e.g., vitamin D or calcium).

[0315] Step 6:

[0316] server:

[0317] Based on the identified nutrient deficiencies, the generative AI model will suggest optimal nutritional supplements and send a smartphone notification to the user.

[0318] Input: List of nutrients you are deficient in

[0319] Output: Nutritional supplement suggestions, notification data

[0320] What it does: Runs a recommendation generation algorithm based on nutrient deficiencies and sends a push notification to the user's smartphone. Example: "Supplement ABC is recommended to replenish vitamin D and calcium."

[0321] Step 7:

[0322] user:

[0323] Review smartphone notifications and agree to suggested dietary supplements.

[0324] Input: Notification data

[0325] Output: User consent data

[0326] Specific actions: Check the proposal on the notification screen and press the agree button.

[0327] Step 8:

[0328] server:

[0329] With the user's consent, instructions are sent to a nutritional supplement dispenser in the physical store to extract the optimal amount of nutritional supplement needed.

[0330] Input: User consent data

[0331] Output: Extraction instructions for dietary supplements

[0332] Specific operation: Using IoT control software (Node.js or Python), it sends extraction instructions to the dispenser. For example, it sends an instruction to extract a predetermined amount of nutritional supplement from a specific cartridge.

[0333] Step 9:

[0334] Device:

[0335] A nutritional supplement dispenser in a physical store receives the instruction, extracts the optimal amount of the specified nutritional supplement, and notifies the server and the user that it is ready to be dispensed.

[0336] Input: Extraction instruction data

[0337] Output: Ready notification

[0338] Specific operation: The device extracts the nutritional supplement and notifies the server that it is ready, which then pushes the notification to the user's device.

[0339] Step 10:

[0340] server:

[0341] It stores the history of the nutritional supplements provided and also collects feedback from users and stores it in a database.

[0342] Input: Provision history data, feedback data

[0343] Output: Updated database

[0344] Specific operation: The provision history and feedback information are stored in a database and used to improve the proposal algorithm in the future.

[0345] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0346] The present invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. This system is comprised of a user, a terminal, a server, and a supplement provider, and has the function of adjusting supplement recommendations based on the user's emotion data. Specific embodiments are described in detail below.

[0347] System configuration

[0348] This system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, an emotion engine, and a supplement delivery device. The role of each element is explained below.

[0349] User operation

[0350] User:

[0351] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[0352] Users take photos of their daily meals using a mobile device and upload the images to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used and the data is synchronized.

[0353] The application periodically checks and collects the user's emotional state to record their emotions.

[0354] Terminal handling

[0355] Device:

[0356] The system receives the food image sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[0357] The user's profile information, health data, and emotional data are transmitted to a server.

[0358] Server Processing

[0359] server:

[0360] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[0361] The system analyzes a user's basic information, health data, and emotional data to identify any nutrient deficiencies.

[0362] The emotional engine tailors supplement recommendations based on the user's emotional state, for example, suggesting a boost in B vitamins if stress levels are high.

[0363] The supplement proposal is sent to the user, and once the user confirms and agrees, the process proceeds to the next step.

[0364] Operating the supplement delivery device

[0365] server:

[0366] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[0367] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[0368] Specific examples

[0369] A specific example will be described below.

[0370] Example 1: Breakfast analysis and supplement provision

[0371] 1. User Operation

[0372] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[0373] Photos of the meal are uploaded to the application and sent to the server along with the profile information entered when logging in.

[0374] 2. Terminal Processing

[0375] The device receives the image of the meal, stores it in memory, and then sends it to the server.

[0376] 3. Server Processing

[0377] The AI ​​engine analyzes the image to determine the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[0378] Basic information, health data, and the user's emotional data (e.g., high stress levels) are combined to provide vitamin D and calcium deficiencies, as well as B vitamins based on the user's emotions.

[0379] The supplement proposal is sent to the user for review and consent.

[0380] 4. Supplement Extraction

[0381] Cartridges containing vitamin D, calcium, and B vitamins are placed into the "supplement delivery device" and the optimal amount is extracted.

[0382] The device notifies the server of the completion of extraction, and then the user is notified.

[0383] As described above, a system is realized that not only allows users to efficiently ingest the nutrients they need on a daily basis, but also allows them to have supplements adjusted according to their emotional state.

[0384] The processing flow will be explained below.

[0385] Step 1:

[0386] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[0387] Step 2:

[0388] Users use their smartphones to take photos of their meals and upload the images to the application. Next, they use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, which is then synchronized with the application. They also input their daily emotional state into the application.

[0389] Step 3:

[0390] The device receives the food image sent by the user and temporarily stores it in memory, then transmits the image data along with the collected profile information, health data, and emotion data to a cloud server.

[0391] Step 4:

[0392] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[0393] Step 5:

[0394] The server analyzes the user's basic information, health data, and emotional data in combination with AI analysis results to identify the need for supplements based on nutrient deficiencies and emotional state.

[0395] Step 6:

[0396] The emotion engine evaluates the user's emotional state and adjusts the supplement suggestions accordingly, for example, adding suggestions including B vitamins if the user's stress level is high.

[0397] Step 7:

[0398] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[0399] Step 8:

[0400] If the user agrees to the suggestion, the server instructs the user to set the cartridge in the "supplement provision device." The supplement provision device checks the set cartridge and extracts the optimal amount of the required supplement.

[0401] Step 9:

[0402] The supplement providing device completes extraction and sends a completion notification to the server. The server then notifies the terminal and notifies the user that the supplement is ready. The user receives and takes the supplement.

[0403] Step 10:

[0404] The server periodically updates the user's health and emotional data, improves supplement recommendations based on the data, and collects user feedback to improve the system's accuracy and user experience.

[0405] In this way, the system comprehensively manages the user's diet, health data, and emotional state, and can suggest and provide personalized supplements that take into account the user's emotional state.

[0406] Example 2

[0407] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0408] In modern society, it is difficult to maintain a balanced diet, but in order to maintain good health, it is important to consume nutrients that are tailored to each individual's dietary content. Furthermore, stress and emotional states also affect appetite and nutrient intake. However, current systems do not provide a means to individually recommend optimal supplements by taking into account both an analysis of dietary content and emotional state. Therefore, a system is needed that can comprehensively analyze a user's dietary content, health data, and emotional data and provide the necessary nutrients.

[0409] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0410] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information, health data, and emotional data of the user, means for identifying nutrients lacking in the user and nutritional elements based on the emotions based on the basic information, health data, and emotional data, means for suggesting optimal supplements based on the nutrients lacking in the user and nutritional elements based on the emotions, means for selecting supplements based on the suggestions, and means for storing a history of the selection of supplements and collecting feedback. This makes it possible to provide optimal supplements individually by comprehensively considering the user's dietary content and emotional state.

[0411] "User" refers to the user of an information system.

[0412] "Meal Image" refers to photographs or image data that visually record a meal consumed by a user.

[0413] "Means for analyzing images" refers to technologies or algorithms for automatically recognizing and identifying food types and nutrients from received image data.

[0414] "Intaken nutrients" refers to the types and amounts of nutrients obtained from the analyzed diet.

[0415] "Basic Information" refers to personal profile data such as a user's age, gender, weight, height, and blood pressure.

[0416] "Health data" refers to data related to a user's health condition, including information collected from IoT devices such as pedometers, body composition monitors, and blood pressure monitors.

[0417] "Emotional data" refers to data that quantifies or classifies a user's stress level or emotional state.

[0418] "Nutrients that are deficient" refers to nutrients that the user should consume but are identified as being deficient in, based on the user's basic information and health data.

[0419] "Emotion-based nutritional components" refer to nutritional components that need to be supplemented based on the user's emotional data.

[0420] "Optimal supplements" refer to supplements that contain nutrients that need to be supplemented based on the user's individual nutritional deficiencies and emotional state.

[0421] "Means for suggesting supplements" refers to the process or function that suggests specific supplements to users based on the analysis results.

[0422] "Means for extracting supplements" refers to a device or mechanism for providing the appropriate amount of the required supplement based on the proposal.

[0423] "Means for storing history and collecting feedback" refers to a system for recording the history of supplement provision and collecting ratings and feedback from users.

[0424] The present invention relates to a system that comprehensively analyzes a user's daily dietary intake and emotional state, and provides optimal supplements to each individual. This system is composed of the following components:

[0425] System configuration

[0426] The system includes a device such as a smartphone or tablet used by the user, a cloud-based server, an emotion engine, and a supplement providing device. The role and operation of each element are described in detail below.

[0427] User operation

[0428] User:

[0429] 1. The user starts the application and logs in to the system by entering their user ID and password on the login screen. The first time they log in, they enter basic information such as their age, gender, weight, and blood pressure.

[0430] 2. Take photos of your daily meals with your smartphone or tablet and upload the images to the application.

[0431] 3. Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync that data to the application.

[0432] 4. The application periodically asks questions on the screen to check the user's emotional state and collects emotional data.

[0433] Terminal handling

[0434] Device:

[0435] 1. The food image sent by the user is temporarily stored in memory.

[0436] 2. The user's profile information, health data, and emotional data are sent to the cloud server along with the meal image.

[0437] Server Processing

[0438] server:

[0439] 1. The server sends the meal image data to an image analysis AI engine to identify the type and amount of ingredients and nutrients.

[0440] 2. Combining profile information, health data, and emotional data to identify nutrient deficiencies.

[0441] 3. Use an emotion engine to tailor supplement recommendations based on the user's emotional state, for example, suggesting more B vitamins if stress levels are high.

[0442] 4. Send the supplement proposal to the user for review and consent.

[0443] Operating the supplement delivery device

[0444] server:

[0445] 1. Give instructions to the dispenser and set the required supplement cartridge.

[0446] 2. The supplement delivery device extracts the optimal amount of supplement and prepares it for delivery in powder form.

[0447] Specific operation example

[0448] Example 1: Breakfast analysis and supplement provision:

[0449] 1. A user has an omelet, salad, and orange juice for breakfast, takes a photo of it, and uploads the image to the application.

[0450] 2. The device temporarily stores the food image and then sends it to a cloud server.

[0451] 3. The server uses an AI engine to analyze the image and identify the ingredients in the omelet (protein, fat, etc.), the ingredients in the salad (vitamins, minerals, etc.), and the ingredients in the orange juice (vitamin C, etc.).

[0452] 4. The basic information, health data, and emotional data are combined to suggest adding vitamin D and calcium, as well as B vitamins, as the user is under a lot of stress.

[0453] 5. The user reviews and agrees to the proposal.

[0454] 6. The required cartridge is inserted into the supplement delivery device, the optimal amount is extracted, the device notifies the user that it is ready, and the user receives it.

[0455] Prompt Sentence Examples

[0456] "A user has an omelet, salad, and orange juice for breakfast and takes a photo of it. Analyze the ingredients of this meal, identify any nutrient deficiencies, and suggest appropriate supplements, taking into account the user's stress level."

[0457] The above is a specific embodiment for carrying out the present invention.

[0458] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0459] Step 1:

[0460] User:

[0461] A user starts an application and logs in by entering their user ID and password on the login screen. The login information is entered and sent to the server.

[0462] Input: User ID, Password

[0463] Output: Login authentication result

[0464] Specific operation: The application sends user information to the server, and the server performs the authentication process. If authentication is successful, a message indicating successful login is displayed and the user proceeds to the next screen. When logging in for the first time, a screen appears where basic information such as age, gender, weight, and blood pressure can be entered.

[0465] Step 2:

[0466] User:

[0467] Users take photos of their daily meals using a smartphone or tablet, and then use the application's upload function to send the images to the server.

[0468] Input: Food image

[0469] Output: Image data sent to the server

[0470] Specific operation: The user uses the "Photo and upload food" function in the application, selects the photo they have taken, and sends it to the server. The photo is temporarily stored in the device's local storage and then uploaded to the server.

[0471] Step 3:

[0472] Device:

[0473] The device receives the image of the meal, temporarily stores it in memory, and then transmits the image data to a cloud server.

[0474] Input: Image data sent by the user

[0475] Output: Image data sent to the cloud server

[0476] Specific operation: The device temporarily stores the received image data, then sends it to the cloud server via an HTTP request. After sending, a success notification is displayed to the user.

[0477] Step 4:

[0478] User:

[0479] Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync the data to the application.

[0480] Input: Health data from IoT devices

[0481] Output: Health data synced to the application

[0482] Specific operation: The IoT device sends health data to the terminal via Bluetooth or Wi-Fi, and the application receives and stores the data. If necessary, the data is sent to the server.

[0483] Step 5:

[0484] Device:

[0485] To periodically check the user's emotional state, emotional questions are displayed on the screen to collect the user's emotional data.

[0486] Input: User sentiment response

[0487] Output: Collected emotion data

[0488] How it works: The application periodically displays a pop-up notification asking for the user's sentiment rating and collects their response. The collected data is temporarily stored within the application and then sent to the server.

[0489] Step 6:

[0490] server:

[0491] The server sends the meal image data to an image analysis AI engine, which identifies the type and amount of ingredients and their nutrients.

[0492] Input: Image data sent to the cloud server

[0493] Output: Analysis of ingredients and nutrients

[0494] How it works: The server sends the received meal image to the AI ​​engine, which uses a deep learning model to analyze the image and identify the types and amounts of ingredients and nutrients. The results are returned to the server in JSON format.

[0495] Step 7:

[0496] server:

[0497] It combines profile information, health data, and emotional data to identify nutrient deficiencies and also determines nutritional factors based on emotional data.

[0498] Input: Profile information, health data, emotion data, image analysis results

[0499] Output: Identification of nutritional deficiencies and emotional nutritional factors

[0500] How it works: The server loads the user's profile information, health data, and emotion data from the database, and integrates them with the image analysis results. It uses an analysis algorithm to identify nutrient deficiencies and emotion-based nutritional factors.

[0501] Step 8:

[0502] server:

[0503] The supplement proposal is sent to the user's device and confirmation and consent are requested.

[0504] Input: Identification of nutritional deficiencies and emotional nutritional factors

[0505] Output: Supplement suggestions sent to the user's device

[0506] Specific operation: The server generates supplement suggestions based on the nutrient deficiency and emotion data, and sends them to the user's device as a notification. The user confirms the suggestions and presses the "Agree" button to proceed to the next step.

[0507] Step 9:

[0508] server:

[0509] Give instructions to the dispenser and load the required supplement cartridge, extracting the appropriate amount of supplement and preparing it for dispensing in powder form.

[0510] Input: User confirmation and consent, supplement suggestions

[0511] Output: Notification that supplement is ready to be extracted

[0512] Specific operation: The server sends a command to the dispenser to automatically load the required supplement cartridge. The dispenser extracts the optimal amount of supplement and notifies the server that it is ready. The server then finally notifies the user that it is ready.

[0513] The above is the flow of specific processing steps of the program of this system.

[0514] (Application example 2)

[0515] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0516] Conventional supplement recommendation systems were based solely on the user's dietary and health data, and were unable to take into account the user's emotional state. This meant that they were unable to meet the nutritional needs of users affected by stress and emotional fluctuations, making it difficult to recommend optimal supplements tailored to individual needs. Another issue was the inability to provide immediate recommendations in physical stores.

[0517] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0518] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for collecting and analyzing emotional information of the user, means for suggesting optimal supplements based on the nutrient deficiencies and emotional information, means for selecting supplements based on the suggestions, and means for storing and collecting a history of supplement selection and user feedback. This enables personalized optimal supplement suggestions based on the user's diet, health data, and emotional state, and immediate provision of supplements in physical stores.

[0519] "Users" refers to people who use the system.

[0520] "Means for receiving images" refers to a device or software that captures images of meals taken or selected by a user into the system.

[0521] "Means for analyzing images to identify nutrients ingested" refers to devices or software that use AI or machine learning to analyze the type of food ingredients and nutrient content from received images.

[0522] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[0523] "Health Data" refers to health-related data such as a user's body composition, blood pressure, heart rate, and number of steps taken.

[0524] "Emotional information" refers to information that indicates the user's emotional state, such as stress or happiness.

[0525] "Missing nutrients" refers to the user's basic information and health data, as well as nutrients necessary for maintaining and improving future health.

[0526] "Supplements" refer to nutritional supplements that are used to supplement the user's nutritional deficiencies.

[0527] "Means for suggesting" refers to a device or software that suggests appropriate supplements based on the analysis results and the user's emotional information.

[0528] "Extracting means" refers to a device or software that provides the user with the appropriate amount of the suggested supplement.

[0529] "Means for storing and collecting history and feedback" refers to devices or software that record and store information about the supplements received by users and their impressions after use.

[0530] "IoT devices" refer to health devices that can be connected to the Internet (such as body composition monitors and blood pressure monitors).

[0531] "Physical store terminals" refers to dedicated terminals or tablets installed in physical stores such as healthcare shops and drugstores.

[0532] This invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. The system is comprised of a user's device, a cloud-based server, an emotion engine, and a supplement provider.

[0533] System configuration

[0534] User device functions

[0535] 1. User authentication and profile management

[0536] The user starts the application and enters their profile information (age, gender, weight, blood pressure, etc.) when using it for the first time. This information is sent to the server and saved.

[0537] 2. Take and upload a photo of your meal

[0538] Users take photos of their daily meals using a device (smartphone, tablet, etc.) and upload the images to the application.

[0539] 3. Collecting Emotional Data

[0540] The application periodically records the user's emotional state, and this emotional data is collected through the user's touch, voice input, or facial recognition technology.

[0541] Server Features

[0542] 1. Image analysis and nutrient identification

[0543] The server passes user-submitted images to an AI model that uses software such as TensorFlow and Keras to identify the type of food and its nutrients.

[0544] 2. Emotion analysis

[0545] The server passes the emotional data sent by the user to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms.

[0546] 3. Recommending the best supplements

[0547] The server identifies nutrient deficiencies based on the user's basic information, health data, and emotional data, and automatically generates optimal supplements.

[0548] Functions of the supplement providing device

[0549] 1. Extraction of Supplements

[0550] The supplement providing device dispenses the appropriate amount of supplements containing the nutrients required by the user based on instructions from the server. The device dispenses only the required amount after inserting a supplement cartridge.

[0551] 2. Submissions and Feedback Notifications

[0552] The extracted supplements are provided to the user, and their history and feedback are stored on a server.

[0553] Specific examples

[0554] Example 1: Breakfast analysis and supplement provision

[0555] 1. User Operation

[0556] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[0557] Photos of the meal are uploaded to the application and sent to the server.

[0558] 2. Image analysis and emotion analysis

[0559] The server uses an AI engine to analyze the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[0560] In addition, it analyzes the user's emotional data to detect high levels of stress.

[0561] 3. Supplement suggestions and selection

[0562] The server suggests supplementing with vitamin D and calcium, which are deficient, as well as B vitamins.

[0563] The supplement delivery device extracts this and provides it to the user.

[0564] Example prompts for generative AI models

[0565] "Build an image classification model to analyze images of meals taken by users and identify the nutrients contained in them. The model should be able to recognize multiple nutrients such as protein, fat, vitamins, and minerals and estimate the amount of each. Also, build an emotion classification model to analyze the user's emotional state from text and identify emotions such as stress or happiness. Integrate this data and generate relevant suggestions for a system that recommends optimal supplements."

[0566] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0567] Step 1:

[0568] User login and profile entry

[0569] Users launch the application and enter their login information to access the system. When using it for the first time, they enter profile information such as age, gender, weight, and blood pressure. This profile information is sent as input data to the cloud server and saved there.

[0570] Input: User login and profile information

[0571] Output: User profile information stored on the server

[0572] Step 2:

[0573] Taking and uploading food photos

[0574] Users take photos of their daily meals with their smartphone or tablet and upload the images to the application, which receives the images and sends them to a cloud server.

[0575] Input: User-taken food image

[0576] Output: Meal images stored on the server

[0577] Step 3:

[0578] Health data collection

[0579] Users measure their own health data using IoT devices such as body composition scales and blood pressure monitors, and synchronize this data with the application. The device receives the health data and sends it to a cloud server.

[0580] Input: Health data collected from IoT devices

[0581] Output: Health data stored on the server

[0582] Step 4:

[0583] Collecting Emotional Data

[0584] The application periodically records the user's emotional state. The user provides emotional information through touch, voice, or facial recognition, and the emotional data is sent from the device to a cloud server.

[0585] Input: User emotion data (touch, voice, facial recognition)

[0586] Output: Emotion data stored on the server

[0587] Step 5:

[0588] Image analysis and nutrient identification

[0589] The server passes the uploaded meal images to an AI model, which analyzes the types of ingredients and nutrient content. It uses TensorFlow and Keras to analyze the images and generate specific nutrient data.

[0590] Input: Food images stored on the server

[0591] Output: Parsed nutrient data

[0592] Step 6:

[0593] Emotion analysis

[0594] The server passes the collected emotional data to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms to generate emotional data.

[0595] Input: Emotion data stored on the server

[0596] Output: Parsed emotional state data

[0597] Step 7:

[0598] Recommendations for optimal supplements

[0599] The server combines the user's basic information, health data, nutrient data, and emotional data, and the system automatically generates optimal supplement recommendations based on the nutrients that are lacking and the user's emotional state.

[0600] Input: Basic information, health data, nutrition data, emotional state data

[0601] Output: Proposed supplemental data

[0602] Step 8:

[0603] Supplement Extract

[0604] The supplement providing device extracts the appropriate amount of supplement containing the necessary nutrients based on the suggested data from the server. The supplement cartridge is set and the extraction process is carried out.

[0605] Input: Suggested supplemental data from the server

[0606] Output: Extracted supplement

[0607] Step 9:

[0608] Offer and Feedback Notification

[0609] The extracted supplements are provided to the user, and their usage history and feedback are collected. The user receives the supplements and provides feedback through the application about their impressions after use. This data is stored on the server.

[0610] Input: User feedback and supplement history

[0611] Output: Feedback and historical data stored on the server

[0612] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0613] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0614] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0615] [Second embodiment]

[0616] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0617] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0618] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0619] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0620] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0621] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0622] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0623] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0624] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0625] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0626] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0627] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0628] The present invention relates to a system that allows users to take photos of their meals, performs nutritional analysis using AI, and provides optimal supplements to individuals. This system is composed of multiple elements, including a user, a terminal, and a server. Specific embodiments are described in detail below.

[0629] System configuration

[0630] The system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, and a supplement delivery device. The role of each element is explained below.

[0631] User operations

[0632] user:

[0633] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[0634] Users take photos of their daily meals using a mobile device and upload them to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used, and the data is synchronized.

[0635] Terminal handling

[0636] Device:

[0637] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[0638] Collects user profile information and health data and sends it to a server.

[0639] Server Processing

[0640] server:

[0641] The server sends the user's meal image to the AI ​​engine for image analysis, which uses image analysis algorithms to identify the ingredients consumed and their nutritional content.

[0642] It analyzes nutrient deficiencies by combining the user's basic information and health data.

[0643] Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[0644] Based on the suggestion, instructions to extract the supplement are sent to the "supplement providing device."

[0645] Operating the supplement delivery device

[0646] server:

[0647] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[0648] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[0649] Specific examples

[0650] A specific example will be described below.

[0651] Example 1: Breakfast analysis and supplement provision

[0652] 1. User Operation

[0653] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[0654] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[0655] 2. Terminal Processing

[0656] The device receives the image of the meal, stores it in memory, and sends it to the server.

[0657] 3. Server Processing

[0658] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[0659] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[0660] The server calculates the optimal combination of supplements, including vitamin D and calcium, and sends the recommendations to the user.

[0661] The user reviews and accepts the proposal.

[0662] 4. Supplement Extraction

[0663] Vitamin D and calcium cartridges are placed in the "supplement supply device" and the optimal amount is extracted.

[0664] The device notifies the server that it is ready, and the user is notified.

[0665] As described above, this system comprehensively manages the user's dietary and health data, and individually replenishes necessary nutrients, thereby achieving balanced nutritional intake.

[0666] The processing flow will be explained below.

[0667] Step 1:

[0668] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[0669] Step 2:

[0670] Users take photos of their meals using their smartphones and upload the images to the application. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, and synchronize that data with the application.

[0671] Step 3:

[0672] The device receives the meal image sent by the user and temporarily stores it in memory. The image data is then sent to a cloud server, along with the user's profile information and health data.

[0673] Step 4:

[0674] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[0675] Step 5:

[0676] The server analyzes the user's basic information and health data in combination with the AI ​​analysis results, identifies any nutrient deficiencies, and calculates the optimal combination of supplements based on that information.

[0677] Step 6:

[0678] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[0679] Step 7:

[0680] If the user agrees to the suggestion, the server instructs the user to insert the cartridge into the "supplement providing device." The device then checks the inserted cartridge and extracts the optimal amount of the required supplement.

[0681] Step 8:

[0682] The supplement providing device completes the extraction and sends a completion notification to the server, which then notifies the terminal that the supplement is ready.

[0683] Step 9:

[0684] The user receives and takes the supplements. This updates the user's health data and history, leading to improved accuracy of the next recommendations. The server periodically collects user feedback and uses it to improve the system.

[0685] In this way, a system is realized that allows users to efficiently take in the nutrients they need on a daily basis.

[0686] Example 1

[0687] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] In today's world, it is difficult to properly manage individual nutritional status and quickly and accurately replenish individual nutrient deficiencies. Conventional systems lack the ability to comprehensively analyze users' dietary and health data, limiting manual data entry and supplement selection. Furthermore, there is no system that utilizes IoT devices to automatically collect and comprehensively analyze health data, resulting in the problem of inappropriate individual nutritional supplementation.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0690] In this invention, the server includes means for receiving meal images from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for identifying nutrient deficiencies in the user based on the basic information and health data, means for suggesting optimal supplements based on the nutrient deficiencies, means for selecting supplements based on the suggestions, means for storing a history of supplement selection and collecting feedback, means for using a cloud-based AI engine to analyze the meal images, means for using the AI ​​engine to compare the image analysis results with a nutritional information database, means for automatically collecting the user's health data via an IoT device, and means for combining and analyzing the user's basic information, health data, and dietary data using a statistical analysis tool. This allows for integrated management of the user's diet and health data, enabling prompt and appropriate individual nutritional supplementation.

[0691] "User" refers to an individual who uses the System.

[0692] "Meal Image" refers to an image file containing visual data of the meal the user consumes.

[0693] "Means for analyzing images to identify ingested nutrients" refers to technologies and algorithms for recognizing ingredients in images of food and identifying the nutritional components of those ingredients.

[0694] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[0695] "Health data" refers to data that indicates a user's health status, such as the number of steps taken, body fat percentage, and blood pressure.

[0696] "Means for identifying nutrient deficiencies" refers to technology that analyzes and identifies nutrients that are lacking in the user's current diet based on the user's basic information and health data.

[0697] "Means for suggesting optimal supplements" refers to technology that calculates the optimal combination of supplements to make up for missing nutrients and suggests them to users.

[0698] "Means for extracting supplements" refers to the equipment and technology used to prepare and deliver the proposed supplements to the user in the appropriate amounts.

[0699] "Means for storing the history of supplement extraction and collecting feedback" refers to the function of storing the history of supplement use and collecting opinions and experiences from users.

[0700] A "cloud-based AI engine" refers to an artificial intelligence processing system that operates in a cloud environment.

[0701] "Nutrition information database" refers to a database that accumulates data on the nutritional components of various food ingredients.

[0702] "IoT devices" refer to health management devices that are capable of transmitting data via the Internet.

[0703] "Statistical analysis tools" refer to software tools used to analyze various types of data and derive specific patterns or trends.

[0704] This invention relates to a system that allows users to take photos of their meals, analyzes their nutritional intake using AI, and provides optimal supplements for each individual. The invention is composed of multiple elements, including the user, a terminal, and a server.

[0705] System configuration

[0706] This system consists of a user device (such as a smartphone or tablet), a cloud-based server, and a supplement delivery device. Each element is described in detail below.

[0707] User operations

[0708] user:

[0709] 1. Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, and blood pressure.

[0710] 2. Take photos of your daily meals with your smartphone camera and upload them to the application.

[0711] 3. Sync IoT devices such as pedometers, body composition monitors, and blood pressure monitors with your smartphone to collect health data.

[0712] Terminal handling

[0713] Device:

[0714] 1. Receives the food image sent by the user and temporarily stores it in memory. The stored image data is then sent to the cloud server.

[0715] 2. It also collects user profile information and health data and sends them to the server.

[0716] Server Processing

[0717] server:

[0718] 1. The received meal image is sent to a cloud-based AI engine for image analysis. Specifically, an image analysis tool (e.g., Google Cloud Vision API) is used to identify the ingredients ingested from the image.

[0719] 2. The image analysis results are compared with a nutritional information database to identify the nutritional components of each ingredient.

[0720] 3. Analyze the user's basic information and health data using statistical analysis tools (e.g., Python, R) to identify nutrient deficiencies.

[0721] 4. Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[0722] 5. Insert the required supplement cartridge into the supplement providing device and send instructions to extract the optimal amount.

[0723] Operating the supplement delivery device

[0724] server:

[0725] 1. Place the cartridge into the supplement dispenser and extract the optimal amount of supplement you need.

[0726] 2. Provides extracted supplements in powder form and notifies you when the device is ready.

[0727] Specific examples

[0728] Specific examples are given below:

[0729] Example 1: Breakfast analysis and supplement provision

[0730] 1. User Action:

[0731] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it with their smartphone.

[0732] Photos of meals are uploaded to the application and user profile information is sent to the server.

[0733] 2. Terminal processing:

[0734] The device receives the image of the meal, stores it in memory, and sends it to a cloud server.

[0735] 3. Server processing:

[0736] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[0737] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[0738] The server calculates the optimal combination of supplements containing vitamin D and calcium and sends the recommendation to the user, who then reviews and accepts the recommendation.

[0739] 4. Supplement Extraction:

[0740] Insert vitamin D and calcium cartridges into the supplement dispenser and extract the optimal amount.

[0741] The device notifies the server that it is ready and notifies the user.

[0742] Prompt Sentence Examples

[0743] Examples of prompts to input to a generative AI model might include:

[0744] "If a user wants to take a photo of an omelet, salad, and orange juice and run a nutritional analysis, what steps should they take?"

[0745] By inputting this prompt, the generative AI model will guide the user through the nutritional analysis process of the meal and generate an optimal supplement delivery flow.

[0746] As described above, this system comprehensively manages the user's dietary and health data, enabling prompt and appropriate individual nutritional supplementation.

[0747] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0748] Step 1: User logs in

[0749] Specific behavior:

[0750] The user launches the app on their smartphone, enters their user ID and password on the login screen, and presses the "Login" button.

[0751] Input: User ID, Password

[0752] Output: User credentials

[0753] Data processing / calculation: The server checks the user ID and password against the database to perform authentication.

[0754] Step 2: User enters profile information

[0755] Specific behavior:

[0756] When logging in for the first time, the user enters personal information such as age, gender, weight, and blood pressure on the profile entry screen that appears, and presses the "Save" button.

[0757] Input: Profile information such as age, gender, weight, blood pressure, etc.

[0758] Output: User profile data

[0759] Data processing / calculation: The terminal formats the input information and sends it to the server.

[0760] Step 3: User takes and uploads a photo of their meal

[0761] Specific behavior:

[0762] Users take a photo of their meal with their smartphone camera and press the "upload" button to upload the image to the app.

[0763] Input: Food image file

[0764] Output: Saved food image data

[0765] Data processing / calculation: The device temporarily stores the image file in memory, compresses it, and then sends it to the cloud server.

[0766] Step 4: User synchronizes data on IoT devices

[0767] Specific behavior:

[0768] The user turns on Bluetooth on each IoT device and smartphone and presses the "Data Sync" button in the app.

[0769] Input: IoT device data (number of steps, body fat percentage, blood pressure, etc.)

[0770] Output: Collected health data

[0771] Data processing / calculation: The terminal receives data from the IoT device and sends it to the server in bulk.

[0772] Step 5: The device temporarily saves the food image and sends it to the server.

[0773] Specific behavior:

[0774] The device temporarily stores the food images sent by the user in memory and sends them to a cloud server.

[0775] Input: Food image data

[0776] Output: Image data stored on a cloud server

[0777] Data processing / calculation: The device compresses the food images and sends them to a cloud server via the network.

[0778] Step 6: The device collects the user's profile information and health data and sends it to the server.

[0779] Specific behavior:

[0780] The device collects the user's profile information and health data from IoT devices and sends it to a cloud server.

[0781] Input: Profile information, health data

[0782] Output: User profile information and health data stored on a cloud server

[0783] Data processing / calculation: The data collected by the terminal is formatted and sent in bulk to the cloud server.

[0784] Step 7: The server sends the food image to the AI ​​engine for image analysis.

[0785] Specific behavior:

[0786] The server sends the received food images to a cloud-based AI engine, which analyzes the images.

[0787] Input: Food image data

[0788] Output: Parsed ingredient information

[0789] Data processing / computation: A cloud-based AI engine (e.g., Google Cloud Vision API) identifies ingredients from images and recognizes their nutritional content.

[0790] Step 8: The server compares the image analysis results with the nutrition information database.

[0791] Specific behavior:

[0792] The server compares the image analysis results with a nutritional information database to identify the nutritional components of each ingredient.

[0793] Input: Ingredient information

[0794] Output: Nutritional information

[0795] Data processing / calculation: The server compares the image analysis results with a database and extracts nutrient data.

[0796] Step 9: The server analyzes the user's basic information and health data to identify nutrient deficiencies.

[0797] Specific behavior:

[0798] The server uses statistical analysis tools (e.g., Python, R) to combine and analyze the user's basic information, health data, and dietary information.

[0799] Input: Basic information, health data, nutritional information

[0800] Output: Missing nutrient information

[0801] Data processing / calculation: Statistical analysis tools analyze the input data and identify nutrient deficiencies.

[0802] Step 10: The server calculates the optimal supplement combination and sends the proposal to the user.

[0803] Specific behavior:

[0804] The server uses an algorithm to generate a list of supplements containing the nutrients to be supplemented and sends suggestions to the user.

[0805] Input: Missing nutrient information

[0806] Output: Supplement suggestion information

[0807] Data processing / calculation: The server calculates the optimal combination of supplements based on the nutrients that are lacking and generates recommendations.

[0808] Step 11: The server sends an instruction to the supplement providing device to extract the supplement.

[0809] Specific behavior:

[0810] The server sets the necessary supplement cartridge in the supplement providing device and sends instructions to extract the optimal amount.

[0811] Input: Supplement suggestion information

[0812] Output: Extracted supplement

[0813] Data processing / calculation: The server sends commands to the provider based on the suggested information and extracts the appropriate supplements.

[0814] The above are the specific processing steps of this system.

[0815] (Application example 1)

[0816] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0817] As health consciousness grows, there is a demand for personalized nutritional management and optimal nutritional supplements. However, conventional systems have made it difficult to provide individually tailored nutritional supplements in physical stores, and they have been unable to respond in real time based on users' health data. This has made it difficult for users to quickly obtain the optimal nutritional supplements tailored to their health condition.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0819] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for extracting optimal amounts of nutritional supplements using a device in a physical store, means used in the physical store for providing the nutritional supplements, and means for storing a history of the provision of nutritional supplements and collecting feedback, thereby enabling users to quickly receive optimal nutritional supplements based on their own health condition at the physical store.

[0820] A "user" is a person who uses the system to take photos of meals and perform nutritional analysis.

[0821] "Meal images" are photographic data that visually record the contents of meals consumed by the user.

[0822] "Nutrients" are components such as vitamins, minerals, and proteins obtained from the food a user consumes.

[0823] "Basic information" refers to personal data such as the user's age, gender, weight, height, and health condition.

[0824] "Health data" refers to data that indicates the user's current health status, such as the number of steps taken, blood pressure, and body composition.

[0825] "Dietary supplements" are foods such as supplements and energy drinks that are suggested or offered to users to supplement nutrients that they are lacking.

[0826] A "brick and mortar store" is a physical store where a user can visit and receive nutritional supplements.

[0827] A "server" is a computer system that receives, analyzes, and processes data from users.

[0828] "Suggestions" are recommendations to provide optimal nutritional supplements to users based on their nutrient deficiencies.

[0829] "Extraction" is the act of extracting a specific amount of a dietary supplement needed based on a proposal.

[0830] "Device" refers to a device that is installed in a physical store and provides nutritional supplements in accordance with instructions from the server.

[0831] "History" refers to the record of dietary supplements provided to a user and their feedback data.

[0832] "Feedback" is information provided by users through comments and ratings about the effectiveness and satisfaction of a dietary supplement.

[0833] MODE FOR CARRYING OUT THE INVENTION

[0834] This invention is a system that allows users to take photos of their meals, upload them to the cloud, perform nutritional analysis, and provide optimal nutritional supplements to individuals in physical stores. Specific embodiments are described below.

[0835] System configuration

[0836] The system consists of a device used by users, such as a smartphone or tablet, a cloud-based server, and a nutritional supplement supply device in a physical store. The role of each element is explained below.

[0837] User operations

[0838] user:

[0839] Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, height, and blood pressure.

[0840] Users take photos of their daily meals with their smartphones and upload them to the app. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data and synchronize the data.

[0841] Terminal handling

[0842] Device:

[0843] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[0844] Collects user profile information and health data and sends it to a server.

[0845] Server Processing

[0846] server:

[0847] The server receives the user's food images and analyzes the meal contents using a generative AI model, which utilizes machine learning libraries such as TensorFlow and PyTorch.

[0848] The system analyzes nutrient deficiencies by combining the user's basic information and health data. It uses Python to process the health data and compare it with known nutritional data stored in a database to identify nutrient deficiencies.

[0849] The server then suggests optimal nutritional supplements based on the nutrients that are lacking and notifies the user, who is then sent a notification to their smartphone.

[0850] Operating a nutritional supplement delivery device

[0851] Device:

[0852] The nutritional supplement supply device installed in the physical store receives instructions from the server and extracts the optimal nutritional supplements. This process is carried out using IoT control software using Node.js and Python.

[0853] The device will then be notified that the correct amount of nutritional supplement is ready to be dispensed to the user.

[0854] Specific examples

[0855] Example: Breakfast analysis and nutritional supplement provision

[0856] 1. User Operation

[0857] The user eats a sandwich, salad, and orange juice for breakfast and takes a photo of it.

[0858] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[0859] 2. Terminal Processing

[0860] The device receives the image of the meal, stores it in memory, and sends it to the server.

[0861] Profile information and health data are also sent to the server.

[0862] 3. Server Processing

[0863] The server sends the images to a generative AI model that analyzes the ingredients (e.g., protein in a sandwich, vitamins in a salad, vitamin C in orange juice).

[0864] This is combined with the user's basic information and health check data to identify vitamin D and calcium deficiencies.

[0865] The server calculates the optimal combination of nutritional supplements containing vitamin D and calcium and sends the proposal to the user, who then reviews and accepts the proposal.

[0866] 4. Nutritional supplements provided

[0867] Vitamin D and calcium ingredients are placed in a dispenser in a physical store, and the optimal amount is extracted.

[0868] The device notifies the server that it is ready, and the user is notified.

[0869] Example prompts to input to a generative AI model:

[0870] Meal image: "https: / / example.com / user_images / breakfast.jpg"

[0871] User profile: Age 30, Gender female, Weight 60kg, Height 160cm, Recent problems: Tired easily

[0872] Health data: Steps 8000, Blood pressure 120 / 80, Body fat 22%

[0873] This prompt text is imported into the server and analyzed to create a system that provides appropriate nutritional supplements.

[0874] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0875] Step 1:

[0876] user:

[0877] The user takes a picture of their meal, such as breakfast, with their smartphone, and the image is saved in an application on their smartphone.

[0878] Input: Food image

[0879] Output: Image data stored on the smartphone

[0880] Specific operation: Take a photo of your meal using the smartphone's camera app, and the image will be automatically saved in the application folder.

[0881] Step 2:

[0882] user:

[0883] Launch the application and enter your login information to log in to the system. The user will then confirm and enter their profile information (age, gender, weight, etc.) and sync their health data (step count, blood pressure, etc.).

[0884] Input: Login information, profile information, health data

[0885] Output: User data stored in the application

[0886] Specific operation: Enter login information on the login screen, confirm and enter profile information, and sync health data from IoT devices to the application.

[0887] Step 3:

[0888] Device:

[0889] The images of the meal taken by the user, along with the profile information and health data entered, are sent to a cloud server.

[0890] Input: Food images, profile information, health data

[0891] Output: Data sent to the cloud server

[0892] Specific operation: Use the application's upload function to send the selected images and synchronized data to the cloud server.

[0893] Step 4:

[0894] server:

[0895] The cloud server inputs the received image data into the generative AI model and performs a nutritional analysis of the meal contents, using libraries such as TensorFlow and PyTorch.

[0896] Input: Food image

[0897] Output: Nutritional data for meals

[0898] How it works: The generative AI model analyzes image data and identifies the nutritional content of each ingredient. For example, it calculates the protein content of an omelet or the vitamin content of a salad.

[0899] Step 5:

[0900] server:

[0901] Using the user's basic information and health data, Python is used to run an algorithm to identify nutrient deficiencies.

[0902] Input: Profile information, health data, dietary nutrient data

[0903] Output: List of nutrients that are lacking

[0904] What it does: It compares the received data with an existing nutrition database and calculates the nutrient deficiencies the user may have (e.g., vitamin D or calcium).

[0905] Step 6:

[0906] server:

[0907] Based on the identified nutrient deficiencies, the generative AI model will suggest optimal nutritional supplements and send a smartphone notification to the user.

[0908] Input: List of nutrients you are deficient in

[0909] Output: Nutritional supplement suggestions, notification data

[0910] What it does: Runs a recommendation generation algorithm based on nutrient deficiencies and sends a push notification to the user's smartphone. Example: "Supplement ABC is recommended to replenish vitamin D and calcium."

[0911] Step 7:

[0912] user:

[0913] Review smartphone notifications and agree to suggested dietary supplements.

[0914] Input: Notification data

[0915] Output: User consent data

[0916] Specific actions: Check the proposal on the notification screen and press the agree button.

[0917] Step 8:

[0918] server:

[0919] With the user's consent, instructions are sent to a nutritional supplement dispenser in the physical store to extract the optimal amount of nutritional supplement needed.

[0920] Input: User consent data

[0921] Output: Extraction instructions for dietary supplements

[0922] Specific operation: Using IoT control software (Node.js or Python), it sends extraction instructions to the dispenser. For example, it sends an instruction to extract a predetermined amount of nutritional supplement from a specific cartridge.

[0923] Step 9:

[0924] Device:

[0925] A nutritional supplement dispenser in a physical store receives the instruction, extracts the optimal amount of the specified nutritional supplement, and notifies the server and the user that it is ready to be dispensed.

[0926] Input: Extraction instruction data

[0927] Output: Ready notification

[0928] Specific operation: The device extracts the nutritional supplement and notifies the server that it is ready, which then pushes the notification to the user's device.

[0929] Step 10:

[0930] server:

[0931] It stores the history of the nutritional supplements provided and also collects feedback from users and stores it in a database.

[0932] Input: Provision history data, feedback data

[0933] Output: Updated database

[0934] Specific operation: The provision history and feedback information are stored in a database and used to improve the proposal algorithm in the future.

[0935] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0936] The present invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. This system is comprised of a user, a terminal, a server, and a supplement provider, and has the function of adjusting supplement recommendations based on the user's emotion data. Specific embodiments are described in detail below.

[0937] System configuration

[0938] This system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, an emotion engine, and a supplement delivery device. The role of each element is explained below.

[0939] User operation

[0940] User:

[0941] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[0942] Users take photos of their daily meals using a mobile device and upload the images to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used and the data is synchronized.

[0943] The application periodically checks and collects the user's emotional state to record their emotions.

[0944] Terminal handling

[0945] Device:

[0946] The system receives the food image sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[0947] The user's profile information, health data, and emotional data are transmitted to a server.

[0948] Server Processing

[0949] server:

[0950] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[0951] The system analyzes a user's basic information, health data, and emotional data to identify any nutrient deficiencies.

[0952] The emotional engine tailors supplement recommendations based on the user's emotional state, for example, suggesting a boost in B vitamins if stress levels are high.

[0953] The supplement proposal is sent to the user, and once the user confirms and agrees, the process proceeds to the next step.

[0954] Operating the supplement delivery device

[0955] server:

[0956] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[0957] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[0958] Specific examples

[0959] A specific example will be described below.

[0960] Example 1: Breakfast analysis and supplement provision

[0961] 1. User Operation

[0962] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[0963] Photos of the meal are uploaded to the application and sent to the server along with the profile information entered when logging in.

[0964] 2. Terminal Processing

[0965] The device receives the image of the meal, stores it in memory, and then sends it to the server.

[0966] 3. Server Processing

[0967] The AI ​​engine analyzes the image to determine the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[0968] Basic information, health data, and the user's emotional data (e.g., high stress levels) are combined to provide vitamin D and calcium deficiencies, as well as B vitamins based on the user's emotions.

[0969] The supplement proposal is sent to the user for review and consent.

[0970] 4. Supplement Extraction

[0971] Cartridges containing vitamin D, calcium, and B vitamins are placed into the "supplement delivery device" and the optimal amount is extracted.

[0972] The device notifies the server of the completion of extraction, and then the user is notified.

[0973] As described above, a system is realized that not only allows users to efficiently ingest the nutrients they need on a daily basis, but also allows them to have supplements adjusted according to their emotional state.

[0974] The processing flow will be explained below.

[0975] Step 1:

[0976] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[0977] Step 2:

[0978] Users use their smartphones to take photos of their meals and upload the images to the application. Next, they use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, which is then synchronized with the application. They also input their daily emotional state into the application.

[0979] Step 3:

[0980] The device receives the food image sent by the user and temporarily stores it in memory, then transmits the image data along with the collected profile information, health data, and emotion data to a cloud server.

[0981] Step 4:

[0982] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[0983] Step 5:

[0984] The server analyzes the user's basic information, health data, and emotional data in combination with AI analysis results to identify the need for supplements based on nutrient deficiencies and emotional state.

[0985] Step 6:

[0986] The emotion engine evaluates the user's emotional state and adjusts the supplement suggestions accordingly, for example, adding suggestions including B vitamins if the user's stress level is high.

[0987] Step 7:

[0988] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[0989] Step 8:

[0990] If the user agrees to the suggestion, the server instructs the user to set the cartridge in the "supplement provision device." The supplement provision device checks the set cartridge and extracts the optimal amount of the required supplement.

[0991] Step 9:

[0992] The supplement providing device completes extraction and sends a completion notification to the server. The server then notifies the terminal and notifies the user that the supplement is ready. The user receives and takes the supplement.

[0993] Step 10:

[0994] The server periodically updates the user's health and emotional data, improves supplement recommendations based on the data, and collects user feedback to improve the system's accuracy and user experience.

[0995] In this way, the system comprehensively manages the user's diet, health data, and emotional state, and can suggest and provide personalized supplements that take into account the user's emotional state.

[0996] Example 2

[0997] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0998] In modern society, it is difficult to maintain a balanced diet, but in order to maintain good health, it is important to consume nutrients that are tailored to each individual's dietary content. Furthermore, stress and emotional states also affect appetite and nutrient intake. However, current systems do not provide a means to individually recommend optimal supplements by taking into account both an analysis of dietary content and emotional state. Therefore, a system is needed that can comprehensively analyze a user's dietary content, health data, and emotional data and provide the necessary nutrients.

[0999] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1000] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information, health data, and emotional data of the user, means for identifying nutrients lacking in the user and nutritional elements based on the emotions based on the basic information, health data, and emotional data, means for suggesting optimal supplements based on the nutrients lacking in the user and nutritional elements based on the emotions, means for selecting supplements based on the suggestions, and means for storing a history of the selection of supplements and collecting feedback. This makes it possible to provide optimal supplements individually by comprehensively considering the user's dietary content and emotional state.

[1001] "User" refers to the user of an information system.

[1002] "Meal Image" refers to photographs or image data that visually record a meal consumed by a user.

[1003] "Means for analyzing images" refers to technologies or algorithms for automatically recognizing and identifying food types and nutrients from received image data.

[1004] "Intaken nutrients" refers to the types and amounts of nutrients obtained from the analyzed diet.

[1005] "Basic Information" refers to personal profile data such as a user's age, gender, weight, height, and blood pressure.

[1006] "Health data" refers to data related to a user's health condition, including information collected from IoT devices such as pedometers, body composition monitors, and blood pressure monitors.

[1007] "Emotional data" refers to data that quantifies or classifies a user's stress level or emotional state.

[1008] "Nutrients that are deficient" refers to nutrients that the user should consume but are identified as being deficient in, based on the user's basic information and health data.

[1009] "Emotion-based nutritional components" refer to nutritional components that need to be supplemented based on the user's emotional data.

[1010] "Optimal supplements" refer to supplements that contain nutrients that need to be supplemented based on the user's individual nutritional deficiencies and emotional state.

[1011] "Means for suggesting supplements" refers to the process or function that suggests specific supplements to users based on the analysis results.

[1012] "Means for extracting supplements" refers to a device or mechanism for providing the appropriate amount of the required supplement based on the proposal.

[1013] "Means for storing history and collecting feedback" refers to a system for recording the history of supplement provision and collecting ratings and feedback from users.

[1014] The present invention relates to a system that comprehensively analyzes a user's daily dietary intake and emotional state, and provides optimal supplements to each individual. This system is composed of the following components:

[1015] System configuration

[1016] The system includes a device such as a smartphone or tablet used by the user, a cloud-based server, an emotion engine, and a supplement providing device. The role and operation of each element are described in detail below.

[1017] User operation

[1018] User:

[1019] 1. The user starts the application and logs in to the system by entering their user ID and password on the login screen. The first time they log in, they enter basic information such as their age, gender, weight, and blood pressure.

[1020] 2. Take photos of your daily meals with your smartphone or tablet and upload the images to the application.

[1021] 3. Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync that data to the application.

[1022] 4. The application periodically asks questions on the screen to check the user's emotional state and collects emotional data.

[1023] Terminal handling

[1024] Device:

[1025] 1. The food image sent by the user is temporarily stored in memory.

[1026] 2. The user's profile information, health data, and emotional data are sent to the cloud server along with the meal image.

[1027] Server Processing

[1028] server:

[1029] 1. The server sends the meal image data to an image analysis AI engine to identify the type and amount of ingredients and nutrients.

[1030] 2. Combining profile information, health data, and emotional data to identify nutrient deficiencies.

[1031] 3. Use an emotion engine to tailor supplement recommendations based on the user's emotional state, for example, suggesting more B vitamins if stress levels are high.

[1032] 4. Send the supplement proposal to the user for review and consent.

[1033] Operating the supplement delivery device

[1034] server:

[1035] 1. Give instructions to the dispenser and set the required supplement cartridge.

[1036] 2. The supplement delivery device extracts the optimal amount of supplement and prepares it for delivery in powder form.

[1037] Specific operation example

[1038] Example 1: Breakfast analysis and supplement provision:

[1039] 1. A user has an omelet, salad, and orange juice for breakfast, takes a photo of it, and uploads the image to the application.

[1040] 2. The device temporarily stores the food image and then sends it to a cloud server.

[1041] 3. The server uses an AI engine to analyze the image and identify the ingredients in the omelet (protein, fat, etc.), the ingredients in the salad (vitamins, minerals, etc.), and the ingredients in the orange juice (vitamin C, etc.).

[1042] 4. The basic information, health data, and emotional data are combined to suggest adding vitamin D and calcium, as well as B vitamins, as the user is under a lot of stress.

[1043] 5. The user reviews and agrees to the proposal.

[1044] 6. The required cartridge is inserted into the supplement delivery device, the optimal amount is extracted, the device notifies the user that it is ready, and the user receives it.

[1045] Prompt Sentence Examples

[1046] "A user has an omelet, salad, and orange juice for breakfast and takes a photo of it. Analyze the ingredients of this meal, identify any nutrient deficiencies, and suggest appropriate supplements, taking into account the user's stress level."

[1047] The above is a specific embodiment for carrying out the present invention.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1:

[1050] User:

[1051] A user starts an application and logs in by entering their user ID and password on the login screen. The login information is entered and sent to the server.

[1052] Input: User ID, Password

[1053] Output: Login authentication result

[1054] Specific operation: The application sends user information to the server, and the server performs the authentication process. If authentication is successful, a message indicating successful login is displayed and the user proceeds to the next screen. When logging in for the first time, a screen appears where basic information such as age, gender, weight, and blood pressure can be entered.

[1055] Step 2:

[1056] User:

[1057] Users take photos of their daily meals using a smartphone or tablet, and then use the application's upload function to send the images to the server.

[1058] Input: Food image

[1059] Output: Image data sent to the server

[1060] Specific operation: The user uses the "Photo and upload food" function in the application, selects the photo they have taken, and sends it to the server. The photo is temporarily stored in the device's local storage and then uploaded to the server.

[1061] Step 3:

[1062] Device:

[1063] The device receives the image of the meal, temporarily stores it in memory, and then transmits the image data to a cloud server.

[1064] Input: Image data sent by the user

[1065] Output: Image data sent to the cloud server

[1066] Specific operation: The device temporarily stores the received image data, then sends it to the cloud server via an HTTP request. After sending, a success notification is displayed to the user.

[1067] Step 4:

[1068] User:

[1069] Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync the data to the application.

[1070] Input: Health data from IoT devices

[1071] Output: Health data synced to the application

[1072] Specific operation: The IoT device sends health data to the terminal via Bluetooth or Wi-Fi, and the application receives and stores the data. If necessary, the data is sent to the server.

[1073] Step 5:

[1074] Device:

[1075] To periodically check the user's emotional state, emotional questions are displayed on the screen to collect the user's emotional data.

[1076] Input: User sentiment response

[1077] Output: Collected emotion data

[1078] How it works: The application periodically displays a pop-up notification asking for the user's sentiment rating and collects their response. The collected data is temporarily stored within the application and then sent to the server.

[1079] Step 6:

[1080] server:

[1081] The server sends the meal image data to an image analysis AI engine, which identifies the type and amount of ingredients and their nutrients.

[1082] Input: Image data sent to the cloud server

[1083] Output: Analysis of ingredients and nutrients

[1084] How it works: The server sends the received meal image to the AI ​​engine, which uses a deep learning model to analyze the image and identify the types and amounts of ingredients and nutrients. The results are returned to the server in JSON format.

[1085] Step 7:

[1086] server:

[1087] It combines profile information, health data, and emotional data to identify nutrient deficiencies and also determines nutritional factors based on emotional data.

[1088] Input: Profile information, health data, emotion data, image analysis results

[1089] Output: Identification of nutritional deficiencies and emotional nutritional factors

[1090] How it works: The server loads the user's profile information, health data, and emotion data from the database, and integrates them with the image analysis results. It uses an analysis algorithm to identify nutrient deficiencies and emotion-based nutritional factors.

[1091] Step 8:

[1092] server:

[1093] The supplement proposal is sent to the user's device and confirmation and consent are requested.

[1094] Input: Identification of nutritional deficiencies and emotional nutritional factors

[1095] Output: Supplement suggestions sent to the user's device

[1096] Specific operation: The server generates supplement suggestions based on the nutrient deficiency and emotion data, and sends them to the user's device as a notification. The user confirms the suggestions and presses the "Agree" button to proceed to the next step.

[1097] Step 9:

[1098] server:

[1099] Give instructions to the dispenser and load the required supplement cartridge, extracting the appropriate amount of supplement and preparing it for dispensing in powder form.

[1100] Input: User confirmation and consent, supplement suggestions

[1101] Output: Notification that supplement is ready to be extracted

[1102] Specific operation: The server sends a command to the dispenser to automatically load the required supplement cartridge. The dispenser extracts the optimal amount of supplement and notifies the server that it is ready. The server then finally notifies the user that it is ready.

[1103] The above is the flow of specific processing steps of the program of this system.

[1104] (Application example 2)

[1105] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1106] Conventional supplement recommendation systems were based solely on the user's dietary and health data, and were unable to take into account the user's emotional state. This meant that they were unable to meet the nutritional needs of users affected by stress and emotional fluctuations, making it difficult to recommend optimal supplements tailored to individual needs. Another issue was the inability to provide immediate recommendations in physical stores.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1108] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for collecting and analyzing emotional information of the user, means for suggesting optimal supplements based on the nutrient deficiencies and emotional information, means for selecting supplements based on the suggestions, and means for storing and collecting a history of supplement selection and user feedback. This enables personalized optimal supplement suggestions based on the user's diet, health data, and emotional state, and immediate provision of supplements in physical stores.

[1109] "Users" refers to people who use the system.

[1110] "Means for receiving images" refers to a device or software that captures images of meals taken or selected by a user into the system.

[1111] "Means for analyzing images to identify nutrients ingested" refers to devices or software that use AI or machine learning to analyze the type of food ingredients and nutrient content from received images.

[1112] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[1113] "Health Data" refers to health-related data such as a user's body composition, blood pressure, heart rate, and number of steps taken.

[1114] "Emotional information" refers to information that indicates the user's emotional state, such as stress or happiness.

[1115] "Missing nutrients" refers to the user's basic information and health data, as well as nutrients necessary for maintaining and improving future health.

[1116] "Supplements" refer to nutritional supplements that are used to supplement the user's nutritional deficiencies.

[1117] "Means for suggesting" refers to a device or software that suggests appropriate supplements based on the analysis results and the user's emotional information.

[1118] "Extracting means" refers to a device or software that provides the user with the appropriate amount of the suggested supplement.

[1119] "Means for storing and collecting history and feedback" refers to devices or software that record and store information about the supplements received by users and their impressions after use.

[1120] "IoT devices" refer to health devices that can be connected to the Internet (such as body composition monitors and blood pressure monitors).

[1121] "Physical store terminals" refers to dedicated terminals or tablets installed in physical stores such as healthcare shops and drugstores.

[1122] This invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. The system is comprised of a user's device, a cloud-based server, an emotion engine, and a supplement provider.

[1123] System configuration

[1124] User device functions

[1125] 1. User authentication and profile management

[1126] The user starts the application and enters their profile information (age, gender, weight, blood pressure, etc.) when using it for the first time. This information is sent to the server and saved.

[1127] 2. Take and upload a photo of your meal

[1128] Users take photos of their daily meals using a device (smartphone, tablet, etc.) and upload the images to the application.

[1129] 3. Collecting Emotional Data

[1130] The application periodically records the user's emotional state, and this emotional data is collected through the user's touch, voice input, or facial recognition technology.

[1131] Server Features

[1132] 1. Image analysis and nutrient identification

[1133] The server passes user-submitted images to an AI model that uses software such as TensorFlow and Keras to identify the type of food and its nutrients.

[1134] 2. Emotion analysis

[1135] The server passes the emotional data sent by the user to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms.

[1136] 3. Recommending the best supplements

[1137] The server identifies nutrient deficiencies based on the user's basic information, health data, and emotional data, and automatically generates optimal supplements.

[1138] Functions of the supplement providing device

[1139] 1. Extraction of Supplements

[1140] The supplement providing device dispenses the appropriate amount of supplements containing the nutrients required by the user based on instructions from the server. The device dispenses only the required amount after inserting a supplement cartridge.

[1141] 2. Submissions and Feedback Notifications

[1142] The extracted supplements are provided to the user, and their history and feedback are stored on a server.

[1143] Specific examples

[1144] Example 1: Breakfast analysis and supplement provision

[1145] 1. User Operation

[1146] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[1147] Photos of the meal are uploaded to the application and sent to the server.

[1148] 2. Image analysis and emotion analysis

[1149] The server uses an AI engine to analyze the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[1150] In addition, it analyzes the user's emotional data to detect high levels of stress.

[1151] 3. Supplement suggestions and selection

[1152] The server suggests supplementing with vitamin D and calcium, which are deficient, as well as B vitamins.

[1153] The supplement delivery device extracts this and provides it to the user.

[1154] Example prompts for generative AI models

[1155] "Build an image classification model to analyze images of meals taken by users and identify the nutrients contained in them. The model should be able to recognize multiple nutrients such as protein, fat, vitamins, and minerals and estimate the amount of each. Also, build an emotion classification model to analyze the user's emotional state from text and identify emotions such as stress or happiness. Integrate this data and generate relevant suggestions for a system that recommends optimal supplements."

[1156] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1157] Step 1:

[1158] User login and profile entry

[1159] Users launch the application and enter their login information to access the system. When using it for the first time, they enter profile information such as age, gender, weight, and blood pressure. This profile information is sent as input data to the cloud server and saved there.

[1160] Input: User login and profile information

[1161] Output: User profile information stored on the server

[1162] Step 2:

[1163] Taking and uploading food photos

[1164] Users take photos of their daily meals with their smartphone or tablet and upload the images to the application, which receives the images and sends them to a cloud server.

[1165] Input: User-taken food image

[1166] Output: Meal images stored on the server

[1167] Step 3:

[1168] Health data collection

[1169] Users measure their own health data using IoT devices such as body composition scales and blood pressure monitors, and synchronize this data with the application. The device receives the health data and sends it to a cloud server.

[1170] Input: Health data collected from IoT devices

[1171] Output: Health data stored on the server

[1172] Step 4:

[1173] Collecting Emotional Data

[1174] The application periodically records the user's emotional state. The user provides emotional information through touch, voice, or facial recognition, and the emotional data is sent from the device to a cloud server.

[1175] Input: User emotion data (touch, voice, facial recognition)

[1176] Output: Emotion data stored on the server

[1177] Step 5:

[1178] Image analysis and nutrient identification

[1179] The server passes the uploaded meal images to an AI model, which analyzes the types of ingredients and nutrient content. It uses TensorFlow and Keras to analyze the images and generate specific nutrient data.

[1180] Input: Food images stored on the server

[1181] Output: Parsed nutrient data

[1182] Step 6:

[1183] Emotion analysis

[1184] The server passes the collected emotional data to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms to generate emotional data.

[1185] Input: Emotion data stored on the server

[1186] Output: Parsed emotional state data

[1187] Step 7:

[1188] Recommendations for optimal supplements

[1189] The server combines the user's basic information, health data, nutrient data, and emotional data, and the system automatically generates optimal supplement recommendations based on the nutrients that are lacking and the user's emotional state.

[1190] Input: Basic information, health data, nutrition data, emotional state data

[1191] Output: Proposed supplemental data

[1192] Step 8:

[1193] Supplement Extract

[1194] The supplement providing device extracts the appropriate amount of supplement containing the necessary nutrients based on the suggested data from the server. The supplement cartridge is set and the extraction process is carried out.

[1195] Input: Suggested supplemental data from the server

[1196] Output: Extracted supplement

[1197] Step 9:

[1198] Offer and Feedback Notification

[1199] The extracted supplements are provided to the user, and their usage history and feedback are collected. The user receives the supplements and provides feedback through the application about their impressions after use. This data is stored on the server.

[1200] Input: User feedback and supplement history

[1201] Output: Feedback and historical data stored on the server

[1202] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1203] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1204] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1205] [Third embodiment]

[1206] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1207] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1208] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1209] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1210] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1211] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1212] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1213] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1214] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1215] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1216] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1217] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1218] The present invention relates to a system that allows users to take photos of their meals, performs nutritional analysis using AI, and provides optimal supplements to individuals. This system is composed of multiple elements, including a user, a terminal, and a server. Specific embodiments are described in detail below.

[1219] System configuration

[1220] The system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, and a supplement delivery device. The role of each element is explained below.

[1221] User operations

[1222] user:

[1223] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[1224] Users take photos of their daily meals using a mobile device and upload them to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used, and the data is synchronized.

[1225] Terminal handling

[1226] Device:

[1227] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[1228] Collects user profile information and health data and sends it to a server.

[1229] Server Processing

[1230] server:

[1231] The server sends the user's meal image to the AI ​​engine for image analysis, which uses image analysis algorithms to identify the ingredients consumed and their nutritional content.

[1232] It analyzes nutrient deficiencies by combining the user's basic information and health data.

[1233] Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[1234] Based on the suggestion, instructions to extract the supplement are sent to the "supplement providing device."

[1235] Operating the supplement delivery device

[1236] server:

[1237] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[1238] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[1239] Specific examples

[1240] A specific example will be described below.

[1241] Example 1: Breakfast analysis and supplement provision

[1242] 1. User Operation

[1243] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[1244] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[1245] 2. Terminal Processing

[1246] The device receives the image of the meal, stores it in memory, and sends it to the server.

[1247] 3. Server Processing

[1248] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[1249] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[1250] The server calculates the optimal combination of supplements, including vitamin D and calcium, and sends the recommendations to the user.

[1251] The user reviews and accepts the proposal.

[1252] 4. Supplement Extraction

[1253] Vitamin D and calcium cartridges are placed in the "supplement supply device" and the optimal amount is extracted.

[1254] The device notifies the server that it is ready, and the user is notified.

[1255] As described above, this system comprehensively manages the user's dietary and health data, and individually replenishes necessary nutrients, thereby achieving balanced nutritional intake.

[1256] The processing flow will be explained below.

[1257] Step 1:

[1258] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[1259] Step 2:

[1260] Users take photos of their meals using their smartphones and upload the images to the application. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, and synchronize that data with the application.

[1261] Step 3:

[1262] The device receives the meal image sent by the user and temporarily stores it in memory. The image data is then sent to a cloud server, along with the user's profile information and health data.

[1263] Step 4:

[1264] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[1265] Step 5:

[1266] The server analyzes the user's basic information and health data in combination with the AI ​​analysis results, identifies any nutrient deficiencies, and calculates the optimal combination of supplements based on that information.

[1267] Step 6:

[1268] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[1269] Step 7:

[1270] If the user agrees to the suggestion, the server instructs the user to insert the cartridge into the "supplement providing device." The device then checks the inserted cartridge and extracts the optimal amount of the required supplement.

[1271] Step 8:

[1272] The supplement providing device completes the extraction and sends a completion notification to the server, which then notifies the terminal that the supplement is ready.

[1273] Step 9:

[1274] The user receives and takes the supplements. This updates the user's health data and history, leading to improved accuracy of the next recommendations. The server periodically collects user feedback and uses it to improve the system.

[1275] In this way, a system is realized that allows users to efficiently take in the nutrients they need on a daily basis.

[1276] Example 1

[1277] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1278] In today's world, it is difficult to properly manage individual nutritional status and quickly and accurately replenish individual nutrient deficiencies. Conventional systems lack the ability to comprehensively analyze users' dietary and health data, limiting manual data entry and supplement selection. Furthermore, there is no system that utilizes IoT devices to automatically collect and comprehensively analyze health data, resulting in the problem of inappropriate individual nutritional supplementation.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1280] In this invention, the server includes means for receiving meal images from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for identifying nutrient deficiencies in the user based on the basic information and health data, means for suggesting optimal supplements based on the nutrient deficiencies, means for selecting supplements based on the suggestions, means for storing a history of supplement selection and collecting feedback, means for using a cloud-based AI engine to analyze the meal images, means for using the AI ​​engine to compare the image analysis results with a nutritional information database, means for automatically collecting the user's health data via an IoT device, and means for combining and analyzing the user's basic information, health data, and dietary data using a statistical analysis tool. This allows for integrated management of the user's diet and health data, enabling prompt and appropriate individual nutritional supplementation.

[1281] "User" refers to an individual who uses the System.

[1282] "Meal Image" refers to an image file containing visual data of the meal the user consumes.

[1283] "Means for analyzing images to identify ingested nutrients" refers to technologies and algorithms for recognizing ingredients in images of food and identifying the nutritional components of those ingredients.

[1284] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[1285] "Health data" refers to data that indicates a user's health status, such as the number of steps taken, body fat percentage, and blood pressure.

[1286] "Means for identifying nutrient deficiencies" refers to technology that analyzes and identifies nutrients that are lacking in the user's current diet based on the user's basic information and health data.

[1287] "Means for suggesting optimal supplements" refers to technology that calculates the optimal combination of supplements to make up for missing nutrients and suggests them to users.

[1288] "Means for extracting supplements" refers to the equipment and technology used to prepare and deliver the proposed supplements to the user in the appropriate amounts.

[1289] "Means for storing the history of supplement extraction and collecting feedback" refers to the function of storing the history of supplement use and collecting opinions and experiences from users.

[1290] A "cloud-based AI engine" refers to an artificial intelligence processing system that operates in a cloud environment.

[1291] "Nutrition information database" refers to a database that accumulates data on the nutritional components of various food ingredients.

[1292] "IoT devices" refer to health management devices that are capable of transmitting data via the Internet.

[1293] "Statistical analysis tools" refer to software tools used to analyze various types of data and derive specific patterns or trends.

[1294] This invention relates to a system that allows users to take photos of their meals, analyzes their nutritional intake using AI, and provides optimal supplements for each individual. The invention is composed of multiple elements, including the user, a terminal, and a server.

[1295] System configuration

[1296] This system consists of a user device (such as a smartphone or tablet), a cloud-based server, and a supplement delivery device. Each element is described in detail below.

[1297] User operations

[1298] user:

[1299] 1. Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, and blood pressure.

[1300] 2. Take photos of your daily meals with your smartphone camera and upload them to the application.

[1301] 3. Sync IoT devices such as pedometers, body composition monitors, and blood pressure monitors with your smartphone to collect health data.

[1302] Terminal handling

[1303] Device:

[1304] 1. Receives the food image sent by the user and temporarily stores it in memory. The stored image data is then sent to the cloud server.

[1305] 2. It also collects user profile information and health data and sends them to the server.

[1306] Server Processing

[1307] server:

[1308] 1. The received meal image is sent to a cloud-based AI engine for image analysis. Specifically, an image analysis tool (e.g., Google Cloud Vision API) is used to identify the ingredients ingested from the image.

[1309] 2. The image analysis results are compared with a nutritional information database to identify the nutritional components of each ingredient.

[1310] 3. Analyze the user's basic information and health data using statistical analysis tools (e.g., Python, R) to identify nutrient deficiencies.

[1311] 4. Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[1312] 5. Insert the required supplement cartridge into the supplement providing device and send instructions to extract the optimal amount.

[1313] Operating the supplement delivery device

[1314] server:

[1315] 1. Place the cartridge into the supplement dispenser and extract the optimal amount of supplement you need.

[1316] 2. Provides extracted supplements in powder form and notifies you when the device is ready.

[1317] Specific examples

[1318] Specific examples are given below:

[1319] Example 1: Breakfast analysis and supplement provision

[1320] 1. User Action:

[1321] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it with their smartphone.

[1322] Photos of meals are uploaded to the application and user profile information is sent to the server.

[1323] 2. Terminal processing:

[1324] The device receives the image of the meal, stores it in memory, and sends it to a cloud server.

[1325] 3. Server processing:

[1326] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[1327] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[1328] The server calculates the optimal combination of supplements containing vitamin D and calcium and sends the recommendation to the user, who then reviews and accepts the recommendation.

[1329] 4. Supplement Extraction:

[1330] Insert vitamin D and calcium cartridges into the supplement dispenser and extract the optimal amount.

[1331] The device notifies the server that it is ready and notifies the user.

[1332] Prompt Sentence Examples

[1333] Examples of prompts to input to a generative AI model might include:

[1334] "If a user wants to take a photo of an omelet, salad, and orange juice and run a nutritional analysis, what steps should they take?"

[1335] By inputting this prompt, the generative AI model will guide the user through the nutritional analysis process of the meal and generate an optimal supplement delivery flow.

[1336] As described above, this system comprehensively manages the user's dietary and health data, enabling prompt and appropriate individual nutritional supplementation.

[1337] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1338] Step 1: User logs in

[1339] Specific behavior:

[1340] The user launches the app on their smartphone, enters their user ID and password on the login screen, and presses the "Login" button.

[1341] Input: User ID, Password

[1342] Output: User credentials

[1343] Data processing / calculation: The server checks the user ID and password against the database to perform authentication.

[1344] Step 2: User enters profile information

[1345] Specific behavior:

[1346] When logging in for the first time, the user enters personal information such as age, gender, weight, and blood pressure on the profile entry screen that appears, and presses the "Save" button.

[1347] Input: Profile information such as age, gender, weight, blood pressure, etc.

[1348] Output: User profile data

[1349] Data processing / calculation: The terminal formats the input information and sends it to the server.

[1350] Step 3: User takes and uploads a photo of their meal

[1351] Specific behavior:

[1352] Users take a photo of their meal with their smartphone camera and press the "upload" button to upload the image to the app.

[1353] Input: Food image file

[1354] Output: Saved food image data

[1355] Data processing / calculation: The device temporarily stores the image file in memory, compresses it, and then sends it to the cloud server.

[1356] Step 4: User synchronizes data on IoT devices

[1357] Specific behavior:

[1358] The user turns on Bluetooth on each IoT device and smartphone and presses the "Data Sync" button in the app.

[1359] Input: IoT device data (number of steps, body fat percentage, blood pressure, etc.)

[1360] Output: Collected health data

[1361] Data processing / calculation: The terminal receives data from the IoT device and sends it to the server in bulk.

[1362] Step 5: The device temporarily saves the food image and sends it to the server.

[1363] Specific behavior:

[1364] The device temporarily stores the food images sent by the user in memory and sends them to a cloud server.

[1365] Input: Food image data

[1366] Output: Image data stored on a cloud server

[1367] Data processing / calculation: The device compresses the food images and sends them to a cloud server via the network.

[1368] Step 6: The device collects the user's profile information and health data and sends it to the server.

[1369] Specific behavior:

[1370] The device collects the user's profile information and health data from IoT devices and sends it to a cloud server.

[1371] Input: Profile information, health data

[1372] Output: User profile information and health data stored on a cloud server

[1373] Data processing / calculation: The data collected by the terminal is formatted and sent in bulk to the cloud server.

[1374] Step 7: The server sends the food image to the AI ​​engine for image analysis.

[1375] Specific behavior:

[1376] The server sends the received food images to a cloud-based AI engine, which analyzes the images.

[1377] Input: Food image data

[1378] Output: Parsed ingredient information

[1379] Data processing / computation: A cloud-based AI engine (e.g., Google Cloud Vision API) identifies ingredients from images and recognizes their nutritional content.

[1380] Step 8: The server compares the image analysis results with the nutrition information database.

[1381] Specific behavior:

[1382] The server compares the image analysis results with a nutritional information database to identify the nutritional components of each ingredient.

[1383] Input: Ingredient information

[1384] Output: Nutritional information

[1385] Data processing / calculation: The server compares the image analysis results with a database and extracts nutrient data.

[1386] Step 9: The server analyzes the user's basic information and health data to identify nutrient deficiencies.

[1387] Specific behavior:

[1388] The server uses statistical analysis tools (e.g., Python, R) to combine and analyze the user's basic information, health data, and dietary information.

[1389] Input: Basic information, health data, nutritional information

[1390] Output: Missing nutrient information

[1391] Data processing / calculation: Statistical analysis tools analyze the input data and identify nutrient deficiencies.

[1392] Step 10: The server calculates the optimal supplement combination and sends the proposal to the user.

[1393] Specific behavior:

[1394] The server uses an algorithm to generate a list of supplements containing the nutrients to be supplemented and sends suggestions to the user.

[1395] Input: Missing nutrient information

[1396] Output: Supplement suggestion information

[1397] Data processing / calculation: The server calculates the optimal combination of supplements based on the nutrients that are lacking and generates recommendations.

[1398] Step 11: The server sends an instruction to the supplement providing device to extract the supplement.

[1399] Specific behavior:

[1400] The server sets the necessary supplement cartridge in the supplement providing device and sends instructions to extract the optimal amount.

[1401] Input: Supplement suggestion information

[1402] Output: Extracted supplement

[1403] Data processing / calculation: The server sends commands to the provider based on the suggested information and extracts the appropriate supplements.

[1404] The above are the specific processing steps of this system.

[1405] (Application example 1)

[1406] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1407] As health consciousness grows, there is a demand for personalized nutritional management and optimal nutritional supplements. However, conventional systems have made it difficult to provide individually tailored nutritional supplements in physical stores, and they have been unable to respond in real time based on users' health data. This has made it difficult for users to quickly obtain the optimal nutritional supplements tailored to their health condition.

[1408] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1409] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for extracting optimal amounts of nutritional supplements using a device in a physical store, means used in the physical store for providing the nutritional supplements, and means for storing a history of the provision of nutritional supplements and collecting feedback, thereby enabling users to quickly receive optimal nutritional supplements based on their own health condition at the physical store.

[1410] A "user" is a person who uses the system to take photos of meals and perform nutritional analysis.

[1411] "Meal images" are photographic data that visually record the contents of meals consumed by the user.

[1412] "Nutrients" are components such as vitamins, minerals, and proteins obtained from the food a user consumes.

[1413] "Basic information" refers to personal data such as the user's age, gender, weight, height, and health condition.

[1414] "Health data" refers to data that indicates the user's current health status, such as the number of steps taken, blood pressure, and body composition.

[1415] "Dietary supplements" are foods such as supplements and energy drinks that are suggested or offered to users to supplement nutrients that they are lacking.

[1416] A "brick and mortar store" is a physical store where a user can visit and receive nutritional supplements.

[1417] A "server" is a computer system that receives, analyzes, and processes data from users.

[1418] "Suggestions" are recommendations to provide optimal nutritional supplements to users based on their nutrient deficiencies.

[1419] "Extraction" is the act of extracting a specific amount of a dietary supplement needed based on a proposal.

[1420] "Device" refers to a device that is installed in a physical store and provides nutritional supplements in accordance with instructions from the server.

[1421] "History" refers to the record of dietary supplements provided to a user and their feedback data.

[1422] "Feedback" is information provided by users through comments and ratings about the effectiveness and satisfaction of a dietary supplement.

[1423] MODE FOR CARRYING OUT THE INVENTION

[1424] This invention is a system that allows users to take photos of their meals, upload them to the cloud, perform nutritional analysis, and provide optimal nutritional supplements to individuals in physical stores. Specific embodiments are described below.

[1425] System configuration

[1426] The system consists of a device used by users, such as a smartphone or tablet, a cloud-based server, and a nutritional supplement supply device in a physical store. The role of each element is explained below.

[1427] User operations

[1428] user:

[1429] Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, height, and blood pressure.

[1430] Users take photos of their daily meals with their smartphones and upload them to the app. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data and synchronize the data.

[1431] Terminal handling

[1432] Device:

[1433] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[1434] Collects user profile information and health data and sends it to a server.

[1435] Server Processing

[1436] server:

[1437] The server receives the user's food images and analyzes the meal contents using a generative AI model, which utilizes machine learning libraries such as TensorFlow and PyTorch.

[1438] The system analyzes nutrient deficiencies by combining the user's basic information and health data. It uses Python to process the health data and compare it with known nutritional data stored in a database to identify nutrient deficiencies.

[1439] The server then suggests optimal nutritional supplements based on the nutrients that are lacking and notifies the user, who is then sent a notification to their smartphone.

[1440] Operating a nutritional supplement delivery device

[1441] Device:

[1442] The nutritional supplement supply device installed in the physical store receives instructions from the server and extracts the optimal nutritional supplements. This process is carried out using IoT control software using Node.js and Python.

[1443] The device will then be notified that the correct amount of nutritional supplement is ready to be dispensed to the user.

[1444] Specific examples

[1445] Example: Breakfast analysis and nutritional supplement provision

[1446] 1. User Operation

[1447] The user eats a sandwich, salad, and orange juice for breakfast and takes a photo of it.

[1448] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[1449] 2. Terminal Processing

[1450] The device receives the image of the meal, stores it in memory, and sends it to the server.

[1451] Profile information and health data are also sent to the server.

[1452] 3. Server Processing

[1453] The server sends the images to a generative AI model that analyzes the ingredients (e.g., protein in a sandwich, vitamins in a salad, vitamin C in orange juice).

[1454] This is combined with the user's basic information and health check data to identify vitamin D and calcium deficiencies.

[1455] The server calculates the optimal combination of nutritional supplements containing vitamin D and calcium and sends the proposal to the user, who then reviews and accepts the proposal.

[1456] 4. Nutritional supplements provided

[1457] Vitamin D and calcium ingredients are placed in a dispenser in a physical store, and the optimal amount is extracted.

[1458] The device notifies the server that it is ready, and the user is notified.

[1459] Example prompts to input to a generative AI model:

[1460] Meal image: "https: / / example.com / user_images / breakfast.jpg"

[1461] User profile: Age 30, Gender female, Weight 60kg, Height 160cm, Recent problems: Tired easily

[1462] Health data: Steps 8000, Blood pressure 120 / 80, Body fat 22%

[1463] This prompt text is imported into the server and analyzed to create a system that provides appropriate nutritional supplements.

[1464] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1465] Step 1:

[1466] user:

[1467] The user takes a picture of their meal, such as breakfast, with their smartphone, and the image is saved in an application on their smartphone.

[1468] Input: Food image

[1469] Output: Image data stored on the smartphone

[1470] Specific operation: Take a photo of your meal using the smartphone's camera app, and the image will be automatically saved in the application folder.

[1471] Step 2:

[1472] user:

[1473] Launch the application and enter your login information to log in to the system. The user will then confirm and enter their profile information (age, gender, weight, etc.) and sync their health data (step count, blood pressure, etc.).

[1474] Input: Login information, profile information, health data

[1475] Output: User data stored in the application

[1476] Specific operation: Enter login information on the login screen, confirm and enter profile information, and sync health data from IoT devices to the application.

[1477] Step 3:

[1478] Device:

[1479] The images of the meal taken by the user, along with the profile information and health data entered, are sent to a cloud server.

[1480] Input: Food images, profile information, health data

[1481] Output: Data sent to the cloud server

[1482] Specific operation: Use the application's upload function to send the selected images and synchronized data to the cloud server.

[1483] Step 4:

[1484] server:

[1485] The cloud server inputs the received image data into the generative AI model and performs a nutritional analysis of the meal contents, using libraries such as TensorFlow and PyTorch.

[1486] Input: Food image

[1487] Output: Nutritional data for meals

[1488] How it works: The generative AI model analyzes image data and identifies the nutritional content of each ingredient. For example, it calculates the protein content of an omelet or the vitamin content of a salad.

[1489] Step 5:

[1490] server:

[1491] Using the user's basic information and health data, Python is used to run an algorithm to identify nutrient deficiencies.

[1492] Input: Profile information, health data, dietary nutrient data

[1493] Output: List of nutrients that are lacking

[1494] What it does: It compares the received data with an existing nutrition database and calculates the nutrient deficiencies the user may have (e.g., vitamin D or calcium).

[1495] Step 6:

[1496] server:

[1497] Based on the identified nutrient deficiencies, the generative AI model will suggest optimal nutritional supplements and send a smartphone notification to the user.

[1498] Input: List of nutrients you are deficient in

[1499] Output: Nutritional supplement suggestions, notification data

[1500] What it does: Runs a recommendation generation algorithm based on nutrient deficiencies and sends a push notification to the user's smartphone. Example: "Supplement ABC is recommended to replenish vitamin D and calcium."

[1501] Step 7:

[1502] user:

[1503] Review smartphone notifications and agree to suggested dietary supplements.

[1504] Input: Notification data

[1505] Output: User consent data

[1506] Specific actions: Check the proposal on the notification screen and press the agree button.

[1507] Step 8:

[1508] server:

[1509] With the user's consent, instructions are sent to a nutritional supplement dispenser in the physical store to extract the optimal amount of nutritional supplement needed.

[1510] Input: User consent data

[1511] Output: Extraction instructions for dietary supplements

[1512] Specific operation: Using IoT control software (Node.js or Python), it sends extraction instructions to the dispenser. For example, it sends an instruction to extract a predetermined amount of nutritional supplement from a specific cartridge.

[1513] Step 9:

[1514] Device:

[1515] A nutritional supplement dispenser in a physical store receives the instruction, extracts the optimal amount of the specified nutritional supplement, and notifies the server and the user that it is ready to be dispensed.

[1516] Input: Extraction instruction data

[1517] Output: Ready notification

[1518] Specific operation: The device extracts the nutritional supplement and notifies the server that it is ready, which then pushes the notification to the user's device.

[1519] Step 10:

[1520] server:

[1521] It stores the history of the nutritional supplements provided and also collects feedback from users and stores it in a database.

[1522] Input: Provision history data, feedback data

[1523] Output: Updated database

[1524] Specific operation: The provision history and feedback information are stored in a database and used to improve the proposal algorithm in the future.

[1525] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1526] The present invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. This system is comprised of a user, a terminal, a server, and a supplement provider, and has the function of adjusting supplement recommendations based on the user's emotion data. Specific embodiments are described in detail below.

[1527] System configuration

[1528] This system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, an emotion engine, and a supplement delivery device. The role of each element is explained below.

[1529] User operation

[1530] User:

[1531] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[1532] Users take photos of their daily meals using a mobile device and upload the images to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used and the data is synchronized.

[1533] The application periodically checks and collects the user's emotional state to record their emotions.

[1534] Terminal handling

[1535] Device:

[1536] The system receives the food image sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[1537] The user's profile information, health data, and emotional data are transmitted to a server.

[1538] Server Processing

[1539] server:

[1540] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[1541] The system analyzes a user's basic information, health data, and emotional data to identify any nutrient deficiencies.

[1542] The emotional engine tailors supplement recommendations based on the user's emotional state, for example, suggesting a boost in B vitamins if stress levels are high.

[1543] The supplement proposal is sent to the user, and once the user confirms and agrees, the process proceeds to the next step.

[1544] Operating the supplement delivery device

[1545] server:

[1546] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[1547] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[1548] Specific examples

[1549] A specific example will be described below.

[1550] Example 1: Breakfast analysis and supplement provision

[1551] 1. User Operation

[1552] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[1553] Photos of the meal are uploaded to the application and sent to the server along with the profile information entered when logging in.

[1554] 2. Terminal Processing

[1555] The device receives the image of the meal, stores it in memory, and then sends it to the server.

[1556] 3. Server Processing

[1557] The AI ​​engine analyzes the image to determine the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[1558] Basic information, health data, and the user's emotional data (e.g., high stress levels) are combined to provide vitamin D and calcium deficiencies, as well as B vitamins based on the user's emotions.

[1559] The supplement proposal is sent to the user for review and consent.

[1560] 4. Supplement Extraction

[1561] Cartridges containing vitamin D, calcium, and B vitamins are placed into the "supplement delivery device" and the optimal amount is extracted.

[1562] The device notifies the server of the completion of extraction, and then the user is notified.

[1563] As described above, a system is realized that not only allows users to efficiently ingest the nutrients they need on a daily basis, but also allows them to have supplements adjusted according to their emotional state.

[1564] The processing flow will be explained below.

[1565] Step 1:

[1566] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[1567] Step 2:

[1568] Users use their smartphones to take photos of their meals and upload the images to the application. Next, they use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, which is then synchronized with the application. They also input their daily emotional state into the application.

[1569] Step 3:

[1570] The device receives the food image sent by the user and temporarily stores it in memory, then transmits the image data along with the collected profile information, health data, and emotion data to a cloud server.

[1571] Step 4:

[1572] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[1573] Step 5:

[1574] The server analyzes the user's basic information, health data, and emotional data in combination with AI analysis results to identify the need for supplements based on nutrient deficiencies and emotional state.

[1575] Step 6:

[1576] The emotion engine evaluates the user's emotional state and adjusts the supplement suggestions accordingly, for example, adding suggestions including B vitamins if the user's stress level is high.

[1577] Step 7:

[1578] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[1579] Step 8:

[1580] If the user agrees to the suggestion, the server instructs the user to set the cartridge in the "supplement provision device." The supplement provision device checks the set cartridge and extracts the optimal amount of the required supplement.

[1581] Step 9:

[1582] The supplement providing device completes extraction and sends a completion notification to the server. The server then notifies the terminal and notifies the user that the supplement is ready. The user receives and takes the supplement.

[1583] Step 10:

[1584] The server periodically updates the user's health and emotional data, improves supplement recommendations based on the data, and collects user feedback to improve the system's accuracy and user experience.

[1585] In this way, the system comprehensively manages the user's diet, health data, and emotional state, and can suggest and provide personalized supplements that take into account the user's emotional state.

[1586] Example 2

[1587] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1588] In modern society, it is difficult to maintain a balanced diet, but in order to maintain good health, it is important to consume nutrients that are tailored to each individual's dietary content. Furthermore, stress and emotional states also affect appetite and nutrient intake. However, current systems do not provide a means to individually recommend optimal supplements by taking into account both an analysis of dietary content and emotional state. Therefore, a system is needed that can comprehensively analyze a user's dietary content, health data, and emotional data and provide the necessary nutrients.

[1589] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1590] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information, health data, and emotional data of the user, means for identifying nutrients lacking in the user and nutritional elements based on the emotions based on the basic information, health data, and emotional data, means for suggesting optimal supplements based on the nutrients lacking in the user and nutritional elements based on the emotions, means for selecting supplements based on the suggestions, and means for storing a history of the selection of supplements and collecting feedback. This makes it possible to provide optimal supplements individually by comprehensively considering the user's dietary content and emotional state.

[1591] "User" refers to the user of an information system.

[1592] "Meal Image" refers to photographs or image data that visually record a meal consumed by a user.

[1593] "Means for analyzing images" refers to technologies or algorithms for automatically recognizing and identifying food types and nutrients from received image data.

[1594] "Intaken nutrients" refers to the types and amounts of nutrients obtained from the analyzed diet.

[1595] "Basic Information" refers to personal profile data such as a user's age, gender, weight, height, and blood pressure.

[1596] "Health data" refers to data related to a user's health condition, including information collected from IoT devices such as pedometers, body composition monitors, and blood pressure monitors.

[1597] "Emotional data" refers to data that quantifies or classifies a user's stress level or emotional state.

[1598] "Nutrients that are deficient" refers to nutrients that the user should consume but are identified as being deficient in, based on the user's basic information and health data.

[1599] "Emotion-based nutritional components" refer to nutritional components that need to be supplemented based on the user's emotional data.

[1600] "Optimal supplements" refer to supplements that contain nutrients that need to be supplemented based on the user's individual nutritional deficiencies and emotional state.

[1601] "Means for suggesting supplements" refers to the process or function that suggests specific supplements to users based on the analysis results.

[1602] "Means for extracting supplements" refers to a device or mechanism for providing the appropriate amount of the required supplement based on the proposal.

[1603] "Means for storing history and collecting feedback" refers to a system for recording the history of supplement provision and collecting ratings and feedback from users.

[1604] The present invention relates to a system that comprehensively analyzes a user's daily dietary intake and emotional state, and provides optimal supplements to each individual. This system is composed of the following components:

[1605] System configuration

[1606] The system includes a device such as a smartphone or tablet used by the user, a cloud-based server, an emotion engine, and a supplement providing device. The role and operation of each element are described in detail below.

[1607] User operation

[1608] User:

[1609] 1. The user starts the application and logs in to the system by entering their user ID and password on the login screen. The first time they log in, they enter basic information such as their age, gender, weight, and blood pressure.

[1610] 2. Take photos of your daily meals with your smartphone or tablet and upload the images to the application.

[1611] 3. Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync that data to the application.

[1612] 4. The application periodically asks questions on the screen to check the user's emotional state and collects emotional data.

[1613] Terminal handling

[1614] Device:

[1615] 1. The food image sent by the user is temporarily stored in memory.

[1616] 2. The user's profile information, health data, and emotional data are sent to the cloud server along with the meal image.

[1617] Server Processing

[1618] server:

[1619] 1. The server sends the meal image data to an image analysis AI engine to identify the type and amount of ingredients and nutrients.

[1620] 2. Combining profile information, health data, and emotional data to identify nutrient deficiencies.

[1621] 3. Use an emotion engine to tailor supplement recommendations based on the user's emotional state, for example, suggesting more B vitamins if stress levels are high.

[1622] 4. Send the supplement proposal to the user for review and consent.

[1623] Operating the supplement delivery device

[1624] server:

[1625] 1. Give instructions to the dispenser and set the required supplement cartridge.

[1626] 2. The supplement delivery device extracts the optimal amount of supplement and prepares it for delivery in powder form.

[1627] Specific operation example

[1628] Example 1: Breakfast analysis and supplement provision:

[1629] 1. A user has an omelet, salad, and orange juice for breakfast, takes a photo of it, and uploads the image to the application.

[1630] 2. The device temporarily stores the food image and then sends it to a cloud server.

[1631] 3. The server uses an AI engine to analyze the image and identify the ingredients in the omelet (protein, fat, etc.), the ingredients in the salad (vitamins, minerals, etc.), and the ingredients in the orange juice (vitamin C, etc.).

[1632] 4. The basic information, health data, and emotional data are combined to suggest adding vitamin D and calcium, as well as B vitamins, as the user is under a lot of stress.

[1633] 5. The user reviews and agrees to the proposal.

[1634] 6. The required cartridge is inserted into the supplement delivery device, the optimal amount is extracted, the device notifies the user that it is ready, and the user receives it.

[1635] Prompt Sentence Examples

[1636] "A user has an omelet, salad, and orange juice for breakfast and takes a photo of it. Analyze the ingredients of this meal, identify any nutrient deficiencies, and suggest appropriate supplements, taking into account the user's stress level."

[1637] The above is a specific embodiment for carrying out the present invention.

[1638] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1639] Step 1:

[1640] User:

[1641] A user starts an application and logs in by entering their user ID and password on the login screen. The login information is entered and sent to the server.

[1642] Input: User ID, Password

[1643] Output: Login authentication result

[1644] Specific operation: The application sends user information to the server, and the server performs the authentication process. If authentication is successful, a message indicating successful login is displayed and the user proceeds to the next screen. When logging in for the first time, a screen appears where basic information such as age, gender, weight, and blood pressure can be entered.

[1645] Step 2:

[1646] User:

[1647] Users take photos of their daily meals using a smartphone or tablet, and then use the application's upload function to send the images to the server.

[1648] Input: Food image

[1649] Output: Image data sent to the server

[1650] Specific operation: The user uses the "Photo and upload food" function in the application, selects the photo they have taken, and sends it to the server. The photo is temporarily stored in the device's local storage and then uploaded to the server.

[1651] Step 3:

[1652] Device:

[1653] The device receives the image of the meal, temporarily stores it in memory, and then transmits the image data to a cloud server.

[1654] Input: Image data sent by the user

[1655] Output: Image data sent to the cloud server

[1656] Specific operation: The device temporarily stores the received image data, then sends it to the cloud server via an HTTP request. After sending, a success notification is displayed to the user.

[1657] Step 4:

[1658] User:

[1659] Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync the data to the application.

[1660] Input: Health data from IoT devices

[1661] Output: Health data synced to the application

[1662] Specific operation: The IoT device sends health data to the terminal via Bluetooth or Wi-Fi, and the application receives and stores the data. If necessary, the data is sent to the server.

[1663] Step 5:

[1664] Device:

[1665] To periodically check the user's emotional state, emotional questions are displayed on the screen to collect the user's emotional data.

[1666] Input: User sentiment response

[1667] Output: Collected emotion data

[1668] How it works: The application periodically displays a pop-up notification asking for the user's sentiment rating and collects their response. The collected data is temporarily stored within the application and then sent to the server.

[1669] Step 6:

[1670] server:

[1671] The server sends the meal image data to an image analysis AI engine, which identifies the type and amount of ingredients and their nutrients.

[1672] Input: Image data sent to the cloud server

[1673] Output: Analysis of ingredients and nutrients

[1674] How it works: The server sends the received meal image to the AI ​​engine, which uses a deep learning model to analyze the image and identify the types and amounts of ingredients and nutrients. The results are returned to the server in JSON format.

[1675] Step 7:

[1676] server:

[1677] It combines profile information, health data, and emotional data to identify nutrient deficiencies and also determines nutritional factors based on emotional data.

[1678] Input: Profile information, health data, emotion data, image analysis results

[1679] Output: Identification of nutritional deficiencies and emotional nutritional factors

[1680] How it works: The server loads the user's profile information, health data, and emotion data from the database, and integrates them with the image analysis results. It uses an analysis algorithm to identify nutrient deficiencies and emotion-based nutritional factors.

[1681] Step 8:

[1682] server:

[1683] The supplement proposal is sent to the user's device and confirmation and consent are requested.

[1684] Input: Identification of nutritional deficiencies and emotional nutritional factors

[1685] Output: Supplement suggestions sent to the user's device

[1686] Specific operation: The server generates supplement suggestions based on the nutrient deficiency and emotion data, and sends them to the user's device as a notification. The user confirms the suggestions and presses the "Agree" button to proceed to the next step.

[1687] Step 9:

[1688] server:

[1689] Give instructions to the dispenser and load the required supplement cartridge, extracting the appropriate amount of supplement and preparing it for dispensing in powder form.

[1690] Input: User confirmation and consent, supplement suggestions

[1691] Output: Notification that supplement is ready to be extracted

[1692] Specific operation: The server sends a command to the dispenser to automatically load the required supplement cartridge. The dispenser extracts the optimal amount of supplement and notifies the server that it is ready. The server then finally notifies the user that it is ready.

[1693] The above is the flow of specific processing steps of the program of this system.

[1694] (Application example 2)

[1695] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1696] Conventional supplement recommendation systems were based solely on the user's dietary and health data, and were unable to take into account the user's emotional state. This meant that they were unable to meet the nutritional needs of users affected by stress and emotional fluctuations, making it difficult to recommend optimal supplements tailored to individual needs. Another issue was the inability to provide immediate recommendations in physical stores.

[1697] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1698] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for collecting and analyzing emotional information of the user, means for suggesting optimal supplements based on the nutrient deficiencies and emotional information, means for selecting supplements based on the suggestions, and means for storing and collecting a history of supplement selection and user feedback. This enables personalized optimal supplement suggestions based on the user's diet, health data, and emotional state, and immediate provision of supplements in physical stores.

[1699] "Users" refers to people who use the system.

[1700] "Means for receiving images" refers to a device or software that captures images of meals taken or selected by a user into the system.

[1701] "Means for analyzing images to identify nutrients ingested" refers to devices or software that use AI or machine learning to analyze the type of food ingredients and nutrient content from received images.

[1702] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[1703] "Health Data" refers to health-related data such as a user's body composition, blood pressure, heart rate, and number of steps taken.

[1704] "Emotional information" refers to information that indicates the user's emotional state, such as stress or happiness.

[1705] "Missing nutrients" refers to the user's basic information and health data, as well as nutrients necessary for maintaining and improving future health.

[1706] "Supplements" refer to nutritional supplements that are used to supplement the user's nutritional deficiencies.

[1707] "Means for suggesting" refers to a device or software that suggests appropriate supplements based on the analysis results and the user's emotional information.

[1708] "Extracting means" refers to a device or software that provides the user with the appropriate amount of the suggested supplement.

[1709] "Means for storing and collecting history and feedback" refers to devices or software that record and store information about the supplements received by users and their impressions after use.

[1710] "IoT devices" refer to health devices that can be connected to the Internet (such as body composition monitors and blood pressure monitors).

[1711] "Physical store terminals" refers to dedicated terminals or tablets installed in physical stores such as healthcare shops and drugstores.

[1712] This invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. The system is comprised of a user's device, a cloud-based server, an emotion engine, and a supplement provider.

[1713] System configuration

[1714] User device functions

[1715] 1. User authentication and profile management

[1716] The user starts the application and enters their profile information (age, gender, weight, blood pressure, etc.) when using it for the first time. This information is sent to the server and saved.

[1717] 2. Take and upload a photo of your meal

[1718] Users take photos of their daily meals using a device (smartphone, tablet, etc.) and upload the images to the application.

[1719] 3. Collecting Emotional Data

[1720] The application periodically records the user's emotional state, and this emotional data is collected through the user's touch, voice input, or facial recognition technology.

[1721] Server Features

[1722] 1. Image analysis and nutrient identification

[1723] The server passes user-submitted images to an AI model that uses software such as TensorFlow and Keras to identify the type of food and its nutrients.

[1724] 2. Emotion analysis

[1725] The server passes the emotional data sent by the user to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms.

[1726] 3. Recommending the best supplements

[1727] The server identifies nutrient deficiencies based on the user's basic information, health data, and emotional data, and automatically generates optimal supplements.

[1728] Functions of the supplement providing device

[1729] 1. Extraction of Supplements

[1730] The supplement providing device dispenses the appropriate amount of supplements containing the nutrients required by the user based on instructions from the server. The device dispenses only the required amount after inserting a supplement cartridge.

[1731] 2. Submissions and Feedback Notifications

[1732] The extracted supplements are provided to the user, and their history and feedback are stored on a server.

[1733] Specific examples

[1734] Example 1: Breakfast analysis and supplement provision

[1735] 1. User Operation

[1736] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[1737] Photos of the meal are uploaded to the application and sent to the server.

[1738] 2. Image analysis and emotion analysis

[1739] The server uses an AI engine to analyze the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[1740] In addition, it analyzes the user's emotional data to detect high levels of stress.

[1741] 3. Supplement suggestions and selection

[1742] The server suggests supplementing with vitamin D and calcium, which are deficient, as well as B vitamins.

[1743] The supplement delivery device extracts this and provides it to the user.

[1744] Example prompts for generative AI models

[1745] "Build an image classification model to analyze images of meals taken by users and identify the nutrients contained in them. The model should be able to recognize multiple nutrients such as protein, fat, vitamins, and minerals and estimate the amount of each. Also, build an emotion classification model to analyze the user's emotional state from text and identify emotions such as stress or happiness. Integrate this data and generate relevant suggestions for a system that recommends optimal supplements."

[1746] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1747] Step 1:

[1748] User login and profile entry

[1749] Users launch the application and enter their login information to access the system. When using it for the first time, they enter profile information such as age, gender, weight, and blood pressure. This profile information is sent as input data to the cloud server and saved there.

[1750] Input: User login and profile information

[1751] Output: User profile information stored on the server

[1752] Step 2:

[1753] Taking and uploading food photos

[1754] Users take photos of their daily meals with their smartphone or tablet and upload the images to the application, which receives the images and sends them to a cloud server.

[1755] Input: User-taken food image

[1756] Output: Meal images stored on the server

[1757] Step 3:

[1758] Health data collection

[1759] Users measure their own health data using IoT devices such as body composition scales and blood pressure monitors, and synchronize this data with the application. The device receives the health data and sends it to a cloud server.

[1760] Input: Health data collected from IoT devices

[1761] Output: Health data stored on the server

[1762] Step 4:

[1763] Collecting Emotional Data

[1764] The application periodically records the user's emotional state. The user provides emotional information through touch, voice, or facial recognition, and the emotional data is sent from the device to a cloud server.

[1765] Input: User emotion data (touch, voice, facial recognition)

[1766] Output: Emotion data stored on the server

[1767] Step 5:

[1768] Image analysis and nutrient identification

[1769] The server passes the uploaded meal images to an AI model, which analyzes the types of ingredients and nutrient content. It uses TensorFlow and Keras to analyze the images and generate specific nutrient data.

[1770] Input: Food images stored on the server

[1771] Output: Parsed nutrient data

[1772] Step 6:

[1773] Emotion analysis

[1774] The server passes the collected emotional data to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms to generate emotional data.

[1775] Input: Emotion data stored on the server

[1776] Output: Parsed emotional state data

[1777] Step 7:

[1778] Recommendations for optimal supplements

[1779] The server combines the user's basic information, health data, nutrient data, and emotional data, and the system automatically generates optimal supplement recommendations based on the nutrients that are lacking and the user's emotional state.

[1780] Input: Basic information, health data, nutrition data, emotional state data

[1781] Output: Proposed supplemental data

[1782] Step 8:

[1783] Supplement Extract

[1784] The supplement providing device extracts the appropriate amount of supplement containing the necessary nutrients based on the suggested data from the server. The supplement cartridge is set and the extraction process is carried out.

[1785] Input: Suggested supplemental data from the server

[1786] Output: Extracted supplement

[1787] Step 9:

[1788] Offer and Feedback Notification

[1789] The extracted supplements are provided to the user, and their usage history and feedback are collected. The user receives the supplements and provides feedback through the application about their impressions after use. This data is stored on the server.

[1790] Input: User feedback and supplement history

[1791] Output: Feedback and historical data stored on the server

[1792] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1793] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1794] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1795] [Fourth embodiment]

[1796] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1797] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1798] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1799] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1800] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1801] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1802] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1803] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1804] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1805] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1806] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1807] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1808] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1809] The present invention relates to a system that allows users to take photos of their meals, performs nutritional analysis using AI, and provides optimal supplements to individuals. This system is composed of multiple elements, including a user, a terminal, and a server. Specific embodiments are described in detail below.

[1810] System configuration

[1811] The system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, and a supplement delivery device. The role of each element is explained below.

[1812] User operations

[1813] user:

[1814] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[1815] Users take photos of their daily meals using a mobile device and upload them to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used, and the data is synchronized.

[1816] Terminal handling

[1817] Device:

[1818] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[1819] Collects user profile information and health data and sends it to a server.

[1820] Server Processing

[1821] server:

[1822] The server sends the user's meal image to the AI ​​engine for image analysis, which uses image analysis algorithms to identify the ingredients consumed and their nutritional content.

[1823] It analyzes nutrient deficiencies by combining the user's basic information and health data.

[1824] Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[1825] Based on the suggestion, instructions to extract the supplement are sent to the "supplement providing device."

[1826] Operating the supplement delivery device

[1827] server:

[1828] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[1829] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[1830] Specific examples

[1831] A specific example will be described below.

[1832] Example 1: Breakfast analysis and supplement provision

[1833] 1. User Operation

[1834] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[1835] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[1836] 2. Terminal Processing

[1837] The device receives the image of the meal, stores it in memory, and sends it to the server.

[1838] 3. Server Processing

[1839] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[1840] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[1841] The server calculates the optimal combination of supplements, including vitamin D and calcium, and sends the recommendations to the user.

[1842] The user reviews and accepts the proposal.

[1843] 4. Supplement Extraction

[1844] Vitamin D and calcium cartridges are placed in the "supplement supply device" and the optimal amount is extracted.

[1845] The device notifies the server that it is ready, and the user is notified.

[1846] As described above, this system comprehensively manages the user's dietary and health data, and individually replenishes necessary nutrients, thereby achieving balanced nutritional intake.

[1847] The processing flow will be explained below.

[1848] Step 1:

[1849] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[1850] Step 2:

[1851] Users take photos of their meals using their smartphones and upload the images to the application. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, and synchronize that data with the application.

[1852] Step 3:

[1853] The device receives the meal image sent by the user and temporarily stores it in memory. The image data is then sent to a cloud server, along with the user's profile information and health data.

[1854] Step 4:

[1855] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[1856] Step 5:

[1857] The server analyzes the user's basic information and health data in combination with the AI ​​analysis results, identifies any nutrient deficiencies, and calculates the optimal combination of supplements based on that information.

[1858] Step 6:

[1859] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[1860] Step 7:

[1861] If the user agrees to the suggestion, the server instructs the user to insert the cartridge into the "supplement providing device." The device then checks the inserted cartridge and extracts the optimal amount of the required supplement.

[1862] Step 8:

[1863] The supplement providing device completes the extraction and sends a completion notification to the server, which then notifies the terminal that the supplement is ready.

[1864] Step 9:

[1865] The user receives and takes the supplements. This updates the user's health data and history, leading to improved accuracy of the next recommendations. The server periodically collects user feedback and uses it to improve the system.

[1866] In this way, a system is realized that allows users to efficiently take in the nutrients they need on a daily basis.

[1867] Example 1

[1868] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] In today's world, it is difficult to properly manage individual nutritional status and quickly and accurately replenish individual nutrient deficiencies. Conventional systems lack the ability to comprehensively analyze users' dietary and health data, limiting manual data entry and supplement selection. Furthermore, there is no system that utilizes IoT devices to automatically collect and comprehensively analyze health data, resulting in the problem of inappropriate individual nutritional supplementation.

[1870] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1871] In this invention, the server includes means for receiving meal images from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for identifying nutrient deficiencies in the user based on the basic information and health data, means for suggesting optimal supplements based on the nutrient deficiencies, means for selecting supplements based on the suggestions, means for storing a history of supplement selection and collecting feedback, means for using a cloud-based AI engine to analyze the meal images, means for using the AI ​​engine to compare the image analysis results with a nutritional information database, means for automatically collecting the user's health data via an IoT device, and means for combining and analyzing the user's basic information, health data, and dietary data using a statistical analysis tool. This allows for integrated management of the user's diet and health data, enabling prompt and appropriate individual nutritional supplementation.

[1872] "User" refers to an individual who uses the System.

[1873] "Meal Image" refers to an image file containing visual data of the meal the user consumes.

[1874] "Means for analyzing images to identify ingested nutrients" refers to technologies and algorithms for recognizing ingredients in images of food and identifying the nutritional components of those ingredients.

[1875] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[1876] "Health data" refers to data that indicates a user's health status, such as the number of steps taken, body fat percentage, and blood pressure.

[1877] "Means for identifying nutrient deficiencies" refers to technology that analyzes and identifies nutrients that are lacking in the user's current diet based on the user's basic information and health data.

[1878] "Means for suggesting optimal supplements" refers to technology that calculates the optimal combination of supplements to make up for missing nutrients and suggests them to users.

[1879] "Means for extracting supplements" refers to the equipment and technology used to prepare and deliver the proposed supplements to the user in the appropriate amounts.

[1880] "Means for storing the history of supplement extraction and collecting feedback" refers to the function of storing the history of supplement use and collecting opinions and experiences from users.

[1881] A "cloud-based AI engine" refers to an artificial intelligence processing system that operates in a cloud environment.

[1882] "Nutrition information database" refers to a database that accumulates data on the nutritional components of various food ingredients.

[1883] "IoT devices" refer to health management devices that are capable of transmitting data via the Internet.

[1884] "Statistical analysis tools" refer to software tools used to analyze various types of data and derive specific patterns or trends.

[1885] This invention relates to a system that allows users to take photos of their meals, analyzes their nutritional intake using AI, and provides optimal supplements for each individual. The invention is composed of multiple elements, including the user, a terminal, and a server.

[1886] System configuration

[1887] This system consists of a user device (such as a smartphone or tablet), a cloud-based server, and a supplement delivery device. Each element is described in detail below.

[1888] User operations

[1889] user:

[1890] 1. Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, and blood pressure.

[1891] 2. Take photos of your daily meals with your smartphone camera and upload them to the application.

[1892] 3. Sync IoT devices such as pedometers, body composition monitors, and blood pressure monitors with your smartphone to collect health data.

[1893] Terminal handling

[1894] Device:

[1895] 1. Receives the food image sent by the user and temporarily stores it in memory. The stored image data is then sent to the cloud server.

[1896] 2. It also collects user profile information and health data and sends them to the server.

[1897] Server Processing

[1898] server:

[1899] 1. The received meal image is sent to a cloud-based AI engine for image analysis. Specifically, an image analysis tool (e.g., Google Cloud Vision API) is used to identify the ingredients ingested from the image.

[1900] 2. The image analysis results are compared with a nutritional information database to identify the nutritional components of each ingredient.

[1901] 3. Analyze the user's basic information and health data using statistical analysis tools (e.g., Python, R) to identify nutrient deficiencies.

[1902] 4. Based on the nutrients that are lacking, the system calculates the optimal combination of supplements and sends a recommendation to the user, who then reviews and accepts the recommendation.

[1903] 5. Insert the required supplement cartridge into the supplement providing device and send instructions to extract the optimal amount.

[1904] Operating the supplement delivery device

[1905] server:

[1906] 1. Place the cartridge into the supplement dispenser and extract the optimal amount of supplement you need.

[1907] 2. Provides extracted supplements in powder form and notifies you when the device is ready.

[1908] Specific examples

[1909] Specific examples are given below:

[1910] Example 1: Breakfast analysis and supplement provision

[1911] 1. User Action:

[1912] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it with their smartphone.

[1913] Photos of meals are uploaded to the application and user profile information is sent to the server.

[1914] 2. Terminal processing:

[1915] The device receives the image of the meal, stores it in memory, and sends it to a cloud server.

[1916] 3. Server processing:

[1917] The server sends the image to an AI engine, which analyzes the ingredients of the omelet (protein, fat, etc.), salad (vitamins, minerals, etc.), and orange juice (vitamin C, etc.).

[1918] The system also combines and analyzes the user's basic information, health checkup data, and pedometer data to identify vitamin D and calcium deficiencies.

[1919] The server calculates the optimal combination of supplements containing vitamin D and calcium and sends the recommendation to the user, who then reviews and accepts the recommendation.

[1920] 4. Supplement Extraction:

[1921] Insert vitamin D and calcium cartridges into the supplement dispenser and extract the optimal amount.

[1922] The device notifies the server that it is ready and notifies the user.

[1923] Prompt Sentence Examples

[1924] Examples of prompts to input to a generative AI model might include:

[1925] "If a user wants to take a photo of an omelet, salad, and orange juice and run a nutritional analysis, what steps should they take?"

[1926] By inputting this prompt, the generative AI model will guide the user through the nutritional analysis process of the meal and generate an optimal supplement delivery flow.

[1927] As described above, this system comprehensively manages the user's dietary and health data, enabling prompt and appropriate individual nutritional supplementation.

[1928] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1929] Step 1: User logs in

[1930] Specific behavior:

[1931] The user launches the app on their smartphone, enters their user ID and password on the login screen, and presses the "Login" button.

[1932] Input: User ID, Password

[1933] Output: User credentials

[1934] Data processing / calculation: The server checks the user ID and password against the database to perform authentication.

[1935] Step 2: User enters profile information

[1936] Specific behavior:

[1937] When logging in for the first time, the user enters personal information such as age, gender, weight, and blood pressure on the profile entry screen that appears, and presses the "Save" button.

[1938] Input: Profile information such as age, gender, weight, blood pressure, etc.

[1939] Output: User profile data

[1940] Data processing / calculation: The terminal formats the input information and sends it to the server.

[1941] Step 3: User takes and uploads a photo of their meal

[1942] Specific behavior:

[1943] Users take a photo of their meal with their smartphone camera and press the "upload" button to upload the image to the app.

[1944] Input: Food image file

[1945] Output: Saved food image data

[1946] Data processing / calculation: The device temporarily stores the image file in memory, compresses it, and then sends it to the cloud server.

[1947] Step 4: User synchronizes data on IoT devices

[1948] Specific behavior:

[1949] The user turns on Bluetooth on each IoT device and smartphone and presses the "Data Sync" button in the app.

[1950] Input: IoT device data (number of steps, body fat percentage, blood pressure, etc.)

[1951] Output: Collected health data

[1952] Data processing / calculation: The terminal receives data from the IoT device and sends it to the server in bulk.

[1953] Step 5: The device temporarily saves the food image and sends it to the server.

[1954] Specific behavior:

[1955] The device temporarily stores the food images sent by the user in memory and sends them to a cloud server.

[1956] Input: Food image data

[1957] Output: Image data stored on a cloud server

[1958] Data processing / calculation: The device compresses the food images and sends them to a cloud server via the network.

[1959] Step 6: The device collects the user's profile information and health data and sends it to the server.

[1960] Specific behavior:

[1961] The device collects the user's profile information and health data from IoT devices and sends it to a cloud server.

[1962] Input: Profile information, health data

[1963] Output: User profile information and health data stored on a cloud server

[1964] Data processing / calculation: The data collected by the terminal is formatted and sent in bulk to the cloud server.

[1965] Step 7: The server sends the food image to the AI ​​engine for image analysis.

[1966] Specific behavior:

[1967] The server sends the received food images to a cloud-based AI engine, which analyzes the images.

[1968] Input: Food image data

[1969] Output: Parsed ingredient information

[1970] Data processing / computation: A cloud-based AI engine (e.g., Google Cloud Vision API) identifies ingredients from images and recognizes their nutritional content.

[1971] Step 8: The server compares the image analysis results with the nutrition information database.

[1972] Specific behavior:

[1973] The server compares the image analysis results with a nutritional information database to identify the nutritional components of each ingredient.

[1974] Input: Ingredient information

[1975] Output: Nutritional information

[1976] Data processing / calculation: The server compares the image analysis results with a database and extracts nutrient data.

[1977] Step 9: The server analyzes the user's basic information and health data to identify nutrient deficiencies.

[1978] Specific behavior:

[1979] The server uses statistical analysis tools (e.g., Python, R) to combine and analyze the user's basic information, health data, and dietary information.

[1980] Input: Basic information, health data, nutritional information

[1981] Output: Missing nutrient information

[1982] Data processing / calculation: Statistical analysis tools analyze the input data and identify nutrient deficiencies.

[1983] Step 10: The server calculates the optimal supplement combination and sends the proposal to the user.

[1984] Specific behavior:

[1985] The server uses an algorithm to generate a list of supplements containing the nutrients to be supplemented and sends suggestions to the user.

[1986] Input: Missing nutrient information

[1987] Output: Supplement suggestion information

[1988] Data processing / calculation: The server calculates the optimal combination of supplements based on the nutrients that are lacking and generates recommendations.

[1989] Step 11: The server sends an instruction to the supplement providing device to extract the supplement.

[1990] Specific behavior:

[1991] The server sets the necessary supplement cartridge in the supplement providing device and sends instructions to extract the optimal amount.

[1992] Input: Supplement suggestion information

[1993] Output: Extracted supplement

[1994] Data processing / calculation: The server sends commands to the provider based on the suggested information and extracts the appropriate supplements.

[1995] The above are the specific processing steps of this system.

[1996] (Application example 1)

[1997] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1998] As health consciousness grows, there is a demand for personalized nutritional management and optimal nutritional supplements. However, conventional systems have made it difficult to provide individually tailored nutritional supplements in physical stores, and they have been unable to respond in real time based on users' health data. This has made it difficult for users to quickly obtain the optimal nutritional supplements tailored to their health condition.

[1999] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2000] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for extracting optimal amounts of nutritional supplements using a device in a physical store, means used in the physical store for providing the nutritional supplements, and means for storing a history of the provision of nutritional supplements and collecting feedback, thereby enabling users to quickly receive optimal nutritional supplements based on their own health condition at the physical store.

[2001] A "user" is a person who uses the system to take photos of meals and perform nutritional analysis.

[2002] "Meal images" are photographic data that visually record the contents of meals consumed by the user.

[2003] "Nutrients" are components such as vitamins, minerals, and proteins obtained from the food a user consumes.

[2004] "Basic information" refers to personal data such as the user's age, gender, weight, height, and health condition.

[2005] "Health data" refers to data that indicates the user's current health status, such as the number of steps taken, blood pressure, and body composition.

[2006] "Dietary supplements" are foods such as supplements and energy drinks that are suggested or offered to users to supplement nutrients that they are lacking.

[2007] A "brick and mortar store" is a physical store where a user can visit and receive nutritional supplements.

[2008] A "server" is a computer system that receives, analyzes, and processes data from users.

[2009] "Suggestions" are recommendations to provide optimal nutritional supplements to users based on their nutrient deficiencies.

[2010] "Extraction" is the act of extracting a specific amount of a dietary supplement needed based on a proposal.

[2011] "Device" refers to a device that is installed in a physical store and provides nutritional supplements in accordance with instructions from the server.

[2012] "History" refers to the record of dietary supplements provided to a user and their feedback data.

[2013] "Feedback" is information provided by users through comments and ratings about the effectiveness and satisfaction of a dietary supplement.

[2014] MODE FOR CARRYING OUT THE INVENTION

[2015] This invention is a system that allows users to take photos of their meals, upload them to the cloud, perform nutritional analysis, and provide optimal nutritional supplements to individuals in physical stores. Specific embodiments are described below.

[2016] System configuration

[2017] The system consists of a device used by users, such as a smartphone or tablet, a cloud-based server, and a nutritional supplement supply device in a physical store. The role of each element is explained below.

[2018] User operations

[2019] user:

[2020] Launch the application and enter your login information to log in to the system. When you log in for the first time, you will be asked to enter your profile information, such as your age, gender, weight, height, and blood pressure.

[2021] Users take photos of their daily meals with their smartphones and upload them to the app. They also use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data and synchronize the data.

[2022] Terminal handling

[2023] Device:

[2024] The system receives the image of the meal sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[2025] Collects user profile information and health data and sends it to a server.

[2026] Server Processing

[2027] server:

[2028] The server receives the user's food images and analyzes the meal contents using a generative AI model, which utilizes machine learning libraries such as TensorFlow and PyTorch.

[2029] The system analyzes nutrient deficiencies by combining the user's basic information and health data. It uses Python to process the health data and compare it with known nutritional data stored in a database to identify nutrient deficiencies.

[2030] The server then suggests optimal nutritional supplements based on the nutrients that are lacking and notifies the user, who is then sent a notification to their smartphone.

[2031] Operating a nutritional supplement delivery device

[2032] Device:

[2033] The nutritional supplement supply device installed in the physical store receives instructions from the server and extracts the optimal nutritional supplements. This process is carried out using IoT control software using Node.js and Python.

[2034] The device will then be notified that the correct amount of nutritional supplement is ready to be dispensed to the user.

[2035] Specific examples

[2036] Example: Breakfast analysis and nutritional supplement provision

[2037] 1. User Operation

[2038] The user eats a sandwich, salad, and orange juice for breakfast and takes a photo of it.

[2039] Photos of meals are uploaded to the application, and the profile information entered when logging in is sent to the server.

[2040] 2. Terminal Processing

[2041] The device receives the image of the meal, stores it in memory, and sends it to the server.

[2042] Profile information and health data are also sent to the server.

[2043] 3. Server Processing

[2044] The server sends the images to a generative AI model that analyzes the ingredients (e.g., protein in a sandwich, vitamins in a salad, vitamin C in orange juice).

[2045] This is combined with the user's basic information and health check data to identify vitamin D and calcium deficiencies.

[2046] The server calculates the optimal combination of nutritional supplements containing vitamin D and calcium and sends the proposal to the user, who then reviews and accepts the proposal.

[2047] 4. Nutritional supplements provided

[2048] Vitamin D and calcium ingredients are placed in a dispenser in a physical store, and the optimal amount is extracted.

[2049] The device notifies the server that it is ready, and the user is notified.

[2050] Example prompts to input to a generative AI model:

[2051] Meal image: "https: / / example.com / user_images / breakfast.jpg"

[2052] User profile: Age 30, Gender female, Weight 60kg, Height 160cm, Recent problems: Tired easily

[2053] Health data: Steps 8000, Blood pressure 120 / 80, Body fat 22%

[2054] This prompt text is imported into the server and analyzed to create a system that provides appropriate nutritional supplements.

[2055] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2056] Step 1:

[2057] user:

[2058] The user takes a picture of their meal, such as breakfast, with their smartphone, and the image is saved in an application on their smartphone.

[2059] Input: Food image

[2060] Output: Image data stored on the smartphone

[2061] Specific operation: Take a photo of your meal using the smartphone's camera app, and the image will be automatically saved in the application folder.

[2062] Step 2:

[2063] user:

[2064] Launch the application and enter your login information to log in to the system. The user will then confirm and enter their profile information (age, gender, weight, etc.) and sync their health data (step count, blood pressure, etc.).

[2065] Input: Login information, profile information, health data

[2066] Output: User data stored in the application

[2067] Specific operation: Enter login information on the login screen, confirm and enter profile information, and sync health data from IoT devices to the application.

[2068] Step 3:

[2069] Device:

[2070] The images of the meal taken by the user, along with the profile information and health data entered, are sent to a cloud server.

[2071] Input: Food images, profile information, health data

[2072] Output: Data sent to the cloud server

[2073] Specific operation: Use the application's upload function to send the selected images and synchronized data to the cloud server.

[2074] Step 4:

[2075] server:

[2076] The cloud server inputs the received image data into the generative AI model and performs a nutritional analysis of the meal contents, using libraries such as TensorFlow and PyTorch.

[2077] Input: Food image

[2078] Output: Nutritional data for meals

[2079] How it works: The generative AI model analyzes image data and identifies the nutritional content of each ingredient. For example, it calculates the protein content of an omelet or the vitamin content of a salad.

[2080] Step 5:

[2081] server:

[2082] Using the user's basic information and health data, Python is used to run an algorithm to identify nutrient deficiencies.

[2083] Input: Profile information, health data, dietary nutrient data

[2084] Output: List of nutrients that are lacking

[2085] What it does: It compares the received data with an existing nutrition database and calculates the nutrient deficiencies the user may have (e.g., vitamin D or calcium).

[2086] Step 6:

[2087] server:

[2088] Based on the identified nutrient deficiencies, the generative AI model will suggest optimal nutritional supplements and send a smartphone notification to the user.

[2089] Input: List of nutrients you are deficient in

[2090] Output: Nutritional supplement suggestions, notification data

[2091] What it does: Runs a recommendation generation algorithm based on nutrient deficiencies and sends a push notification to the user's smartphone. Example: "Supplement ABC is recommended to replenish vitamin D and calcium."

[2092] Step 7:

[2093] user:

[2094] Review smartphone notifications and agree to suggested dietary supplements.

[2095] Input: Notification data

[2096] Output: User consent data

[2097] Specific actions: Check the proposal on the notification screen and press the agree button.

[2098] Step 8:

[2099] server:

[2100] With the user's consent, instructions are sent to a nutritional supplement dispenser in the physical store to extract the optimal amount of nutritional supplement needed.

[2101] Input: User consent data

[2102] Output: Extraction instructions for dietary supplements

[2103] Specific operation: Using IoT control software (Node.js or Python), it sends extraction instructions to the dispenser. For example, it sends an instruction to extract a predetermined amount of nutritional supplement from a specific cartridge.

[2104] Step 9:

[2105] Device:

[2106] A nutritional supplement dispenser in a physical store receives the instruction, extracts the optimal amount of the specified nutritional supplement, and notifies the server and the user that it is ready to be dispensed.

[2107] Input: Extraction instruction data

[2108] Output: Ready notification

[2109] Specific operation: The device extracts the nutritional supplement and notifies the server that it is ready, which then pushes the notification to the user's device.

[2110] Step 10:

[2111] server:

[2112] It stores the history of the nutritional supplements provided and also collects feedback from users and stores it in a database.

[2113] Input: Provision history data, feedback data

[2114] Output: Updated database

[2115] Specific operation: The provision history and feedback information are stored in a database and used to improve the proposal algorithm in the future.

[2116] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2117] The present invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. This system is comprised of a user, a terminal, a server, and a supplement provider, and has the function of adjusting supplement recommendations based on the user's emotion data. Specific embodiments are described in detail below.

[2118] System configuration

[2119] This system consists of a device used by the user, such as a smartphone or tablet, a cloud-based server, an emotion engine, and a supplement delivery device. The role of each element is explained below.

[2120] User operation

[2121] User:

[2122] Launch the application and enter your login information to log in to the system. When logging in for the first time, you will be asked to enter profile information such as age, gender, weight, and blood pressure.

[2123] Users take photos of their daily meals using a mobile device and upload the images to the application. Furthermore, to collect health data, IoT devices such as pedometers, body composition monitors, and blood pressure monitors are used and the data is synchronized.

[2124] The application periodically checks and collects the user's emotional state to record their emotions.

[2125] Terminal handling

[2126] Device:

[2127] The system receives the food image sent by the user, temporarily stores it in memory, and then sends the image data to a cloud server.

[2128] The user's profile information, health data, and emotional data are transmitted to a server.

[2129] Server Processing

[2130] server:

[2131] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[2132] The system analyzes a user's basic information, health data, and emotional data to identify any nutrient deficiencies.

[2133] The emotional engine tailors supplement recommendations based on the user's emotional state, for example, suggesting a boost in B vitamins if stress levels are high.

[2134] The supplement proposal is sent to the user, and once the user confirms and agrees, the process proceeds to the next step.

[2135] Operating the supplement delivery device

[2136] server:

[2137] Insert the cartridge into the "supplement providing device" and extract the optimal amount of supplement you need.

[2138] The extracted supplement is provided in powder form and the device notifies you when it is ready.

[2139] Specific examples

[2140] A specific example will be described below.

[2141] Example 1: Breakfast analysis and supplement provision

[2142] 1. User Operation

[2143] A user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[2144] Photos of the meal are uploaded to the application and sent to the server along with the profile information entered when logging in.

[2145] 2. Terminal Processing

[2146] The device receives the image of the meal, stores it in memory, and then sends it to the server.

[2147] 3. Server Processing

[2148] The AI ​​engine analyzes the image to determine the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[2149] Basic information, health data, and the user's emotional data (e.g., high stress levels) are combined to provide vitamin D and calcium deficiencies, as well as B vitamins based on the user's emotions.

[2150] The supplement proposal is sent to the user for review and consent.

[2151] 4. Supplement Extraction

[2152] Cartridges containing vitamin D, calcium, and B vitamins are placed into the "supplement delivery device" and the optimal amount is extracted.

[2153] The device notifies the server of the completion of extraction, and then the user is notified.

[2154] As described above, a system is realized that not only allows users to efficiently ingest the nutrients they need on a daily basis, but also allows them to have supplements adjusted according to their emotional state.

[2155] The processing flow will be explained below.

[2156] Step 1:

[2157] A user starts the application and enters login information to log in to the system. When logging in for the first time, the user enters profile information such as age, sex, weight, and blood pressure.

[2158] Step 2:

[2159] Users use their smartphones to take photos of their meals and upload the images to the application. Next, they use IoT devices such as pedometers, body composition monitors, and blood pressure monitors to collect health data, which is then synchronized with the application. They also input their daily emotional state into the application.

[2160] Step 3:

[2161] The device receives the food image sent by the user and temporarily stores it in memory, then transmits the image data along with the collected profile information, health data, and emotion data to a cloud server.

[2162] Step 4:

[2163] The server sends the received meal image to an AI engine for image analysis, which then uses the image to identify the types and amounts of ingredients consumed and the nutrients they contain.

[2164] Step 5:

[2165] The server analyzes the user's basic information, health data, and emotional data in combination with AI analysis results to identify the need for supplements based on nutrient deficiencies and emotional state.

[2166] Step 6:

[2167] The emotion engine evaluates the user's emotional state and adjusts the supplement suggestions accordingly, for example, adding suggestions including B vitamins if the user's stress level is high.

[2168] Step 7:

[2169] The server sends the supplement proposal to the terminal, which displays it to the user. The user then checks the proposal and agrees or modifies it.

[2170] Step 8:

[2171] If the user agrees to the suggestion, the server instructs the user to set the cartridge in the "supplement provision device." The supplement provision device checks the set cartridge and extracts the optimal amount of the required supplement.

[2172] Step 9:

[2173] The supplement providing device completes extraction and sends a completion notification to the server. The server then notifies the terminal and notifies the user that the supplement is ready. The user receives and takes the supplement.

[2174] Step 10:

[2175] The server periodically updates the user's health and emotional data, improves supplement recommendations based on the data, and collects user feedback to improve the system's accuracy and user experience.

[2176] In this way, the system comprehensively manages the user's diet, health data, and emotional state, and can suggest and provide personalized supplements that take into account the user's emotional state.

[2177] Example 2

[2178] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2179] In modern society, it is difficult to maintain a balanced diet, but in order to maintain good health, it is important to consume nutrients that are tailored to each individual's dietary content. Furthermore, stress and emotional states also affect appetite and nutrient intake. However, current systems do not provide a means to individually recommend optimal supplements by taking into account both an analysis of dietary content and emotional state. Therefore, a system is needed that can comprehensively analyze a user's dietary content, health data, and emotional data and provide the necessary nutrients.

[2180] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2181] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information, health data, and emotional data of the user, means for identifying nutrients lacking in the user and nutritional elements based on the emotions based on the basic information, health data, and emotional data, means for suggesting optimal supplements based on the nutrients lacking in the user and nutritional elements based on the emotions, means for selecting supplements based on the suggestions, and means for storing a history of the selection of supplements and collecting feedback. This makes it possible to provide optimal supplements individually by comprehensively considering the user's dietary content and emotional state.

[2182] "User" refers to the user of an information system.

[2183] "Meal Image" refers to photographs or image data that visually record a meal consumed by a user.

[2184] "Means for analyzing images" refers to technologies or algorithms for automatically recognizing and identifying food types and nutrients from received image data.

[2185] "Intaken nutrients" refers to the types and amounts of nutrients obtained from the analyzed diet.

[2186] "Basic Information" refers to personal profile data such as a user's age, gender, weight, height, and blood pressure.

[2187] "Health data" refers to data related to a user's health condition, including information collected from IoT devices such as pedometers, body composition monitors, and blood pressure monitors.

[2188] "Emotional data" refers to data that quantifies or classifies a user's stress level or emotional state.

[2189] "Nutrients that are deficient" refers to nutrients that the user should consume but are identified as being deficient in, based on the user's basic information and health data.

[2190] "Emotion-based nutritional components" refer to nutritional components that need to be supplemented based on the user's emotional data.

[2191] "Optimal supplements" refer to supplements that contain nutrients that need to be supplemented based on the user's individual nutritional deficiencies and emotional state.

[2192] "Means for suggesting supplements" refers to the process or function that suggests specific supplements to users based on the analysis results.

[2193] "Means for extracting supplements" refers to a device or mechanism for providing the appropriate amount of the required supplement based on the proposal.

[2194] "Means for storing history and collecting feedback" refers to a system for recording the history of supplement provision and collecting ratings and feedback from users.

[2195] The present invention relates to a system that comprehensively analyzes a user's daily dietary intake and emotional state, and provides optimal supplements to each individual. This system is composed of the following components:

[2196] System configuration

[2197] The system includes a device such as a smartphone or tablet used by the user, a cloud-based server, an emotion engine, and a supplement providing device. The role and operation of each element are described in detail below.

[2198] User operation

[2199] User:

[2200] 1. The user starts the application and logs in to the system by entering their user ID and password on the login screen. The first time they log in, they enter basic information such as their age, gender, weight, and blood pressure.

[2201] 2. Take photos of your daily meals with your smartphone or tablet and upload the images to the application.

[2202] 3. Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync that data to the application.

[2203] 4. The application periodically asks questions on the screen to check the user's emotional state and collects emotional data.

[2204] Terminal handling

[2205] Device:

[2206] 1. The food image sent by the user is temporarily stored in memory.

[2207] 2. The user's profile information, health data, and emotional data are sent to the cloud server along with the meal image.

[2208] Server Processing

[2209] server:

[2210] 1. The server sends the meal image data to an image analysis AI engine to identify the type and amount of ingredients and nutrients.

[2211] 2. Combining profile information, health data, and emotional data to identify nutrient deficiencies.

[2212] 3. Use an emotion engine to tailor supplement recommendations based on the user's emotional state, for example, suggesting more B vitamins if stress levels are high.

[2213] 4. Send the supplement proposal to the user for review and consent.

[2214] Operating the supplement delivery device

[2215] server:

[2216] 1. Give instructions to the dispenser and set the required supplement cartridge.

[2217] 2. The supplement delivery device extracts the optimal amount of supplement and prepares it for delivery in powder form.

[2218] Specific operation example

[2219] Example 1: Breakfast analysis and supplement provision:

[2220] 1. A user has an omelet, salad, and orange juice for breakfast, takes a photo of it, and uploads the image to the application.

[2221] 2. The device temporarily stores the food image and then sends it to a cloud server.

[2222] 3. The server uses an AI engine to analyze the image and identify the ingredients in the omelet (protein, fat, etc.), the ingredients in the salad (vitamins, minerals, etc.), and the ingredients in the orange juice (vitamin C, etc.).

[2223] 4. The basic information, health data, and emotional data are combined to suggest adding vitamin D and calcium, as well as B vitamins, as the user is under a lot of stress.

[2224] 5. The user reviews and agrees to the proposal.

[2225] 6. The required cartridge is inserted into the supplement delivery device, the optimal amount is extracted, the device notifies the user that it is ready, and the user receives it.

[2226] Prompt Sentence Examples

[2227] "A user has an omelet, salad, and orange juice for breakfast and takes a photo of it. Analyze the ingredients of this meal, identify any nutrient deficiencies, and suggest appropriate supplements, taking into account the user's stress level."

[2228] The above is a specific embodiment for carrying out the present invention.

[2229] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2230] Step 1:

[2231] User:

[2232] A user starts an application and logs in by entering their user ID and password on the login screen. The login information is entered and sent to the server.

[2233] Input: User ID, Password

[2234] Output: Login authentication result

[2235] Specific operation: The application sends user information to the server, and the server performs the authentication process. If authentication is successful, a message indicating successful login is displayed and the user proceeds to the next screen. When logging in for the first time, a screen appears where basic information such as age, gender, weight, and blood pressure can be entered.

[2236] Step 2:

[2237] User:

[2238] Users take photos of their daily meals using a smartphone or tablet, and then use the application's upload function to send the images to the server.

[2239] Input: Food image

[2240] Output: Image data sent to the server

[2241] Specific operation: The user uses the "Photo and upload food" function in the application, selects the photo they have taken, and sends it to the server. The photo is temporarily stored in the device's local storage and then uploaded to the server.

[2242] Step 3:

[2243] Device:

[2244] The device receives the image of the meal, temporarily stores it in memory, and then transmits the image data to a cloud server.

[2245] Input: Image data sent by the user

[2246] Output: Image data sent to the cloud server

[2247] Specific operation: The device temporarily stores the received image data, then sends it to the cloud server via an HTTP request. After sending, a success notification is displayed to the user.

[2248] Step 4:

[2249] User:

[2250] Use IoT devices that collect health data, such as pedometers, body composition monitors, and blood pressure monitors, and sync the data to the application.

[2251] Input: Health data from IoT devices

[2252] Output: Health data synced to the application

[2253] Specific operation: The IoT device sends health data to the terminal via Bluetooth or Wi-Fi, and the application receives and stores the data. If necessary, the data is sent to the server.

[2254] Step 5:

[2255] Device:

[2256] To periodically check the user's emotional state, emotional questions are displayed on the screen to collect the user's emotional data.

[2257] Input: User sentiment response

[2258] Output: Collected emotion data

[2259] How it works: The application periodically displays a pop-up notification asking for the user's sentiment rating and collects their response. The collected data is temporarily stored within the application and then sent to the server.

[2260] Step 6:

[2261] server:

[2262] The server sends the meal image data to an image analysis AI engine, which identifies the type and amount of ingredients and their nutrients.

[2263] Input: Image data sent to the cloud server

[2264] Output: Analysis of ingredients and nutrients

[2265] How it works: The server sends the received meal image to the AI ​​engine, which uses a deep learning model to analyze the image and identify the types and amounts of ingredients and nutrients. The results are returned to the server in JSON format.

[2266] Step 7:

[2267] server:

[2268] It combines profile information, health data, and emotional data to identify nutrient deficiencies and also determines nutritional factors based on emotional data.

[2269] Input: Profile information, health data, emotion data, image analysis results

[2270] Output: Identification of nutritional deficiencies and emotional nutritional factors

[2271] How it works: The server loads the user's profile information, health data, and emotion data from the database, and integrates them with the image analysis results. It uses an analysis algorithm to identify nutrient deficiencies and emotion-based nutritional factors.

[2272] Step 8:

[2273] server:

[2274] The supplement proposal is sent to the user's device and confirmation and consent are requested.

[2275] Input: Identification of nutritional deficiencies and emotional nutritional factors

[2276] Output: Supplement suggestions sent to the user's device

[2277] Specific operation: The server generates supplement suggestions based on the nutrient deficiency and emotion data, and sends them to the user's device as a notification. The user confirms the suggestions and presses the "Agree" button to proceed to the next step.

[2278] Step 9:

[2279] server:

[2280] Give instructions to the dispenser and load the required supplement cartridge, extracting the appropriate amount of supplement and preparing it for dispensing in powder form.

[2281] Input: User confirmation and consent, supplement suggestions

[2282] Output: Notification that supplement is ready to be extracted

[2283] Specific operation: The server sends a command to the dispenser to automatically load the required supplement cartridge. The dispenser extracts the optimal amount of supplement and notifies the server that it is ready. The server then finally notifies the user that it is ready.

[2284] The above is the flow of specific processing steps of the program of this system.

[2285] (Application example 2)

[2286] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2287] Conventional supplement recommendation systems were based solely on the user's dietary and health data, and were unable to take into account the user's emotional state. This meant that they were unable to meet the nutritional needs of users affected by stress and emotional fluctuations, making it difficult to recommend optimal supplements tailored to individual needs. Another issue was the inability to provide immediate recommendations in physical stores.

[2288] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2289] In this invention, the server includes means for receiving images of meals from a user, means for analyzing the images to identify ingested nutrients, means for collecting basic information and health data of the user, means for collecting and analyzing emotional information of the user, means for suggesting optimal supplements based on the nutrient deficiencies and emotional information, means for selecting supplements based on the suggestions, and means for storing and collecting a history of supplement selection and user feedback. This enables personalized optimal supplement suggestions based on the user's diet, health data, and emotional state, and immediate provision of supplements in physical stores.

[2290] "Users" refers to people who use the system.

[2291] "Means for receiving images" refers to a device or software that captures images of meals taken or selected by a user into the system.

[2292] "Means for analyzing images to identify nutrients ingested" refers to devices or software that use AI or machine learning to analyze the type of food ingredients and nutrient content from received images.

[2293] "Basic information" refers to personal data such as the user's age, gender, weight, and blood pressure.

[2294] "Health Data" refers to health-related data such as a user's body composition, blood pressure, heart rate, and number of steps taken.

[2295] "Emotional information" refers to information that indicates the user's emotional state, such as stress or happiness.

[2296] "Missing nutrients" refers to the user's basic information and health data, as well as nutrients necessary for maintaining and improving future health.

[2297] "Supplements" refer to nutritional supplements that are used to supplement the user's nutritional deficiencies.

[2298] "Means for suggesting" refers to a device or software that suggests appropriate supplements based on the analysis results and the user's emotional information.

[2299] "Extracting means" refers to a device or software that provides the user with the appropriate amount of the suggested supplement.

[2300] "Means for storing and collecting history and feedback" refers to devices or software that record and store information about the supplements received by users and their impressions after use.

[2301] "IoT devices" refer to health devices that can be connected to the Internet (such as body composition monitors and blood pressure monitors).

[2302] "Physical store terminals" refers to dedicated terminals or tablets installed in physical stores such as healthcare shops and drugstores.

[2303] This invention relates to a system that allows users to take photos of their meals, analyze their nutritional intake using AI, and then combines this with an emotion engine that recognizes the user's emotions to provide optimal supplements to individuals. The system is comprised of a user's device, a cloud-based server, an emotion engine, and a supplement provider.

[2304] System configuration

[2305] User device functions

[2306] 1. User authentication and profile management

[2307] The user starts the application and enters their profile information (age, gender, weight, blood pressure, etc.) when using it for the first time. This information is sent to the server and saved.

[2308] 2. Take and upload a photo of your meal

[2309] Users take photos of their daily meals using a device (smartphone, tablet, etc.) and upload the images to the application.

[2310] 3. Collecting Emotional Data

[2311] The application periodically records the user's emotional state, and this emotional data is collected through the user's touch, voice input, or facial recognition technology.

[2312] Server Features

[2313] 1. Image analysis and nutrient identification

[2314] The server passes user-submitted images to an AI model that uses software such as TensorFlow and Keras to identify the type of food and its nutrients.

[2315] 2. Emotion analysis

[2316] The server passes the emotional data sent by the user to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms.

[2317] 3. Recommending the best supplements

[2318] The server identifies nutrient deficiencies based on the user's basic information, health data, and emotional data, and automatically generates optimal supplements.

[2319] Functions of the supplement providing device

[2320] 1. Extraction of Supplements

[2321] The supplement providing device dispenses the appropriate amount of supplements containing the nutrients required by the user based on instructions from the server. The device dispenses only the required amount after inserting a supplement cartridge.

[2322] 2. Submissions and Feedback Notifications

[2323] The extracted supplements are provided to the user, and their history and feedback are stored on a server.

[2324] Specific examples

[2325] Example 1: Breakfast analysis and supplement provision

[2326] 1. User Operation

[2327] The user has an omelet, salad, and orange juice for breakfast and takes a photo of it.

[2328] Photos of the meal are uploaded to the application and sent to the server.

[2329] 2. Image analysis and emotion analysis

[2330] The server uses an AI engine to analyze the ingredients of the omelet (protein, fat, etc.), the salad (vitamins, minerals, etc.), and the orange juice (vitamin C, etc.).

[2331] In addition, it analyzes the user's emotional data to detect high levels of stress.

[2332] 3. Supplement suggestions and selection

[2333] The server suggests supplementing with vitamin D and calcium, which are deficient, as well as B vitamins.

[2334] The supplement delivery device extracts this and provides it to the user.

[2335] Example prompts for generative AI models

[2336] "Build an image classification model to analyze images of meals taken by users and identify the nutrients contained in them. The model should be able to recognize multiple nutrients such as protein, fat, vitamins, and minerals and estimate the amount of each. Also, build an emotion classification model to analyze the user's emotional state from text and identify emotions such as stress or happiness. Integrate this data and generate relevant suggestions for a system that recommends optimal supplements."

[2337] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2338] Step 1:

[2339] User login and profile entry

[2340] Users launch the application and enter their login information to access the system. When using it for the first time, they enter profile information such as age, gender, weight, and blood pressure. This profile information is sent as input data to the cloud server and saved there.

[2341] Input: User login and profile information

[2342] Output: User profile information stored on the server

[2343] Step 2:

[2344] Taking and uploading food photos

[2345] Users take photos of their daily meals with their smartphone or tablet and upload the images to the application, which receives the images and sends them to a cloud server.

[2346] Input: User-taken food image

[2347] Output: Meal images stored on the server

[2348] Step 3:

[2349] Health data collection

[2350] Users measure their own health data using IoT devices such as body composition scales and blood pressure monitors, and synchronize this data with the application. The device receives the health data and sends it to a cloud server.

[2351] Input: Health data collected from IoT devices

[2352] Output: Health data stored on the server

[2353] Step 4:

[2354] Collecting Emotional Data

[2355] The application periodically records the user's emotional state. The user provides emotional information through touch, voice, or facial recognition, and the emotional data is sent from the device to a cloud server.

[2356] Input: User emotion data (touch, voice, facial recognition)

[2357] Output: Emotion data stored on the server

[2358] Step 5:

[2359] Image analysis and nutrient identification

[2360] The server passes the uploaded meal images to an AI model, which analyzes the types of ingredients and nutrient content. It uses TensorFlow and Keras to analyze the images and generate specific nutrient data.

[2361] Input: Food images stored on the server

[2362] Output: Parsed nutrient data

[2363] Step 6:

[2364] Emotion analysis

[2365] The server passes the collected emotional data to an emotion recognition engine, which analyzes the user's emotional state using natural language processing technology and machine learning algorithms to generate emotional data.

[2366] Input: Emotion data stored on the server

[2367] Output: Parsed emotional state data

[2368] Step 7:

[2369] Recommendations for optimal supplements

[2370] The server combines the user's basic information, health data, nutrient data, and emotional data, and the system automatically generates optimal supplement recommendations based on the nutrients that are lacking and the user's emotional state.

[2371] Input: Basic information, health data, nutrition data, emotional state data

[2372] Output: Proposed supplemental data

[2373] Step 8:

[2374] Supplement Extract

[2375] The supplement providing device extracts the appropriate amount of supplement containing the necessary nutrients based on the suggested data from the server. The supplement cartridge is set and the extraction process is carried out.

[2376] Input: Suggested supplemental data from the server

[2377] Output: Extracted supplement

[2378] Step 9:

[2379] Offer and Feedback Notification

[2380] The extracted supplements are provided to the user, and their usage history and feedback are collected. The user receives the supplements and provides feedback through the application about their impressions after use. This data is stored on the server.

[2381] Input: User feedback and supplement history

[2382] Output: Feedback and historical data stored on the server

[2383] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2384] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2385] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2386] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2387] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2388] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate betwe...

Claims

1. means for receiving an image of a meal from a user; means for analyzing the image to identify ingested nutrients; means of collecting basic information and health data of users; A means for identifying nutrients that are lacking in the user based on the basic information and health data; A means for suggesting optimal supplements based on the nutrients that are lacking; A means for extracting a supplement based on the proposal; A means of storing history and collecting feedback on supplement extractions; A system including:

2. The system according to claim 1, wherein personal data is used to suggest supplements.

3. The system of claim 1, which automatically collects health data from IoT devices.

Images