System

Abstract

A system is provided.SOLUTION: This system includes a means for receiving image and text data inputted by the user, a means for analyzing the received image and text data and recognizing food materials and their nutrients, a means for evaluating the degree of nutritional conditions of the user on the basis of the recognized food materials and nutrients, and a means for displaying evaluation results in a score form.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, achieving a nutritionally balanced diet is extremely difficult, especially for people living alone or housewives and househusbands who plan their daily meals. There is a demand for a method to easily determine nutrient deficiencies and excesses and efficiently consume necessary nutrients. Furthermore, there is a lack of easy ways to select appropriate meals when cooking at home takes time and effort. Against this background, the objective of this invention is to provide a system that accurately grasps the user's nutritional status and suggests optimal meals. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for receiving image and text data input by a user, a means for analyzing the received image and text data to recognize ingredients and their nutrients, a means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, and a means for displaying the evaluation results in the form of a score. Furthermore, based on the evaluation results, the system suggests optimal menus and recipes for the user, and provides a means for creating and ordering meal kits based on the suggested menus and recipes, thereby reducing the effort required for home cooking. Furthermore, based on the evaluation results, the system suggests restaurants that can provide missing nutrients and provides a means for searching for restaurants based on the user's location information, allowing users to easily select appropriate meals even when cooking at home is difficult.

[0006] "User" refers to an individual who uses the system to understand their own nutritional status and receive suggestions for optimal menus and dining out options.

[0007] "Images" and "text data" are information input by the user that indicates the meal contents, with the images being photographs of the meal and the text data including a list of ingredients and menu items.

[0008] "Image recognition AI" is an artificial intelligence technology that analyzes input images and identifies ingredients within them.

[0009] "Text analysis AI" is an artificial intelligence technology that analyzes input text data and extracts information about ingredients and nutrients.

[0010] "Nutritional status" refers to the total amount of nutrients such as calories, protein, vitamins, and minerals that a user consumes each day.

[0011] "Assessment" refers to the process of comparing the intake of each nutrient with the recommended intake based on the user's current nutritional status to determine whether it is insufficient or excessive.

[0012] The "score format" is a scoring method that visually expresses the user's nutritional status in an easy-to-understand manner.

[0013] "Menu" refers to the combination of meals that a user should consume in a day, including specific menus.

[0014] A "recipe" is a detailed cooking guide that includes the steps and list of ingredients a user needs to prepare based on a suggested menu.

[0015] A "meal kit" refers to a set that provides all the necessary ingredients and quantities based on a specific menu.

[0016] The "dining out destination" refers to a restaurant or eatery that the user selects when dining out, and is a place that offers a menu containing the suggested nutrients.

[0017] "Location information" is information that indicates the user's current location or a specified location, and is used to suggest and search for places to eat out. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention is a system that accurately grasps the user's nutritional status and suggests optimal meal plans and dining out options. Below, we will create a program for the system and explain its processing in natural language. We will also provide specific examples.

[0040] Overall flow

[0041] The system operates through the following major steps:

[0042] 1. Image / text data input

[0043] 2. Image Recognition and Text Analysis

[0044] 3. Nutritional status assessment and labelling

[0045] 4. Menu and recipe suggestions (for home cooking)

[0046] 5. Dining out suggestions (if you don't cook at home)

[0047] Program processing

[0048] 1. Image / text data input

[0049] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app. Alternatively, the user can enter the ingredients and meal details in text format.

[0050] User: Daily meal information can be provided to the system with simple operations.

[0051] 2. Image Recognition and Text Analysis

[0052] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo shows toast and bananas.

[0053] Server: The text data is processed by text analysis AI to extract ingredients and cooked dishes from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0054] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[0055] 3. Nutritional status assessment and labelling

[0056] Server: Collects the user's daily dietary information and calculates the total intake of calories, protein, vitamins, minerals, etc. For example, it calculates that the calories are 1820 kcal, the vitamin C is 45 mg, and the iron is 7 mg.

[0057] Server: Compare with recommended intake and assess whether there is a surplus or deficiency.

[0058] On your device: The evaluation results are displayed in the form of a score. For example, your overall nutritional balance is 90 points, your vitamin C intake is 70 points, and your iron intake is 60 points.

[0059] User: Checks nutritional status and chooses next action (cooking at home or eating out).

[0060] 4. Menu and recipe suggestions (for home cooking)

[0061] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[0062] Server: Generates and displays detailed recipes based on the proposed menu. For example, it presents a recipe for "Lemon and Chicken Stir-fry."

[0063] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0064] Server: Works with the meal kit generation system to allow users to easily order.

[0065] User: Order a meal kit and have it delivered to your home.

[0066] 5. Dining out suggestions (if you don't cook at home)

[0067] Server: Based on the user's nutritional assessment results and location information, searches for restaurants that can help fill in any missing nutrients. For example, it searches for restaurants with menus rich in vitamin C.

[0068] Device: Displays a list of nearby restaurants and provides nutritional information for each location.

[0069] User: Selects a dining location and eats at the designated restaurant.

[0070] Specific examples

[0071] User's first day

[0072] Breakfast: User uploads a photo of toast and bananas to the app.

[0073] The server analyzes the image and recognizes toast and bananas.

[0074] The server compares the data with a nutritional database to obtain calorie, vitamin, and mineral information.

[0075] The server calculates intake and evaluates nutritional balance.

[0076] The device will display the score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[0077] The user chooses to cook for themselves the next day.

[0078] The server will suggest the best menu and provide detailed recipes.

[0079] Users order a meal kit and the ingredients arrive the next morning.

[0080] User's second day

[0081] Lunch: Users upload photos of their salad and soup to the app.

[0082] The server analyzes the image and recognizes the ingredients.

[0083] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[0084] If the user does not cook for themselves, they will look for nearby restaurants.

[0085] The device will suggest restaurants with menu items that provide vitamin C supplements.

[0086] The user selects a restaurant and enjoys dining out.

[0087] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[0088] The processing flow will be explained below.

[0089] Step 1:

[0090] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[0091] Step 2:

[0092] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[0093] Step 3:

[0094] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0095] Step 4:

[0096] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates and a small amount of protein, and that bananas contain vitamin C and dietary fiber.

[0097] Step 5:

[0098] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[0099] Step 6:

[0100] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[0101] Step 7:

[0102] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[0103] Step 8:

[0104] User: Check nutritional status and choose whether to cook at home or eat out.

[0105] Step 9:

[0106] Server: If you choose to cook at home, the server will generate optimal menus to supplement any missing nutrients. For example, it will suggest dishes using lemon to supplement the missing vitamin C.

[0107] Step 10:

[0108] Server: Generates detailed recipes based on the menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[0109] Step 11:

[0110] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0111] Step 12:

[0112] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[0113] Step 13:

[0114] User: Orders a meal kit and chooses to have it delivered to their home.

[0115] Step 14:

[0116] Server: If you choose not to cook at home, the app will search for restaurants that can help you meet your nutritional needs based on your rating and location. For example, it will search for nearby restaurants with menus rich in vitamin C.

[0117] Step 15:

[0118] On your device: Based on your location, it will display a list of appropriate restaurants and provide nutritional information for each restaurant.

[0119] Step 16:

[0120] User: Choose from suggested dining options and dine at the specified restaurant.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] Nutritional management of meals is difficult for many people, and especially for busy modern people, there is a need to accurately and easily understand what they eat every day and ensure adequate nutritional intake. However, current methods require the time-consuming manual recording of meal contents, and without specialized knowledge, it is difficult to maintain an appropriate nutritional balance. Furthermore, there is a problem in that it is difficult to consider nutritional balance when choosing meals to eat out.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes a means for receiving image and text data entered by the user, a means for analyzing the received image data with a generative AI model to identify ingredients in the photo, a means for analyzing the received text data with a generative AI model to extract ingredients and menu items, a means for comparing the data with a nutritional value database of ingredients to obtain nutrient information, a means for evaluating the user's nutritional status based on the obtained nutrient information, and a means for displaying the evaluation results in the form of a score. This allows users to easily record their daily diet and obtain specific guidelines for maintaining nutritional balance. This system also provides an efficient and easy nutritional management system for busy modern people.

[0126] "User" refers to an individual who uses the system.

[0127] "Image and text data" refers to photographic data and text information that records meal details entered by the user.

[0128] A "generative AI model" refers to an artificial intelligence algorithm that analyzes received image and text data to identify ingredients and menu items.

[0129] "Ingredients" are raw materials used in cooking, specifically vegetables, meat, fruits, etc.

[0130] "Nutrients" refer to the nutritional components contained in food ingredients, such as calories, protein, vitamins, and minerals.

[0131] A "nutritional value database" refers to a database that accumulates nutritional information for each food ingredient.

[0132] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[0133] The "evaluation results" are scores indicating the nutritional balance calculated based on the dietary content.

[0134] "Score format" refers to a method in which evaluation results are expressed numerically as a score out of 100.

[0135] A "menu" refers to the combination of dishes served at one meal or one day.

[0136] A "recipe" is a detailed list of ingredients and steps for making a particular dish.

[0137] A "meal kit" is a package that brings together all the ingredients and cooking instructions needed for a specific menu.

[0138] "Dining out" refers to places where you can eat and drink outside of the home, such as restaurants and cafes.

[0139] "Location information" refers to digital data that indicates a user's current location.

[0140] This invention is a system that accurately assesses a user's nutritional status and suggests optimal meal plans and dining options. Below, we will create a program for the system and explain its processing in natural language. We will also provide names and specific examples of the hardware and software used.

[0141] Overall flow

[0142] The system operates through the following major steps:

[0143] 1. Image / text data input

[0144] 2. Image Recognition and Text Analysis

[0145] 3. Nutritional status assessment and labelling

[0146] 4. Menu and recipe suggestions (for home cooking)

[0147] 5. Dining out suggestions (if you don't cook at home)

[0148] Hardware and software used

[0149] Device: An input device such as a smartphone, tablet, or PC. The user takes photos of food and inputs text.

[0150] Server: Cloud service or dedicated server. Analyzes received data and evaluates nutritional status.

[0151] Generative AI models: AI models for image recognition and text analysis, for example, using frameworks such as TensorFlow and PyTorch.

[0152] Database: Nutritional value database such as food composition tables. Use a relational database such as MySQL or PostgreSQL.

[0153] Specific examples of programs

[0154] Image / text data input

[0155] Device: Users can take photos of their meals using their smartphone camera and upload them to the server via the app. They can also input ingredients and menu items in text format.

[0156] User: With simple operations, users can provide their daily dietary information to the system.

[0157] for example:

[0158] "Upload a photo of toast and bananas."

[0159] "Breakfast: toast, banana"

[0160] Image recognition and text analysis

[0161] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana."

[0162] Server: The text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, ingredients are recognized from "Breakfast: toast, banana."

[0163] Server: Compares with a database of food ingredients' nutritional values ​​to obtain calorie and nutrient information.

[0164] for example:

[0165] "Get nutritional information for the ingredients in this photo."

[0166] "Extract ingredients from text data and collect nutritional information."

[0167] Nutritional status assessment and labeling

[0168] Server: Collects the user's daily dietary information and calculates intake of calories, protein, vitamins, minerals, etc.

[0169] Server: Based on this data, we compare it with the recommended intake and evaluate whether it is excessive or insufficient.

[0170] Device: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is 90 points, vitamin C intake is 70 points, etc.

[0171] User: Based on the evaluation results, choose the next action to take: cooking at home or eating out.

[0172] for example:

[0173] "Please rate your nutritional status today and display it in the form of a score."

[0174] "You are deficient in Vitamin C. Please select your next action."

[0175] Menu and recipe suggestions (for home cooking)

[0176] Server: Suggests optimal menus to supplement missing nutrients. For example, if you are lacking in vitamin C, suggest dishes using lemon.

[0177] Server: Based on the proposed menu, generates a specific recipe and presents it to the user.

[0178] Device: Displays a weekly meal plan with a list of ingredients and cooking instructions.

[0179] Server: If necessary, it will link with meal kit providers and provide a function that allows users to easily order meal kits.

[0180] User: Orders the suggested meal kit, has it delivered to their home, and cooks it.

[0181] for example:

[0182] "Please suggest some recipes using lemon as I am lacking in Vitamin C."

[0183] "Order your meal kit and view a week's worth of meal plans."

[0184] Dining out suggestions (if you don't cook at home)

[0185] Server: Based on the user's location information and nutritional assessment results, search for restaurants that can replenish missing nutrients. For example, search for restaurants rich in vitamin C.

[0186] Terminal: Displays a list of suitable restaurants in the vicinity to the user and provides information on the nutritional intake at each restaurant.

[0187] User: Chooses a restaurant and decides to take action to eat out.

[0188] for example:

[0189] "Find restaurants that have menu items rich in vitamin C."

[0190] "View a list of nearby restaurants and choose the one that's right for you."

[0191] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[0192] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0193] Step 1: Input image / text data

[0194] Device: Users take photos of their meals using their smartphone camera and upload them to the server via the app. They can also enter meal details in text format. Input data includes meal photos and text information such as "Breakfast: toast, banana."

[0195] User: The app is easy to use, and taking photos and entering information can be completed with just a few taps. Once entered, images and text data of the meal contents are generated.

[0196] Step 2: Image recognition and text analysis

[0197] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana" from an uploaded photo. The input is the image data, and the output is a list of recognized ingredients.

[0198] Server: The received text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, the text "Breakfast: toast, banana" is used to recognize "toast" and "banana." The input is text data, and the output is a list of extracted ingredients.

[0199] Server: Compares with the nutritional value database of ingredients to obtain the necessary nutritional information. For example, obtain the calorie and vitamin content of toast and banana from the database. The input is a list of recognized ingredients, and the output is a list of nutritional information.

[0200] Step 3: Nutritional status assessment and labelling

[0201] Server: Aggregates the user's daily dietary information and calculates the intake of calories, protein, vitamins, minerals, etc. For example, calculate the total intake by adding up each nutrient in breakfast and lunch. The input is a list of nutrient information, and the output is the total intake.

[0202] Server: Based on these total intakes, compare them with the recommended intake and evaluate whether there are any nutritional deficiencies or excesses. For example, judge based on the Japanese Dietary Reference Intakes. The input is the total intake, and the output is the evaluation result.

[0203] Terminal: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is displayed as 90 out of 100, and vitamin C intake is displayed as 70. The input is the evaluation results, and the output is a score display.

[0204] User: Based on the evaluation results, the user chooses whether to cook at home or eat out. The results are used as an indicator to determine the next action.

[0205] Step 4: Menu and recipe suggestions (for home cooking)

[0206] Server: Suggests the optimal menu to supplement missing nutrients. For example, if you are lacking in vitamin C, it suggests dishes using lemon. The input is the evaluation result, and the output is the suggested menu.

[0207] Server: Generates a specific recipe based on the proposed menu and presents it to the user. For example, it provides a recipe for "stir-fried chicken with lemon." The input is the proposed menu, and the output is a detailed recipe.

[0208] Terminal: Displays a weekly meal plan with a list of ingredients and cooking instructions, allowing users to use it as a shopping list. The input is a detailed recipe, and the output is a meal plan and an ingredient list.

[0209] Server: If necessary, it connects with meal kit providers and provides a function that allows users to easily order meal kits. The input is a detailed recipe, and the output is meal kit ordering information.

[0210] User: Order a meal kit, have it delivered to your home, and then cook it. Easily prepare a nutritiously balanced meal.

[0211] Step 5: Suggesting places to eat out (if you don't cook at home)

[0212] Server: Based on the user's location information and nutritional assessment results, searches for restaurants that can replenish missing nutrients. For example, searching for restaurants with menus rich in vitamin C. The input is the location information and the assessment results, and the output is the suggested restaurants.

[0213] Terminal: Displays a list of nearby restaurants to the user and provides information on the nutritional value of each restaurant. The input is the suggested dining out location, and the output is the restaurant list.

[0214] User: Chooses a restaurant and decides to dine out. Dining options are presented in an easy-to-understand way.

[0215] In this way, each processing step is clearly separated, and the necessary input data is acquired, appropriate data processing and calculations are performed based on that data, and output is obtained. This allows users to easily understand their nutritional status and implement an accurate meal plan. The entire system efficiently and effectively supports users' nutritional management.

[0216] (Application example 1)

[0217] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0218] There is a need for a system that can accurately grasp the nutrients that users are consuming in their daily meals and receive specific suggestions to supplement any nutrient deficiencies. In particular, there is a problem in that there is a lack of ways to check nutritional status and efficiently purchase necessary ingredients when shopping in physical stores, making it difficult for users to manage their health.

[0219] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0220] In this invention, the server includes means for receiving image and text data input by a user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, means for displaying the evaluation results in the form of a score, means for taking images of ingredients purchased in the store, and means for suggesting foods in the store that will supplement any nutrient deficiencies based on the nutritional evaluation results. This allows users to check their nutritional status in real time while shopping and purchase recommended foods that will efficiently supplement any nutrient deficiencies.

[0221] A "user" is an individual who uses the system to understand their own nutritional status and receive appropriate meal plans and food recommendations.

[0222] "Image and text data" refers to photographs and text information of meal contents and ingredients that users input into the system.

[0223] "Analysis" is the process by which the system automatically recognizes and understands the content based on the input image and text data.

[0224] "Ingredients" are elements of food that are recognized based on images taken by the user or text data entered by the user.

[0225] "Nutrients" are components contained in foodstuffs that are necessary for the human body, such as proteins, lipids, vitamins, and minerals.

[0226] "Recognition" is the process by which the system analyzes image and text data to identify ingredients and nutrients and extract them as information.

[0227] "Evaluation" refers to quantifying and analyzing the amount and balance of nutrients ingested by the user based on the information on recognized ingredients and nutrients.

[0228] The "score format" is a method of converting evaluation results into numerical values ​​and displaying them in a way that users can understand at a glance.

[0229] "Display" means visually showing the evaluation results and proposals on the screen of the user's terminal.

[0230] "In-store" refers to the indoor area of ​​a physical store where food is purchased.

[0231] "Photographing" refers to the act of a user taking a picture of an ingredient with a smartphone or camera device.

[0232] "Complementary foods" are foods that the system suggests to supplement missing nutrients.

[0233] "Suggestion" refers to the system recommending the most suitable meals or foods for the user.

[0234] "Nutritional assessment results" refer to the results obtained by analyzing the quantity and quality of nutrients ingested by the user.

[0235] This invention is a system that grasps the user's nutritional status and suggests optimal meal plans and dining out locations. This system mainly includes the following means:

[0236] 1. Means for receiving image and text data entered by the user:

[0237] Users can take photos of their meals using their smartphone camera and upload them to the server using a dedicated application. They can also enter the meal contents as text.

[0238] 2. A means for analyzing the received image and text data to recognize ingredients and their nutrients:

[0239] The server uses image recognition software (e.g., OpenCV) to identify ingredients from image data, and natural language processing (NLP) techniques (e.g., the NLTK library) for text data to analyze meal details.

[0240] 3. A means of assessing the user's nutritional status based on perceived ingredients and nutrients:

[0241] The server references a food nutritional value database (e.g., the USDA nutrition database) to obtain the nutritional information of the photographed food. Based on this, it calculates the daily intake and evaluates the user's nutritional balance.

[0242] 4. How to display the evaluation results in the form of a score:

[0243] The server converts the evaluation results into numerical values ​​and calculates the nutritional balance and intake level of each nutrient in the form of a score, which is then displayed on the user's smartphone via a dedicated app.

[0244] 5. How to take pictures of ingredients purchased in-store:

[0245] Users take photos of ingredients they have purchased at a physical store and upload them to the server through the application, where image recognition technology is used to identify the ingredients they have purchased.

[0246] 6. Based on the nutritional assessment results, a method for proposing foods to supplement nutrients that are lacking in the store:

[0247] The server identifies any nutrient deficiencies based on the nutritional balance assessment and suggests foods to supplement them. These suggestions are displayed on the user's smartphone via the application. For example, if a person is lacking in vitamin C, it will suggest purchasing "oranges."

[0248] Specific examples

[0249] While shopping at a physical store, a user purchases "toast" and "banana" and uploads a photo of them to the application. The server uses image recognition to identify the toast and banana. It then refers to a nutritional value database and compiles the nutritional information for each ingredient. If the result shows that the user is deficient in vitamin C, the server suggests purchasing "orange juice" and displays this to the user through the application.

[0250] This process can be started with the following prompt:

[0251] "I had toast and a banana for breakfast. I'd like to upload a photo and receive a nutritional assessment and suggestions to fill in any missing nutrients."

[0252] By using the above means, this system helps users to efficiently understand their nutritional status and effectively take in the necessary nutrients.

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] Receives image and text data entered by the user

[0256] The user uses their smartphone to take a photo of their meal or input the ingredients and meal contents in text format. The device sends the data to the server via the app. The input data is received by the server as an image file and text data. This input data is analyzed in the next step.

[0257] Step 2:

[0258] Analyzes received image and text data to recognize ingredients and their nutrients

[0259] The server uses image recognition software (e.g., OpenCV) to analyze the received image data. This analysis identifies ingredients in the image and generates an ingredient list. The text data is also analyzed using natural language processing (NLP) techniques (e.g., the NLTK library) to extract ingredient information. As a result, a list of identified ingredients and their nutritional information is output.

[0260] Step 3:

[0261] Evaluate the user's nutritional status based on recognized ingredients and nutrients

[0262] The server references a nutritional value database (e.g., the USDA database) to obtain the nutrient information for the ingredients identified in step 2. Based on this data, it calculates the amount of nutrients consumed per day and evaluates the user's nutritional balance. The calculation results are output as intake amounts for calories, protein, vitamins, minerals, etc.

[0263] Step 4:

[0264] Display the evaluation results in score format

[0265] The server converts the nutritional balance calculated in step 3 into a numerical score. This score is calculated based on a comparison with the target intake amount for each nutrient. The evaluation results are sent to the device as a numerical score and displayed on the user's smartphone via a dedicated app. Users can visually grasp the nutritional balance.

[0266] Step 5:

[0267] Take a photo of the food you purchased in the store

[0268] A user purchases ingredients at a physical store and takes a photo of the ingredients with their smartphone. This photo data is then uploaded to the server. The input data is received by the server as an image file. This input data is analyzed in the next step.

[0269] Step 6:

[0270] Based on the nutritional assessment results, we will suggest foods to supplement the nutrients lacking in the store.

[0271] The server compares the nutritional assessment results obtained in step 3 with the food ingredient information acquired in step 5 to identify any nutrients that are lacking. Based on the identified nutrients that are lacking, it selects and generates a list of foods to recommend. These suggestions are displayed on the user's smartphone via the application. For example, if the user is lacking in vitamin C, it will suggest purchasing "oranges."

[0272] Through the above steps, this system helps users efficiently understand their nutritional status and effectively ingest the necessary nutrients.

[0273] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0274] This invention combines a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out options with an emotion engine that recognizes the user's emotions. Below, we will create a program for the system and explain its processing in natural language. Specific examples will also be included.

[0275] Overall flow

[0276] The system operates through the following major steps:

[0277] 1. Image / text data input

[0278] 2. Entering Emotion Data

[0279] 3. Image Recognition and Text Analysis

[0280] 4. Nutritional status assessment and labelling

[0281] 5. Menu and recipe suggestions (for home cooking)

[0282] 6. Dining out suggestions (if you don't cook at home)

[0283] Program processing

[0284] 1. Image / text data input

[0285] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[0286] User: Provides daily meal information to the system.

[0287] 2. Entering Emotion Data

[0288] Device: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze the user's emotions from their voice and facial expressions.

[0289] Server: The emotion engine analyzes the user's input data and voice data to obtain emotional information.

[0290] 3. Image Recognition and Text Analysis

[0291] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[0292] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0293] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[0294] 4. Nutritional status assessment and labelling

[0295] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[0296] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[0297] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[0298] Server: Adjusts the displayed score of the evaluation results based on the user's emotional information. For example, if the user is under high stress, the server may prioritize vitamin C intake.

[0299] 5. Menu and recipe suggestions (for home cooking)

[0300] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[0301] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[0302] Server: Considers the user's emotional information and suggests menus that match their preferences and mood. For example, if the user is under a lot of stress, it suggests recipes that use ingredients that have a relaxing effect.

[0303] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0304] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[0305] User: Orders a meal kit and chooses to have it delivered to their home.

[0306] 6. Dining out suggestions (if you don't cook at home)

[0307] Server: Based on the evaluation results and emotional information, the server searches for restaurants that will replenish nutrients and match your mood. For example, it searches for nearby restaurants with menus rich in vitamin C.

[0308] On your device: Based on your location, it will display a list of restaurants that are relevant to you, with nutritionally and emotionally conscious menus for each restaurant.

[0309] User: Choose from suggested dining options and dine at the specified restaurant.

[0310] Specific examples

[0311] User's first day

[0312] Breakfast: User uploads a photo of toast and bananas to the app.

[0313] Emotion data: User enters "I feel depressed."

[0314] The server analyzes the image and recognizes toast and bananas.

[0315] The server checks the nutritional value database and obtains the nutritional information.

[0316] The server calculates intake and evaluates nutritional balance.

[0317] The device will display the evaluation score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[0318] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[0319] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[0320] User's second day

[0321] Lunch: Users upload photos of their salad and soup to the app.

[0322] Emotion data: User inputs "I feel stressed."

[0323] The server analyzes the image and recognizes the ingredients.

[0324] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[0325] The server suggests recipes that include foods that are effective in reducing stress.

[0326] If the user does not cook, the server will search for nearby restaurants.

[0327] The device will suggest restaurants with menus that are effective in replenishing vitamin C and reducing stress.

[0328] The user selects a restaurant and enjoys dining out.

[0329] This system allows users to easily understand their nutritional status and obtain an optimal meal plan that not only efficiently ingests the necessary nutrients but also takes into account their emotional state, contributing to both the user's health and mental well-being.

[0330] The processing flow will be explained below.

[0331] Step 1:

[0332] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[0333] Step 2:

[0334] Device: The user inputs their mood for the day from a selection of options, or their voice and facial expressions are recorded using the device's camera and microphone and sent to the emotion engine.

[0335] Step 3:

[0336] Server: The received image is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo contains toast and bananas.

[0337] Step 4:

[0338] Server: The received text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0339] Step 5:

[0340] Server: Using the emotion engine, analyzes the user's input data and voice data to extract the emotional information of the day. For example, it identifies the user as "feeling depressed" or "highly stressed."

[0341] Step 6:

[0342] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates, and bananas contain vitamin C and dietary fiber.

[0343] Step 7:

[0344] Server: Calculates the user's total daily calorie intake, protein, vitamins, minerals, etc. For example, calculates the total daily calories as 1820 kcal, vitamin C as 45 mg, and iron as 7 mg.

[0345] Step 8:

[0346] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[0347] Step 9:

[0348] Server: Adjust the score display of the evaluation results taking into account emotional information. For example, if stress is high, add points for vitamin C.

[0349] Step 10:

[0350] On-device: Displays nutritional status in the form of a score. For example, overall nutritional balance is 90 points, vitamin C intake is 70 points, and iron intake is 60 points.

[0351] Step 11:

[0352] User: Check nutritional status and choose whether to cook at home or eat out.

[0353] Step 12:

[0354] Server: If you choose to cook at home, the server will suggest the best meal plan to fill in any missing nutrients and emotional information. For example, if you are feeling depressed due to a lack of vitamin C, the server will suggest dishes that use lemon.

[0355] Step 13:

[0356] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[0357] Step 14:

[0358] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0359] Step 15:

[0360] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[0361] Step 16:

[0362] User: Orders a meal kit and chooses to have it delivered to their home.

[0363] Step 17:

[0364] Server: If you choose not to cook at home, the server uses your location information to supplement any missing nutrients and search for dining options that match your mood, based on your evaluation results and emotional information. For example, it searches for nearby restaurants with menus rich in vitamin C and that are relaxing.

[0365] Step 18:

[0366] On your device: Based on your location, it will display a list of restaurants that are suitable for you, and each restaurant will offer menus that take into consideration your nutritional needs and mood.

[0367] Step 19:

[0368] User: Choose from suggested dining options and dine at the specified restaurant.

[0369] In this way, the system comprehensively analyzes the user's nutritional status and emotional information to suggest optimal meal plans and dining options, allowing users to easily choose the right meal for their health and mental state.

[0370] Example 2

[0371] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] Conventional nutrition management systems perform nutritional assessments based solely on the dietary information entered by the user, making it difficult to propose optimal meal plans that take the user's emotional state into account. They also need to accurately recognize the nutritional information of ingredients and make specific suggestions tailored to each user's nutritional status. Furthermore, they need to ensure that the proposed menus and dining options are actually satisfying for the user both nutritionally and emotionally. Therefore, there is a need for a system that provides more comprehensive and accurate nutritional management and can balance the user's health and mental well-being.

[0373] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0374] In this invention, the server includes means for receiving image and text data input by the user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for analyzing the user's emotions and obtaining emotion data, means for identifying ingredients and nutrients based on the image and text data and comparing that information with an ingredient database, means for evaluating the user's nutritional status based on the identified information and determining whether nutrients are in excess or deficiency, and means for displaying the evaluation results in a score format taking into account the evaluation results and emotion data. This enables more accurate nutritional evaluation and meal plans that take the user's emotional state into consideration.

[0375] "User" refers to a person who uses this system.

[0376] "Image and text data" refers to photos and text information of meal contents that users enter into the system.

[0377] "Emotion data" refers to information that indicates the user's emotional state.

[0378] "Image recognition" refers to the technology of analyzing image data to identify ingredients and objects.

[0379] "Text analysis" refers to the technology of analyzing text data and extracting necessary information.

[0380] "Nutrients" refers to the calories, vitamins, minerals, and other components contained in food ingredients.

[0381] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[0382] "Evaluation results" refers to the results obtained by the system analyzing the user's nutritional status and indicating any deficiencies or excesses in the form of a score.

[0383] "Emotion engine" refers to technology that analyzes emotional data from the user's voice and facial expressions.

[0384] "Menu" refers to the meal menu proposed to the user.

[0385] A "recipe" refers to information that shows how to make a dish or the steps involved.

[0386] A "meal kit" is a product that includes a set of ingredients and cooking instructions needed for a specific menu.

[0387] "Dining out" refers to a restaurant that the user uses to eat out.

[0388] The present invention relates to a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out locations. This system is combined with an emotion engine that recognizes the user's emotions, enabling more comprehensive and personalized suggestions.

[0389] System configuration

[0390] The system uses the following main hardware and software:

[0391] Device: A mobile device such as a smartphone or tablet on which a dedicated application is installed.

[0392] Server: The central processing unit that collects, analyzes, and serves data. Uses a cloud-based server.

[0393] Image recognition AI: For example, use Google's Cloud Vision API.

[0394] Text analysis AI: For example, use OpenAI's GPT-3.

[0395] Emotion engine: An AI engine that performs voice analysis and facial expression analysis.

[0396] Entering data

[0397] Device: The user takes a photo of the meal with the device's camera and uploads it to the server using a dedicated application, or enters the ingredients and meal contents in text format. The system then receives the entered data.

[0398] Data analysis

[0399] Server: Analyzes the received image and text data. Image data is used to identify ingredients using image recognition AI (Google's Cloud Vision API). Text data is used to extract ingredients and menu items using text analysis AI (OpenAI's GPT-3).

[0400] Nutritional status assessment

[0401] Server: Compares the nutritional information of ingredients with the food database and collects the necessary nutritional information. For example, it obtains information such as the calories, vitamins, and minerals of toast and bananas. It then calculates the user's total daily calorie intake and intake of protein, vitamins, minerals, etc. This determines whether the user is consuming too many or too few nutrients.

[0402] Quantitative evaluation and display

[0403] Device: Displays the nutritional status assessment results in the form of a score. The assessment results include an overall score and scores for specific nutrients. For example, calorie intake score, vitamin C intake score, etc.

[0404] Acquiring and adjusting emotion data

[0405] Server: Analyzes the user's emotional data and adjusts the evaluation results based on their emotional state. For example, if the user is under high stress, the server may prioritize vitamin C intake. Emotional data is acquired using an emotion engine that analyzes voice and facial expressions.

[0406] Menu and recipe suggestions

[0407] Server: Based on the evaluation results and emotional data, the server proposes optimal menus and recipes for the user. Specific recipes to supplement missing nutrients and recipes that correspond to the user's emotional state are proposed. Detailed recipes are generated based on the proposed menus.

[0408] Meal kit generation

[0409] Server: Generates meal kits based on suggested menus and recipes and makes them available for selection by the user. The user can then order the suggested meal kit and have it delivered to their home.

[0410] Dining out suggestions

[0411] Server: To supplement missing nutrients in ingredients, the server recommends optimal dining options based on the evaluation results and emotion data. Based on the user's location information, the server searches for the nearest restaurant and lists dining options that offer appropriate menus. The user selects from the suggested dining options and eats at the specified restaurant.

[0412] Examples and prompts

[0413] Example of a user on day one

[0414] Breakfast: User uploads a photo of toast and bananas to the app.

[0415] Emotion data: User enters "I feel depressed."

[0416] The server analyzes the image and obtains nutritional information for the toast and banana.

[0417] The server calculates the intake amount and displays the evaluation score on the device (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[0418] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[0419] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[0420] Examples of prompt statements

[0421] Obtaining Nutrition Information

[0422] "Assigned task: Identify the nutritional content of the following foods: toast, banana."

[0423] Emotion-based regulation

[0424] "Assigned task: Adjust the nutritional evaluation based on the user's emotional state: sad. Increase emphasis on Vitamin C."

[0425] Menu suggestions

[0426] "Assigned task: Suggest a meal plan that includes ingredients to meet the following nutritional needs: Vitamin C. Consider user preference for chicken and stress-relief."

[0427] This system allows users to easily understand their nutritional status and efficiently consume the nutrients they need. It also provides an optimal meal plan that takes into account their emotional state, contributing to the user's health and mental well-being.

[0428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0429] Step 1:

[0430] Entering data

[0431] Input: The user takes a photo of the meal and uploads it through the app, or enters the ingredients and meal details in text format.

[0432] Specific operation: The user takes a photo of the meal using the device's camera function, then uploads the photo to the server using the app's "meal record" function, or enters the meal contents as text and sends it to the server.

[0433] Output: The device sends image and text data to the server.

[0434] Step 2:

[0435] Entering emotion data

[0436] Input: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze emotions from the user's voice and facial expressions.

[0437] Specific actions: The user selects an option such as "depressed" on the app's "emotion input" screen, and also uses the voice input function to say, "I'm feeling stressed today."

[0438] Output: The device sends emotion data to the server.

[0439] Step 3:

[0440] Image Recognition and Text Analysis

[0441] Input: Image and text data received by the server.

[0442] Specific operation: The server uses Google's Cloud Vision API to identify ingredients from the received image. For example, it recognizes "toast" and "banana" in the photo. It also uses OpenAI's GPT-3 to analyze text data and extract the ingredients and menu items listed. For example, it recognizes toast and banana from the text "Breakfast: toast, banana."

[0443] Output: Data containing identified ingredients and their nutritional information.

[0444] Step 4:

[0445] Obtaining and verifying nutrition information

[0446] Input: The ingredients identified by the server in step 3 and their nutritional information.

[0447] Specific operation: The server checks the food database (e.g., a public database) to obtain the necessary nutritional information. For example, it collects information such as the calories, vitamins, and minerals of toast and banana.

[0448] Output: Data containing nutritional information for each ingredient.

[0449] Step 5:

[0450] Nutritional status assessment and labeling

[0451] Input: Nutrition information and emotion data obtained by the server.

[0452] Specific operation: The server calculates and evaluates the user's total daily calorie intake, protein intake, vitamin intake, mineral intake, etc. For example, the total intake of toast and banana is 300 kcal and the total intake of vitamin C is 10 mg. The server also compares this with the recommended intake amount to determine whether there is a nutrient deficiency or excess. Furthermore, the server adjusts the evaluation result based on emotional data, increasing the importance of vitamin C.

[0453] Output: Data showing the adjusted assessment results in the form of scores.

[0454] Step 6:

[0455] Menu and recipe suggestions

[0456] Input: Server-adjusted nutritional assessment results and emotion data.

[0457] Specific operation: The server uses the assessment results to suggest the optimal menu to supplement the missing nutrients. For example, if you are lacking in vitamin C, it suggests "stir-fried chicken with lemon." It then generates a detailed recipe and provides it to the user.

[0458] Output: Data containing suggested meals and recipes.

[0459] Step 7:

[0460] Generate and order meal kits

[0461] Input: Server-generated recipe data.

[0462] What it does: The server generates meal kits based on the menu and recipes and makes them available for ordering. The meal kits include the necessary ingredients and cooking instructions. The user can order the meal kit and have it delivered to their home.

[0463] Output: Data containing meal kit order information.

[0464] Step 8:

[0465] Dining out suggestions

[0466] Input: Server receives evaluation results and emotion data.

[0467] Specific operation: The server searches for the best place to eat out based on the evaluation results and emotion data to supplement the missing nutrients. Based on the user's location information, it searches for nearby restaurants and lists stores that offer suitable menus.

[0468] Output: Data containing suggested dining locations and corresponding menus.

[0469] This system allows users to gain a detailed understanding of their nutritional status and receive optimal meal plans and dining out suggestions that take into account their emotional state, which is expected to promote health and mental well-being.

[0470] (Application example 2)

[0471] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0472] Conventional nutrition management systems generally assess a user's nutritional status based on their dietary data, but they are unable to optimize meal plans taking into account the user's emotional state. Therefore, nutritional assessment alone can be insufficient, and there is a particular need for meal suggestions tailored to the user's emotional state. Meanwhile, when choosing to eat out, it is difficult to suggest restaurants that combine emotional state and nutritional balance, creating a need for a system that comprehensively supports the user's health and mental state.

[0473] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring emotional data from the user's voice and facial expression, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, and means for proposing an optimal delivery menu based on the nutritional status evaluation result and the emotional data. This makes it possible to propose meals that comprehensively consider the user's nutritional balance and emotional state.

[0474] "User" means an individual who uses the system to manage their own diet and nutrition.

[0475] "Image and text data" refers to information entered by the user in the form of photos and text about meal contents and ingredients.

[0476] "Ingredients" refers to foods and ingredients used in cooking and eating.

[0477] "Nutrients" refer to the components of food that are necessary for the human body, such as proteins, vitamins, and minerals.

[0478] "Voice and facial expressions" refers to the tone of voice and facial expressions used by the user to express their emotional state.

[0479] "Emotion data" is data obtained as a result of analyzing the user's emotions.

[0480] "Nutritional status" refers to the total amount and balance of nutrients a user consumes each day.

[0481] "Evaluation results" refers to the evaluation value or score that the system gives based on the nutritional status and emotional data analyzed.

[0482] "Delivery menu" refers to the meal menu delivered to the user, taking into consideration nutritional balance and emotional state.

[0483] A "menu" is a specific combination of ingredients and dishes used to determine a meal menu or schedule.

[0484] A "recipe" is a description of the steps and ingredients for making a particular dish.

[0485] A "meal kit" is a set of necessary ingredients based on the menu provided.

[0486] "Dining out" refers to a restaurant where the user can eat meals outside the home.

[0487] "Location information" is geographical data about the user's current location.

[0488] "Optimization" refers to making adjustments or improvements to achieve the most effective state or result according to a purpose.

[0489] This invention is a system that proposes an optimal delivery menu by combining a user's dietary management and emotional data. A specific system configuration for implementing this invention will be described below.

[0490] Hardware and software used

[0491] Hardware:

[0492] Smartphone: Used by the user to take photos of meals and input text data.

[0493] Camera: Built into the smartphone, it is used to take photos of food and obtain emotional data.

[0494] Microphone: Built into the smartphone and used to capture voice data.

[0495] software:

[0496] Image Recognition AI: For example, Google Cloud Vision API is used to analyze ingredients in a meal photo.

[0497] Text analysis AI: For example, the BERT model is used to analyze input text data.

[0498] Emotion analysis engine: For example, the Microsoft Azure Emotion API is used to analyze emotional data from voice and facial expressions.

[0499] Nutritional Assessment Program: A system that assesses a user's nutritional status based on a custom database (e.g., USDA's FoodData Central).

[0500] Recommendation engine: Suggests optimal delivery menus based on nutritional assessment results and emotional data.

[0501] Data processing and calculation

[0502] Image / Text Data Input:

[0503] Photos of food taken with a smartphone and text data entered are sent to a server.

[0504] Emotion data input:

[0505] Using the smartphone's camera and microphone, the user's voice and facial expression data are captured, and emotional data is obtained through an emotion analysis engine.

[0506] Image Recognition and Text Analysis:

[0507] The server uses the received image data with the Google Cloud Vision API to identify ingredients, and the text data is analyzed using the BERT model.

[0508] Nutritional status assessment and labeling:

[0509] The server compares the nutritional information of the recognized ingredients with a custom database and calculates the user's daily nutritional intake. The evaluation results are displayed in the form of a score on the smartphone.

[0510] Delivery menu suggestions:

[0511] The recommendation engine will suggest the most suitable delivery menu for the user based on the nutritional evaluation results and emotional data, and the user can select from the suggested menu and order via smartphone.

[0512] Specific examples

[0513] Example 1:

[0514] The user uploads a photo of their breakfast of "omelette and salad" to their smartphone and enters the phrase "I'm feeling stressed." Analysis of this data reveals a lack of B vitamins, and suggests a delivery menu including stir-fried chicken and vegetables. Chamomile tea is also recommended to help reduce stress.

[0515] Example 2:

[0516] After eating a sandwich and fruit for lunch, the user enters that they are feeling "low energy." Based on this data, a nutritional assessment is performed, identifying that they are iron deficient. Delivery menus including iron-rich hijiki rice and spinach salad are suggested. A protein smoothie can also be added to provide energy.

[0517] Example prompt sentence:

[0518] "I uploaded a photo of an omelet and salad for breakfast. I'm feeling stressed today. What delivery option would be best?"

[0519] "I had a sandwich and fruit for lunch, but I'm still not feeling well. What's the best nutritional option for me?"

[0520] In this way, the system can comprehensively manage the user's nutritional balance and emotional state and suggest the optimal delivery menu.

[0521] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0522] Step 1:

[0523] The user inputs image and text data.

[0524] Input: Meal photo, text description of meal

[0525] How it works: Users can take photos of their meals using their smartphone camera and upload them to the app, and can also enter the meal contents and ingredients in text format.

[0526] Output: Image data and text data are sent to the server.

[0527] Step 2:

[0528] Emotional data is obtained from the user's voice and facial expressions.

[0529] Input: Voice data, facial expression data

[0530] How it works: The user inputs voice through the smartphone's microphone and records facial expressions with the camera. This data is sent to the emotion analysis engine for analysis.

[0531] Output: Emotion data is sent to the server.

[0532] Step 3:

[0533] Analyze data using image recognition AI and text analysis AI.

[0534] Input: Image data, text data

[0535] How it works: The server uses the Google Cloud Vision API to analyze image data and identify ingredients in the image, and also uses the BERT model to analyze text data and extract information about ingredients and menu items.

[0536] Output: Recognized ingredients and nutrition information

[0537] Step 4:

[0538] Evaluate nutritional status and display the results in score format.

[0539] Input: Recognized ingredients and nutrient information

[0540] How it works: The server compares the nutritional value information of the recognized ingredients with a custom database (such as USDA's FoodData Central) to calculate the user's total daily calorie and nutrient intake, then calculates a nutritional status assessment and displays it on the smartphone in the form of a score.

[0541] Output: Nutritional status assessment results (score format)

[0542] Step 5:

[0543] Use a recommendation engine to suggest delivery menus.

[0544] Input: Nutritional status assessment results, emotion data

[0545] How it works: The server's recommendation engine generates the optimal delivery menu for the user based on the nutritional assessment results and emotional data. For example, if you are deficient in vitamin C, it will suggest dishes rich in vitamin C. If you are under a lot of stress, it will suggest dishes with a relaxing effect.

[0546] Output: Delivery menu suggestions for user

[0547] Step 6:

[0548] The user selects a delivery menu and places an order.

[0549] Input: Suggested delivery menu

[0550] Specific operation: The user selects from the suggested delivery menu through the smartphone app and confirms the order.

[0551] Output: Menu order information is sent to the server and delivery is arranged.

[0552] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0553] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0554] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0555] [Second embodiment]

[0556] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0557] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0558] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0559] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0560] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0561] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0562] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0563] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0564] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0565] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0566] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0567] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0568] The present invention is a system that accurately grasps the user's nutritional status and suggests optimal meal plans and dining out options. Below, we will create a program for the system and explain its processing in natural language. We will also provide specific examples.

[0569] Overall flow

[0570] The system operates through the following major steps:

[0571] 1. Image / text data input

[0572] 2. Image Recognition and Text Analysis

[0573] 3. Nutritional status assessment and labelling

[0574] 4. Menu and recipe suggestions (for home cooking)

[0575] 5. Dining out suggestions (if you don't cook at home)

[0576] Program processing

[0577] 1. Image / text data input

[0578] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app. Alternatively, the user can enter the ingredients and meal details in text format.

[0579] User: Daily meal information can be provided to the system with simple operations.

[0580] 2. Image Recognition and Text Analysis

[0581] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo shows toast and bananas.

[0582] Server: The text data is processed by text analysis AI to extract ingredients and cooked dishes from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0583] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[0584] 3. Nutritional status assessment and labelling

[0585] Server: Collects the user's daily dietary information and calculates the total intake of calories, protein, vitamins, minerals, etc. For example, it calculates that the calories are 1820 kcal, the vitamin C is 45 mg, and the iron is 7 mg.

[0586] Server: Compare with recommended intake and assess whether there is a surplus or deficiency.

[0587] On your device: The evaluation results are displayed in the form of a score. For example, your overall nutritional balance is 90 points, your vitamin C intake is 70 points, and your iron intake is 60 points.

[0588] User: Checks nutritional status and chooses next action (cooking at home or eating out).

[0589] 4. Menu and recipe suggestions (for home cooking)

[0590] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[0591] Server: Generates and displays detailed recipes based on the proposed menu. For example, it presents a recipe for "Lemon and Chicken Stir-fry."

[0592] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0593] Server: Works with the meal kit generation system to allow users to easily order.

[0594] User: Order a meal kit and have it delivered to your home.

[0595] 5. Dining out suggestions (if you don't cook at home)

[0596] Server: Based on the user's nutritional assessment results and location information, searches for restaurants that can help fill in any missing nutrients. For example, it searches for restaurants with menus rich in vitamin C.

[0597] Device: Displays a list of nearby restaurants and provides nutritional information for each location.

[0598] User: Selects a dining location and eats at the designated restaurant.

[0599] Specific examples

[0600] User's first day

[0601] Breakfast: User uploads a photo of toast and bananas to the app.

[0602] The server analyzes the image and recognizes toast and bananas.

[0603] The server compares the data with a nutritional database to obtain calorie, vitamin, and mineral information.

[0604] The server calculates intake and evaluates nutritional balance.

[0605] The device will display the score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[0606] The user chooses to cook for themselves the next day.

[0607] The server will suggest the best menu and provide detailed recipes.

[0608] Users order a meal kit and the ingredients arrive the next morning.

[0609] User's second day

[0610] Lunch: Users upload photos of their salad and soup to the app.

[0611] The server analyzes the image and recognizes the ingredients.

[0612] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[0613] If the user does not cook for themselves, they will look for nearby restaurants.

[0614] The device will suggest restaurants with menu items that provide vitamin C supplements.

[0615] The user selects a restaurant and enjoys dining out.

[0616] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[0617] The processing flow will be explained below.

[0618] Step 1:

[0619] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[0620] Step 2:

[0621] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[0622] Step 3:

[0623] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0624] Step 4:

[0625] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates and a small amount of protein, and that bananas contain vitamin C and dietary fiber.

[0626] Step 5:

[0627] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[0628] Step 6:

[0629] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[0630] Step 7:

[0631] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[0632] Step 8:

[0633] User: Check nutritional status and choose whether to cook at home or eat out.

[0634] Step 9:

[0635] Server: If you choose to cook at home, the server will generate optimal menus to supplement any missing nutrients. For example, it will suggest dishes using lemon to supplement the missing vitamin C.

[0636] Step 10:

[0637] Server: Generates detailed recipes based on the menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[0638] Step 11:

[0639] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0640] Step 12:

[0641] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[0642] Step 13:

[0643] User: Orders a meal kit and chooses to have it delivered to their home.

[0644] Step 14:

[0645] Server: If you choose not to cook at home, the app will search for restaurants that can help you meet your nutritional needs based on your rating and location. For example, it will search for nearby restaurants with menus rich in vitamin C.

[0646] Step 15:

[0647] On your device: Based on your location, it will display a list of appropriate restaurants and provide nutritional information for each restaurant.

[0648] Step 16:

[0649] User: Choose from suggested dining options and dine at the specified restaurant.

[0650] Example 1

[0651] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0652] Nutritional management of meals is difficult for many people, and especially for busy modern people, there is a need to accurately and easily understand what they eat every day and ensure adequate nutritional intake. However, current methods require the time-consuming manual recording of meal contents, and without specialized knowledge, it is difficult to maintain an appropriate nutritional balance. Furthermore, there is a problem in that it is difficult to consider nutritional balance when choosing meals to eat out.

[0653] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0654] In this invention, the server includes a means for receiving image and text data entered by the user, a means for analyzing the received image data with a generative AI model to identify ingredients in the photo, a means for analyzing the received text data with a generative AI model to extract ingredients and menu items, a means for comparing the data with a nutritional value database of ingredients to obtain nutrient information, a means for evaluating the user's nutritional status based on the obtained nutrient information, and a means for displaying the evaluation results in the form of a score. This allows users to easily record their daily diet and obtain specific guidelines for maintaining nutritional balance. This system also provides an efficient and easy nutritional management system for busy modern people.

[0655] "User" refers to an individual who uses the system.

[0656] "Image and text data" refers to photographic data and text information that records meal details entered by the user.

[0657] A "generative AI model" refers to an artificial intelligence algorithm that analyzes received image and text data to identify ingredients and menu items.

[0658] "Ingredients" are raw materials used in cooking, specifically vegetables, meat, fruits, etc.

[0659] "Nutrients" refer to the nutritional components contained in food ingredients, such as calories, protein, vitamins, and minerals.

[0660] A "nutritional value database" refers to a database that accumulates nutritional information for each food ingredient.

[0661] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[0662] The "evaluation results" are scores indicating the nutritional balance calculated based on the dietary content.

[0663] "Score format" refers to a method in which evaluation results are expressed numerically as a score out of 100.

[0664] A "menu" refers to the combination of dishes served at one meal or one day.

[0665] A "recipe" is a detailed list of ingredients and steps for making a particular dish.

[0666] A "meal kit" is a package that brings together all the ingredients and cooking instructions needed for a specific menu.

[0667] "Dining out" refers to places where you can eat and drink outside of the home, such as restaurants and cafes.

[0668] "Location information" refers to digital data that indicates a user's current location.

[0669] This invention is a system that accurately assesses a user's nutritional status and suggests optimal meal plans and dining options. Below, we will create a program for the system and explain its processing in natural language. We will also provide names and specific examples of the hardware and software used.

[0670] Overall flow

[0671] The system operates through the following major steps:

[0672] 1. Image / text data input

[0673] 2. Image Recognition and Text Analysis

[0674] 3. Nutritional status assessment and labelling

[0675] 4. Menu and recipe suggestions (for home cooking)

[0676] 5. Dining out suggestions (if you don't cook at home)

[0677] Hardware and software used

[0678] Device: An input device such as a smartphone, tablet, or PC. The user takes photos of food and inputs text.

[0679] Server: Cloud service or dedicated server. Analyzes received data and evaluates nutritional status.

[0680] Generative AI models: AI models for image recognition and text analysis, for example, using frameworks such as TensorFlow and PyTorch.

[0681] Database: Nutritional value database such as food composition tables. Use a relational database such as MySQL or PostgreSQL.

[0682] Specific examples of programs

[0683] Image / text data input

[0684] Device: Users can take photos of their meals using their smartphone camera and upload them to the server via the app. They can also input ingredients and menu items in text format.

[0685] User: With simple operations, users can provide their daily dietary information to the system.

[0686] for example:

[0687] "Upload a photo of toast and bananas."

[0688] "Breakfast: toast, banana"

[0689] Image recognition and text analysis

[0690] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana."

[0691] Server: The text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, ingredients are recognized from "Breakfast: toast, banana."

[0692] Server: Compares with a database of food ingredients' nutritional values ​​to obtain calorie and nutrient information.

[0693] for example:

[0694] "Get nutritional information for the ingredients in this photo."

[0695] "Extract ingredients from text data and collect nutritional information."

[0696] Nutritional status assessment and labeling

[0697] Server: Collects the user's daily dietary information and calculates intake of calories, protein, vitamins, minerals, etc.

[0698] Server: Based on this data, we compare it with the recommended intake and evaluate whether it is excessive or insufficient.

[0699] Device: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is 90 points, vitamin C intake is 70 points, etc.

[0700] User: Based on the evaluation results, choose the next action to take: cooking at home or eating out.

[0701] for example:

[0702] "Please rate your nutritional status today and display it in the form of a score."

[0703] "You are deficient in Vitamin C. Please select your next action."

[0704] Menu and recipe suggestions (for home cooking)

[0705] Server: Suggests optimal menus to supplement missing nutrients. For example, if you are lacking in vitamin C, suggest dishes using lemon.

[0706] Server: Based on the proposed menu, generates a specific recipe and presents it to the user.

[0707] Device: Displays a weekly meal plan with a list of ingredients and cooking instructions.

[0708] Server: If necessary, it will link with meal kit providers and provide a function that allows users to easily order meal kits.

[0709] User: Orders the suggested meal kit, has it delivered to their home, and cooks it.

[0710] for example:

[0711] "Please suggest some recipes using lemon as I am lacking in Vitamin C."

[0712] "Order your meal kit and view a week's worth of meal plans."

[0713] Dining out suggestions (if you don't cook at home)

[0714] Server: Based on the user's location information and nutritional assessment results, search for restaurants that can replenish missing nutrients. For example, search for restaurants rich in vitamin C.

[0715] Terminal: Displays a list of suitable restaurants in the vicinity to the user and provides information on the nutritional intake at each restaurant.

[0716] User: Chooses a restaurant and decides to take action to eat out.

[0717] for example:

[0718] "Find restaurants that have menu items rich in vitamin C."

[0719] "View a list of nearby restaurants and choose the one that's right for you."

[0720] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[0721] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0722] Step 1: Input image / text data

[0723] Device: Users take photos of their meals using their smartphone camera and upload them to the server via the app. They can also enter meal details in text format. Input data includes meal photos and text information such as "Breakfast: toast, banana."

[0724] User: The app is easy to use, and taking photos and entering information can be completed with just a few taps. Once entered, images and text data of the meal contents are generated.

[0725] Step 2: Image recognition and text analysis

[0726] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana" from an uploaded photo. The input is the image data, and the output is a list of recognized ingredients.

[0727] Server: The received text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, the text "Breakfast: toast, banana" is used to recognize "toast" and "banana." The input is text data, and the output is a list of extracted ingredients.

[0728] Server: Compares with the nutritional value database of ingredients to obtain the necessary nutritional information. For example, obtain the calorie and vitamin content of toast and banana from the database. The input is a list of recognized ingredients, and the output is a list of nutritional information.

[0729] Step 3: Nutritional status assessment and labelling

[0730] Server: Aggregates the user's daily dietary information and calculates the intake of calories, protein, vitamins, minerals, etc. For example, calculate the total intake by adding up each nutrient in breakfast and lunch. The input is a list of nutrient information, and the output is the total intake.

[0731] Server: Based on these total intakes, compare them with the recommended intake and evaluate whether there are any nutritional deficiencies or excesses. For example, judge based on the Japanese Dietary Reference Intakes. The input is the total intake, and the output is the evaluation result.

[0732] Terminal: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is displayed as 90 out of 100, and vitamin C intake is displayed as 70. The input is the evaluation results, and the output is a score display.

[0733] User: Based on the evaluation results, the user chooses whether to cook at home or eat out. The results are used as an indicator to determine the next action.

[0734] Step 4: Menu and recipe suggestions (for home cooking)

[0735] Server: Suggests the optimal menu to supplement missing nutrients. For example, if you are lacking in vitamin C, it suggests dishes using lemon. The input is the evaluation result, and the output is the suggested menu.

[0736] Server: Generates a specific recipe based on the proposed menu and presents it to the user. For example, it provides a recipe for "stir-fried chicken with lemon." The input is the proposed menu, and the output is a detailed recipe.

[0737] Terminal: Displays a weekly meal plan with a list of ingredients and cooking instructions, allowing users to use it as a shopping list. The input is a detailed recipe, and the output is a meal plan and an ingredient list.

[0738] Server: If necessary, it connects with meal kit providers and provides a function that allows users to easily order meal kits. The input is a detailed recipe, and the output is meal kit ordering information.

[0739] User: Order a meal kit, have it delivered to your home, and then cook it. Easily prepare a nutritiously balanced meal.

[0740] Step 5: Suggesting places to eat out (if you don't cook at home)

[0741] Server: Based on the user's location information and nutritional assessment results, searches for restaurants that can replenish missing nutrients. For example, searching for restaurants with menus rich in vitamin C. The input is the location information and the assessment results, and the output is the suggested restaurants.

[0742] Terminal: Displays a list of nearby restaurants to the user and provides information on the nutritional value of each restaurant. The input is the suggested dining out location, and the output is the restaurant list.

[0743] User: Chooses a restaurant and decides to dine out. Dining options are presented in an easy-to-understand way.

[0744] In this way, each processing step is clearly separated, and the necessary input data is acquired, appropriate data processing and calculations are performed based on that data, and output is obtained. This allows users to easily understand their nutritional status and implement an accurate meal plan. The entire system efficiently and effectively supports users' nutritional management.

[0745] (Application example 1)

[0746] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] There is a need for a system that can accurately grasp the nutrients that users are consuming in their daily meals and receive specific suggestions to supplement any nutrient deficiencies. In particular, there is a problem in that there is a lack of ways to check nutritional status and efficiently purchase necessary ingredients when shopping in physical stores, making it difficult for users to manage their health.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0749] In this invention, the server includes means for receiving image and text data input by a user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, means for displaying the evaluation results in the form of a score, means for taking images of ingredients purchased in the store, and means for suggesting foods in the store that will supplement any nutrient deficiencies based on the nutritional evaluation results. This allows users to check their nutritional status in real time while shopping and purchase recommended foods that will efficiently supplement any nutrient deficiencies.

[0750] A "user" is an individual who uses the system to understand their own nutritional status and receive appropriate meal plans and food recommendations.

[0751] "Image and text data" refers to photographs and text information of meal contents and ingredients that users input into the system.

[0752] "Analysis" is the process by which the system automatically recognizes and understands the content based on the input image and text data.

[0753] "Ingredients" are elements of food that are recognized based on images taken by the user or text data entered by the user.

[0754] "Nutrients" are components contained in foodstuffs that are necessary for the human body, such as proteins, lipids, vitamins, and minerals.

[0755] "Recognition" is the process by which the system analyzes image and text data to identify ingredients and nutrients and extract them as information.

[0756] "Evaluation" refers to quantifying and analyzing the amount and balance of nutrients ingested by the user based on the information on recognized ingredients and nutrients.

[0757] The "score format" is a method of converting evaluation results into numerical values ​​and displaying them in a way that users can understand at a glance.

[0758] "Display" means visually showing the evaluation results and proposals on the screen of the user's terminal.

[0759] "In-store" refers to the indoor area of ​​a physical store where food is purchased.

[0760] "Photographing" refers to the act of a user taking a picture of an ingredient with a smartphone or camera device.

[0761] "Complementary foods" are foods that the system suggests to supplement missing nutrients.

[0762] "Suggestion" refers to the system recommending the most suitable meals or foods for the user.

[0763] "Nutritional assessment results" refer to the results obtained by analyzing the quantity and quality of nutrients ingested by the user.

[0764] This invention is a system that grasps the user's nutritional status and suggests optimal meal plans and dining out locations. This system mainly includes the following means:

[0765] 1. Means for receiving image and text data entered by the user:

[0766] Users can take photos of their meals using their smartphone camera and upload them to the server using a dedicated application. They can also enter the meal contents as text.

[0767] 2. A means for analyzing the received image and text data to recognize ingredients and their nutrients:

[0768] The server uses image recognition software (e.g., OpenCV) to identify ingredients from image data, and natural language processing (NLP) techniques (e.g., the NLTK library) for text data to analyze meal details.

[0769] 3. A means of assessing the user's nutritional status based on perceived ingredients and nutrients:

[0770] The server references a food nutritional value database (e.g., the USDA nutrition database) to obtain the nutritional information of the photographed food. Based on this, it calculates the daily intake and evaluates the user's nutritional balance.

[0771] 4. How to display the evaluation results in the form of a score:

[0772] The server converts the evaluation results into numerical values ​​and calculates the nutritional balance and intake level of each nutrient in the form of a score, which is then displayed on the user's smartphone via a dedicated app.

[0773] 5. How to take pictures of ingredients purchased in-store:

[0774] Users take photos of ingredients they have purchased at a physical store and upload them to the server through the application, where image recognition technology is used to identify the ingredients they have purchased.

[0775] 6. Based on the nutritional assessment results, a method for proposing foods to supplement nutrients that are lacking in the store:

[0776] The server identifies any nutrient deficiencies based on the nutritional balance assessment and suggests foods to supplement them. These suggestions are displayed on the user's smartphone via the application. For example, if a person is lacking in vitamin C, it will suggest purchasing "oranges."

[0777] Specific examples

[0778] While shopping at a physical store, a user purchases "toast" and "banana" and uploads a photo of them to the application. The server uses image recognition to identify the toast and banana. It then refers to a nutritional value database and compiles the nutritional information for each ingredient. If the result shows that the user is deficient in vitamin C, the server suggests purchasing "orange juice" and displays this to the user through the application.

[0779] This process can be started with the following prompt:

[0780] "I had toast and a banana for breakfast. I'd like to upload a photo and receive a nutritional assessment and suggestions to fill in any missing nutrients."

[0781] By using the above means, this system helps users to efficiently understand their nutritional status and effectively take in the necessary nutrients.

[0782] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0783] Step 1:

[0784] Receives image and text data entered by the user

[0785] The user uses their smartphone to take a photo of their meal or input the ingredients and meal contents in text format. The device sends the data to the server via the app. The input data is received by the server as an image file and text data. This input data is analyzed in the next step.

[0786] Step 2:

[0787] Analyzes received image and text data to recognize ingredients and their nutrients

[0788] The server uses image recognition software (e.g., OpenCV) to analyze the received image data. This analysis identifies ingredients in the image and generates an ingredient list. The text data is also analyzed using natural language processing (NLP) techniques (e.g., the NLTK library) to extract ingredient information. As a result, a list of identified ingredients and their nutritional information is output.

[0789] Step 3:

[0790] Evaluate the user's nutritional status based on recognized ingredients and nutrients

[0791] The server references a nutritional value database (e.g., the USDA database) to obtain the nutrient information for the ingredients identified in step 2. Based on this data, it calculates the amount of nutrients consumed per day and evaluates the user's nutritional balance. The calculation results are output as intake amounts for calories, protein, vitamins, minerals, etc.

[0792] Step 4:

[0793] Display the evaluation results in score format

[0794] The server converts the nutritional balance calculated in step 3 into a numerical score. This score is calculated based on a comparison with the target intake amount for each nutrient. The evaluation results are sent to the device as a numerical score and displayed on the user's smartphone via a dedicated app. Users can visually grasp the nutritional balance.

[0795] Step 5:

[0796] Take a photo of the food you purchased in the store

[0797] A user purchases ingredients at a physical store and takes a photo of the ingredients with their smartphone. This photo data is then uploaded to the server. The input data is received by the server as an image file. This input data is analyzed in the next step.

[0798] Step 6:

[0799] Based on the nutritional assessment results, we will suggest foods to supplement the nutrients lacking in the store.

[0800] The server compares the nutritional assessment results obtained in step 3 with the food ingredient information acquired in step 5 to identify any nutrients that are lacking. Based on the identified nutrients that are lacking, it selects and generates a list of foods to recommend. These suggestions are displayed on the user's smartphone via the application. For example, if the user is lacking in vitamin C, it will suggest purchasing "oranges."

[0801] Through the above steps, this system helps users efficiently understand their nutritional status and effectively ingest the necessary nutrients.

[0802] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0803] This invention combines a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out options with an emotion engine that recognizes the user's emotions. Below, we will create a program for the system and explain its processing in natural language. Specific examples will also be included.

[0804] Overall flow

[0805] The system operates through the following major steps:

[0806] 1. Image / text data input

[0807] 2. Entering Emotion Data

[0808] 3. Image Recognition and Text Analysis

[0809] 4. Nutritional status assessment and labelling

[0810] 5. Menu and recipe suggestions (for home cooking)

[0811] 6. Dining out suggestions (if you don't cook at home)

[0812] Program processing

[0813] 1. Image / text data input

[0814] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[0815] User: Provides daily meal information to the system.

[0816] 2. Entering Emotion Data

[0817] Device: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze the user's emotions from their voice and facial expressions.

[0818] Server: The emotion engine analyzes the user's input data and voice data to obtain emotional information.

[0819] 3. Image Recognition and Text Analysis

[0820] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[0821] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0822] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[0823] 4. Nutritional status assessment and labelling

[0824] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[0825] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[0826] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[0827] Server: Adjusts the displayed score of the evaluation results based on the user's emotional information. For example, if the user is under high stress, the server may prioritize vitamin C intake.

[0828] 5. Menu and recipe suggestions (for home cooking)

[0829] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[0830] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[0831] Server: Considers the user's emotional information and suggests menus that match their preferences and mood. For example, if the user is under a lot of stress, it suggests recipes that use ingredients that have a relaxing effect.

[0832] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0833] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[0834] User: Orders a meal kit and chooses to have it delivered to their home.

[0835] 6. Dining out suggestions (if you don't cook at home)

[0836] Server: Based on the evaluation results and emotional information, the server searches for restaurants that will replenish nutrients and match your mood. For example, it searches for nearby restaurants with menus rich in vitamin C.

[0837] On your device: Based on your location, it will display a list of restaurants that are relevant to you, with nutritionally and emotionally conscious menus for each restaurant.

[0838] User: Choose from suggested dining options and dine at the specified restaurant.

[0839] Specific examples

[0840] User's first day

[0841] Breakfast: User uploads a photo of toast and bananas to the app.

[0842] Emotion data: User enters "I feel depressed."

[0843] The server analyzes the image and recognizes toast and bananas.

[0844] The server checks the nutritional value database and obtains the nutritional information.

[0845] The server calculates intake and evaluates nutritional balance.

[0846] The device will display the evaluation score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[0847] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[0848] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[0849] User's second day

[0850] Lunch: Users upload photos of their salad and soup to the app.

[0851] Emotion data: User inputs "I feel stressed."

[0852] The server analyzes the image and recognizes the ingredients.

[0853] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[0854] The server suggests recipes that include foods that are effective in reducing stress.

[0855] If the user does not cook, the server will search for nearby restaurants.

[0856] The device will suggest restaurants with menus that are effective in replenishing vitamin C and reducing stress.

[0857] The user selects a restaurant and enjoys dining out.

[0858] This system allows users to easily understand their nutritional status and obtain an optimal meal plan that not only efficiently ingests the necessary nutrients but also takes into account their emotional state, contributing to both the user's health and mental well-being.

[0859] The processing flow will be explained below.

[0860] Step 1:

[0861] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[0862] Step 2:

[0863] Device: The user inputs their mood for the day from a selection of options, or their voice and facial expressions are recorded using the device's camera and microphone and sent to the emotion engine.

[0864] Step 3:

[0865] Server: The received image is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo contains toast and bananas.

[0866] Step 4:

[0867] Server: The received text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[0868] Step 5:

[0869] Server: Using the emotion engine, analyzes the user's input data and voice data to extract the emotional information of the day. For example, it identifies the user as "feeling depressed" or "highly stressed."

[0870] Step 6:

[0871] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates, and bananas contain vitamin C and dietary fiber.

[0872] Step 7:

[0873] Server: Calculates the user's total daily calorie intake, protein, vitamins, minerals, etc. For example, calculates the total daily calories as 1820 kcal, vitamin C as 45 mg, and iron as 7 mg.

[0874] Step 8:

[0875] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[0876] Step 9:

[0877] Server: Adjust the score display of the evaluation results taking into account emotional information. For example, if stress is high, add points for vitamin C.

[0878] Step 10:

[0879] On-device: Displays nutritional status in the form of a score. For example, overall nutritional balance is 90 points, vitamin C intake is 70 points, and iron intake is 60 points.

[0880] Step 11:

[0881] User: Check nutritional status and choose whether to cook at home or eat out.

[0882] Step 12:

[0883] Server: If you choose to cook at home, the server will suggest the best meal plan to fill in any missing nutrients and emotional information. For example, if you are feeling depressed due to a lack of vitamin C, the server will suggest dishes that use lemon.

[0884] Step 13:

[0885] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[0886] Step 14:

[0887] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[0888] Step 15:

[0889] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[0890] Step 16:

[0891] User: Orders a meal kit and chooses to have it delivered to their home.

[0892] Step 17:

[0893] Server: If you choose not to cook at home, the server uses your location information to supplement any missing nutrients and search for dining options that match your mood, based on your evaluation results and emotional information. For example, it searches for nearby restaurants with menus rich in vitamin C and that are relaxing.

[0894] Step 18:

[0895] On your device: Based on your location, it will display a list of restaurants that are suitable for you, and each restaurant will offer menus that take into consideration your nutritional needs and mood.

[0896] Step 19:

[0897] User: Choose from suggested dining options and dine at the specified restaurant.

[0898] In this way, the system comprehensively analyzes the user's nutritional status and emotional information to suggest optimal meal plans and dining options, allowing users to easily choose the right meal for their health and mental state.

[0899] Example 2

[0900] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0901] Conventional nutrition management systems perform nutritional assessments based solely on the dietary information entered by the user, making it difficult to propose optimal meal plans that take the user's emotional state into account. They also need to accurately recognize the nutritional information of ingredients and make specific suggestions tailored to each user's nutritional status. Furthermore, they need to ensure that the proposed menus and dining options are actually satisfying for the user both nutritionally and emotionally. Therefore, there is a need for a system that provides more comprehensive and accurate nutritional management and can balance the user's health and mental well-being.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0903] In this invention, the server includes means for receiving image and text data input by the user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for analyzing the user's emotions and obtaining emotion data, means for identifying ingredients and nutrients based on the image and text data and comparing that information with an ingredient database, means for evaluating the user's nutritional status based on the identified information and determining whether nutrients are in excess or deficiency, and means for displaying the evaluation results in a score format taking into account the evaluation results and emotion data. This enables more accurate nutritional evaluation and meal plans that take the user's emotional state into consideration.

[0904] "User" refers to a person who uses this system.

[0905] "Image and text data" refers to photos and text information of meal contents that users enter into the system.

[0906] "Emotion data" refers to information that indicates the user's emotional state.

[0907] "Image recognition" refers to the technology of analyzing image data to identify ingredients and objects.

[0908] "Text analysis" refers to the technology of analyzing text data and extracting necessary information.

[0909] "Nutrients" refers to the calories, vitamins, minerals, and other components contained in food ingredients.

[0910] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[0911] "Evaluation results" refers to the results obtained by the system analyzing the user's nutritional status and indicating any deficiencies or excesses in the form of a score.

[0912] "Emotion engine" refers to technology that analyzes emotional data from the user's voice and facial expressions.

[0913] "Menu" refers to the meal menu proposed to the user.

[0914] A "recipe" refers to information that shows how to make a dish or the steps involved.

[0915] A "meal kit" is a product that includes a set of ingredients and cooking instructions needed for a specific menu.

[0916] "Dining out" refers to a restaurant that the user uses to eat out.

[0917] The present invention relates to a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out locations. This system is combined with an emotion engine that recognizes the user's emotions, enabling more comprehensive and personalized suggestions.

[0918] System configuration

[0919] The system uses the following main hardware and software:

[0920] Device: A mobile device such as a smartphone or tablet on which a dedicated application is installed.

[0921] Server: The central processing unit that collects, analyzes, and serves data. Uses a cloud-based server.

[0922] Image recognition AI: For example, use Google's Cloud Vision API.

[0923] Text analysis AI: For example, use OpenAI's GPT-3.

[0924] Emotion engine: An AI engine that performs voice analysis and facial expression analysis.

[0925] Entering data

[0926] Device: The user takes a photo of the meal with the device's camera and uploads it to the server using a dedicated application, or enters the ingredients and meal contents in text format. The system then receives the entered data.

[0927] Data analysis

[0928] Server: Analyzes the received image and text data. Image data is used to identify ingredients using image recognition AI (Google's Cloud Vision API). Text data is used to extract ingredients and menu items using text analysis AI (OpenAI's GPT-3).

[0929] Nutritional status assessment

[0930] Server: Compares the nutritional information of ingredients with the food database and collects the necessary nutritional information. For example, it obtains information such as the calories, vitamins, and minerals of toast and bananas. It then calculates the user's total daily calorie intake and intake of protein, vitamins, minerals, etc. This determines whether the user is consuming too many or too few nutrients.

[0931] Quantitative evaluation and display

[0932] Device: Displays the nutritional status assessment results in the form of a score. The assessment results include an overall score and scores for specific nutrients. For example, calorie intake score, vitamin C intake score, etc.

[0933] Acquiring and adjusting emotion data

[0934] Server: Analyzes the user's emotional data and adjusts the evaluation results based on their emotional state. For example, if the user is under high stress, the server may prioritize vitamin C intake. Emotional data is acquired using an emotion engine that analyzes voice and facial expressions.

[0935] Menu and recipe suggestions

[0936] Server: Based on the evaluation results and emotional data, the server proposes optimal menus and recipes for the user. Specific recipes to supplement missing nutrients and recipes that correspond to the user's emotional state are proposed. Detailed recipes are generated based on the proposed menus.

[0937] Meal kit generation

[0938] Server: Generates meal kits based on suggested menus and recipes and makes them available for selection by the user. The user can then order the suggested meal kit and have it delivered to their home.

[0939] Dining out suggestions

[0940] Server: To supplement missing nutrients in ingredients, the server recommends optimal dining options based on the evaluation results and emotion data. Based on the user's location information, the server searches for the nearest restaurant and lists dining options that offer appropriate menus. The user selects from the suggested dining options and eats at the specified restaurant.

[0941] Examples and prompts

[0942] Example of a user on day one

[0943] Breakfast: User uploads a photo of toast and bananas to the app.

[0944] Emotion data: User enters "I feel depressed."

[0945] The server analyzes the image and obtains nutritional information for the toast and banana.

[0946] The server calculates the intake amount and displays the evaluation score on the device (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[0947] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[0948] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[0949] Examples of prompt statements

[0950] Obtaining Nutrition Information

[0951] "Assigned task: Identify the nutritional content of the following foods: toast, banana."

[0952] Emotion-based regulation

[0953] "Assigned task: Adjust the nutritional evaluation based on the user's emotional state: sad. Increase emphasis on Vitamin C."

[0954] Menu suggestions

[0955] "Assigned task: Suggest a meal plan that includes ingredients to meet the following nutritional needs: Vitamin C. Consider user preference for chicken and stress-relief."

[0956] This system allows users to easily understand their nutritional status and efficiently consume the nutrients they need. It also provides an optimal meal plan that takes into account their emotional state, contributing to the user's health and mental well-being.

[0957] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0958] Step 1:

[0959] Entering data

[0960] Input: The user takes a photo of the meal and uploads it through the app, or enters the ingredients and meal details in text format.

[0961] Specific operation: The user takes a photo of the meal using the device's camera function, then uploads the photo to the server using the app's "meal record" function, or enters the meal contents as text and sends it to the server.

[0962] Output: The device sends image and text data to the server.

[0963] Step 2:

[0964] Entering emotion data

[0965] Input: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze emotions from the user's voice and facial expressions.

[0966] Specific actions: The user selects an option such as "depressed" on the app's "emotion input" screen, and also uses the voice input function to say, "I'm feeling stressed today."

[0967] Output: The device sends emotion data to the server.

[0968] Step 3:

[0969] Image Recognition and Text Analysis

[0970] Input: Image and text data received by the server.

[0971] Specific operation: The server uses Google's Cloud Vision API to identify ingredients from the received image. For example, it recognizes "toast" and "banana" in the photo. It also uses OpenAI's GPT-3 to analyze text data and extract the ingredients and menu items listed. For example, it recognizes toast and banana from the text "Breakfast: toast, banana."

[0972] Output: Data containing identified ingredients and their nutritional information.

[0973] Step 4:

[0974] Obtaining and verifying nutrition information

[0975] Input: The ingredients identified by the server in step 3 and their nutritional information.

[0976] Specific operation: The server checks the food database (e.g., a public database) to obtain the necessary nutritional information. For example, it collects information such as the calories, vitamins, and minerals of toast and banana.

[0977] Output: Data containing nutritional information for each ingredient.

[0978] Step 5:

[0979] Nutritional status assessment and labeling

[0980] Input: Nutrition information and emotion data obtained by the server.

[0981] Specific operation: The server calculates and evaluates the user's total daily calorie intake, protein intake, vitamin intake, mineral intake, etc. For example, the total intake of toast and banana is 300 kcal and the total intake of vitamin C is 10 mg. The server also compares this with the recommended intake amount to determine whether there is a nutrient deficiency or excess. Furthermore, the server adjusts the evaluation result based on emotional data, increasing the importance of vitamin C.

[0982] Output: Data showing the adjusted assessment results in the form of scores.

[0983] Step 6:

[0984] Menu and recipe suggestions

[0985] Input: Server-adjusted nutritional assessment results and emotion data.

[0986] Specific operation: The server uses the assessment results to suggest the optimal menu to supplement the missing nutrients. For example, if you are lacking in vitamin C, it suggests "stir-fried chicken with lemon." It then generates a detailed recipe and provides it to the user.

[0987] Output: Data containing suggested meals and recipes.

[0988] Step 7:

[0989] Generate and order meal kits

[0990] Input: Server-generated recipe data.

[0991] What it does: The server generates meal kits based on the menu and recipes and makes them available for ordering. The meal kits include the necessary ingredients and cooking instructions. The user can order the meal kit and have it delivered to their home.

[0992] Output: Data containing meal kit order information.

[0993] Step 8:

[0994] Dining out suggestions

[0995] Input: Server receives evaluation results and emotion data.

[0996] Specific operation: The server searches for the best place to eat out based on the evaluation results and emotion data to supplement the missing nutrients. Based on the user's location information, it searches for nearby restaurants and lists stores that offer suitable menus.

[0997] Output: Data containing suggested dining locations and corresponding menus.

[0998] This system allows users to gain a detailed understanding of their nutritional status and receive optimal meal plans and dining out suggestions that take into account their emotional state, which is expected to promote health and mental well-being.

[0999] (Application example 2)

[1000] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1001] Conventional nutrition management systems generally assess a user's nutritional status based on their dietary data, but they are unable to optimize meal plans taking into account the user's emotional state. Therefore, nutritional assessment alone can be insufficient, and there is a particular need for meal suggestions tailored to the user's emotional state. Meanwhile, when choosing to eat out, it is difficult to suggest restaurants that combine emotional state and nutritional balance, creating a need for a system that comprehensively supports the user's health and mental state.

[1002] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring emotional data from the user's voice and facial expression, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, and means for proposing an optimal delivery menu based on the nutritional status evaluation result and the emotional data. This makes it possible to propose meals that comprehensively consider the user's nutritional balance and emotional state.

[1003] "User" means an individual who uses the system to manage their own diet and nutrition.

[1004] "Image and text data" refers to information entered by the user in the form of photos and text about meal contents and ingredients.

[1005] "Ingredients" refers to foods and ingredients used in cooking and eating.

[1006] "Nutrients" refer to the components of food that are necessary for the human body, such as proteins, vitamins, and minerals.

[1007] "Voice and facial expressions" refers to the tone of voice and facial expressions used by the user to express their emotional state.

[1008] "Emotion data" is data obtained as a result of analyzing the user's emotions.

[1009] "Nutritional status" refers to the total amount and balance of nutrients a user consumes each day.

[1010] "Evaluation results" refers to the evaluation value or score that the system gives based on the nutritional status and emotional data analyzed.

[1011] "Delivery menu" refers to the meal menu delivered to the user, taking into consideration nutritional balance and emotional state.

[1012] A "menu" is a specific combination of ingredients and dishes used to determine a meal menu or schedule.

[1013] A "recipe" is a description of the steps and ingredients for making a particular dish.

[1014] A "meal kit" is a set of necessary ingredients based on the menu provided.

[1015] "Dining out" refers to a restaurant where the user can eat meals outside the home.

[1016] "Location information" is geographical data about the user's current location.

[1017] "Optimization" refers to making adjustments or improvements to achieve the most effective state or result according to a purpose.

[1018] This invention is a system that proposes an optimal delivery menu by combining a user's dietary management and emotional data. A specific system configuration for implementing this invention will be described below.

[1019] Hardware and software used

[1020] Hardware:

[1021] Smartphone: Used by the user to take photos of meals and input text data.

[1022] Camera: Built into the smartphone, it is used to take photos of food and obtain emotional data.

[1023] Microphone: Built into the smartphone and used to capture voice data.

[1024] software:

[1025] Image Recognition AI: For example, Google Cloud Vision API is used to analyze ingredients in a meal photo.

[1026] Text analysis AI: For example, the BERT model is used to analyze input text data.

[1027] Emotion analysis engine: For example, the Microsoft Azure Emotion API is used to analyze emotional data from voice and facial expressions.

[1028] Nutritional Assessment Program: A system that assesses a user's nutritional status based on a custom database (e.g., USDA's FoodData Central).

[1029] Recommendation engine: Suggests optimal delivery menus based on nutritional assessment results and emotional data.

[1030] Data processing and calculation

[1031] Image / Text Data Input:

[1032] Photos of food taken with a smartphone and text data entered are sent to a server.

[1033] Emotion data input:

[1034] Using the smartphone's camera and microphone, the user's voice and facial expression data are captured, and emotional data is obtained through an emotion analysis engine.

[1035] Image Recognition and Text Analysis:

[1036] The server uses the received image data with the Google Cloud Vision API to identify ingredients, and the text data is analyzed using the BERT model.

[1037] Nutritional status assessment and labeling:

[1038] The server compares the nutritional information of the recognized ingredients with a custom database and calculates the user's daily nutritional intake. The evaluation results are displayed in the form of a score on the smartphone.

[1039] Delivery menu suggestions:

[1040] The recommendation engine will suggest the most suitable delivery menu for the user based on the nutritional evaluation results and emotional data, and the user can select from the suggested menu and order via smartphone.

[1041] Specific examples

[1042] Example 1:

[1043] The user uploads a photo of their breakfast of "omelette and salad" to their smartphone and enters the phrase "I'm feeling stressed." Analysis of this data reveals a lack of B vitamins, and suggests a delivery menu including stir-fried chicken and vegetables. Chamomile tea is also recommended to help reduce stress.

[1044] Example 2:

[1045] After eating a sandwich and fruit for lunch, the user enters that they are feeling "low energy." Based on this data, a nutritional assessment is performed, identifying that they are iron deficient. Delivery menus including iron-rich hijiki rice and spinach salad are suggested. A protein smoothie can also be added to provide energy.

[1046] Example prompt sentence:

[1047] "I uploaded a photo of an omelet and salad for breakfast. I'm feeling stressed today. What delivery option would be best?"

[1048] "I had a sandwich and fruit for lunch, but I'm still not feeling well. What's the best nutritional option for me?"

[1049] In this way, the system can comprehensively manage the user's nutritional balance and emotional state and suggest the optimal delivery menu.

[1050] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1051] Step 1:

[1052] The user inputs image and text data.

[1053] Input: Meal photo, text description of meal

[1054] How it works: Users can take photos of their meals using their smartphone camera and upload them to the app, and can also enter the meal contents and ingredients in text format.

[1055] Output: Image data and text data are sent to the server.

[1056] Step 2:

[1057] Emotional data is obtained from the user's voice and facial expressions.

[1058] Input: Voice data, facial expression data

[1059] How it works: The user inputs voice through the smartphone's microphone and records facial expressions with the camera. This data is sent to the emotion analysis engine for analysis.

[1060] Output: Emotion data is sent to the server.

[1061] Step 3:

[1062] Analyze data using image recognition AI and text analysis AI.

[1063] Input: Image data, text data

[1064] How it works: The server uses the Google Cloud Vision API to analyze image data and identify ingredients in the image, and also uses the BERT model to analyze text data and extract information about ingredients and menu items.

[1065] Output: Recognized ingredients and nutrition information

[1066] Step 4:

[1067] Evaluate nutritional status and display the results in score format.

[1068] Input: Recognized ingredients and nutrient information

[1069] How it works: The server compares the nutritional value information of the recognized ingredients with a custom database (such as USDA's FoodData Central) to calculate the user's total daily calorie and nutrient intake, then calculates a nutritional status assessment and displays it on the smartphone in the form of a score.

[1070] Output: Nutritional status assessment results (score format)

[1071] Step 5:

[1072] Use a recommendation engine to suggest delivery menus.

[1073] Input: Nutritional status assessment results, emotion data

[1074] How it works: The server's recommendation engine generates the optimal delivery menu for the user based on the nutritional assessment results and emotional data. For example, if you are deficient in vitamin C, it will suggest dishes rich in vitamin C. If you are under a lot of stress, it will suggest dishes with a relaxing effect.

[1075] Output: Delivery menu suggestions for user

[1076] Step 6:

[1077] The user selects a delivery menu and places an order.

[1078] Input: Suggested delivery menu

[1079] Specific operation: The user selects from the suggested delivery menu through the smartphone app and confirms the order.

[1080] Output: Menu order information is sent to the server and delivery is arranged.

[1081] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1082] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1083] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1084] [Third embodiment]

[1085] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1086] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1087] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1088] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1089] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1090] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1091] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1092] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1093] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1094] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1095] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1096] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1097] The present invention is a system that accurately grasps the user's nutritional status and suggests optimal meal plans and dining out options. Below, we will create a program for the system and explain its processing in natural language. We will also provide specific examples.

[1098] Overall flow

[1099] The system operates through the following major steps:

[1100] 1. Image / text data input

[1101] 2. Image Recognition and Text Analysis

[1102] 3. Nutritional status assessment and labelling

[1103] 4. Menu and recipe suggestions (for home cooking)

[1104] 5. Dining out suggestions (if you don't cook at home)

[1105] Program processing

[1106] 1. Image / text data input

[1107] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app. Alternatively, the user can enter the ingredients and meal details in text format.

[1108] User: Daily meal information can be provided to the system with simple operations.

[1109] 2. Image Recognition and Text Analysis

[1110] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo shows toast and bananas.

[1111] Server: The text data is processed by text analysis AI to extract ingredients and cooked dishes from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1112] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[1113] 3. Nutritional status assessment and labelling

[1114] Server: Collects the user's daily dietary information and calculates the total intake of calories, protein, vitamins, minerals, etc. For example, it calculates that the calories are 1820 kcal, the vitamin C is 45 mg, and the iron is 7 mg.

[1115] Server: Compare with recommended intake and assess whether there is a surplus or deficiency.

[1116] On your device: The evaluation results are displayed in the form of a score. For example, your overall nutritional balance is 90 points, your vitamin C intake is 70 points, and your iron intake is 60 points.

[1117] User: Checks nutritional status and chooses next action (cooking at home or eating out).

[1118] 4. Menu and recipe suggestions (for home cooking)

[1119] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[1120] Server: Generates and displays detailed recipes based on the proposed menu. For example, it presents a recipe for "Lemon and Chicken Stir-fry."

[1121] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1122] Server: Works with the meal kit generation system to allow users to easily order.

[1123] User: Order a meal kit and have it delivered to your home.

[1124] 5. Dining out suggestions (if you don't cook at home)

[1125] Server: Based on the user's nutritional assessment results and location information, searches for restaurants that can help fill in any missing nutrients. For example, it searches for restaurants with menus rich in vitamin C.

[1126] Device: Displays a list of nearby restaurants and provides nutritional information for each location.

[1127] User: Selects a dining location and eats at the designated restaurant.

[1128] Specific examples

[1129] User's first day

[1130] Breakfast: User uploads a photo of toast and bananas to the app.

[1131] The server analyzes the image and recognizes toast and bananas.

[1132] The server compares the data with a nutritional database to obtain calorie, vitamin, and mineral information.

[1133] The server calculates intake and evaluates nutritional balance.

[1134] The device will display the score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[1135] The user chooses to cook for themselves the next day.

[1136] The server will suggest the best menu and provide detailed recipes.

[1137] Users order a meal kit and the ingredients arrive the next morning.

[1138] User's second day

[1139] Lunch: Users upload photos of their salad and soup to the app.

[1140] The server analyzes the image and recognizes the ingredients.

[1141] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[1142] If the user does not cook for themselves, they will look for nearby restaurants.

[1143] The device will suggest restaurants with menu items that provide vitamin C supplements.

[1144] The user selects a restaurant and enjoys dining out.

[1145] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[1146] The processing flow will be explained below.

[1147] Step 1:

[1148] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[1149] Step 2:

[1150] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[1151] Step 3:

[1152] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1153] Step 4:

[1154] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates and a small amount of protein, and that bananas contain vitamin C and dietary fiber.

[1155] Step 5:

[1156] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[1157] Step 6:

[1158] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[1159] Step 7:

[1160] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[1161] Step 8:

[1162] User: Check nutritional status and choose whether to cook at home or eat out.

[1163] Step 9:

[1164] Server: If you choose to cook at home, the server will generate optimal menus to supplement any missing nutrients. For example, it will suggest dishes using lemon to supplement the missing vitamin C.

[1165] Step 10:

[1166] Server: Generates detailed recipes based on the menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[1167] Step 11:

[1168] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1169] Step 12:

[1170] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[1171] Step 13:

[1172] User: Orders a meal kit and chooses to have it delivered to their home.

[1173] Step 14:

[1174] Server: If you choose not to cook at home, the app will search for restaurants that can help you meet your nutritional needs based on your rating and location. For example, it will search for nearby restaurants with menus rich in vitamin C.

[1175] Step 15:

[1176] On your device: Based on your location, it will display a list of appropriate restaurants and provide nutritional information for each restaurant.

[1177] Step 16:

[1178] User: Choose from suggested dining options and dine at the specified restaurant.

[1179] Example 1

[1180] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Nutritional management of meals is difficult for many people, and especially for busy modern people, there is a need to accurately and easily understand what they eat every day and ensure adequate nutritional intake. However, current methods require the time-consuming manual recording of meal contents, and without specialized knowledge, it is difficult to maintain an appropriate nutritional balance. Furthermore, there is a problem in that it is difficult to consider nutritional balance when choosing meals to eat out.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1183] In this invention, the server includes a means for receiving image and text data entered by the user, a means for analyzing the received image data with a generative AI model to identify ingredients in the photo, a means for analyzing the received text data with a generative AI model to extract ingredients and menu items, a means for comparing the data with a nutritional value database of ingredients to obtain nutrient information, a means for evaluating the user's nutritional status based on the obtained nutrient information, and a means for displaying the evaluation results in the form of a score. This allows users to easily record their daily diet and obtain specific guidelines for maintaining nutritional balance. This system also provides an efficient and easy nutritional management system for busy modern people.

[1184] "User" refers to an individual who uses the system.

[1185] "Image and text data" refers to photographic data and text information that records meal details entered by the user.

[1186] A "generative AI model" refers to an artificial intelligence algorithm that analyzes received image and text data to identify ingredients and menu items.

[1187] "Ingredients" are raw materials used in cooking, specifically vegetables, meat, fruits, etc.

[1188] "Nutrients" refer to the nutritional components contained in food ingredients, such as calories, protein, vitamins, and minerals.

[1189] A "nutritional value database" refers to a database that accumulates nutritional information for each food ingredient.

[1190] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[1191] The "evaluation results" are scores indicating the nutritional balance calculated based on the dietary content.

[1192] "Score format" refers to a method in which evaluation results are expressed numerically as a score out of 100.

[1193] A "menu" refers to the combination of dishes served at one meal or one day.

[1194] A "recipe" is a detailed list of ingredients and steps for making a particular dish.

[1195] A "meal kit" is a package that brings together all the ingredients and cooking instructions needed for a specific menu.

[1196] "Dining out" refers to places where you can eat and drink outside of the home, such as restaurants and cafes.

[1197] "Location information" refers to digital data that indicates a user's current location.

[1198] This invention is a system that accurately assesses a user's nutritional status and suggests optimal meal plans and dining options. Below, we will create a program for the system and explain its processing in natural language. We will also provide names and specific examples of the hardware and software used.

[1199] Overall flow

[1200] The system operates through the following major steps:

[1201] 1. Image / text data input

[1202] 2. Image Recognition and Text Analysis

[1203] 3. Nutritional status assessment and labelling

[1204] 4. Menu and recipe suggestions (for home cooking)

[1205] 5. Dining out suggestions (if you don't cook at home)

[1206] Hardware and software used

[1207] Device: An input device such as a smartphone, tablet, or PC. The user takes photos of food and inputs text.

[1208] Server: Cloud service or dedicated server. Analyzes received data and evaluates nutritional status.

[1209] Generative AI models: AI models for image recognition and text analysis, for example, using frameworks such as TensorFlow and PyTorch.

[1210] Database: Nutritional value database such as food composition tables. Use a relational database such as MySQL or PostgreSQL.

[1211] Specific examples of programs

[1212] Image / text data input

[1213] Device: Users can take photos of their meals using their smartphone camera and upload them to the server via the app. They can also input ingredients and menu items in text format.

[1214] User: With simple operations, users can provide their daily dietary information to the system.

[1215] for example:

[1216] "Upload a photo of toast and bananas."

[1217] "Breakfast: toast, banana"

[1218] Image recognition and text analysis

[1219] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana."

[1220] Server: The text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, ingredients are recognized from "Breakfast: toast, banana."

[1221] Server: Compares with a database of food ingredients' nutritional values ​​to obtain calorie and nutrient information.

[1222] for example:

[1223] "Get nutritional information for the ingredients in this photo."

[1224] "Extract ingredients from text data and collect nutritional information."

[1225] Nutritional status assessment and labeling

[1226] Server: Collects the user's daily dietary information and calculates intake of calories, protein, vitamins, minerals, etc.

[1227] Server: Based on this data, we compare it with the recommended intake and evaluate whether it is excessive or insufficient.

[1228] Device: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is 90 points, vitamin C intake is 70 points, etc.

[1229] User: Based on the evaluation results, choose the next action to take: cooking at home or eating out.

[1230] for example:

[1231] "Please rate your nutritional status today and display it in the form of a score."

[1232] "You are deficient in Vitamin C. Please select your next action."

[1233] Menu and recipe suggestions (for home cooking)

[1234] Server: Suggests optimal menus to supplement missing nutrients. For example, if you are lacking in vitamin C, suggest dishes using lemon.

[1235] Server: Based on the proposed menu, generates a specific recipe and presents it to the user.

[1236] Device: Displays a weekly meal plan with a list of ingredients and cooking instructions.

[1237] Server: If necessary, it will link with meal kit providers and provide a function that allows users to easily order meal kits.

[1238] User: Orders the suggested meal kit, has it delivered to their home, and cooks it.

[1239] for example:

[1240] "Please suggest some recipes using lemon as I am lacking in Vitamin C."

[1241] "Order your meal kit and view a week's worth of meal plans."

[1242] Dining out suggestions (if you don't cook at home)

[1243] Server: Based on the user's location information and nutritional assessment results, search for restaurants that can replenish missing nutrients. For example, search for restaurants rich in vitamin C.

[1244] Terminal: Displays a list of suitable restaurants in the vicinity to the user and provides information on the nutritional intake at each restaurant.

[1245] User: Chooses a restaurant and decides to take action to eat out.

[1246] for example:

[1247] "Find restaurants that have menu items rich in vitamin C."

[1248] "View a list of nearby restaurants and choose the one that's right for you."

[1249] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[1250] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1251] Step 1: Input image / text data

[1252] Device: Users take photos of their meals using their smartphone camera and upload them to the server via the app. They can also enter meal details in text format. Input data includes meal photos and text information such as "Breakfast: toast, banana."

[1253] User: The app is easy to use, and taking photos and entering information can be completed with just a few taps. Once entered, images and text data of the meal contents are generated.

[1254] Step 2: Image recognition and text analysis

[1255] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana" from an uploaded photo. The input is the image data, and the output is a list of recognized ingredients.

[1256] Server: The received text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, the text "Breakfast: toast, banana" is used to recognize "toast" and "banana." The input is text data, and the output is a list of extracted ingredients.

[1257] Server: Compares with the nutritional value database of ingredients to obtain the necessary nutritional information. For example, obtain the calorie and vitamin content of toast and banana from the database. The input is a list of recognized ingredients, and the output is a list of nutritional information.

[1258] Step 3: Nutritional status assessment and labelling

[1259] Server: Aggregates the user's daily dietary information and calculates the intake of calories, protein, vitamins, minerals, etc. For example, calculate the total intake by adding up each nutrient in breakfast and lunch. The input is a list of nutrient information, and the output is the total intake.

[1260] Server: Based on these total intakes, compare them with the recommended intake and evaluate whether there are any nutritional deficiencies or excesses. For example, judge based on the Japanese Dietary Reference Intakes. The input is the total intake, and the output is the evaluation result.

[1261] Terminal: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is displayed as 90 out of 100, and vitamin C intake is displayed as 70. The input is the evaluation results, and the output is a score display.

[1262] User: Based on the evaluation results, the user chooses whether to cook at home or eat out. The results are used as an indicator to determine the next action.

[1263] Step 4: Menu and recipe suggestions (for home cooking)

[1264] Server: Suggests the optimal menu to supplement missing nutrients. For example, if you are lacking in vitamin C, it suggests dishes using lemon. The input is the evaluation result, and the output is the suggested menu.

[1265] Server: Generates a specific recipe based on the proposed menu and presents it to the user. For example, it provides a recipe for "stir-fried chicken with lemon." The input is the proposed menu, and the output is a detailed recipe.

[1266] Terminal: Displays a weekly meal plan with a list of ingredients and cooking instructions, allowing users to use it as a shopping list. The input is a detailed recipe, and the output is a meal plan and an ingredient list.

[1267] Server: If necessary, it connects with meal kit providers and provides a function that allows users to easily order meal kits. The input is a detailed recipe, and the output is meal kit ordering information.

[1268] User: Order a meal kit, have it delivered to your home, and then cook it. Easily prepare a nutritiously balanced meal.

[1269] Step 5: Suggesting places to eat out (if you don't cook at home)

[1270] Server: Based on the user's location information and nutritional assessment results, searches for restaurants that can replenish missing nutrients. For example, searching for restaurants with menus rich in vitamin C. The input is the location information and the assessment results, and the output is the suggested restaurants.

[1271] Terminal: Displays a list of nearby restaurants to the user and provides information on the nutritional value of each restaurant. The input is the suggested dining out location, and the output is the restaurant list.

[1272] User: Chooses a restaurant and decides to dine out. Dining options are presented in an easy-to-understand way.

[1273] In this way, each processing step is clearly separated, and the necessary input data is acquired, appropriate data processing and calculations are performed based on that data, and output is obtained. This allows users to easily understand their nutritional status and implement an accurate meal plan. The entire system efficiently and effectively supports users' nutritional management.

[1274] (Application example 1)

[1275] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1276] There is a need for a system that can accurately grasp the nutrients that users are consuming in their daily meals and receive specific suggestions to supplement any nutrient deficiencies. In particular, there is a problem in that there is a lack of ways to check nutritional status and efficiently purchase necessary ingredients when shopping in physical stores, making it difficult for users to manage their health.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1278] In this invention, the server includes means for receiving image and text data input by a user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, means for displaying the evaluation results in the form of a score, means for taking images of ingredients purchased in the store, and means for suggesting foods in the store that will supplement any nutrient deficiencies based on the nutritional evaluation results. This allows users to check their nutritional status in real time while shopping and purchase recommended foods that will efficiently supplement any nutrient deficiencies.

[1279] A "user" is an individual who uses the system to understand their own nutritional status and receive appropriate meal plans and food recommendations.

[1280] "Image and text data" refers to photographs and text information of meal contents and ingredients that users input into the system.

[1281] "Analysis" is the process by which the system automatically recognizes and understands the content based on the input image and text data.

[1282] "Ingredients" are elements of food that are recognized based on images taken by the user or text data entered by the user.

[1283] "Nutrients" are components contained in foodstuffs that are necessary for the human body, such as proteins, lipids, vitamins, and minerals.

[1284] "Recognition" is the process by which the system analyzes image and text data to identify ingredients and nutrients and extract them as information.

[1285] "Evaluation" refers to quantifying and analyzing the amount and balance of nutrients ingested by the user based on the information on recognized ingredients and nutrients.

[1286] The "score format" is a method of converting evaluation results into numerical values ​​and displaying them in a way that users can understand at a glance.

[1287] "Display" means visually showing the evaluation results and proposals on the screen of the user's terminal.

[1288] "In-store" refers to the indoor area of ​​a physical store where food is purchased.

[1289] "Photographing" refers to the act of a user taking a picture of an ingredient with a smartphone or camera device.

[1290] "Complementary foods" are foods that the system suggests to supplement missing nutrients.

[1291] "Suggestion" refers to the system recommending the most suitable meals or foods for the user.

[1292] "Nutritional assessment results" refer to the results obtained by analyzing the quantity and quality of nutrients ingested by the user.

[1293] This invention is a system that grasps the user's nutritional status and suggests optimal meal plans and dining out locations. This system mainly includes the following means:

[1294] 1. Means for receiving image and text data entered by the user:

[1295] Users can take photos of their meals using their smartphone camera and upload them to the server using a dedicated application. They can also enter the meal contents as text.

[1296] 2. A means for analyzing the received image and text data to recognize ingredients and their nutrients:

[1297] The server uses image recognition software (e.g., OpenCV) to identify ingredients from image data, and natural language processing (NLP) techniques (e.g., the NLTK library) for text data to analyze meal details.

[1298] 3. A means of assessing the user's nutritional status based on perceived ingredients and nutrients:

[1299] The server references a food nutritional value database (e.g., the USDA nutrition database) to obtain the nutritional information of the photographed food. Based on this, it calculates the daily intake and evaluates the user's nutritional balance.

[1300] 4. How to display the evaluation results in the form of a score:

[1301] The server converts the evaluation results into numerical values ​​and calculates the nutritional balance and intake level of each nutrient in the form of a score, which is then displayed on the user's smartphone via a dedicated app.

[1302] 5. How to take pictures of ingredients purchased in-store:

[1303] Users take photos of ingredients they have purchased at a physical store and upload them to the server through the application, where image recognition technology is used to identify the ingredients they have purchased.

[1304] 6. Based on the nutritional assessment results, a method for proposing foods to supplement nutrients that are lacking in the store:

[1305] The server identifies any nutrient deficiencies based on the nutritional balance assessment and suggests foods to supplement them. These suggestions are displayed on the user's smartphone via the application. For example, if a person is lacking in vitamin C, it will suggest purchasing "oranges."

[1306] Specific examples

[1307] While shopping at a physical store, a user purchases "toast" and "banana" and uploads a photo of them to the application. The server uses image recognition to identify the toast and banana. It then refers to a nutritional value database and compiles the nutritional information for each ingredient. If the result shows that the user is deficient in vitamin C, the server suggests purchasing "orange juice" and displays this to the user through the application.

[1308] This process can be started with the following prompt:

[1309] "I had toast and a banana for breakfast. I'd like to upload a photo and receive a nutritional assessment and suggestions to fill in any missing nutrients."

[1310] By using the above means, this system helps users to efficiently understand their nutritional status and effectively take in the necessary nutrients.

[1311] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1312] Step 1:

[1313] Receives image and text data entered by the user

[1314] The user uses their smartphone to take a photo of their meal or input the ingredients and meal contents in text format. The device sends the data to the server via the app. The input data is received by the server as an image file and text data. This input data is analyzed in the next step.

[1315] Step 2:

[1316] Analyzes received image and text data to recognize ingredients and their nutrients

[1317] The server uses image recognition software (e.g., OpenCV) to analyze the received image data. This analysis identifies ingredients in the image and generates an ingredient list. The text data is also analyzed using natural language processing (NLP) techniques (e.g., the NLTK library) to extract ingredient information. As a result, a list of identified ingredients and their nutritional information is output.

[1318] Step 3:

[1319] Evaluate the user's nutritional status based on recognized ingredients and nutrients

[1320] The server references a nutritional value database (e.g., the USDA database) to obtain the nutrient information for the ingredients identified in step 2. Based on this data, it calculates the amount of nutrients consumed per day and evaluates the user's nutritional balance. The calculation results are output as intake amounts for calories, protein, vitamins, minerals, etc.

[1321] Step 4:

[1322] Display the evaluation results in score format

[1323] The server converts the nutritional balance calculated in step 3 into a numerical score. This score is calculated based on a comparison with the target intake amount for each nutrient. The evaluation results are sent to the device as a numerical score and displayed on the user's smartphone via a dedicated app. Users can visually grasp the nutritional balance.

[1324] Step 5:

[1325] Take a photo of the food you purchased in the store

[1326] A user purchases ingredients at a physical store and takes a photo of the ingredients with their smartphone. This photo data is then uploaded to the server. The input data is received by the server as an image file. This input data is analyzed in the next step.

[1327] Step 6:

[1328] Based on the nutritional assessment results, we will suggest foods to supplement the nutrients lacking in the store.

[1329] The server compares the nutritional assessment results obtained in step 3 with the food ingredient information acquired in step 5 to identify any nutrients that are lacking. Based on the identified nutrients that are lacking, it selects and generates a list of foods to recommend. These suggestions are displayed on the user's smartphone via the application. For example, if the user is lacking in vitamin C, it will suggest purchasing "oranges."

[1330] Through the above steps, this system helps users efficiently understand their nutritional status and effectively ingest the necessary nutrients.

[1331] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1332] This invention combines a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out options with an emotion engine that recognizes the user's emotions. Below, we will create a program for the system and explain its processing in natural language. Specific examples will also be included.

[1333] Overall flow

[1334] The system operates through the following major steps:

[1335] 1. Image / text data input

[1336] 2. Entering Emotion Data

[1337] 3. Image Recognition and Text Analysis

[1338] 4. Nutritional status assessment and labelling

[1339] 5. Menu and recipe suggestions (for home cooking)

[1340] 6. Dining out suggestions (if you don't cook at home)

[1341] Program processing

[1342] 1. Image / text data input

[1343] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[1344] User: Provides daily meal information to the system.

[1345] 2. Entering Emotion Data

[1346] Device: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze the user's emotions from their voice and facial expressions.

[1347] Server: The emotion engine analyzes the user's input data and voice data to obtain emotional information.

[1348] 3. Image Recognition and Text Analysis

[1349] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[1350] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1351] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[1352] 4. Nutritional status assessment and labelling

[1353] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[1354] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[1355] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[1356] Server: Adjusts the displayed score of the evaluation results based on the user's emotional information. For example, if the user is under high stress, the server may prioritize vitamin C intake.

[1357] 5. Menu and recipe suggestions (for home cooking)

[1358] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[1359] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[1360] Server: Considers the user's emotional information and suggests menus that match their preferences and mood. For example, if the user is under a lot of stress, it suggests recipes that use ingredients that have a relaxing effect.

[1361] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1362] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[1363] User: Orders a meal kit and chooses to have it delivered to their home.

[1364] 6. Dining out suggestions (if you don't cook at home)

[1365] Server: Based on the evaluation results and emotional information, the server searches for restaurants that will replenish nutrients and match your mood. For example, it searches for nearby restaurants with menus rich in vitamin C.

[1366] On your device: Based on your location, it will display a list of restaurants that are relevant to you, with nutritionally and emotionally conscious menus for each restaurant.

[1367] User: Choose from suggested dining options and dine at the specified restaurant.

[1368] Specific examples

[1369] User's first day

[1370] Breakfast: User uploads a photo of toast and bananas to the app.

[1371] Emotion data: User enters "I feel depressed."

[1372] The server analyzes the image and recognizes toast and bananas.

[1373] The server checks the nutritional value database and obtains the nutritional information.

[1374] The server calculates intake and evaluates nutritional balance.

[1375] The device will display the evaluation score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[1376] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[1377] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[1378] User's second day

[1379] Lunch: Users upload photos of their salad and soup to the app.

[1380] Emotion data: User inputs "I feel stressed."

[1381] The server analyzes the image and recognizes the ingredients.

[1382] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[1383] The server suggests recipes that include foods that are effective in reducing stress.

[1384] If the user does not cook, the server will search for nearby restaurants.

[1385] The device will suggest restaurants with menus that are effective in replenishing vitamin C and reducing stress.

[1386] The user selects a restaurant and enjoys dining out.

[1387] This system allows users to easily understand their nutritional status and obtain an optimal meal plan that not only efficiently ingests the necessary nutrients but also takes into account their emotional state, contributing to both the user's health and mental well-being.

[1388] The processing flow will be explained below.

[1389] Step 1:

[1390] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[1391] Step 2:

[1392] Device: The user inputs their mood for the day from a selection of options, or their voice and facial expressions are recorded using the device's camera and microphone and sent to the emotion engine.

[1393] Step 3:

[1394] Server: The received image is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo contains toast and bananas.

[1395] Step 4:

[1396] Server: The received text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1397] Step 5:

[1398] Server: Using the emotion engine, analyzes the user's input data and voice data to extract the emotional information of the day. For example, it identifies the user as "feeling depressed" or "highly stressed."

[1399] Step 6:

[1400] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates, and bananas contain vitamin C and dietary fiber.

[1401] Step 7:

[1402] Server: Calculates the user's total daily calorie intake, protein, vitamins, minerals, etc. For example, calculates the total daily calories as 1820 kcal, vitamin C as 45 mg, and iron as 7 mg.

[1403] Step 8:

[1404] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[1405] Step 9:

[1406] Server: Adjust the score display of the evaluation results taking into account emotional information. For example, if stress is high, add points for vitamin C.

[1407] Step 10:

[1408] On-device: Displays nutritional status in the form of a score. For example, overall nutritional balance is 90 points, vitamin C intake is 70 points, and iron intake is 60 points.

[1409] Step 11:

[1410] User: Check nutritional status and choose whether to cook at home or eat out.

[1411] Step 12:

[1412] Server: If you choose to cook at home, the server will suggest the best meal plan to fill in any missing nutrients and emotional information. For example, if you are feeling depressed due to a lack of vitamin C, the server will suggest dishes that use lemon.

[1413] Step 13:

[1414] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[1415] Step 14:

[1416] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1417] Step 15:

[1418] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[1419] Step 16:

[1420] User: Orders a meal kit and chooses to have it delivered to their home.

[1421] Step 17:

[1422] Server: If you choose not to cook at home, the server uses your location information to supplement any missing nutrients and search for dining options that match your mood, based on your evaluation results and emotional information. For example, it searches for nearby restaurants with menus rich in vitamin C and that are relaxing.

[1423] Step 18:

[1424] On your device: Based on your location, it will display a list of restaurants that are suitable for you, and each restaurant will offer menus that take into consideration your nutritional needs and mood.

[1425] Step 19:

[1426] User: Choose from suggested dining options and dine at the specified restaurant.

[1427] In this way, the system comprehensively analyzes the user's nutritional status and emotional information to suggest optimal meal plans and dining options, allowing users to easily choose the right meal for their health and mental state.

[1428] Example 2

[1429] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1430] Conventional nutrition management systems perform nutritional assessments based solely on the dietary information entered by the user, making it difficult to propose optimal meal plans that take the user's emotional state into account. They also need to accurately recognize the nutritional information of ingredients and make specific suggestions tailored to each user's nutritional status. Furthermore, they need to ensure that the proposed menus and dining options are actually satisfying for the user both nutritionally and emotionally. Therefore, there is a need for a system that provides more comprehensive and accurate nutritional management and can balance the user's health and mental well-being.

[1431] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1432] In this invention, the server includes means for receiving image and text data input by the user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for analyzing the user's emotions and obtaining emotion data, means for identifying ingredients and nutrients based on the image and text data and comparing that information with an ingredient database, means for evaluating the user's nutritional status based on the identified information and determining whether nutrients are in excess or deficiency, and means for displaying the evaluation results in a score format taking into account the evaluation results and emotion data. This enables more accurate nutritional evaluation and meal plans that take the user's emotional state into consideration.

[1433] "User" refers to a person who uses this system.

[1434] "Image and text data" refers to photos and text information of meal contents that users enter into the system.

[1435] "Emotion data" refers to information that indicates the user's emotional state.

[1436] "Image recognition" refers to the technology of analyzing image data to identify ingredients and objects.

[1437] "Text analysis" refers to the technology of analyzing text data and extracting necessary information.

[1438] "Nutrients" refers to the calories, vitamins, minerals, and other components contained in food ingredients.

[1439] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[1440] "Evaluation results" refers to the results obtained by the system analyzing the user's nutritional status and indicating any deficiencies or excesses in the form of a score.

[1441] "Emotion engine" refers to technology that analyzes emotional data from the user's voice and facial expressions.

[1442] "Menu" refers to the meal menu proposed to the user.

[1443] A "recipe" refers to information that shows how to make a dish or the steps involved.

[1444] A "meal kit" is a product that includes a set of ingredients and cooking instructions needed for a specific menu.

[1445] "Dining out" refers to a restaurant that the user uses to eat out.

[1446] The present invention relates to a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out locations. This system is combined with an emotion engine that recognizes the user's emotions, enabling more comprehensive and personalized suggestions.

[1447] System configuration

[1448] The system uses the following main hardware and software:

[1449] Device: A mobile device such as a smartphone or tablet on which a dedicated application is installed.

[1450] Server: The central processing unit that collects, analyzes, and serves data. Uses a cloud-based server.

[1451] Image recognition AI: For example, use Google's Cloud Vision API.

[1452] Text analysis AI: For example, use OpenAI's GPT-3.

[1453] Emotion engine: An AI engine that performs voice analysis and facial expression analysis.

[1454] Entering data

[1455] Device: The user takes a photo of the meal with the device's camera and uploads it to the server using a dedicated application, or enters the ingredients and meal contents in text format. The system then receives the entered data.

[1456] Data analysis

[1457] Server: Analyzes the received image and text data. Image data is used to identify ingredients using image recognition AI (Google's Cloud Vision API). Text data is used to extract ingredients and menu items using text analysis AI (OpenAI's GPT-3).

[1458] Nutritional status assessment

[1459] Server: Compares the nutritional information of ingredients with the food database and collects the necessary nutritional information. For example, it obtains information such as the calories, vitamins, and minerals of toast and bananas. It then calculates the user's total daily calorie intake and intake of protein, vitamins, minerals, etc. This determines whether the user is consuming too many or too few nutrients.

[1460] Quantitative evaluation and display

[1461] Device: Displays the nutritional status assessment results in the form of a score. The assessment results include an overall score and scores for specific nutrients. For example, calorie intake score, vitamin C intake score, etc.

[1462] Acquiring and adjusting emotion data

[1463] Server: Analyzes the user's emotional data and adjusts the evaluation results based on their emotional state. For example, if the user is under high stress, the server may prioritize vitamin C intake. Emotional data is acquired using an emotion engine that analyzes voice and facial expressions.

[1464] Menu and recipe suggestions

[1465] Server: Based on the evaluation results and emotional data, the server proposes optimal menus and recipes for the user. Specific recipes to supplement missing nutrients and recipes that correspond to the user's emotional state are proposed. Detailed recipes are generated based on the proposed menus.

[1466] Meal kit generation

[1467] Server: Generates meal kits based on suggested menus and recipes and makes them available for selection by the user. The user can then order the suggested meal kit and have it delivered to their home.

[1468] Dining out suggestions

[1469] Server: To supplement missing nutrients in ingredients, the server recommends optimal dining options based on the evaluation results and emotion data. Based on the user's location information, the server searches for the nearest restaurant and lists dining options that offer appropriate menus. The user selects from the suggested dining options and eats at the specified restaurant.

[1470] Examples and prompts

[1471] Example of a user on day one

[1472] Breakfast: User uploads a photo of toast and bananas to the app.

[1473] Emotion data: User enters "I feel depressed."

[1474] The server analyzes the image and obtains nutritional information for the toast and banana.

[1475] The server calculates the intake amount and displays the evaluation score on the device (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[1476] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[1477] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[1478] Examples of prompt statements

[1479] Obtaining Nutrition Information

[1480] "Assigned task: Identify the nutritional content of the following foods: toast, banana."

[1481] Emotion-based regulation

[1482] "Assigned task: Adjust the nutritional evaluation based on the user's emotional state: sad. Increase emphasis on Vitamin C."

[1483] Menu suggestions

[1484] "Assigned task: Suggest a meal plan that includes ingredients to meet the following nutritional needs: Vitamin C. Consider user preference for chicken and stress-relief."

[1485] This system allows users to easily understand their nutritional status and efficiently consume the nutrients they need. It also provides an optimal meal plan that takes into account their emotional state, contributing to the user's health and mental well-being.

[1486] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1487] Step 1:

[1488] Entering data

[1489] Input: The user takes a photo of the meal and uploads it through the app, or enters the ingredients and meal details in text format.

[1490] Specific operation: The user takes a photo of the meal using the device's camera function, then uploads the photo to the server using the app's "meal record" function, or enters the meal contents as text and sends it to the server.

[1491] Output: The device sends image and text data to the server.

[1492] Step 2:

[1493] Entering emotion data

[1494] Input: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze emotions from the user's voice and facial expressions.

[1495] Specific actions: The user selects an option such as "depressed" on the app's "emotion input" screen, and also uses the voice input function to say, "I'm feeling stressed today."

[1496] Output: The device sends emotion data to the server.

[1497] Step 3:

[1498] Image Recognition and Text Analysis

[1499] Input: Image and text data received by the server.

[1500] Specific operation: The server uses Google's Cloud Vision API to identify ingredients from the received image. For example, it recognizes "toast" and "banana" in the photo. It also uses OpenAI's GPT-3 to analyze text data and extract the ingredients and menu items listed. For example, it recognizes toast and banana from the text "Breakfast: toast, banana."

[1501] Output: Data containing identified ingredients and their nutritional information.

[1502] Step 4:

[1503] Obtaining and verifying nutrition information

[1504] Input: The ingredients identified by the server in step 3 and their nutritional information.

[1505] Specific operation: The server checks the food database (e.g., a public database) to obtain the necessary nutritional information. For example, it collects information such as the calories, vitamins, and minerals of toast and banana.

[1506] Output: Data containing nutritional information for each ingredient.

[1507] Step 5:

[1508] Nutritional status assessment and labeling

[1509] Input: Nutrition information and emotion data obtained by the server.

[1510] Specific operation: The server calculates and evaluates the user's total daily calorie intake, protein intake, vitamin intake, mineral intake, etc. For example, the total intake of toast and banana is 300 kcal and the total intake of vitamin C is 10 mg. The server also compares this with the recommended intake amount to determine whether there is a nutrient deficiency or excess. Furthermore, the server adjusts the evaluation result based on emotional data, increasing the importance of vitamin C.

[1511] Output: Data showing the adjusted assessment results in the form of scores.

[1512] Step 6:

[1513] Menu and recipe suggestions

[1514] Input: Server-adjusted nutritional assessment results and emotion data.

[1515] Specific operation: The server uses the assessment results to suggest the optimal menu to supplement the missing nutrients. For example, if you are lacking in vitamin C, it suggests "stir-fried chicken with lemon." It then generates a detailed recipe and provides it to the user.

[1516] Output: Data containing suggested meals and recipes.

[1517] Step 7:

[1518] Generate and order meal kits

[1519] Input: Server-generated recipe data.

[1520] What it does: The server generates meal kits based on the menu and recipes and makes them available for ordering. The meal kits include the necessary ingredients and cooking instructions. The user can order the meal kit and have it delivered to their home.

[1521] Output: Data containing meal kit order information.

[1522] Step 8:

[1523] Dining out suggestions

[1524] Input: Server receives evaluation results and emotion data.

[1525] Specific operation: The server searches for the best place to eat out based on the evaluation results and emotion data to supplement the missing nutrients. Based on the user's location information, it searches for nearby restaurants and lists stores that offer suitable menus.

[1526] Output: Data containing suggested dining locations and corresponding menus.

[1527] This system allows users to gain a detailed understanding of their nutritional status and receive optimal meal plans and dining out suggestions that take into account their emotional state, which is expected to promote health and mental well-being.

[1528] (Application example 2)

[1529] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1530] Conventional nutrition management systems generally assess a user's nutritional status based on their dietary data, but they are unable to optimize meal plans taking into account the user's emotional state. Therefore, nutritional assessment alone can be insufficient, and there is a particular need for meal suggestions tailored to the user's emotional state. Meanwhile, when choosing to eat out, it is difficult to suggest restaurants that combine emotional state and nutritional balance, creating a need for a system that comprehensively supports the user's health and mental state.

[1531] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring emotional data from the user's voice and facial expression, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, and means for proposing an optimal delivery menu based on the nutritional status evaluation result and the emotional data. This makes it possible to propose meals that comprehensively consider the user's nutritional balance and emotional state.

[1532] "User" means an individual who uses the system to manage their own diet and nutrition.

[1533] "Image and text data" refers to information entered by the user in the form of photos and text about meal contents and ingredients.

[1534] "Ingredients" refers to foods and ingredients used in cooking and eating.

[1535] "Nutrients" refer to the components of food that are necessary for the human body, such as proteins, vitamins, and minerals.

[1536] "Voice and facial expressions" refers to the tone of voice and facial expressions used by the user to express their emotional state.

[1537] "Emotion data" is data obtained as a result of analyzing the user's emotions.

[1538] "Nutritional status" refers to the total amount and balance of nutrients a user consumes each day.

[1539] "Evaluation results" refers to the evaluation value or score that the system gives based on the nutritional status and emotional data analyzed.

[1540] "Delivery menu" refers to the meal menu delivered to the user, taking into consideration nutritional balance and emotional state.

[1541] A "menu" is a specific combination of ingredients and dishes used to determine a meal menu or schedule.

[1542] A "recipe" is a description of the steps and ingredients for making a particular dish.

[1543] A "meal kit" is a set of necessary ingredients based on the menu provided.

[1544] "Dining out" refers to a restaurant where the user can eat meals outside the home.

[1545] "Location information" is geographical data about the user's current location.

[1546] "Optimization" refers to making adjustments or improvements to achieve the most effective state or result according to a purpose.

[1547] This invention is a system that proposes an optimal delivery menu by combining a user's dietary management and emotional data. A specific system configuration for implementing this invention will be described below.

[1548] Hardware and software used

[1549] Hardware:

[1550] Smartphone: Used by the user to take photos of meals and input text data.

[1551] Camera: Built into the smartphone, it is used to take photos of food and obtain emotional data.

[1552] Microphone: Built into the smartphone and used to capture voice data.

[1553] software:

[1554] Image Recognition AI: For example, Google Cloud Vision API is used to analyze ingredients in a meal photo.

[1555] Text analysis AI: For example, the BERT model is used to analyze input text data.

[1556] Emotion analysis engine: For example, the Microsoft Azure Emotion API is used to analyze emotional data from voice and facial expressions.

[1557] Nutritional Assessment Program: A system that assesses a user's nutritional status based on a custom database (e.g., USDA's FoodData Central).

[1558] Recommendation engine: Suggests optimal delivery menus based on nutritional assessment results and emotional data.

[1559] Data processing and calculation

[1560] Image / Text Data Input:

[1561] Photos of food taken with a smartphone and text data entered are sent to a server.

[1562] Emotion data input:

[1563] Using the smartphone's camera and microphone, the user's voice and facial expression data are captured, and emotional data is obtained through an emotion analysis engine.

[1564] Image Recognition and Text Analysis:

[1565] The server uses the received image data with the Google Cloud Vision API to identify ingredients, and the text data is analyzed using the BERT model.

[1566] Nutritional status assessment and labeling:

[1567] The server compares the nutritional information of the recognized ingredients with a custom database and calculates the user's daily nutritional intake. The evaluation results are displayed in the form of a score on the smartphone.

[1568] Delivery menu suggestions:

[1569] The recommendation engine will suggest the most suitable delivery menu for the user based on the nutritional evaluation results and emotional data, and the user can select from the suggested menu and order via smartphone.

[1570] Specific examples

[1571] Example 1:

[1572] The user uploads a photo of their breakfast of "omelette and salad" to their smartphone and enters the phrase "I'm feeling stressed." Analysis of this data reveals a lack of B vitamins, and suggests a delivery menu including stir-fried chicken and vegetables. Chamomile tea is also recommended to help reduce stress.

[1573] Example 2:

[1574] After eating a sandwich and fruit for lunch, the user enters that they are feeling "low energy." Based on this data, a nutritional assessment is performed, identifying that they are iron deficient. Delivery menus including iron-rich hijiki rice and spinach salad are suggested. A protein smoothie can also be added to provide energy.

[1575] Example prompt sentence:

[1576] "I uploaded a photo of an omelet and salad for breakfast. I'm feeling stressed today. What delivery option would be best?"

[1577] "I had a sandwich and fruit for lunch, but I'm still not feeling well. What's the best nutritional option for me?"

[1578] In this way, the system can comprehensively manage the user's nutritional balance and emotional state and suggest the optimal delivery menu.

[1579] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1580] Step 1:

[1581] The user inputs image and text data.

[1582] Input: Meal photo, text description of meal

[1583] How it works: Users can take photos of their meals using their smartphone camera and upload them to the app, and can also enter the meal contents and ingredients in text format.

[1584] Output: Image data and text data are sent to the server.

[1585] Step 2:

[1586] Emotional data is obtained from the user's voice and facial expressions.

[1587] Input: Voice data, facial expression data

[1588] How it works: The user inputs voice through the smartphone's microphone and records facial expressions with the camera. This data is sent to the emotion analysis engine for analysis.

[1589] Output: Emotion data is sent to the server.

[1590] Step 3:

[1591] Analyze data using image recognition AI and text analysis AI.

[1592] Input: Image data, text data

[1593] How it works: The server uses the Google Cloud Vision API to analyze image data and identify ingredients in the image, and also uses the BERT model to analyze text data and extract information about ingredients and menu items.

[1594] Output: Recognized ingredients and nutrition information

[1595] Step 4:

[1596] Evaluate nutritional status and display the results in score format.

[1597] Input: Recognized ingredients and nutrient information

[1598] How it works: The server compares the nutritional value information of the recognized ingredients with a custom database (such as USDA's FoodData Central) to calculate the user's total daily calorie and nutrient intake, then calculates a nutritional status assessment and displays it on the smartphone in the form of a score.

[1599] Output: Nutritional status assessment results (score format)

[1600] Step 5:

[1601] Use a recommendation engine to suggest delivery menus.

[1602] Input: Nutritional status assessment results, emotion data

[1603] How it works: The server's recommendation engine generates the optimal delivery menu for the user based on the nutritional assessment results and emotional data. For example, if you are deficient in vitamin C, it will suggest dishes rich in vitamin C. If you are under a lot of stress, it will suggest dishes with a relaxing effect.

[1604] Output: Delivery menu suggestions for user

[1605] Step 6:

[1606] The user selects a delivery menu and places an order.

[1607] Input: Suggested delivery menu

[1608] Specific operation: The user selects from the suggested delivery menu through the smartphone app and confirms the order.

[1609] Output: Menu order information is sent to the server and delivery is arranged.

[1610] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1611] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1612] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1613] [Fourth embodiment]

[1614] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1615] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1616] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1617] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1618] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1619] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1620] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1621] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1622] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1623] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1624] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1625] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1626] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1627] The present invention is a system that accurately grasps the user's nutritional status and suggests optimal meal plans and dining out options. Below, we will create a program for the system and explain its processing in natural language. We will also provide specific examples.

[1628] Overall flow

[1629] The system operates through the following major steps:

[1630] 1. Image / text data input

[1631] 2. Image Recognition and Text Analysis

[1632] 3. Nutritional status assessment and labelling

[1633] 4. Menu and recipe suggestions (for home cooking)

[1634] 5. Dining out suggestions (if you don't cook at home)

[1635] Program processing

[1636] 1. Image / text data input

[1637] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app. Alternatively, the user can enter the ingredients and meal details in text format.

[1638] User: Daily meal information can be provided to the system with simple operations.

[1639] 2. Image Recognition and Text Analysis

[1640] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo shows toast and bananas.

[1641] Server: The text data is processed by text analysis AI to extract ingredients and cooked dishes from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1642] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[1643] 3. Nutritional status assessment and labelling

[1644] Server: Collects the user's daily dietary information and calculates the total intake of calories, protein, vitamins, minerals, etc. For example, it calculates that the calories are 1820 kcal, the vitamin C is 45 mg, and the iron is 7 mg.

[1645] Server: Compare with recommended intake and assess whether there is a surplus or deficiency.

[1646] On your device: The evaluation results are displayed in the form of a score. For example, your overall nutritional balance is 90 points, your vitamin C intake is 70 points, and your iron intake is 60 points.

[1647] User: Checks nutritional status and chooses next action (cooking at home or eating out).

[1648] 4. Menu and recipe suggestions (for home cooking)

[1649] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[1650] Server: Generates and displays detailed recipes based on the proposed menu. For example, it presents a recipe for "Lemon and Chicken Stir-fry."

[1651] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1652] Server: Works with the meal kit generation system to allow users to easily order.

[1653] User: Order a meal kit and have it delivered to your home.

[1654] 5. Dining out suggestions (if you don't cook at home)

[1655] Server: Based on the user's nutritional assessment results and location information, searches for restaurants that can help fill in any missing nutrients. For example, it searches for restaurants with menus rich in vitamin C.

[1656] Device: Displays a list of nearby restaurants and provides nutritional information for each location.

[1657] User: Selects a dining location and eats at the designated restaurant.

[1658] Specific examples

[1659] User's first day

[1660] Breakfast: User uploads a photo of toast and bananas to the app.

[1661] The server analyzes the image and recognizes toast and bananas.

[1662] The server compares the data with a nutritional database to obtain calorie, vitamin, and mineral information.

[1663] The server calculates intake and evaluates nutritional balance.

[1664] The device will display the score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[1665] The user chooses to cook for themselves the next day.

[1666] The server will suggest the best menu and provide detailed recipes.

[1667] Users order a meal kit and the ingredients arrive the next morning.

[1668] User's second day

[1669] Lunch: Users upload photos of their salad and soup to the app.

[1670] The server analyzes the image and recognizes the ingredients.

[1671] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[1672] If the user does not cook for themselves, they will look for nearby restaurants.

[1673] The device will suggest restaurants with menu items that provide vitamin C supplements.

[1674] The user selects a restaurant and enjoys dining out.

[1675] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[1676] The processing flow will be explained below.

[1677] Step 1:

[1678] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[1679] Step 2:

[1680] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[1681] Step 3:

[1682] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1683] Step 4:

[1684] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates and a small amount of protein, and that bananas contain vitamin C and dietary fiber.

[1685] Step 5:

[1686] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[1687] Step 6:

[1688] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[1689] Step 7:

[1690] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[1691] Step 8:

[1692] User: Check nutritional status and choose whether to cook at home or eat out.

[1693] Step 9:

[1694] Server: If you choose to cook at home, the server will generate optimal menus to supplement any missing nutrients. For example, it will suggest dishes using lemon to supplement the missing vitamin C.

[1695] Step 10:

[1696] Server: Generates detailed recipes based on the menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[1697] Step 11:

[1698] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1699] Step 12:

[1700] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[1701] Step 13:

[1702] User: Orders a meal kit and chooses to have it delivered to their home.

[1703] Step 14:

[1704] Server: If you choose not to cook at home, the app will search for restaurants that can help you meet your nutritional needs based on your rating and location. For example, it will search for nearby restaurants with menus rich in vitamin C.

[1705] Step 15:

[1706] On your device: Based on your location, it will display a list of appropriate restaurants and provide nutritional information for each restaurant.

[1707] Step 16:

[1708] User: Choose from suggested dining options and dine at the specified restaurant.

[1709] Example 1

[1710] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1711] Nutritional management of meals is difficult for many people, and especially for busy modern people, there is a need to accurately and easily understand what they eat every day and ensure adequate nutritional intake. However, current methods require the time-consuming manual recording of meal contents, and without specialized knowledge, it is difficult to maintain an appropriate nutritional balance. Furthermore, there is a problem in that it is difficult to consider nutritional balance when choosing meals to eat out.

[1712] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1713] In this invention, the server includes a means for receiving image and text data entered by the user, a means for analyzing the received image data with a generative AI model to identify ingredients in the photo, a means for analyzing the received text data with a generative AI model to extract ingredients and menu items, a means for comparing the data with a nutritional value database of ingredients to obtain nutrient information, a means for evaluating the user's nutritional status based on the obtained nutrient information, and a means for displaying the evaluation results in the form of a score. This allows users to easily record their daily diet and obtain specific guidelines for maintaining nutritional balance. This system also provides an efficient and easy nutritional management system for busy modern people.

[1714] "User" refers to an individual who uses the system.

[1715] "Image and text data" refers to photographic data and text information that records meal details entered by the user.

[1716] A "generative AI model" refers to an artificial intelligence algorithm that analyzes received image and text data to identify ingredients and menu items.

[1717] "Ingredients" are raw materials used in cooking, specifically vegetables, meat, fruits, etc.

[1718] "Nutrients" refer to the nutritional components contained in food ingredients, such as calories, protein, vitamins, and minerals.

[1719] A "nutritional value database" refers to a database that accumulates nutritional information for each food ingredient.

[1720] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[1721] The "evaluation results" are scores indicating the nutritional balance calculated based on the dietary content.

[1722] "Score format" refers to a method in which evaluation results are expressed numerically as a score out of 100.

[1723] A "menu" refers to the combination of dishes served at one meal or one day.

[1724] A "recipe" is a detailed list of ingredients and steps for making a particular dish.

[1725] A "meal kit" is a package that brings together all the ingredients and cooking instructions needed for a specific menu.

[1726] "Dining out" refers to places where you can eat and drink outside of the home, such as restaurants and cafes.

[1727] "Location information" refers to digital data that indicates a user's current location.

[1728] This invention is a system that accurately assesses a user's nutritional status and suggests optimal meal plans and dining options. Below, we will create a program for the system and explain its processing in natural language. We will also provide names and specific examples of the hardware and software used.

[1729] Overall flow

[1730] The system operates through the following major steps:

[1731] 1. Image / text data input

[1732] 2. Image Recognition and Text Analysis

[1733] 3. Nutritional status assessment and labelling

[1734] 4. Menu and recipe suggestions (for home cooking)

[1735] 5. Dining out suggestions (if you don't cook at home)

[1736] Hardware and software used

[1737] Device: An input device such as a smartphone, tablet, or PC. The user takes photos of food and inputs text.

[1738] Server: Cloud service or dedicated server. Analyzes received data and evaluates nutritional status.

[1739] Generative AI models: AI models for image recognition and text analysis, for example, using frameworks such as TensorFlow and PyTorch.

[1740] Database: Nutritional value database such as food composition tables. Use a relational database such as MySQL or PostgreSQL.

[1741] Specific examples of programs

[1742] Image / text data input

[1743] Device: Users can take photos of their meals using their smartphone camera and upload them to the server via the app. They can also input ingredients and menu items in text format.

[1744] User: With simple operations, users can provide their daily dietary information to the system.

[1745] for example:

[1746] "Upload a photo of toast and bananas."

[1747] "Breakfast: toast, banana"

[1748] Image recognition and text analysis

[1749] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana."

[1750] Server: The text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, ingredients are recognized from "Breakfast: toast, banana."

[1751] Server: Compares with a database of food ingredients' nutritional values ​​to obtain calorie and nutrient information.

[1752] for example:

[1753] "Get nutritional information for the ingredients in this photo."

[1754] "Extract ingredients from text data and collect nutritional information."

[1755] Nutritional status assessment and labeling

[1756] Server: Collects the user's daily dietary information and calculates intake of calories, protein, vitamins, minerals, etc.

[1757] Server: Based on this data, we compare it with the recommended intake and evaluate whether it is excessive or insufficient.

[1758] Device: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is 90 points, vitamin C intake is 70 points, etc.

[1759] User: Based on the evaluation results, choose the next action to take: cooking at home or eating out.

[1760] for example:

[1761] "Please rate your nutritional status today and display it in the form of a score."

[1762] "You are deficient in Vitamin C. Please select your next action."

[1763] Menu and recipe suggestions (for home cooking)

[1764] Server: Suggests optimal menus to supplement missing nutrients. For example, if you are lacking in vitamin C, suggest dishes using lemon.

[1765] Server: Based on the proposed menu, generates a specific recipe and presents it to the user.

[1766] Device: Displays a weekly meal plan with a list of ingredients and cooking instructions.

[1767] Server: If necessary, it will link with meal kit providers and provide a function that allows users to easily order meal kits.

[1768] User: Orders the suggested meal kit, has it delivered to their home, and cooks it.

[1769] for example:

[1770] "Please suggest some recipes using lemon as I am lacking in Vitamin C."

[1771] "Order your meal kit and view a week's worth of meal plans."

[1772] Dining out suggestions (if you don't cook at home)

[1773] Server: Based on the user's location information and nutritional assessment results, search for restaurants that can replenish missing nutrients. For example, search for restaurants rich in vitamin C.

[1774] Terminal: Displays a list of suitable restaurants in the vicinity to the user and provides information on the nutritional intake at each restaurant.

[1775] User: Chooses a restaurant and decides to take action to eat out.

[1776] for example:

[1777] "Find restaurants that have menu items rich in vitamin C."

[1778] "View a list of nearby restaurants and choose the one that's right for you."

[1779] Through these processes, users can easily understand their nutritional status and efficiently consume the nutrients they need. The system contributes to improving the users' health.

[1780] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1781] Step 1: Input image / text data

[1782] Device: Users take photos of their meals using their smartphone camera and upload them to the server via the app. They can also enter meal details in text format. Input data includes meal photos and text information such as "Breakfast: toast, banana."

[1783] User: The app is easy to use, and taking photos and entering information can be completed with just a few taps. Once entered, images and text data of the meal contents are generated.

[1784] Step 2: Image recognition and text analysis

[1785] Server: The received image data is passed through a generative AI model to identify the ingredients in the photo. For example, it recognizes "toast" and "banana" from an uploaded photo. The input is the image data, and the output is a list of recognized ingredients.

[1786] Server: The received text data is passed through a generative AI model to extract ingredients and menu items from the input. For example, the text "Breakfast: toast, banana" is used to recognize "toast" and "banana." The input is text data, and the output is a list of extracted ingredients.

[1787] Server: Compares with the nutritional value database of ingredients to obtain the necessary nutritional information. For example, obtain the calorie and vitamin content of toast and banana from the database. The input is a list of recognized ingredients, and the output is a list of nutritional information.

[1788] Step 3: Nutritional status assessment and labelling

[1789] Server: Aggregates the user's daily dietary information and calculates the intake of calories, protein, vitamins, minerals, etc. For example, calculate the total intake by adding up each nutrient in breakfast and lunch. The input is a list of nutrient information, and the output is the total intake.

[1790] Server: Based on these total intakes, compare them with the recommended intake and evaluate whether there are any nutritional deficiencies or excesses. For example, judge based on the Japanese Dietary Reference Intakes. The input is the total intake, and the output is the evaluation result.

[1791] Terminal: The evaluation results are displayed to the user in the form of a score. For example, nutritional balance is displayed as 90 out of 100, and vitamin C intake is displayed as 70. The input is the evaluation results, and the output is a score display.

[1792] User: Based on the evaluation results, the user chooses whether to cook at home or eat out. The results are used as an indicator to determine the next action.

[1793] Step 4: Menu and recipe suggestions (for home cooking)

[1794] Server: Suggests the optimal menu to supplement missing nutrients. For example, if you are lacking in vitamin C, it suggests dishes using lemon. The input is the evaluation result, and the output is the suggested menu.

[1795] Server: Generates a specific recipe based on the proposed menu and presents it to the user. For example, it provides a recipe for "stir-fried chicken with lemon." The input is the proposed menu, and the output is a detailed recipe.

[1796] Terminal: Displays a weekly meal plan with a list of ingredients and cooking instructions, allowing users to use it as a shopping list. The input is a detailed recipe, and the output is a meal plan and an ingredient list.

[1797] Server: If necessary, it connects with meal kit providers and provides a function that allows users to easily order meal kits. The input is a detailed recipe, and the output is meal kit ordering information.

[1798] User: Order a meal kit, have it delivered to your home, and then cook it. Easily prepare a nutritiously balanced meal.

[1799] Step 5: Suggesting places to eat out (if you don't cook at home)

[1800] Server: Based on the user's location information and nutritional assessment results, searches for restaurants that can replenish missing nutrients. For example, searching for restaurants with menus rich in vitamin C. The input is the location information and the assessment results, and the output is the suggested restaurants.

[1801] Terminal: Displays a list of nearby restaurants to the user and provides information on the nutritional value of each restaurant. The input is the suggested dining out location, and the output is the restaurant list.

[1802] User: Chooses a restaurant and decides to dine out. Dining options are presented in an easy-to-understand way.

[1803] In this way, each processing step is clearly separated, and the necessary input data is acquired, appropriate data processing and calculations are performed based on that data, and output is obtained. This allows users to easily understand their nutritional status and implement an accurate meal plan. The entire system efficiently and effectively supports users' nutritional management.

[1804] (Application example 1)

[1805] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1806] There is a need for a system that can accurately grasp the nutrients that users are consuming in their daily meals and receive specific suggestions to supplement any nutrient deficiencies. In particular, there is a problem in that there is a lack of ways to check nutritional status and efficiently purchase necessary ingredients when shopping in physical stores, making it difficult for users to manage their health.

[1807] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1808] In this invention, the server includes means for receiving image and text data input by a user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, means for displaying the evaluation results in the form of a score, means for taking images of ingredients purchased in the store, and means for suggesting foods in the store that will supplement any nutrient deficiencies based on the nutritional evaluation results. This allows users to check their nutritional status in real time while shopping and purchase recommended foods that will efficiently supplement any nutrient deficiencies.

[1809] A "user" is an individual who uses the system to understand their own nutritional status and receive appropriate meal plans and food recommendations.

[1810] "Image and text data" refers to photographs and text information of meal contents and ingredients that users input into the system.

[1811] "Analysis" is the process by which the system automatically recognizes and understands the content based on the input image and text data.

[1812] "Ingredients" are elements of food that are recognized based on images taken by the user or text data entered by the user.

[1813] "Nutrients" are components contained in foodstuffs that are necessary for the human body, such as proteins, lipids, vitamins, and minerals.

[1814] "Recognition" is the process by which the system analyzes image and text data to identify ingredients and nutrients and extract them as information.

[1815] "Evaluation" refers to quantifying and analyzing the amount and balance of nutrients ingested by the user based on the information on recognized ingredients and nutrients.

[1816] The "score format" is a method of converting evaluation results into numerical values ​​and displaying them in a way that users can understand at a glance.

[1817] "Display" means visually showing the evaluation results and proposals on the screen of the user's terminal.

[1818] "In-store" refers to the indoor area of ​​a physical store where food is purchased.

[1819] "Photographing" refers to the act of a user taking a picture of an ingredient with a smartphone or camera device.

[1820] "Complementary foods" are foods that the system suggests to supplement missing nutrients.

[1821] "Suggestion" refers to the system recommending the most suitable meals or foods for the user.

[1822] "Nutritional assessment results" refer to the results obtained by analyzing the quantity and quality of nutrients ingested by the user.

[1823] This invention is a system that grasps the user's nutritional status and suggests optimal meal plans and dining out locations. This system mainly includes the following means:

[1824] 1. Means for receiving image and text data entered by the user:

[1825] Users can take photos of their meals using their smartphone camera and upload them to the server using a dedicated application. They can also enter the meal contents as text.

[1826] 2. A means for analyzing the received image and text data to recognize ingredients and their nutrients:

[1827] The server uses image recognition software (e.g., OpenCV) to identify ingredients from image data, and natural language processing (NLP) techniques (e.g., the NLTK library) for text data to analyze meal details.

[1828] 3. A means of assessing the user's nutritional status based on perceived ingredients and nutrients:

[1829] The server references a food nutritional value database (e.g., the USDA nutrition database) to obtain the nutritional information of the photographed food. Based on this, it calculates the daily intake and evaluates the user's nutritional balance.

[1830] 4. How to display the evaluation results in the form of a score:

[1831] The server converts the evaluation results into numerical values ​​and calculates the nutritional balance and intake level of each nutrient in the form of a score, which is then displayed on the user's smartphone via a dedicated app.

[1832] 5. How to take pictures of ingredients purchased in-store:

[1833] Users take photos of ingredients they have purchased at a physical store and upload them to the server through the application, where image recognition technology is used to identify the ingredients they have purchased.

[1834] 6. Based on the nutritional assessment results, a method for proposing foods to supplement nutrients that are lacking in the store:

[1835] The server identifies any nutrient deficiencies based on the nutritional balance assessment and suggests foods to supplement them. These suggestions are displayed on the user's smartphone via the application. For example, if a person is lacking in vitamin C, it will suggest purchasing "oranges."

[1836] Specific examples

[1837] While shopping at a physical store, a user purchases "toast" and "banana" and uploads a photo of them to the application. The server uses image recognition to identify the toast and banana. It then refers to a nutritional value database and compiles the nutritional information for each ingredient. If the result shows that the user is deficient in vitamin C, the server suggests purchasing "orange juice" and displays this to the user through the application.

[1838] This process can be started with the following prompt:

[1839] "I had toast and a banana for breakfast. I'd like to upload a photo and receive a nutritional assessment and suggestions to fill in any missing nutrients."

[1840] By using the above means, this system helps users to efficiently understand their nutritional status and effectively take in the necessary nutrients.

[1841] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1842] Step 1:

[1843] Receives image and text data entered by the user

[1844] The user uses their smartphone to take a photo of their meal or input the ingredients and meal contents in text format. The device sends the data to the server via the app. The input data is received by the server as an image file and text data. This input data is analyzed in the next step.

[1845] Step 2:

[1846] Analyzes received image and text data to recognize ingredients and their nutrients

[1847] The server uses image recognition software (e.g., OpenCV) to analyze the received image data. This analysis identifies ingredients in the image and generates an ingredient list. The text data is also analyzed using natural language processing (NLP) techniques (e.g., the NLTK library) to extract ingredient information. As a result, a list of identified ingredients and their nutritional information is output.

[1848] Step 3:

[1849] Evaluate the user's nutritional status based on recognized ingredients and nutrients

[1850] The server references a nutritional value database (e.g., the USDA database) to obtain the nutrient information for the ingredients identified in step 2. Based on this data, it calculates the amount of nutrients consumed per day and evaluates the user's nutritional balance. The calculation results are output as intake amounts for calories, protein, vitamins, minerals, etc.

[1851] Step 4:

[1852] Display the evaluation results in score format

[1853] The server converts the nutritional balance calculated in step 3 into a numerical score. This score is calculated based on a comparison with the target intake amount for each nutrient. The evaluation results are sent to the device as a numerical score and displayed on the user's smartphone via a dedicated app. Users can visually grasp the nutritional balance.

[1854] Step 5:

[1855] Take a photo of the food you purchased in the store

[1856] A user purchases ingredients at a physical store and takes a photo of the ingredients with their smartphone. This photo data is then uploaded to the server. The input data is received by the server as an image file. This input data is analyzed in the next step.

[1857] Step 6:

[1858] Based on the nutritional assessment results, we will suggest foods to supplement the nutrients lacking in the store.

[1859] The server compares the nutritional assessment results obtained in step 3 with the food ingredient information acquired in step 5 to identify any nutrients that are lacking. Based on the identified nutrients that are lacking, it selects and generates a list of foods to recommend. These suggestions are displayed on the user's smartphone via the application. For example, if the user is lacking in vitamin C, it will suggest purchasing "oranges."

[1860] Through the above steps, this system helps users efficiently understand their nutritional status and effectively ingest the necessary nutrients.

[1861] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1862] This invention combines a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out options with an emotion engine that recognizes the user's emotions. Below, we will create a program for the system and explain its processing in natural language. Specific examples will also be included.

[1863] Overall flow

[1864] The system operates through the following major steps:

[1865] 1. Image / text data input

[1866] 2. Entering Emotion Data

[1867] 3. Image Recognition and Text Analysis

[1868] 4. Nutritional status assessment and labelling

[1869] 5. Menu and recipe suggestions (for home cooking)

[1870] 6. Dining out suggestions (if you don't cook at home)

[1871] Program processing

[1872] 1. Image / text data input

[1873] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[1874] User: Provides daily meal information to the system.

[1875] 2. Entering Emotion Data

[1876] Device: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze the user's emotions from their voice and facial expressions.

[1877] Server: The emotion engine analyzes the user's input data and voice data to obtain emotional information.

[1878] 3. Image Recognition and Text Analysis

[1879] Server: The received image data is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that a photo contains toast and a banana.

[1880] Server: The text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1881] Server: Compares with a database of food nutritional values ​​to obtain the necessary nutritional information.

[1882] 4. Nutritional status assessment and labelling

[1883] Server: Calculates the user's total daily intake of calories, protein, vitamins, minerals, etc. For example, it calculates that calories are 1820 kcal, vitamin C is 45 mg, and iron is 7 mg.

[1884] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[1885] On the device: The results of the nutritional assessment are displayed in the form of a score. For example, the overall nutritional balance is 90 points, the vitamin C intake is 70 points, and the iron intake is 60 points.

[1886] Server: Adjusts the displayed score of the evaluation results based on the user's emotional information. For example, if the user is under high stress, the server may prioritize vitamin C intake.

[1887] 5. Menu and recipe suggestions (for home cooking)

[1888] Server: Providing optimal menu suggestions to users to supplement missing nutrients. For example, suggesting dishes using lemon to supplement missing vitamin C.

[1889] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[1890] Server: Considers the user's emotional information and suggests menus that match their preferences and mood. For example, if the user is under a lot of stress, it suggests recipes that use ingredients that have a relaxing effect.

[1891] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1892] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[1893] User: Orders a meal kit and chooses to have it delivered to their home.

[1894] 6. Dining out suggestions (if you don't cook at home)

[1895] Server: Based on the evaluation results and emotional information, the server searches for restaurants that will replenish nutrients and match your mood. For example, it searches for nearby restaurants with menus rich in vitamin C.

[1896] On your device: Based on your location, it will display a list of restaurants that are relevant to you, with nutritionally and emotionally conscious menus for each restaurant.

[1897] User: Choose from suggested dining options and dine at the specified restaurant.

[1898] Specific examples

[1899] User's first day

[1900] Breakfast: User uploads a photo of toast and bananas to the app.

[1901] Emotion data: User enters "I feel depressed."

[1902] The server analyzes the image and recognizes toast and bananas.

[1903] The server checks the nutritional value database and obtains the nutritional information.

[1904] The server calculates intake and evaluates nutritional balance.

[1905] The device will display the evaluation score (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[1906] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[1907] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[1908] User's second day

[1909] Lunch: Users upload photos of their salad and soup to the app.

[1910] Emotion data: User inputs "I feel stressed."

[1911] The server analyzes the image and recognizes the ingredients.

[1912] The server updates the nutritional assessment and displays the results (e.g., nutritional balance 85 points, iron intake 60 points).

[1913] The server suggests recipes that include foods that are effective in reducing stress.

[1914] If the user does not cook, the server will search for nearby restaurants.

[1915] The device will suggest restaurants with menus that are effective in replenishing vitamin C and reducing stress.

[1916] The user selects a restaurant and enjoys dining out.

[1917] This system allows users to easily understand their nutritional status and obtain an optimal meal plan that not only efficiently ingests the necessary nutrients but also takes into account their emotional state, contributing to both the user's health and mental well-being.

[1918] The processing flow will be explained below.

[1919] Step 1:

[1920] Device: The user takes a photo of the meal using the device's camera and uploads it to the server via the app, or enters the ingredients and meal details in text format.

[1921] Step 2:

[1922] Device: The user inputs their mood for the day from a selection of options, or their voice and facial expressions are recorded using the device's camera and microphone and sent to the emotion engine.

[1923] Step 3:

[1924] Server: The received image is passed through image recognition AI to identify the ingredients in the photo. For example, it recognizes that the photo contains toast and bananas.

[1925] Step 4:

[1926] Server: The received text data is processed by text analysis AI to extract ingredients and menu items from the list. For example, nutritional information is collected from the input "Breakfast: toast, banana."

[1927] Step 5:

[1928] Server: Using the emotion engine, analyzes the user's input data and voice data to extract the emotional information of the day. For example, it identifies the user as "feeling depressed" or "highly stressed."

[1929] Step 6:

[1930] Server: Compares the recognized ingredients with a nutritional value database to obtain nutritional information for each ingredient. For example, it determines that toast contains carbohydrates, and bananas contain vitamin C and dietary fiber.

[1931] Step 7:

[1932] Server: Calculates the user's total daily calorie intake, protein, vitamins, minerals, etc. For example, calculates the total daily calories as 1820 kcal, vitamin C as 45 mg, and iron as 7 mg.

[1933] Step 8:

[1934] Server: Compare the calculated intake with the recommended intake to determine whether you are deficient or over-qualified for each nutrient. For example, determine whether you are deficient in vitamin C.

[1935] Step 9:

[1936] Server: Adjust the score display of the evaluation results taking into account emotional information. For example, if stress is high, add points for vitamin C.

[1937] Step 10:

[1938] On-device: Displays nutritional status in the form of a score. For example, overall nutritional balance is 90 points, vitamin C intake is 70 points, and iron intake is 60 points.

[1939] Step 11:

[1940] User: Check nutritional status and choose whether to cook at home or eat out.

[1941] Step 12:

[1942] Server: If you choose to cook at home, the server will suggest the best meal plan to fill in any missing nutrients and emotional information. For example, if you are feeling depressed due to a lack of vitamin C, the server will suggest dishes that use lemon.

[1943] Step 13:

[1944] Server: Generates detailed recipes based on the proposed menu and provides them to the device. For example, it presents a recipe for "stir-fried chicken with lemon."

[1945] Step 14:

[1946] Device: Displays a weekly meal plan, showing ingredients needed and cooking instructions.

[1947] Step 15:

[1948] Server: Works with the meal kit generation system to provide users with the option to generate a meal kit based on their selected menu.

[1949] Step 16:

[1950] User: Orders a meal kit and chooses to have it delivered to their home.

[1951] Step 17:

[1952] Server: If you choose not to cook at home, the server uses your location information to supplement any missing nutrients and search for dining options that match your mood, based on your evaluation results and emotional information. For example, it searches for nearby restaurants with menus rich in vitamin C and that are relaxing.

[1953] Step 18:

[1954] On your device: Based on your location, it will display a list of restaurants that are suitable for you, and each restaurant will offer menus that take into consideration your nutritional needs and mood.

[1955] Step 19:

[1956] User: Choose from suggested dining options and dine at the specified restaurant.

[1957] In this way, the system comprehensively analyzes the user's nutritional status and emotional information to suggest optimal meal plans and dining options, allowing users to easily choose the right meal for their health and mental state.

[1958] Example 2

[1959] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1960] Conventional nutrition management systems perform nutritional assessments based solely on the dietary information entered by the user, making it difficult to propose optimal meal plans that take the user's emotional state into account. They also need to accurately recognize the nutritional information of ingredients and make specific suggestions tailored to each user's nutritional status. Furthermore, they need to ensure that the proposed menus and dining options are actually satisfying for the user both nutritionally and emotionally. Therefore, there is a need for a system that provides more comprehensive and accurate nutritional management and can balance the user's health and mental well-being.

[1961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1962] In this invention, the server includes means for receiving image and text data input by the user, means for analyzing the received image and text data to recognize ingredients and their nutrients, means for analyzing the user's emotions and obtaining emotion data, means for identifying ingredients and nutrients based on the image and text data and comparing that information with an ingredient database, means for evaluating the user's nutritional status based on the identified information and determining whether nutrients are in excess or deficiency, and means for displaying the evaluation results in a score format taking into account the evaluation results and emotion data. This enables more accurate nutritional evaluation and meal plans that take the user's emotional state into consideration.

[1963] "User" refers to a person who uses this system.

[1964] "Image and text data" refers to photos and text information of meal contents that users enter into the system.

[1965] "Emotion data" refers to information that indicates the user's emotional state.

[1966] "Image recognition" refers to the technology of analyzing image data to identify ingredients and objects.

[1967] "Text analysis" refers to the technology of analyzing text data and extracting necessary information.

[1968] "Nutrients" refers to the calories, vitamins, minerals, and other components contained in food ingredients.

[1969] "Nutritional status" refers to the total amount and balance of nutrients consumed by a user.

[1970] "Evaluation results" refers to the results obtained by the system analyzing the user's nutritional status and indicating any deficiencies or excesses in the form of a score.

[1971] "Emotion engine" refers to technology that analyzes emotional data from the user's voice and facial expressions.

[1972] "Menu" refers to the meal menu proposed to the user.

[1973] A "recipe" refers to information that shows how to make a dish or the steps involved.

[1974] A "meal kit" is a product that includes a set of ingredients and cooking instructions needed for a specific menu.

[1975] "Dining out" refers to a restaurant that the user uses to eat out.

[1976] The present invention relates to a system that accurately grasps a user's nutritional status and suggests optimal meal plans and dining out locations. This system is combined with an emotion engine that recognizes the user's emotions, enabling more comprehensive and personalized suggestions.

[1977] System configuration

[1978] The system uses the following main hardware and software:

[1979] Device: A mobile device such as a smartphone or tablet on which a dedicated application is installed.

[1980] Server: The central processing unit that collects, analyzes, and serves data. Uses a cloud-based server.

[1981] Image recognition AI: For example, use Google's Cloud Vision API.

[1982] Text analysis AI: For example, use OpenAI's GPT-3.

[1983] Emotion engine: An AI engine that performs voice analysis and facial expression analysis.

[1984] Entering data

[1985] Device: The user takes a photo of the meal with the device's camera and uploads it to the server using a dedicated application, or enters the ingredients and meal contents in text format. The system then receives the entered data.

[1986] Data analysis

[1987] Server: Analyzes the received image and text data. Image data is used to identify ingredients using image recognition AI (Google's Cloud Vision API). Text data is used to extract ingredients and menu items using text analysis AI (OpenAI's GPT-3).

[1988] Nutritional status assessment

[1989] Server: Compares the nutritional information of ingredients with the food database and collects the necessary nutritional information. For example, it obtains information such as the calories, vitamins, and minerals of toast and bananas. It then calculates the user's total daily calorie intake and intake of protein, vitamins, minerals, etc. This determines whether the user is consuming too many or too few nutrients.

[1990] Quantitative evaluation and display

[1991] Device: Displays the nutritional status assessment results in the form of a score. The assessment results include an overall score and scores for specific nutrients. For example, calorie intake score, vitamin C intake score, etc.

[1992] Acquiring and adjusting emotion data

[1993] Server: Analyzes the user's emotional data and adjusts the evaluation results based on their emotional state. For example, if the user is under high stress, the server may prioritize vitamin C intake. Emotional data is acquired using an emotion engine that analyzes voice and facial expressions.

[1994] Menu and recipe suggestions

[1995] Server: Based on the evaluation results and emotional data, the server proposes optimal menus and recipes for the user. Specific recipes to supplement missing nutrients and recipes that correspond to the user's emotional state are proposed. Detailed recipes are generated based on the proposed menus.

[1996] Meal kit generation

[1997] Server: Generates meal kits based on suggested menus and recipes and makes them available for selection by the user. The user can then order the suggested meal kit and have it delivered to their home.

[1998] Dining out suggestions

[1999] Server: To supplement missing nutrients in ingredients, the server recommends optimal dining options based on the evaluation results and emotion data. Based on the user's location information, the server searches for the nearest restaurant and lists dining options that offer appropriate menus. The user selects from the suggested dining options and eats at the specified restaurant.

[2000] Examples and prompts

[2001] Example of a user on day one

[2002] Breakfast: User uploads a photo of toast and bananas to the app.

[2003] Emotion data: User enters "I feel depressed."

[2004] The server analyzes the image and obtains nutritional information for the toast and banana.

[2005] The server calculates the intake amount and displays the evaluation score on the device (e.g., nutritional balance 90 points, vitamin C intake 70 points).

[2006] The server takes into account the user's emotional information and suggests recipes that are rich in vitamin C.

[2007] The user orders the suggested meal kit, and the ingredients arrive the next morning.

[2008] Examples of prompt statements

[2009] Obtaining Nutrition Information

[2010] "Assigned task: Identify the nutritional content of the following foods: toast, banana."

[2011] Emotion-based regulation

[2012] "Assigned task: Adjust the nutritional evaluation based on the user's emotional state: sad. Increase emphasis on Vitamin C."

[2013] Menu suggestions

[2014] "Assigned task: Suggest a meal plan that includes ingredients to meet the following nutritional needs: Vitamin C. Consider user preference for chicken and stress-relief."

[2015] This system allows users to easily understand their nutritional status and efficiently consume the nutrients they need. It also provides an optimal meal plan that takes into account their emotional state, contributing to the user's health and mental well-being.

[2016] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2017] Step 1:

[2018] Entering data

[2019] Input: The user takes a photo of the meal and uploads it through the app, or enters the ingredients and meal details in text format.

[2020] Specific operation: The user takes a photo of the meal using the device's camera function, then uploads the photo to the server using the app's "meal record" function, or enters the meal contents as text and sends it to the server.

[2021] Output: The device sends image and text data to the server.

[2022] Step 2:

[2023] Entering emotion data

[2024] Input: The user inputs their mood for the day from a selection of options. The emotion engine may also analyze emotions from the user's voice and facial expressions.

[2025] Specific actions: The user selects an option such as "depressed" on the app's "emotion input" screen, and also uses the voice input function to say, "I'm feeling stressed today."

[2026] Output: The device sends emotion data to the server.

[2027] Step 3:

[2028] Image Recognition and Text Analysis

[2029] Input: Image and text data received by the server.

[2030] Specific operation: The server uses Google's Cloud Vision API to identify ingredients from the received image. For example, it recognizes "toast" and "banana" in the photo. It also uses OpenAI's GPT-3 to analyze text data and extract the ingredients and menu items listed. For example, it recognizes toast and banana from the text "Breakfast: toast, banana."

[2031] Output: Data containing identified ingredients and their nutritional information.

[2032] Step 4:

[2033] Obtaining and verifying nutrition information

[2034] Input: The ingredients identified by the server in step 3 and their nutritional information.

[2035] Specific operation: The server checks the food database (e.g., a public database) to obtain the necessary nutritional information. For example, it collects information such as the calories, vitamins, and minerals of toast and banana.

[2036] Output: Data containing nutritional information for each ingredient.

[2037] Step 5:

[2038] Nutritional status assessment and labeling

[2039] Input: Nutrition information and emotion data obtained by the server.

[2040] Specific operation: The server calculates and evaluates the user's total daily calorie intake, protein intake, vitamin intake, mineral intake, etc. For example, the total intake of toast and banana is 300 kcal and the total intake of vitamin C is 10 mg. The server also compares this with the recommended intake amount to determine whether there is a nutrient deficiency or excess. Furthermore, the server adjusts the evaluation result based on emotional data, increasing the importance of vitamin C.

[2041] Output: Data showing the adjusted assessment results in the form of scores.

[2042] Step 6:

[2043] Menu and recipe suggestions

[2044] Input: Server-adjusted nutritional assessment results and emotion data.

[2045] Specific operation: The server uses the assessment results to suggest the optimal menu to supplement the missing nutrients. For example, if you are lacking in vitamin C, it suggests "stir-fried chicken with lemon." It then generates a detailed recipe and provides it to the user.

[2046] Output: Data containing suggested meals and recipes.

[2047] Step 7:

[2048] Generate and order meal kits

[2049] Input: Server-generated recipe data.

[2050] What it does: The server generates meal kits based on the menu and recipes and makes them available for ordering. The meal kits include the necessary ingredients and cooking instructions. The user can order the meal kit and have it delivered to their home.

[2051] Output: Data containing meal kit order information.

[2052] Step 8:

[2053] Dining out suggestions

[2054] Input: Server receives evaluation results and emotion data.

[2055] Specific operation: The server searches for the best place to eat out based on the evaluation results and emotion data to supplement the missing nutrients. Based on the user's location information, it searches for nearby restaurants and lists stores that offer suitable menus.

[2056] Output: Data containing suggested dining locations and corresponding menus.

[2057] This system allows users to gain a detailed understanding of their nutritional status and receive optimal meal plans and dining out suggestions that take into account their emotional state, which is expected to promote health and mental well-being.

[2058] (Application example 2)

[2059] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2060] Conventional nutrition management systems generally assess a user's nutritional status based on their dietary data, but they are unable to optimize meal plans taking into account the user's emotional state. Therefore, nutritional assessment alone can be insufficient, and there is a particular need for meal suggestions tailored to the user's emotional state. Meanwhile, when choosing to eat out, it is difficult to suggest restaurants that combine emotional state and nutritional balance, creating a need for a system that comprehensively supports the user's health and mental state.

[2061] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring emotional data from the user's voice and facial expression, means for evaluating the user's nutritional status based on the recognized ingredients and nutrients, and means for proposing an optimal delivery menu based on the nutritional status evaluation result and the emotional data. This makes it possible to propose meals that comprehensively consider the user's nutritional balance and emotional state.

[2062] "User" means an individual who uses the system to manage their own diet and nutrition.

[2063] "Image and text data" refers to information entered by the user in the form of photos and text about meal contents and ingredients.

[2064] "Ingredients" refers to foods and ingredients used in cooking and eating.

[2065] "Nutrients" refer to the components of food that are necessary for the human body, such as proteins, vitamins, and minerals.

[2066] "Voice and facial expressions" refers to the tone of voice and facial expressions used by the user to express their emotional state.

[2067] "Emotion data" is data obtained as a result of analyzing the user's emotions.

[2068] "Nutritional status" refers to the total amount and balance of nutrients a user consumes each day.

[2069] "Evaluation results" refers to the evaluation value or score that the system gives based on the nutritional status and emotional data analyzed.

[2070] "Delivery menu" refers to the meal menu delivered to the user, taking into consideration nutritional balance and emotional state.

[2071] A "menu" is a specific combination of ingredients and dishes used to determine a meal menu or schedule.

[2072] A "recipe" is a description of the steps and ingredients for making a particular dish.

[2073] A "meal kit" is a set of necessary ingredients based on the menu provided.

[2074] "Dining out" refers to a restaurant where the user can eat meals outside the home.

[2075] "Location information" is geographical data about the user's current location.

[2076] "Optimization" refers to making adjustments or improvements to achieve the most effective state or result according to a purpose.

[2077] This invention is a system that proposes an optimal delivery menu by combining a user's dietary management and emotional data. A specific system configuration for implementing this invention will be described below.

[2078] Hardware and software used

[2079] Hardware:

[2080] Smartphone: Used by the user to take photos of meals and input text data.

[2081] Camera: Built into the smartphone, it is used to take photos of food and obtain emotional data.

[2082] Microphone: Built into the smartphone and used to capture voice data.

[2083] software:

[2084] Image Recognition AI: For example, Google Cloud Vision API is used to analyze ingredients in a meal photo.

[2085] Text analysis AI: For example, the BERT model is used to analyze input text data.

[2086] Emotion analysis engine: For example, the Microsoft Azure Emotion API is used to analyze emotional data from voice and facial expressions.

[2087] Nutritional Assessment Program: A system that assesses a user's nutritional status based on a custom database (e.g., USDA's FoodData Central).

[2088] Recommendation engine: Suggests optimal delivery menus based on nutritional assessment results and emotional data.

[2089] Data processing and calculation

[2090] Image / Text Data Input:

[2091] Photos of food taken with a smartphone and text data entered are sent to a server.

[2092] Emotion data input:

[2093] Using the smartphone's camera and microphone, the user's voice and facial expression data are captured, and emotional data is obtained through an emotion analysis engine.

[2094] Image Recognition and Text Analysis:

[2095] The server uses the received image data with the Google Cloud Vision API to identify ingredients, and the text data is analyzed using the BERT model.

[2096] Nutritional status assessment and labeling:

[2097] The server compares the nutritional information of the recognized ingredients with a custom database and calculates the user's daily nutritional intake. The evaluation results are displayed in the form of a score on the smartphone.

[2098] Delivery menu suggestions:

[2099] The recommendation engine will suggest the most suitable delivery menu for the user based on the nutritional evaluation results and emotional data, and the user can select from the suggested menu and order via smartphone.

[2100] Specific examples

[2101] Example 1:

[2102] The user uploads a photo of their breakfast of "omelette and salad" to their smartphone and enters the phrase "I'm feeling stressed." Analysis of this data reveals a lack of B vitamins, and suggests a delivery menu including stir-fried chicken and vegetables. Chamomile tea is also recommended to help reduce stress.

[2103] Example 2:

[2104] After eating a sandwich and fruit for lunch, the user enters that they are feeling "low energy." Based on this data, a nutritional assessment is performed, identifying that they are iron deficient. Delivery menus including iron-rich hijiki rice and spinach salad are suggested. A protein smoothie can also be added to provide energy.

[2105] Example prompt sentence:

[2106] "I uploaded a photo of an omelet and salad for breakfast. I'm feeling stressed today. What delivery option would be best?"

[2107] "I had a sandwich and fruit for lunch, but I'm still not feeling well. What's the best nutritional option for me?"

[2108] In this way, the system can comprehensively manage the user's nutritional balance and emotional state and suggest the optimal delivery menu.

[2109] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2110] Step 1:

[2111] The user inputs image and text data.

[2112] Input: Meal photo, text description of meal

[2113] How it works: Users can take photos of their meals using their smartphone camera and upload them to the app, and can also enter the meal contents and ingredients in text format.

[2114] Output: Image data and text data are sent to the server.

[2115] Step 2:

[2116] Emotional data is obtained from the user's voice and facial expressions.

[2117] Input: Voice data, facial expression data

[2118] How it works: The user inputs voice through the smartphone's microphone and records facial expressions with the camera. This data is sent to the emotion analysis engine for analysis.

[2119] Output: Emotion data is sent to the server.

[2120] Step 3:

[2121] Analyze data using image recognition AI and text analysis AI.

[2122] Input: Image data, text data

[2123] How it works: The server uses the Google Cloud Vision API to analyze image data and identify ingredients in the image, and also uses the BERT model to analyze text data and extract information about ingredients and menu items.

[2124] Output: Recognized ingredients and nutrition information

[2125] Step 4:

[2126] Evaluate nutritional status and display the results in score format.

[2127] Input: Recognized ingredients and nutrient information

[2128] How it works: The server compares the nutritional value information of the recognized ingredients with a custom database (such as USDA's FoodData Central) to calculate the user's total daily calorie and nutrient intake, then calculates a nutritional status assessment and displays it on the smartphone in the form of a score.

[2129] Output: Nutritional status assessment results (score format)

[2130] Step 5:

[2131] Use a recommendation engine to suggest delivery menus.

[2132] Input: Nutritional status assessment results, emotion data

[2133] How it works: The server's recommendation engine generates the optimal delivery menu for the user based on the nutritional assessment results and emotional data. For example, if you are deficient in vitamin C, it will suggest dishes rich in vitamin C. If you are under a lot of stress, it will suggest dishes with a relaxing effect.

[2134] Output: Delivery menu suggestions for user

[2135] Step 6:

[2136] The user selects a delivery menu and places an order.

[2137] Input: Suggested delivery menu

[2138] Specific operation: The user selects from the suggested delivery menu through the smartphone app and confirms the order.

[2139] Output: Menu order information is sent to the server and delivery is arranged.

[2140] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2141] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2142] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2143] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2144] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includ...

Claims

1. means for receiving image and text data input by a user; means for analyzing the received image and text data to recognize ingredients and their nutrients; A means for assessing the user's nutritional status based on the recognized ingredients and nutrients; A means for displaying the evaluation results in the form of a score; A system including:

2. A means for proposing optimal menus and recipes to users based on the evaluation results; means for generating meal kits based on the suggested menu and recipes; A means for ordering the generated meal kit; The system of claim 1 further comprising:

3. Based on the evaluation results, a method is provided to suggest restaurants that can supplement the nutritional deficiencies. A means for searching for dining out locations based on the user's location information; The system of claim 1 further comprising:

Images